Principal lecturer: Prof Neil Lawrence
Taken by: MPhil ACS, Part III
Code: L172
Term: Michaelmas
Hours: 16 (8 × 2hr lectures)
When: Tuesdays 10:00–12:00, 13 October – 1 December 2026
Where: FW26, William Gates Building
Class limit: max. 10 students
Prerequisites: Undergraduate-level probability and statistics (distributions, Bayes theorem), linear algebra (matrix operations, eigenvalues), and basic multivariate calculus. No prior physics or thermodynamics assumed.

Aim

Develop the mathematical connections between thermodynamics, information theory, and Bayesian inference, and understand how they provide a toolkit for reasoning about the foundations of intelligent systems.

Syllabus

Entropy appears in three apparently separate traditions — thermodynamics (Boltzmann, Gibbs), information theory (Shannon), and Bayesian inference (Jaynes) — and turns out to be the same mathematical object viewed from different operational assumptions. The operational split is: entropy forbids, probability prescribes. The course covers:

  1. probability and entropy review; Socratic LLM method; that theme via perpetual motion and the human–machine bandwidth gap of The Atomic Human; then the Boltzmann distribution and free energy,
  2. Shannon entropy and its formal equivalence to thermodynamic entropy; grouping / missing measure; MaxEnt and the exponential family,
  3. Maxwell’s demon as compartments and two ledgers; Landauer’s principle as the thermodynamic cost of decision-making,
  4. exponential family and Fisher $I(\theta)$ versus Shannon $H$ (Week 6 as taught); Shannon capacity and DPI recovered in Week 7,
  5. Good Regulator overview ($D,S,A,Z$), goals as change of measure, and channel no-gos ($R\le C$, DPI statement),
  6. after Quiz 4: classical limits synthesis, then no-barber motivation for von Neumann / quantum information, and the course close (unassessed).

Four ten-minute in-class quizzes sit at the start of lectures 2, 4, 6 and 8.

Worksheets 1–4 are Socratic LLM dialogues (curiosity then skepticism) plus a handwritten reflection. Due at the start of weeks 2, 4, 6 and 8. Moodle filenames use L172_<CRSid>_worksheet…; identify work by CRSid, not by full name.

Learning outcomes

ID Students will be able to… Week
LO1 Decompose the Boltzmann distribution into internal energy, entropy, and Helmholtz free energy, and interpret each term 2
LO2 Derive Shannon entropy as uncertainty and prove its formal equivalence to thermodynamic (Gibbs) entropy in equilibrium 3
LO3 Derive the canonical ensemble and use the partition function to compute mean energy, entropy, and free energy 3
LO4 Explain Maxwell’s demon, locate the apparent second-law violation, and apply Landauer’s principle 4
LO5 Apply Jaynes’ maximum entropy principle with Lagrange multipliers under moment constraints 5
LO6 Identify the exponential family as the MaxEnt family (canonical, Gaussian, Bernoulli, …) 6
LO7 Compare information-theoretic, thermodynamic, and Bayesian perspectives on entropy 5
LO8 Distinguish Fisher information $I(\theta)$ from Shannon entropy $H$ 6
LO9 Identify MaxEnt as an $m$-projection (no natural gradients this term) 7
LO10 State $R\le C$, define $I+H=C$, apply DPI, and explain the information bottleneck 6–7
LO11 Explain why classical $I=C$ forces $H=0$, and show von Neumann entropy resolves nontrivial marginals 8
LO12 Articulate agency via Shannon / DPI / GRT (not transport geometry) 7–8
LO13 Evaluate capability claims with Landauer, $R\le C$, $I+H=C$, DPI, GRT, and human bandwidth 6+8

Assessment

Four take-home worksheets (15% each, 60% total) and four short in-class quizzes (10% each, 40% total).

Component Due Form Outcomes
Worksheet 1 Start of week 2 Socratic dialogue + handwritten reflection LO1, LO2
Quiz 1 Start of week 2 lecture 10 min invigilated (paper) LO1, LO2, LO3
Worksheet 2 Start of week 4 Socratic dialogue + handwritten reflection LO2, LO3
Quiz 2 Start of week 4 lecture 10 min invigilated (paper) LO1, LO2, LO3
Worksheet 3 Start of week 6 Socratic dialogue + handwritten reflection LO4
Quiz 3 Start of week 6 lecture 10 min invigilated (paper) LO4, LO5, LO6, LO7
Worksheet 4 Start of week 8 Socratic dialogue + handwritten reflection LO12, LO13
Quiz 4 Start of week 8 lecture 10 min invigilated (paper) LO8–LO13

Details and marking guidance: assessment.

Generative AI

How the lecturer uses generative AI to write and maintain the module, and the student-facing rule for worksheets versus quizzes, is on the generative AI page.

Getting help

Ask module questions on GitHub Discussions. Start from the pinned Course Q&A — start here post, then open a new Q&A discussion for each question. Keep personal or confidential matters (extensions, marks, candidate numbers) off Discussions — use email or the course office instead.

Shannon (1948), Bell System Technical Journal; Jaynes (1957), Physical Review; Landauer (1961), IBM Journal; Cover & Thomas (2006) and MacKay (2003) as background. Conant & Ashby (1970) for the Good Regulator. The exponential family and its Fisher metric are live in Week 6. Thermodynamic length (Crooks 2007) is dropped. Welling, Lu and Holdijk (2026), Generative AI and Stochastic Thermodynamics (GAIST), is a supplementary monograph: Chapter 3 for free energy, Chapter 5 for the ELBO; Chapters 14 and 22 (Wasserstein / Schrödinger) are optional colour, not on the live Weeks 6–8 spine this term.