Principal lecturer: Prof Neil Lawrence
Taken by: MPhil ACS, Part III
Code: L172
Term: Michaelmas
Hours: 16 (8 × 2hr lectures)
When: Tuesdays 10:00–12:00, 13 October – 1 December 2026
Where: FW26, William Gates Building
Class limit: max. 10 students
Prerequisites: Undergraduate-level probability and statistics (distributions, Bayes theorem), linear algebra (matrix operations, eigenvalues), and basic multivariate calculus. No prior physics or thermodynamics assumed.
Aim
Develop the mathematical connections between thermodynamics, information theory, and Bayesian inference, and understand how they provide a toolkit for reasoning about the foundations of intelligent systems.
Syllabus
Entropy appears in three apparently separate traditions — thermodynamics (Boltzmann, Gibbs), information theory (Shannon), and Bayesian inference (Jaynes) — and turns out to be the same mathematical object viewed from different operational assumptions. The operational split is: entropy forbids, probability prescribes. The course covers:
- probability and entropy review; Socratic LLM method; that theme via perpetual motion and the human–machine bandwidth gap of The Atomic Human; then the Boltzmann distribution and free energy,
- Shannon entropy and its formal equivalence to thermodynamic entropy; grouping / missing measure; MaxEnt and the exponential family,
- Maxwell’s demon as compartments and two ledgers; Landauer’s principle as the thermodynamic cost of decision-making,
- exponential family and Fisher $I(\theta)$ versus Shannon $H$ (Week 6 as taught); Shannon capacity and DPI recovered in Week 7,
- Good Regulator overview ($D,S,A,Z$), goals as change of measure, and channel no-gos ($R\le C$, DPI statement),
- after Quiz 4: classical limits synthesis, then no-barber motivation for von Neumann / quantum information, and the course close (unassessed).
Four ten-minute in-class quizzes sit at the start of lectures 2, 4, 6 and 8.
Worksheets 1–4 are Socratic LLM dialogues (curiosity then skepticism) plus a handwritten reflection. Due at the start of weeks 2, 4, 6 and 8. Moodle filenames use L172_<CRSid>_worksheet…; identify work by CRSid, not by full name.
Learning outcomes
| ID | Students will be able to… | Week |
|---|---|---|
| LO1 | Decompose the Boltzmann distribution into internal energy, entropy, and Helmholtz free energy, and interpret each term | 2 |
| LO2 | Derive Shannon entropy as uncertainty and prove its formal equivalence to thermodynamic (Gibbs) entropy in equilibrium | 3 |
| LO3 | Derive the canonical ensemble and use the partition function to compute mean energy, entropy, and free energy | 3 |
| LO4 | Explain Maxwell’s demon, locate the apparent second-law violation, and apply Landauer’s principle | 4 |
| LO5 | Apply Jaynes’ maximum entropy principle with Lagrange multipliers under moment constraints | 5 |
| LO6 | Identify the exponential family as the MaxEnt family (canonical, Gaussian, Bernoulli, …) | 6 |
| LO7 | Compare information-theoretic, thermodynamic, and Bayesian perspectives on entropy | 5 |
| LO8 | Distinguish Fisher information $I(\theta)$ from Shannon entropy $H$ | 6 |
| LO9 | Identify MaxEnt as an $m$-projection (no natural gradients this term) | 7 |
| LO10 | State $R\le C$, define $I+H=C$, apply DPI, and explain the information bottleneck | 6–7 |
| LO11 | Explain why classical $I=C$ forces $H=0$, and show von Neumann entropy resolves nontrivial marginals | 8 |
| LO12 | Articulate agency via Shannon / DPI / GRT (not transport geometry) | 7–8 |
| LO13 | Evaluate capability claims with Landauer, $R\le C$, $I+H=C$, DPI, GRT, and human bandwidth | 6+8 |
Assessment
Four take-home worksheets (15% each, 60% total) and four short in-class quizzes (10% each, 40% total).
| Component | Due | Form | Outcomes |
|---|---|---|---|
| Worksheet 1 | Start of week 2 | Socratic dialogue + handwritten reflection | LO1, LO2 |
| Quiz 1 | Start of week 2 lecture | 10 min invigilated (paper) | LO1, LO2, LO3 |
| Worksheet 2 | Start of week 4 | Socratic dialogue + handwritten reflection | LO2, LO3 |
| Quiz 2 | Start of week 4 lecture | 10 min invigilated (paper) | LO1, LO2, LO3 |
| Worksheet 3 | Start of week 6 | Socratic dialogue + handwritten reflection | LO4 |
| Quiz 3 | Start of week 6 lecture | 10 min invigilated (paper) | LO4, LO5, LO6, LO7 |
| Worksheet 4 | Start of week 8 | Socratic dialogue + handwritten reflection | LO12, LO13 |
| Quiz 4 | Start of week 8 lecture | 10 min invigilated (paper) | LO8–LO13 |
Details and marking guidance: assessment.
Generative AI
How the lecturer uses generative AI to write and maintain the module, and the student-facing rule for worksheets versus quizzes, is on the generative AI page.
Getting help
Ask module questions on GitHub Discussions. Start from the pinned Course Q&A — start here post, then open a new Q&A discussion for each question. Keep personal or confidential matters (extensions, marks, candidate numbers) off Discussions — use email or the course office instead.
Recommended Reading
Shannon (1948), Bell System Technical Journal; Jaynes (1957), Physical Review; Landauer (1961), IBM Journal; Cover & Thomas (2006) and MacKay (2003) as background. Conant & Ashby (1970) for the Good Regulator. The exponential family and its Fisher metric are live in Week 6. Thermodynamic length (Crooks 2007) is dropped. Welling, Lu and Holdijk (2026), Generative AI and Stochastic Thermodynamics (GAIST), is a supplementary monograph: Chapter 3 for free energy, Chapter 5 for the ELBO; Chapters 14 and 22 (Wasserstein / Schrödinger) are optional colour, not on the live Weeks 6–8 spine this term.