Boltzmann, Free Energy, and Entropy

Neil D. Lawrence

FW26, William Gates Building

This Session

  • Quiz 1 (ten minutes)
  • Gibbs–Boltzmann form; free energy \(F=U-TS\)
  • Bath, Schottky, finite-time dissipation named

Quiz 1

  • Device required; Moodle open
  • No notes, no network, no LLMs
  • Ten minutes

From the Seed to the Formula

Perpetual Motion and Superintelligence

  • 1925: Motor vehicles (no perpetual motion)
  • 2025: Promises of superintelligence singularity
  • Same fundamental impossibility?

Why Perpetual Motion Failed

\[\frac{\text{d}H}{\text{d}t} \geq 0\]

  • Entropy always increases
  • No motion without entropy production
  • No work without energy input

An Equivalent Statement for Intelligence?

Maxwell’s Demon:

  • “Intelligent” entity that violates 2nd law
  • Resolution: Landauer’s principle?
  • Information erasure requires energy

Implication

  • Intelligence has thermodynamic cost
  • Information processing has physical limits

Carnot and Clausius

  • Carnot (1824): no real engine beats a reversible cycle between two baths
  • Clausius (1850s): heat cannot flow from cold to hot without work
  • Clausius (1865): names entropy — the state’s transformation content

Clausius: the entropy of the universe tends to a maximum.

  • You cannot run a cyclic engine that converts heat entirely into work
  • Similar for perpetual motion — Clausius makes the prohibition explicit
  • Prescription comes later: Boltzmann weights, then Shannon/Jaynes (this term)
  • Macroscopic: Carnot \(\to\) Clausius (second law, entropy named)
  • Microscopic: Maxwell, Boltzmann, Gibbs (same \(S\), counted states)
  • Information: Shannon, Jaynes (same \(H\), different job)

Information, entropy and intelligence course notebook setup

The Gibbs–Boltzmann Distribution

Same formula, three names: Boltzmann weights, Gibbs distribution, canonical ensemble.

  • \(p_i = e^{-\beta E_i}/Z\), \(\quad Z(\beta)=\sum_j e^{-\beta E_j}\)
  • \(\beta=1/k_B T\): coldness; \(T\): bath temperature
  • \(\log Z\) is the cumulant generating function of the energy

The same energy-and-occupation grammar reappears in associative memory and early deep learning.

  • Hopfield (1982): energy over binary configurations; recall as settling
  • Boltzmann machine (1985): make those weights learnable
  • Week 4: one physical bit as an equal double well (Landauer)

Coldness and Temperature

  • \(T\): the bath, the thermometer
  • \(\beta = 1/k_B T\): coldness
  • Same occupation; entropy sits in a different place

Two Representations

  • \(T\)-first: \(F = U - TS\)
  • \(\beta\)-first: \(Z(\beta)\), then \(U = -\partial_\beta\log Z\)
  • Week 5: \(\beta\) is the Lagrange multiplier

Free Energy Decomposition

Once \(p_i = e^{-\beta E_i}/Z\) is fixed, thermodynamics is bookkeeping.

  • Mean energy: \(U = \langle E\rangle = \sum_i p_i E_i\)
  • Entropy: \(S = -k\sum_i p_i \log p_i\)
  • Helmholtz free energy: \(F = U - TS = -kT\log Z\)
  • Prescription: \(p_i = e^{-\beta E_i}/Z\)
  • Accounting: \(F = U - TS\)

Thermodynamic Bath and Schottky’s Anomaly

A thermodynamic bath is the large system that justifies the canonical ensemble: it fixes \(T\) and exchanges energy with a small system.

  • Two-state system: \(E\in\{0,\varepsilon\}\)
  • Heat capacity: \(C = \partial U/\partial T\) peaks at intermediate \(T\)
  • Schottky’s anomaly: both states populated; maximal thermal response
  • Bath: why the canonical ensemble exists
  • Two-state system: Schottky peak in heat capacity
  • Entropic reading of Schottky: not today

Finite Time Costs More Than \(\Delta F\)

Equilibrium thermodynamics gives a prescription for reversible work. Real protocols take time.

  • Quasi-static: \(W = \Delta F\)
  • Finite time: extra dissipation
  • Length comes in week 6

Define This Week

  • How was entropy discovered?
  • What is the Gibbs–Boltzmann distribution?
  • Energy and entropy?
  • What is a thermodynamic bath?
  • What is Schottky’s anomaly?

After This Lecture

  • Next: Shannon \(H\) and \(Z\) as generating function
  • LLM probe: is free energy a bound or a recipe?

Further Reading

  • Boltzmann machines (optional colour) of Ackley et al. (1985)

  • Hopfield networks (optional colour) of Hopfield (1982)

  • Why Stochastic Thermodynamics of Machine Learning? of Welling et al. (2026)

  • Chapter 3 of Welling et al. (2026)

  • Chapters 1–4 of Callen (1985)

  • Chapters 5–6 of Callen (1985)

Thanks!

References

Ackley, D., Hinton, G.E., Sejnowski, T.J., 1985. A learning algorithm for Boltzmann machines. Cognitive Science 9, 147–169.
Callen, H.B., 1985. Thermodynamics and an introduction to thermostatistics, 2nd ed. Wiley, New York.
Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 79, 2554–2558.
Welling, M., Lu, S., Holdijk, L., 2026. Generative AI and stochastic thermodynamics: A tale of free energies. Cambridge University Press, Cambridge, U.K.