Week 2: Boltzmann, Free Energy, and Entropy

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

This week we’ll map the statistical notions of entropy we reviewed in Week 1 onto the physical notions of entropy through the Gibbs–Boltzmann distribution. We’ll show how this equation can be decomposed to give the so-called free energy \(F = U - TS\). We’ll introduce the notion of a thermodynamic bath and review Schottky’s anomaly. Worksheet 1 is due and the session will start with Quiz 1.

Worksheet 1 is due at the start of this session. Quiz 1 occupies the first ten minutes: probability, elementary entropy, and Week 1 seeds. Then 110 minutes on the Gibbs–Boltzmann form, free energy, bath, Schottky, and finite time. Do not derive the occupation from MaxEnt today — that is week 5.

This Session

Time plan (120 minutes)

Quiz 1

Invigilated Moodle, own device, no notes, no network, no LLMs. Ten questions on last week’s probability and entropy review and the Boltzmann / entropy seeds pressed in Worksheet 1.

From the Seed to the Formula

Last week we introduced energy and entropy from a machine learning perspective. That allowed us to write down the Gibbs distribution, [ p_i = Z = _i e^{-E_i}. ] Today we’re going deeper into Boltzman’s physical interpretation of this equation, and the decomposition that underpins the famous equation [ U = W + TS ] where \(U=\left\langle E\right\rangle\) is total energy (or inner energy), \(F\) is the free engineer, \(T\) is temperature and \(S\) is physical entropy, i.e. \(k_B H = -k_B \sum_i p_i \log p_i\).

Information, energy and intelligence course notebook setup

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util

cmd = install_command('pods')

%system {cmd}

cmd = install_command('notutils')

%system {cmd}

cmd = install_command('mlai')

%system {cmd}

Carnot and Clausius

Sadi Carnot (1796–1832) asked, in 1824, what limits the efficiency of a heat engine. Rudolf Clausius (1822–1888) built on Carnot and Kelvin to state the second law of thermodynamics in several equivalent forms, and in 1865 he coined the name entropy for the state function that tracks irreversibility. Boltzmann and Gibbs, later in the same century, gave the microscopic count behind Clausius’s macroscopic \(S\). Shannon and Jaynes, in the twentieth century, reuse the same functional form with different operational readings. The historical chain is: engines first, then entropy as a named quantity, then statistics, then information.

Clausius did not give the Boltzmann distribution. He gave the macroscopic balance that any prescription must respect. When we write \(p_i \propto e^{-\beta E_i}/Z\) and later derive it from MaxEnt, read it as the statistical answer to a constraint Clausius already framed: fixed mean energy, maximum entropy, no perpetual motion.

The Gibbs–Boltzmann Distribution

Name the object and fix the formula. The MaxEnt derivation that earns these weights is a separate step: deriving the occupation from Lagrange multipliers before Shannon \(H\) and Jaynes would put the cart before the horse.

Write the equilibrium occupation of a discrete system with energies \(\{E_i\}\) as \[ p_i = \frac{e^{-\beta E_i}}{Z(\beta)}, \qquad Z(\beta)=\sum_j e^{-\beta E_j}. \] Physicists call \(p_i\) the Boltzmann distribution (or Boltzmann weights) and, for a system exchanging energy with a bath at fixed \(T\), the Gibbs distribution or canonical ensemble. The three names point at the same formula. The normalisation \(Z(\beta)\) is the partition function. Its logarithm \(\log Z(\beta)\) is the cumulant generating function for the energy under this exponential family: derivatives of \(\log Z\) recover the mean energy, the variance (heat capacity, up to factors of \(\beta\)), and higher cumulants. That generating-function reading is why \(Z\) is more than a normalisation constant.

Figure: Occupation of a two-state system as coldness increases. At low \(\beta\) both states are populated; at high \(\beta\) the ground state dominates.

The Gibbs–Boltzmann occupation is not only a statement about gases and magnets. Hopfield networks (Hopfield, 1982) assign an energy to every binary configuration of a recurrent net and treat recall as a descent toward low-energy states — equilibrium statistics are again Gibbs. Ackley, Hinton and Sejnowski (Ackley et al., 1985) made the weights of that energy learnable: a Boltzmann machine is an undirected model whose distribution over configurations is exactly \(p(s)\propto e^{-E(s)/T}\). The 2024 Nobel Prize in Physics, awarded to John Hopfield and Geoffrey Hinton, recognised that physical-systems reading of computation and learning. Name the lineage so the formula does not feel confined to nineteenth-century heat baths; do not divert into training algorithms.

Coldness and Temperature

The Boltzmann occupation can be written in two ways, \[ p_i = \frac{e^{-E_i/k_B T}}{Z} = \frac{e^{-\beta E_i}}{Z}, \qquad \beta = \frac{1}{k_B T}. \] The mathematics is the same, but the emphasis is different.

\(T\) is the variable of the bath. It is what a thermometer reports, and it is the intensive parameter in the Helmholtz accounting \(F = U - TS\). In that representation energy comes first and entropy is the correction: \(TS\) is the cut the second law takes from \(U\).

\(\beta\) is coldness. It is the variable conjugate to energy. In that representation entropy comes first: you maximise \(S\) (or write a generating function in \(\beta\)) and energy is the constraint. Temperature is then a derived reading, \(T = 1/k_B\beta = (\partial S/\partial U)^{-1}\).

We will use both. Weeks 1–2 keep \(T\) in view so the bath is familiar, and they already compute in \(\beta\) because that is the natural argument of \(Z\). Week 5 makes the \(\beta\)-first order honest: the Lagrange multiplier on a mean-energy constraint is coldness, and it is the natural parameter of the exponential family. Weeks 6–7 then treat \(\beta\) as a coordinate on that family. A student who still hears \(\beta\) as ``one over the thermometer’’ will miss why the geometry is written in those coordinates.

Free Energy Decomposition

Helmholtz free energy \(F=U-TS=-kT\log Z\) accounts for what remains after the entropy takes its cut: \(U\) is total energy, \(TS\) is unavailable, \(F\) is what the bath still allows you to do.

Figure: \(U\), \(S\), and \(F\) for the Worksheet 1 three-state system. Verify \(F=-\log Z/\beta\) numerically.

Welling, Lu and Holdijk

Only this year, Welling, Lu and Holdijk’s Generative AI and Stochastic Thermodynamics (GAIST, Welling et al. (2026)) was published. Although their focus is on free energy we will now see that free energy and entropy are two sides of the same thermodynamic coin. GAIST explores the shared mathematics of generative models and nonequilibrium thermodynamics are the same mathematics. Although the mathematics is broadly the same ur philosophy is slightly different. GAIST has the slogan free energy is all you need, if this course had a slogan it would more likely be “entropy is all you need.”

GAIST on Boltzmann and Free Energy

The opening essay, ``Why Stochastic Thermodynamics of Machine Learning?’’ (Welling et al., 2026), states the free-energy decomposition we use today: entropy measures the information we are missing; we subtract it from the energy and call what remains free energy, the energy still free to perform work. Maxwell’s demon appears there as a knowledge engine — not yet as Landauer’s cost. That cost is week 3.

Chapter 3 is the physics we need for LO1. Section 3.2 introduces the Hamiltonian, the microcanonical and canonical ensembles, and the passage from the partition function to \(U\), \(S\), and \(F\). Section 3.3 writes the first law and splits an energy change into heat and work: \[ \delta U = \int \delta\rho\,\mathrm{H} + \int \rho\,\delta\mathrm{H}. \] The first term is heat (\(\rho\) changes at fixed Hamiltonian); the second is work (the Hamiltonian changes at fixed \(\rho\)). That is the same accounting as \(F = U - TS\): \(TS\) is the cut taken by ignorance, \(F\) is what remains.

Read Chapter 3, not the whole book. Callen remains the undergraduate thermodynamics reference. GAIST does not discuss perpetual motion or the human bandwidth constraint; those motivations are ours.

Thermodynamic Bath and Schottky’s Anomaly

A thermodynamic bath fixes \(T\) and exchanges energy with a small system, justifying the canonical ensemble. Heat capacity \(C=\partial U/\partial T\) peaks when both states are equally populated — Schottky’s anomaly.

Figure: Mean energy and heat capacity of a two-state system. The Schottky peak appears when \(p_0\approx p_1\approx\frac12\).

Finite Time Costs More Than \(\Delta F\)

Quasi-static processes achieve reversible work \(W=\Delta F\). Finite-time driving dissipates additional energy beyond the equilibrium bound.

Figure: Cartoon of quasi-static versus finite-time driving between two equilibrium states.

Define This Week

Interpret later: how entropy is understood today; equilibrium versus non-equilibrium; the purely entropic Schottky reading. Define-stage answer to “How was entropy discovered?”: Carnot on engines; Clausius names entropy and states the second law (1865); Boltzmann and Gibbs give the statistical count; Shannon and Jaynes reuse \(H\) with different operational readings.

After This Lecture

Next week: Shannon entropy and the partition function as a generating function. Optional LLM probe: is free energy a constraint or a recipe?

Further Reading

• Boltzmann machines (optional colour) of Ackley et al. (1985)

• Hopfield networks (optional colour) of Hopfield (1982)

• Why Stochastic Thermodynamics of Machine Learning? of Welling et al. (2026)

• Chapter 3 of Welling et al. (2026)

• Chapters 1–4 of Callen (1985)

• Chapters 5–6 of Callen (1985)

Thanks!

For more information on these subjects and more you might want to check the following resources.

• company: Trent AI
• book: The Atomic Human
• twitter: @lawrennd
• podcast: The Talking Machines
• newspaper: Guardian Profile Page
• blog: http://inverseprobability.com

References