Free Energy and Shannon Entropy

This Session

No laptops; homework from Weeks 1–2 (Quiz 1 / Worksheet 1)
Heat as far as Joule before the break
Helmholtz \(F=U-TS\) on the board after the break (LO1)
Clausius after Helmholtz; Shannon / Jaynes / Wiener as a preview

Homework from Weeks 1–2

No Quiz Today

Worksheet 1 and Quiz 1 are marked. Today we spend a few minutes on what the scripts showed, then finish Week 2.

Comments already sent by email;
Quiz 1: fifteen papers from people who had Week 1; mean \(8.2/10\)
Worksheet 1: thirteen scripts; mean \(12.2/15\)

Quiz 1

Quiz 1: Where Marks Moved

Mean \(8.2/10\). Two items accounted for most of the misses.

Demon versus gremlin: about three in five missed it
Energy as a machine-learning score: about half got this wrong
Entropy as limit versus probability as prescription: one in five

Gremlin, Not Demon

Maxwell’s demon sorts. Laplace’s gremlin is the admission that complete knowledge is not available in practice.

The demon is an intelligent mechanism at a trap door
The gremlin is not a mind in the box; it is a limit on what we can know
Shannon’s \(H\), in an hour, is a calculus for that incomplete knowledge

Energy Is a Score

In machine learning, energy is a preference: lower is more probable.

\(p_i \propto e^{-\beta E_i}\): the occupation from Week 1
Quadratic loss and cross-entropy are energy-like objectives
The word energy does not, by itself, buy you a heat bath

Worksheet 1

Worksheet 1: The Habit

Curiosity ran slightly ahead of skepticism. Reflection was where the conversation was least often turned back on itself.

Mean \(12.2/15\); the dialogues did the job
Strongest work expanded after the first thread, then challenged without collapsing the chat
Weaker work ratholed, or let the model agree and stop

The See-Saw

A direct challenge is often met by sycophancy: the model agrees, and the conversation has nowhere to go.

Concede the part you accept; then ask what follows
Admitting “this may be my misunderstanding” denies the model the chance to merely please
Grasshopper is not sycophantic … the models are.

Expand, Then Return

Once a thread is established, step back. Ten turns is enough to open a second front and still close.

Early precision is good; staying there for six turns is not
State the model’s point in your prompt, then ask the adjacent question
Smaller models sometimes left more to work with than verbose ones

Who Made Entropy Uncertainty?

Clausius did not. A question most chats approached and few quite landed.

Clausius: a thermodynamic accounting identity
Uncertainty as entropy is already in the statistical tradition — Boltzmann, Gibbs, then Wiener’s “ignorance”
Shannon’s footnote is Tolman, not the cocktail-party story

Worksheet 2 Is Already Out

Same habit, different claim: is Shannon’s \(H\) Boltzmann’s \(S\), or an analogy?

Due 10:00 at the start of lecture 4 (3 November)
Open on the equivalence; then press sample space, discrete vs continuous, what \(p\) is over
Use the see-saw lesson: concede, then move. Do not rathole on the first definition.

From Heat to Helmholtz

The Story of the Lecture

Heat first. Information after Helmholtz.

Black \(\to\) Carnot \(\to\) Rumford \(\to\) air thermometer \(\to\) Kelvin \(\to\) Joule
Helmholtz on the board after the break; Clausius after Helmholtz
Shannon / Jaynes / Wiener as a preview; entropy game / MacKay next week

Joseph Black and Heat as a Quantity

Joseph Black

The Problem

Ice melts. The thermometer does not move.

Heat is entering the mixture
Temperature stays at the melting point until the ice is gone
If heat were temperature, that would be a contradiction

Heat Is Not Temperature

Temperature is the reading. Heat is what you add.

Mix two bodies of known mass and starting temperature; one final reading
Equal heat, equal mass: mercury rises more than water — different capacity
Ice melts with the thermometer stuck — heat can be hidden

Keep mechanism and maths apart. Mechanism: mix, or melt ice, or time a kettle. Maths: heat as an additive quantity independent of the thermometer. Capacity first, then latent. Students often still hear heat'' ashow hot’’.

The Ice Calorimeter

Black mixed and timed. This box weighs meltwater.

Heat leaving the sample melts ice in the jacket
You weigh what melted — that is the heat
The conversion is Black’s latent heat of fusion

Latent Heat of Steam

A kettle on a steady fire. Time is the meter.

Clock the water from ice-cold to the boil; then clock it boiling dry
Boiling dry takes about four and a half times as long
That extra heat does not move the thermometer. Black calls it latent

Sadi Carnot and the Fall of Heat

Sadi Carnot

Why Look at Heat Engines?

Steam already drives mills, mines, and looms. Nobody has a theory of it.

Watt’s engines are a British industrial fact
Engineers try higher pressure, and other fluids than steam
Carnot: is there a limit? Does the choice of fluid matter?

The Réflexions

A Heat Engine Is a Mill

Caloric falls from hot to cold. The same caloric arrives at the bottom.

Water mill: quantity of water, height of the fall; the same water arrives below
Heat engine: quantity of caloric, difference of temperature; the same caloric arrives in the condenser
Two reservoirs. Nothing is used up.

A Short Life

Carnot dies in the 1832 cholera epidemic, aged 36. The book almost vanishes with him.

Six hundred copies, paid for by the author, unread
Clapeyron recasts the argument in 1834

Count Rumford’s Cannon

Count Rumford

The Cannon

A blunt borer, a brass cylinder, a box of water, horses turning the gun.

About \(18.8\,\mathrm{lb}\) of water, starting at \(60^\circ\mathrm{F}\)
The cylinder at about \(32\) turns a minute
After two hours and thirty minutes the water boils — with no fire

Rumford’s Apparatus

Inexhaustible

Furnishing heat without limitation …

The chips do not lose a hidden store of caloric
The heat continues as long as the work continues
Rumford’s conclusion: heat is a mode of motion

Mercury, Ice, Steam, and Air

Fahrenheit and Celsius

A degree is an equal expansion of mercury. Ice and steam are the pins.

Fahrenheit (1724): mercury; ice at \(32\), steam at \(212\)
Celsius (1742): the same two pins, later numbered \(0\) and \(100\)
Neither scale is heat. Both are the expansion of one liquid

The Air Thermometer

The laboratory standard is air, not mercury. Pressure is held at the room’s.

Trap a mass of air behind mercury that is open to the atmosphere
Slide the thread, or lower the cistern, until the mercury surfaces are level
Volume is then the reading; ice and steam still pin \(0\) and \(100\)

Thomson’s Scale from the Mill

William Thomson

What a Degree Is

The pins are a century old. He is defining the degree.

Fahrenheit, Celsius, air: a degree is equal expansion of one substance
Thomson, aged 24: a degree is an equal fall of caloric — the same motive power from the same quantity of heat, anywhere on the scale
Ice remains \(0\), steam remains \(100\). He has not moved the pins. He has defined the rungs.

An Absolute Thermometer

A degree is a height of fall of conserved heat. Ice is still \(0\), steam still \(100\).

Take a unit of heat from a body at \(T\) on this scale
Let it fall to a body at \(T-1\); the same unit arrives
A perfect mill in between always gives the same motive power, whatever \(T\) is

Is There a Bottom?

In 1848 the mill has no bottom. Infinite cold is infinitely far down.

Air already suggests a wall: its volume would vanish at about \(-273^\circ\)
That number is Gay-Lussac’s expansion, not Carnot’s mill
Thomson still pins \(0\) at ice; on his mill scale you can always fall one more equal degree

Heat Is Not Destroyed

He still thinks with Carnot. Heat is conserved. The mill turns because caloric falls.

A unit of heat that leaves the hot body arrives at the cold
Motive power is the mill, not heat being used up
``The conversion of heat into mechanical effect is probably impossible, certainly undiscovered’’

Joule and the Mechanical Equivalent

James Prescott Joule

The Paddle Wheel

A Number

A definite amount of work makes a definite amount of heat.

1845, British Association at Cambridge: the paddle already gives an equivalent, still scattered
1850, Philosophical Transactions: \(772\) foot-pounds to raise \(1\,\mathrm{lb}\) of water by \(1^\circ\mathrm{F}\)
Modern, on the same scale: about \(778\) foot-pounds, or \(4.186\,\mathrm{J}\) per calorie

Not Carnot’s Question

Joule is converting work into heat. He is not running a mill between two baths.

Carnot: maximum work from a fall of conserved caloric
Joule: how much work makes how much heat
Conservation of heat cannot be the general law if the paddle is right

The Contradiction

You cannot keep Carnot’s conserved heat and Joule’s equivalent as general laws.

If heat is conserved, none of it becomes work
If Joule is right, some of it does
What can be kept: a limit that depends only on the two temperatures

Clausius and \(\mathrm{d}Q/T\)

Rudolf Clausius

Two Laws

Keep Joule’s equivalent. Keep Carnot’s limit. Drop conservation of heat.

First law: heat and work are interchangeable, at Joule’s rate
Second law: heat does not flow from cold to hot without a cost
A cycle cannot turn the heat of one bath entirely into work

The Ratio that Survives

After Joule, Carnot’s mill is a ratio of heats, not a conserved fluid.

Reversible engine: \(Q_h/T_h = Q_c/T_c\)
\(Q_h\) taken from the hot bath, \(Q_c\) delivered to the cold
Heat does not balance. Heat over temperature does

The Loop

A reversible cycle is a closed path. If it is reversible, the integral is zero.

\[\oint \frac{\mathrm{d}Q}{T} = 0 \qquad \text{(reversible cycle)}\]

Every reversible loop integrates to zero
So there is a state function \(S\) with \(\mathrm{d}S = \mathrm{d}Q_{\mathrm{rev}}/T\)

A Reversible Cycle Produces No Entropy

You return to the start. Nothing has been created.

The working substance: \(S\) is a state function, so \(\Delta S = 0\) around any cycle
The baths: the entropy that left the hot bath arrives at the cold bath
Universe: \(\Delta S = 0\). A reversible cycle produces no entropy

When the Cycle Is Irreversible

The working substance still comes home. The books do not.

Friction, a finite drop of temperature, a paddle in the mechanism
More heat is dumped to the cold bath than the reversible case
\(\displaystyle\oint \mathrm{d}Q/T < 0\), and the entropy of the universe increases

The Name

Not Uncertainty

Clausius’s \(S\) is the piece that fits the puzzle: heat divided by temperature.

Macroscopic: \(S\) from \(\mathrm{d}Q_{\mathrm{rev}}/T\); a reversible cycle produces none
Microscopic, later: Maxwell, Boltzmann, Gibbs count the same \(S\)
Information, later: Shannon, Jaynes reuse the functional form for a different job

Thomson Returns: The Scale After Joule

Thomson, 1851

The 1848 scale assumed conserved heat. After Joule that assumption is gone.

Clausius, 1850: two laws; Carnot’s ceiling without caloric conservation
Thomson, 1851: accepts Joule; heat is not a conserved fluid
What survives: a reversible engine still has a limit that depends only on the two temperatures

The Ratio, Not the Fall

In 1854 Thomson and Joule replace equal mill-degrees by a proportion of the two heats.

Reversible engine: \(\displaystyle\frac{Q_h}{T_h} = \frac{Q_c}{T_c}\)
The same identity Clausius wrote as \(\mathrm{d}S = \mathrm{d}Q_{\mathrm{rev}}/T\)
Absolute temperatures are now those for which this ratio holds

Now There Is a Bottom

Absolute zero is where a reversible engine would reject no heat.

\(T=0\) is \(Q_c=0\) for a perfect mill between \(T_h\) and the floor
The air thermometer’s \(-273^\circ\) was vanishing volume; this is vanishing rejected heat
Infinite cold is no longer infinitely far down the mill

Molecules

Maxwell–Boltzmann: Components Versus Speed

Each Cartesian velocity component is Gaussian. Speed \(v=|\mathbf{v}|\) is not.

Component: \(f(v_x)=\sqrt{m/(2\pi kT)}\,e^{-mv_x^2/2kT}\)
Speed: integrate the isotropic 3D Gaussian over angles
Jacobian \(d^3v = v^2\,dv\,d\Omega\) supplies the \(v^2\) factor

Same Physics, Different Variable

\[ f(v_x)=\sqrt{\tfrac{m}{2\pi kT}}\,e^{-mv_x^2/2kT} \qquad\text{versus}\qquad f(v)\propto v^2\,e^{-mv^2/2kT} \]

The \(v^2\) is not an energy correction — it is geometry
Each component is the quadratic \(E=mv_x^2/2\) whose expectation we split next

The Gibbs–Boltzmann Distribution

Gibbs–Boltzmann Occupation

Same formula, three names: Boltzmann weights, Gibbs distribution, canonical ensemble.

\(p_i = e^{-\beta E_i}/Z\), \(\quad Z(\beta)=\sum_j e^{-\beta E_j}\)
\(\beta=1/k_B T\): coldness; \(T\): bath temperature
\(\log Z\) is the cumulant generating function of the energy

Aside: Hopfield, Boltzmann Machines, Nobel 2024

The same energy-and-occupation grammar reappears in associative memory and early deep learning.

Hopfield (1982): fixed-weight energy; recall as settling
Boltzmann machine (1985): make those couplings learnable
Pairwise energy: \(E(\mathbf{s})=-\sum_i b_i s_i-\sum_{i<j}W_{ij}s_i s_j\)

Coldness and Temperature

\(T\): the bath, the thermometer
\(\beta = 1/k_B T\): coldness
Same occupation; entropy sits in a different place

Two Representations

\(T\)-first: \(F = U - TS\)
\(\beta\)-first: \(Z(\beta)\), then \(U = -\partial_\beta\log Z\)
Week 5: \(\beta\) is the Lagrange multiplier

Free Energy Decomposition

From Occupation to Accounting

From Gibbs distribution to energy bookkeeping.

\(p(x)=e^{-\beta E(x)}/Z\)
Equilibrium: inner energy is an expectation, \(U=\left\langle E\right\rangle\)
Algebra then defines \(F\) and \(S\)

Take the Logarithm

\[ \log p(x) = -\beta E(x) - \log Z \]

Solve for the microscopic score: \(E(x)=-k_BT\log p(x)-k_BT\log Z\)
\(1/\beta=k_BT\) converts coldness back to the bath’s temperature

Equilibrium Identifies \(U\) with \(\left\langle E\right\rangle\)

\[ U \equiv \left\langle E\right\rangle = \sum_x p(x)\, E(x) \]

\(E(x)\) is a score on configurations
\(U\) is what a calorimeter would call the energy of the system
The move is the ensemble: replace a number by an expectation

Name the Two Remaining Terms

\[ U = \underbrace{-k_BT\log Z}_{F} + T\underbrace{\bigl(-k_B\sum_x p(x)\log p(x)\bigr)}_{S} \]

Helmholtz free energy: \(F=-k_BT\log Z\)
Boltzmann entropy: \(S=-k_B\sum_x p(x)\log p(x)=k_B H\)
Hence \(U=F+TS\), or \(F=U-TS\)

Two Temperatures, One Identity

\(T\)-first: \(F=U-TS\) — energy, then entropy’s cut
\(\beta\)-first: \(F=-\beta^{-1}\log Z\), then \(U=-\partial_\beta\log Z\)
Same numbers; week 5 makes \(\beta\) a Lagrange multiplier

Three-State Check

Prescription: \(p(x)=e^{-\beta E(x)}/Z\)
Accounting: \(F=U-TS\) with \(U=\left\langle E\right\rangle\)

As Taught: Helmholtz on the Board

Occupation to accounting, written by hand. Clausius slides come after.

\(p_i = e^{-\beta E_i}/Z\); \(Z\) is Summe der Zustände
Take the log; take the expectation; \(U\) splits into entropy and available energy
Maxwell’s demon, here, is a warning about crossing coarse-graining levels — Week 4 is the treatment

The Carnot Cycle on \(T\)-\(S\)

The Fall of Caloric

Carnot compared the motive power of heat to a waterfall. He did not know whether the work was proportional to the height.

The Mill, Drawn as a Rectangle

Replace height by temperature and mass of water by entropy, and the mill is already a rectangle.

Height \(h\) \(\leftrightarrow\) temperature \(T\)
Mass of water \(\Delta m\) \(\leftrightarrow\) entropy transferred \(\Delta S\)
Fill at the top \(\leftrightarrow\) take in entropy at \(T_h\)
Loaded bucket descends \(\leftrightarrow\) isolated expansion (the working body cools; no heat transfer)
Empty at the bottom \(\leftrightarrow\) reject entropy at \(T_c\)
Empty bucket returns \(\leftrightarrow\) isolated compression
Work \(g\,\Delta h\,\Delta m\) \(\leftrightarrow\) work \(\Delta T\times\Delta S\)

The Square

The ideal Carnot cycle on a \(T\)–\(S\) diagram is a square.

Expansion at \(T_h\): entropy increases, volume increases
Isolated cool: adiabatic expansion, entropy holds
Contraction at \(T_c\): entropy comes back down (Watt’s condenser)
Isolated heat: adiabatic compression, the square closes
Work is the area: \(W=\Delta T\times\Delta S\)

The Fall Is Not the Downward Stroke

Carnot’s fall is the difference between the reservoirs. The downward adiabat is the working body cooling with no heat transfer.

Four-Stroke, and the Smaller Loop

The four-stroke is the implementation. The square is the theory. Same \(\Delta T\), same \(\Delta S\): you cut two corners.

The \(p\)-\(V\) Chart

Engineers often draw the same strokes on a pressure-volume chart. In that case it no longer looks like a square.

Two Sides of the Same Coin

Welling’s Book, This Course

They named the book for \(F\). We named the course for \(S\). The hinge is \(U\).

GAIST: A Tale of Free Energies — variational \(F\), ELBO, work and heat
This course: entropy as the no-go; \(F\) as what remains after the cut \(TS\)
Fix the inner energy \(U=\left\langle E\right\rangle\), and maximising \(S\) is minimising \(F\)

Preview: Shannon, Jaynes, Wiener

The same functional form, later, for a different job. We will try to push the jobs back together.

Shannon and Jaynes reuse the Helmholtz / Gibbs form
Shannon is discrete; Wiener is continuous
Wiener never drops the mechanism; Shannon, you just say it is a probability
Entropy game / MacKay: not reached on 27 October — next week

After Helmholtz: Not Reached

Define This Week

Heat thread: Black \(\to\) Carnot \(\to\) Rumford \(\to\) air thermometer \(\to\) Kelvin \(\to\) Joule
Helmholtz \(F=U-TS\); Clausius after; Welling reads \(F\), we read \(S\)
Shannon / Jaynes / Wiener preview; entropy game / MacKay next week

After This Lecture

Next: class test (Quiz 2), then grouping, then the demon
Worksheet 2: \(H\) vs \(S\) — identity or metaphor?

Thanks!

References

Ackley, D., Hinton, G.E., Sejnowski, T.J., 1985. A learning algorithm for Boltzmann machines. Cognitive Science 9, 147–169.

Callen, H.B., 1985. Thermodynamics and an introduction to thermostatistics, 2nd ed. Wiley, New York.

Carnot, S., 1824. Réflexions sur la puissance motrice du feu et sur les machines propres à développer cette puissance. Bachelier, Paris.

Falk, G., 1985. Entropy, a resurrection of caloric — a look at the history of thermodynamics. European Journal of Physics 6, 108–115. https://doi.org/10.1088/0143-0807/6/2/009

Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 79, 2554–2558.

Meyn, J.-P., 2024. A contemporary view on Carnot’s réflexions. Entropy 26, 1002. https://doi.org/10.3390/e26121002

Welling, M., Lu, S., Holdijk, L., 2026. Generative AI and stochastic thermodynamics: A tale of free energies. Cambridge University Press, Cambridge, U.K.