Grouping and Maxwell’s Demon

Neil D. Lawrence

FW26, William Gates Building

Define This Week

  • Questions; MacKay’s balance; die grouping \(H=H(w)+\sum_i w_i H(p_i)\)
  • Gaussian vs Maxwell: \(H\) without a measure \(m\)
  • The demon: compartments and two ledgers
  • Worksheet 3: where can the entropy sit?

This Session

  • Quiz 2 (ten minutes); collect Worksheet 2
  • Questions / MacKay / grouping; then the missing measure
  • Compartments and two ledgers
  • Worksheet 3 after the fork

After This Lecture

  • Worksheet 2 due today, before the quiz
  • Worksheet 3 released (due 17 November, with Quiz 3)

Quiz 2

  • Paper; no notes, no network, no LLMs
  • Ten minutes
  • Worksheet 2 in as you sit

Questions, a Balance, and a Die

The Information Game

The Information Game

One player has chosen an object from a known finite list. The other may ask only yes/no questions.

  • Goal: name the object
  • Cost: number of questions (worst case, or on average)
  • Optimal play halves the remaining possibilities each time

Bits as Balanced Questions

A fair yes/no answer distinguishes two equally likely cases — one bit.

  • \(N\) equally likely objects \(\Rightarrow\) worst case \(\lceil\log_2 N\rceil\) bits
  • Twenty questions reach about a million hypotheses (\(2^{20}\))
  • Unequal priors: best questions still split probability as evenly as possible

The same idea, with a different instrument, comes next — MacKay’s balance.

MacKay’s Balance and Balls

Twelve Balls, One Odd

MacKay’s weighing problem, Exercise 4.1 (MacKay, 2003): twelve balls; exactly one is odd; it may be heavier or lighter.

  • Instrument: a two-pan balance
  • Task: identify the odd ball and whether it is heavy or light

How Should We Proceed?

We have a finite list of hypotheses and a physical instrument. How should we think?

  • What, exactly, do we need to name?
  • What would count as a useful use of the balance?
  • When would we know we were done?

Information at the Start

Maths. Before any weighing, how many hypotheses?

  • 12 choices for which ball is odd
  • \(\times\) 2 directions (heavy or light)
  • \(= 24\) equally likely states of the world

Try Two Outcomes

Let’s Try Two Outcomes

Treat the balance as the information game: left heavy or right heavy only.

  • Two readings \(\Rightarrow\) at most \(\log_2 2\) bits per use
  • Mechanism next: what we actually put on the pans
  • Maths stays on its own slides — the two accounts can disagree

Maths: Two Readings

Maths. Start from \(\log_2 24\). Each use returns at most \(\log_2 2\).

\[ \log_2 24 \approx 4.58,\qquad 2^4 = 16 < 24 \le 32 = 2^5 \]

  • Four uses cannot name 24 hypotheses
  • Five uses are not ruled out, if each use really returns one bit
  • That ``if’’ is mechanism, not maths

Mechanism: 6 vs 6

Mechanism. Put six against six.

  • The pans will tip — the odd ball is on one side
  • Follow the down pan, as if the odd ball were heavier
  • Six balls remain under that story

Maths: What 6 vs 6 Bought

Maths. Score the same physical act on the 24 hypotheses.

  • 24 \(\to\) 12 remaining hypotheses: exactly \(\log_2 2\) bits
  • Zero balls eliminated — each ball is still possibly odd
  • The bit went into a coupling of group and direction

Mechanism: 3 vs 3

Mechanism. Keep going as a binary search of the balls: three against three.

  • Under the heavier-side story, three balls remain
  • Same move, smaller pile
  • We are still not asking about light versus heavy

Mechanism: Three Left

Mechanism. Three balls remain. We cannot put one and a half on a pan.

  • 2 vs 2 needs a fourth ball
  • Borrow one of known normal mass
  • Then 1 vs 1 on whatever the 2 vs 2 leaves

Maths: The Known Ball Taxes the Bit

Maths. Three remaining heavier-suspects; pans are 2 vs 1+known.

\[ \left(\tfrac{2}{3},\tfrac{1}{3}\right) \quad\Rightarrow\quad H = h_2\!\left(\tfrac{2}{3}\right) \approx 0.92 < 1 \text{ bit} \]

  • The even split was \(\log_2 2\); the pans cannot make it
  • Padding with a known ball is a mechanism patch
  • Information gain falls because the probabilities are no longer equal

Mechanism: 1 vs 1

Mechanism. If two suspects remain, weigh them against each other.

  • Now the split can be even
  • Under the heavier story, this names the ball
  • We still have not asked light versus heavy

Mechanism: Lighter or Heavier?

Mechanism. We have a candidate ball. Can we name the direction?

  • Weigh the candidate against a known normal
  • Two outcomes name heavy versus light — if it tips
  • If it balances, our heavier-side story was wrong

Two Accounts, One Experiment

Keep the columns separate.

  • Maths: \(\log_2 24\) to close; at most \(\log_2 2\) per two-outcome use
  • Mechanism: 6 vs 6, 3 vs 3, 2 vs 2 with a known, 1 vs 1, then direction
  • The known ball, and the unused ``level’’ reading, are mechanism — they change the gain

Three Readings

Aha: It Balances Three Ways

The pans also sit level. Left heavy, right heavy, or balance.

  • The instrument was never a yes/no question
  • 6 vs 6 made the third reading impossible
  • That was a mechanism choice that threw away a symbol

Maths: Three Readings

Maths. Each use now returns at most \(\log_2 3\).

\[ \log_2 3 \approx 1.58,\qquad 3^2 = 9 < 24 \le 27 = 3^3 \]

  • Two uses cannot name 24 hypotheses
  • Three uses can, if each use really returns \(\log_2 3\)
  • Same ``if’’ as before: mechanism must realise the split

Mechanism Must Collect \(\log_2 3\)

Mechanism. To earn \(\log_2 3\), all three readings must be possible and near equal.

  • First weighing: 4 vs 4, not 6 vs 6 — so level can happen
  • A known ball, or emptying a reading, is again a tax
  • Optimal play keeps the three readings as near equiprobable as possible

Doable in Three — Strategy Deferred

The counting argument shows three weighings must be enough in principle.

  • We do not give the sequence of groupings today
  • Info gained and mechanism are separate ledgers — score both
  • Same moral as twenty questions: split the prior evenly, now three ways

Coarse-Graining a Die

A fair die has \(H(X)=\log_2 6\approx 2.58\) bits.

  • First cut: odd or even?
  • Second cut: is it a six?
  • Same total either way — different prices for the first answer

Odd or Even: Exactly One Bit

Groups \(\{1,3,5\}\) and \(\{2,4,6\}\) are equiprobable.

\[ H(G_{\mathrm{parity}})=1 \text{ bit}, \qquad H(X\mid G_{\mathrm{parity}})=\log_2 3 \approx 1.585 \text{ bits} \]

  • Group label: fair coin — one bit
  • Inside either group: three equally likely faces
  • Total: \(1+\log_2 3=\log_2 6\)

“Is It a Six?”: Not One Bit

A yes/no question is one bit only when the two answers are equiprobable.

\[ w=\Bigl(\tfrac16,\tfrac56\Bigr), \qquad H(G_{\mathrm{six}})=h_2\Bigl(\tfrac16\Bigr)\approx 0.65 \text{ bits} \]

  • Yes (\(X=6\)): no residual uncertainty
  • No: five faces left, \(H=\log_2 5\)
  • Remaining: \(\tfrac56\log_2 5\approx 1.93\) bits
  • Total again \(\approx 0.65+1.93=2.58=\log_2 6\)

Same Total, Different First Cut

However you slice the questions, you pay \(\log_2 6\) in total.

  • Odd/even first: pay \(1\), then \(\log_2 3\)
  • Six-or-not first: pay \(\approx 0.65\), then \(\approx 1.93\)
  • Coarse-graining keeps only the group term and throws away the rest

Grouping Composition

Uncertainty splits into stages: which group, then where inside the group.

\[ H(X)=H(w)+\sum_i w_i\, H(p_i) \]

  • \(w_i\): probability of group \(i\)
  • \(p_i\): distribution inside group \(i\)
  • First term: cost of the coarse label; second: average residual detail

Reverse Direction

Learning the face determines the group for free. What is left to find after the group?

  • \(H(G\mid X)=0\): the coarse label is redundant given \(X\)
  • Information extracted by the group: \(I(X;G)=H(G)\)
  • Information still to find: \(H(X\mid G)\)

Why the Total Is Invariant

Entropy is the uncertainty that is consistent no matter how you decompose a question into stages.

  • Any partition: \(H(w)+\sum_i w_i H(p_i)\) returns the same \(H(X)\)
  • That consistency is special to the Shannon form (Khinchin; later Baez–Fritz–Leinster)
  • Pointer only today — grouping is Shannon’s third requirement in 1948; the axioms stay parked

Grouping Is a Choice of Abstraction

Keeping \(\{1,2,3\}\) versus \(\{4,5,6\}\) and refusing the face is the same move as a macrostate.

  • The group label is what we keep; \(\sum_i w_i H(p_i)\) is what we refuse
  • Macroscopic variables: extensive \(N,V,E,S\); intensive \(T,P\)
  • We cannot follow Laplace’s curves, so we keep a few numbers

Gaussian, Maxwell–Boltzmann, and the Missing Measure

Entropy of a Gaussian Velocity Histogram

Equal-width bins of one Cartesian component. Discrete \(H\) is a sum over masses.

  • \(p_i = n_i/N\) from equal-\(\Delta v\) bins of \(v_x\sim\mathscr{N}\left(0,\sigma^2\right)\)
  • \(H=-\sum_i p_i \log p_i\) depends only on those masses
  • Analytic \(h=\tfrac12\log(2\pi e \sigma^2)\); the histogram tracks \(h-\log\Delta\)

Histograms as Riemann Sums

A histogram replaces \(f(x)\) by a piecewise-constant function: height \(f(x_i)\) on each bin of width \(\Delta\).

  • Bin mass \(p_i = \int_{\text{bin } i} f(x)\,\text{d}x \approx f(x_i)\Delta\) (midpoint rule)
  • \(\sum_i g(x_i)\Delta \to \int g(x)\,\text{d}x\) as \(\Delta\to 0\)
  • Used twice: \(\sum_i f_i\Delta \approx 1\) and \(-\sum_i f_i\log f_i\,\Delta \approx h\)
  • So \(H \approx h - \log\Delta\): the \(-\log\Delta\) comes from the \(\Delta\) inside the log

Pushed Bins Keep the Same Entropy

The same cells, written on the speed axis: masses unchanged, widths change.

  • Equal-volume shells in 3D velocity: equal width in \(u=s^3\)
  • Push the edges to \(s=u^{1/3}\); \(\Delta s_i\) now grows then shrinks
  • \(H\) is a functional of the masses — it does not see the widths

The Continuous Limit Drops \(\log\Delta\)

As the bins shrink, discrete \(H\) tracks \(-\log\Delta\) to infinity. Differential \(h\) is what remains.

  • \(H(X^\Delta)\approx h-\log\Delta\)
  • \(-\log\Delta\to+\infty\) as \(\Delta\to 0\)
  • Passing to a density throws that term away

Entropy of the Maxwell Speed Law

The speed \(s=|\mathbf{v}|\) is a different random variable, with a different \(h\).

  • \(f(s)=4\pi s^2(2\pi\sigma^2)^{-3/2}e^{-s^2/2\sigma^2}\)
  • \(h(s)=\log(\sigma\sqrt{2\pi})+\gamma-\tfrac12\), not \(\tfrac12\log(2\pi e\sigma^2)\)
  • Why is this number not the Gaussian’s \(h\)?

Two Discretisations of the Same Speeds

Equal \(\Delta s\) is a different partition from the pushed equal-volume cells. Discrete \(H\) changes.

  • Same samples \(s=|\mathbf{v}|\), two partitions
  • Pushed: equal 3D volume (\(u=s^3\)); uniform: equal \(\Delta s\)
  • The discretisation is the measure

Bin Widths Are a Radon-Nikodym Derivative

The ratio of widths is the discrete Jacobian. It converts one \(h\) into the other.

  • Same cells: \(H\) is shared; \(h\approx H+\sum_i p_i\log\Delta_i\)
  • \(\Delta s_i/\Delta u_i \to |{\mathrm{d}s}/{\mathrm{d}u}|=1/(3s^2)\)
  • Continuum name: Radon-Nikodym derivative between the two measures

Jaynes: Promote the Bins to a Measure

The well-defined continuum object is relative entropy against a declared \(m\).

\[ h_m(f)=-\int f\log\frac{f}{m}\,{\mathrm{d}}x=-\text{KL}\left( f\,\|\,m \right) \]

  • \(m\) is Jaynes’ limiting density of discrete points — the binning, written as a density
  • Uniform \(m\) in \(v_x\) recovers ordinary \(h(v_x)\)
  • Uniform \(m\) in 3D velocity is \(m(s)\propto s^2\) in speed

Maxwell’s Demon, Compartments, Two Ledgers

Maxwell’s Demon

Velocity-bin entropy:

Detailed Balance Across the Membrane

What Does the Membrane Do?

The JavaScript demon is a selective membrane: one speed threshold \(v_\star\), opposite rules on each side.

  • Left \(\to\) right: pass only if \(v > v_\star\) (hot)
  • Right \(\to\) left: pass only if \(v \le v_\star\) (cold)
  • Elastic collisions remix velocities; the membrane only gates crossings

Steady State \(=\) Balanced Fluxes

In a steady state the number of balls leaving left equals the number leaving right — detailed balance of membrane crossings.

  • Assume each side thermalises to a 2D Maxwellian at its own \(T_L\), \(T_R\)
  • Particle flux \(L\to R\): only the hot tail, \(v>v_\star\)
  • Particle flux \(R\to L\): only the cold body, \(v\le v_\star\)
  • Steady state: \(\Phi_L^{\mathrm{hot}}(n_L,T_L)=\Phi_R^{\mathrm{cold}}(n_R,T_R)\)

Temperatures Split; Entropy Falls

For the simulation budget, particle-flux balance at equal \(N\) yields \(T_L\approx 1.9\), \(T_R\approx 23\) (\(k_B=1\)) — a large split.

  • Unconstrained equilibrium: one \(T_0=\langle\mathrm{KE}\rangle\), entropy \(S_0\)
  • Membrane steady state: \(S_{\mathrm{ss}}<S_0\) (here \(\Delta S\approx -24\) nats at \(N=39\))
  • The gas alone looks like a second-law violation

Maxwellians Are Only an Approximation

Particle balance and energy balance cannot both hold for two Maxwellians — the true steady state depletes velocity tails.

  • Hot crossings carry more energy per ball than cold crossings
  • Equal number fluxes still leave a net energy flux under Maxwellians
  • Collisions rebuild the tails; the membrane continually sculpts them
  • Local-\(T\) picture still predicts the direction and an entropy deficit

Where Did the Entropy Go?

The membrane implements Maxwell’s sorting policy. The gas entropy falls; Clausius is not repealed.

  • Prescription: the velocity gate (who may cross)
  • No-go: you cannot harvest that \(\Delta S\) for free in a cycle
  • Next: Szilard priced measurement; Landauer priced erasure; Bennett (1982) made the erasure resolution

The Demon Sorts Groups

Hot versus cold is a coarse label — this lecture’s grouping, now with a policy.

  • Group \(G\): pass / no-pass under the membrane rule (tail vs body)
  • Within group: collisions try to restore a Maxwellian \(\pi_G\)
  • Local-equilibrium assumption: mixing inside each side is fast vs crossings

Lumpability (Pointer)

When do the groups themselves form a Markov process?

  • If exit rates agree for every microstate in a group \(\Rightarrow\) coarse chain is Markov (lumpable)
  • Otherwise coarse-graining leaves memory — projection operators (Zwanzig/Mori), not this course’s core
  • Demon membrane: approximately lumpable only while local equilibrium holds

Further Reading

  • Exercise 4.1 and the discussion on pp. 69–70 of MacKay (2003)

  • limiting density of discrete points of Jaynes (1963)

Thanks!

References

Jaynes, E.T., 1963. Information theory and statistical mechanics, in: Ford, K.W. (Ed.), Brandeis University Summer Institute Lectures in Theoretical Physics, Vol. 3: Statistical Physics. W. A. Benjamin, Inc., New York, pp. 181–218.
MacKay, D.J.C., 2003. Information theory, inference and learning algorithms. Cambridge University Press, Cambridge, U.K.