Week 4: Grouping and Maxwell’s Demon

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

Worksheet 2 is due and the session starts with Quiz 2. Then the Week 3 overflow: questions, MacKay’s balance, die grouping. Gaussian versus Maxwell–Boltzmann makes the missing measure visible. Maxwell’s demon as compartments and two ledgers; the resolution families are Worksheet 3.

Worksheet 2 is due at the start of this session. Quiz 2 occupies the first ten minutes from 10:05 — Weeks 2–3 as delivered (ensemble, the heat thread through Joule, Helmholtz on the board, Shannon \(H\) versus Boltzmann \(S\)). Collect the worksheets as they sit. After the quiz the three composition wrappers: questions–MacKay–die, Gaussian–Maxwell missing measure, then the demon as compartments and two ledgers. The fork is named, not resolved; Worksheet 3 is released. Shannon axioms, HMC, \(Z\), KL, Schottky and finite time stay parked; Szilard, Landauer and Ellis wait for Worksheet 3.

Define This Week

• Questions; MacKay’s balance; die grouping \(H=H(w)+\sum_i w_i H(p_i)\)
• Gaussian vs Maxwell: \(H\) without a measure \(m\)
• The demon: compartments and two ledgers
• Worksheet 3: where can the entropy sit?

This Session

Time plan. The slot is 10:05–11:55 (110 minutes, not 120). Quiz 2 is paper, as Quiz 1 was. Protect grouping, the missing-measure figures, and Maxwell’s sentence; stop MacKay after two-versus-three readings if the clock is tight. Do not open Szilard or Landauer.

After This Lecture

Worksheet 2 was due at the start of this lecture, before Quiz 2. Worksheet 3 is released today; due 17 November (start of lecture 6), the same morning as Quiz 3.

Quiz 2

Invigilated paper test, no notes, no network, no LLMs. Ten questions on Weeks 2–3 as delivered: Bernoulli / ensemble, the heat thread (Black through Joule and Clausius), Helmholtz \(F=U-TS\) and \(\log Z\), Shannon \(H\) versus Boltzmann \(S\). Not the information game, not MacKay’s balance, not Maxwell’s demon — those start after the quiz. Worksheet 2 is collected as they sit.

Class test timer: 10:05

Information, energy and intelligence course notebook setup

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util

cmd = install_command('pods')

%system {cmd}

cmd = install_command('notutils')

%system {cmd}

cmd = install_command('mlai')

%system {cmd}

Questions, a Balance, and a Die

The Information Game

Before writing a formula for entropy, play a game. One player thinks of an object drawn from a known finite list. The other may ask only yes/no questions. How many questions do you need in the worst case? That count is already a measure of information: it is how much uncertainty you must remove to name the object.

With a list of size \(N\) and equal prior probability on each item, an ideal strategy halves the search space at every step. After \(k\) answers you can distinguish at most \(2^k\) hypotheses, so you need \[ k \ge \lceil\log_2 N\rceil \] questions in the worst case. Twenty questions reach a million possibilities: \(2^{20} = 1\,048\,576\). A fair yes/no answer therefore carries one bit of information. That is Shannon’s unit before Shannon’s formula: information as the number of balanced binary distinctions required to resolve uncertainty.

If the prior is not uniform, the best question is no longer a literal half-split of the count of remaining objects. It is a half-split of their probability. The Shannon entropy \(H = -\sum_i p_i\log_2 p_i\) is the information budget, not the question count. The optimal expected number of yes/no questions \(L\) satisfies \(H\le L<H+1\); equality \(L=H\) holds only when every \(p_i\) is a power of \(1/2\). The lecture’s own die is the exhibit: \(H=\log_2 6\approx 2.585\) bits, but a Huffman strategy needs \(8/3\approx 2.667\) questions on average. We will meet the formula next; the game already shows what the number is counting.

MacKay’s Balance and Balls

David MacKay opens the compression half of Information Theory, Inference, and Learning Algorithms with a weighing puzzle, Exercise 4.1 (MacKay, 2003). State the puzzle first. Do not yet say that the instrument is ternary. The information-game habit is to ask how we should proceed — what we must name — and then to try the instrument as if it had two readings, like a yes/no question. Keep the mechanism (what we put on the pans) on a different slide from the maths (how many bits that use can return). Only after that sequence do we notice that the balance has a third reading.

Figure: A two-pan balance and twelve identical balls.

You are given twelve balls, identical in appearance. Exactly one differs in weight from the others; you do not know whether that odd ball is heavier or lighter. A two-pan balance compares two equal-sized groups. Design a strategy that finds the odd ball and the direction of the defect.

Resist jumping to a particular grouping. Count first: how many states of the world. Then try the instrument as a two-outcome device, because that is the game we already know. Keep what we put on the pans separate from the information budget those pans can return.

Under a uniform prior over those twenty-four possibilities, the Shannon uncertainty we must remove is \[ H_{\mathrm{start}} = \log_2 24 \approx 4.58 \text{ bits}. \] At the end we need a single named hypothesis: \(H_{\mathrm{end}} = 0\). The deficit to close is therefore about \(4.58\) bits. That difference — start minus end — is the information the experiment must supply. We have not yet said what the balance returns.

Try Two Outcomes

A two-outcome instrument, used optimally, returns at most \(\log_2 2 = 1\) bit per measurement. The information budget then says \[ 2^4 = 16 < 24 \le 32 = 2^5, \] so five uses are not ruled out on paper and four cannot suffice. Whether a physical balance achieves that one bit is a different question: it depends on what we put on the pans.

\[ \log_2 24 \approx 4.58,\qquad 2^4 = 16 < 24 \le 32 = 2^5 \]

Write two columns if you can: left, the information budget; right, the pans. The left column does not know about equal group sizes, leftover balls, or borrowing a ball of known mass. Those constraints live on the right. The rest of this block is the right-hand column, scored afterwards by the left.

The move 6 vs 6 is the binary-search instinct. If we already knew the odd ball was heavier, it would be a perfect split of twelve balls. We do not know that. A down-left reading means ``one of the left six is heavy, or one of the right six is light’’: twelve of the original twenty-four hypotheses remain. One bit toward the 24, and no ball has been eliminated. Mechanism and maths have already come apart: the pans moved, the pile of balls did not shrink.

The information gain can be perfect for the two-outcome model — an even split of 24 — while the mechanism fails to name a set of six suspect balls. What we carry forward is six maybe-heavy and six maybe-light. If we now pretend it was heavier and keep only the down pan, we have thrown away half of the remaining hypotheses by assumption, not by measurement.

This is the sequence people sketch on the board: 6 vs 6, then 3 vs 3. It is the right mechanism for a different puzzle — twelve balls, one heavier. Hold that sequence, because the next move is where the pans refuse to cooperate.

With three suspects we have two mechanical choices. Weigh 1 vs 1 and leave one off: the left-off ball is the odd one if the pans balance. That reading is not in the two-outcome story we are trying to tell. Or weigh 2 vs 2 by padding with a ball already known to be normal: A,B against C plus a normal. The pans are now full, so in the heavier-only story they must tip, and we have not used a third symbol. We have changed the split.

\[ \left(\tfrac{2}{3},\tfrac{1}{3}\right) \quad\Rightarrow\quad H = h_2\!\left(\tfrac{2}{3}\right) \approx 0.92 < 1 \text{ bit} \]

If A,B,C are the three remaining heavier-suspects and N is known normal, the reading A,B down leaves two hypotheses and has probability \(2/3\); C,N down names C and has probability \(1/3\). Binary entropy \(h_2(2/3)\approx 0.92\) bits. The deficit \(1-0.92\) is exactly the price of the mechanical constraint ``equal numbers on the pans, and no third reading’’. We did not lose those bits because the mathematics of two outcomes changed. We lost them because the mechanism could not realise an equiprobable two-way split of three objects.

1 vs 1 of two remaining heavier-suspects returns the missing bit: the down pan is the odd ball, if the heavier-side story is true. The sequence of acts was 6 vs 6, 3 vs 3, 2 vs 2 with a known, then 1 vs 1. The sequence of gains was 1, 1, \(0.92\), 1 — already not four bits. And this named a ball only inside the heavier-side story.

To finish the original task we must name the direction as well as the ball. The obvious mechanism is candidate versus known normal. If the two-outcome story were true, this use would return the last bit: heavy or light. But if we followed the down pan all the way and the odd ball was actually light, the candidate may be normal — and then the pans balance. The third reading has been sitting in the mechanism whenever a known ball appears. We cannot keep it out without lying about what the device does. That is the cue to abandon the two-outcome model, not to patch it again.

The moral is not that the binary budget is wrong. It is that an information budget is silent about pans. Equal group sizes, leftover balls, and borrowed normals are constraints on the channel we actually have. Score every weighing twice: what the device did, then how many bits that split was worth.

Three Readings

A two-pan balance has a ternary alphabet. The reading “level” is not an annoyance to be designed out; it is a symbol that can carry information. The first move of the two-outcome story, 6 vs 6, sets the probability of that symbol to zero: with exactly one odd ball among the twelve, the pans cannot balance. We spent a mechanism choice to turn a three-outcome device into a two-outcome device, and then paid further mechanism taxes (the known ball) to keep the third symbol from coming back.

\[ \log_2 3 \approx 1.58,\qquad 3^2 = 9 < 24 \le 27 = 3^3 \]

A binary question yields at most one bit. If the three readings are equiprobable, one weighing contributes \[ \log_2 3 \approx 1.585 \text{ bits}. \] After \(k\) weighings the strategy tree has at most \(3^k\) leaves, so it can name at most \(3^k\) hypotheses. Compare budgets: \[ \begin{align*} 3^2 &= 9 < 24,\\ 3^3 &= 27 \ge 24. \end{align*} \] Two weighings cannot suffice no matter how clever the groupings. Three weighings can: after naming the 24 hypotheses, three paths are left over (\(27-24=3\)). Those leftover paths are not a thirteenth ball — \(13\times 2=26\) still sits under \(27\), but three weighings cannot identify an odd ball among thirteen without a known genuine. In bits, \[ 3\log_2 3 \approx 4.75 > \log_2 24 \approx 4.58. \] The gap is small. That is why the puzzle feels tight. It is the same comparison we already made with \(\log_2 2\), with a larger per-use budget.

The first weighing that realises the ternary budget puts four against four and leaves four off. Level then means the odd ball is in the four left off; a tip couples a group of four with a direction, as 6 vs 6 did, but now the third symbol is alive. We do not give the rest of the tree. The design principle is the one already present in the information game: choose each measurement so that its outcomes are as nearly equiprobable as possible under the current posterior. That maximises expected information gain per weighing and is what makes the \(3^3 \ge 24\) budget achievable in practice. Adding a known-normal ball, or arranging a weighing that can never balance, is the same kind of mechanism tax we already computed for \(h_2(2/3)\).

We stop at the existence argument. Constructing the adaptive tree — which balls on which pan at each step — is MacKay’s exercise, and solving it is the best way to feel what “informative experiment” means. What the lecture owns is the split between ledgers: \(\log_2 24\) versus \(\log_2 2\) or \(\log_2 3\) is the information account; 6 vs 6, 4 vs 4, and padding with a known ball are the mechanical account. The two are not the same, and the known-ball tax is the exhibit.

Coarse-Graining a Die

Take a fair six-sided die, \(X\in\{1,\ldots,6\}\) with \(p(x)=1/6\), so \[\begin{align} H(X)=\log_2 6 = 1 + \log_2 3 \approx 2.585 \text{ bits}. \end{align}\] Ask questions in two stages: first learn which group the face falls in, then resolve the face inside the group.

\[ H(G_{\mathrm{parity}})=1 \text{ bit}, \qquad H(X\mid G_{\mathrm{parity}})=\log_2 3 \approx 1.585 \text{ bits} \]

Let \(G_{\mathrm{parity}}=0\) on even faces and \(1\) on odd. Then \(w=(1/2,1/2)\), so \(H(G_{\mathrm{parity}})=1\). Conditional on the group, three faces remain, each with probability \(1/3\), so \(H(X\mid G_{\mathrm{parity}})=\log_2 3\). The stages add: \[\begin{align} H(G_{\mathrm{parity}})+H(X\mid G_{\mathrm{parity}}) = 1+\log_2 3 = \log_2 6 = H(X). \end{align}\]

\[ w=\Bigl(\tfrac16,\tfrac56\Bigr), \qquad H(G_{\mathrm{six}})=h_2\Bigl(\tfrac16\Bigr)\approx 0.65 \text{ bits} \]

Let \(G_{\mathrm{six}}=\mathbf{1}_{\{X=6\}}\). The group weights are unbalanced, \(w=(1/6,5/6)\), so the binary entropy of the first answer is \[\begin{align} H(G_{\mathrm{six}}) = h_2\Bigl(\tfrac16\Bigr) = -\tfrac16\log_2\tfrac16-\tfrac56\log_2\tfrac56 \approx 0.650 \text{ bits}. \end{align}\] That is not one bit. After a “yes” the face is known (\(H=0\)). After a “no” five faces remain (\(H=\log_2 5\)). Averaging, \[\begin{align} H(X\mid G_{\mathrm{six}}) = \tfrac16\cdot 0 + \tfrac56\log_2 5 \approx 1.935 \text{ bits}. \end{align}\] Again \(H(G_{\mathrm{six}})+H(X\mid G_{\mathrm{six}})=H(X)\). Note that “\(\ge 4\)?” on a fair die is also a balanced cut (\(\{1,2,3\}\) vs \(\{4,5,6\}\)), so it is one bit — the unbalanced example has to be chosen carefully.

Nothing about the cube stops you asking “odd?” after “low?” The question can be asked; it cannot be balanced (\(h_2(1/3)\approx 0.918\) bits). A Huffman strategy on the six faces needs \(8/3\approx 2.667\) questions on average against \(H=\log_2 6\approx 2.585\). The physical point about the balance is an instrument with a fixed alphabet. The die point is atoms that are not a power of two.

Grouping Composition

\[ H(X)=H(w)+\sum_i w_i\, H(p_i) \]

Partition the outcomes of \(X\) into groups labelled by \(i\), with group probabilities \(w_i\) and within-group distributions \(p_i\). Then \[\begin{align} H(X)=H(w)+\sum_i w_i\, H(p_i). \end{align}\] The die examples are the case check: \(H(w)\) is the entropy of the coarse question; \(\sum_i w_i H(p_i)\) is what remains once the group is known. If \(G=g(X)\) is the group label, this identity is exactly the chain rule \[\begin{align} H(X)=H(G)+H(X\mid G), \end{align}\] because \(G\) is a function of \(X\) (so \(H(G\mid X)=0\)) and \(H(X\mid G)=\sum_i w_i H(p_i)\).

In the reverse direction, observing \(X\) fixes \(G\) with no leftover uncertainty. The mutual information \(I(X;G)=H(G)-H(G\mid X)=H(G)\) is exactly the price of the coarse question. After you know the group, the remaining work is \(H(X\mid G)\) — which of the fine outcomes inside that group. Coarse-graining keeps \(H(G)\) as your description and discards \(H(X\mid G)\): \[\begin{align} \text{information lost by coarse-graining} = H(X\mid G) = \sum_i w_i\, H(p_i). \end{align}\]

The die paid \(\log_2 6\) whichever cut came first. That invariance under regrouping is not an accident of base-two logarithms; it is part of what characterises Shannon entropy among candidate uncertainty measures (Shannon’s third requirement in 1948, (Shannon, 1948); (Khinchin, 1957)). Categorical accounts make the same demand precise (Baez, Fritz and Leinster). The axioms themselves stay parked. For this lecture it is enough that staged questions add, so coarse-graining has a well-defined price.

Grouping Is a Choice of Abstraction

The idealised information game fails when we turn to the mechanism. But we do have some control. We can coarse-grain, and reduce the search space. For example, the low/high grouping is a choice of abstraction. We keep a two-valued group label and throw away which of the three faces occurred. Coarse graining in thermodynamics can be seen similarly. We cannot track Laplace’s curves for every molecule, instead we keep macroscopic numbers. Those macroscopic numbers split into extensive quantities that scale with the system (e.g. \(N\), \(V\), energy, entropy) and intensive quantities that do not (e.g. temperature, pressure). The grouping property constrains the form of \(H\); it is not that the form of \(H\) creates the grouping. Accounting for information works through the groupings: the macrostate is the group label \(G\), its entropy is \(H(w)\), and the unresolved motion inside it is \(\sum_i w_i H(p_i)\).

Gaussian, Maxwell–Boltzmann, and the Missing Measure

import numpy as np

rng = np.random.default_rng(0)
n_velocities = 10000
velocity_sigma = 1.0
n_histogram_bins = 40
velocities = rng.normal(0.0, velocity_sigma, size=(n_velocities, 3))
speed = np.linalg.norm(velocities, axis=1)
velocity_x = velocities[:, 0]


def discrete_entropy(counts):
    n = counts.sum()
    p = counts[counts > 0] / float(n)
    return -np.sum(p * np.log(p))


def gaussian_differential_entropy(sigma):
    return 0.5 * np.log(2.0 * np.pi * np.e * sigma**2)


def maxwell_speed_pdf(s, sigma):
    return (
        np.sqrt(2.0 / np.pi)
        * (s**2 / sigma**3)
        * np.exp(-s**2 / (2.0 * sigma**2))
    )


def maxwell_speed_differential_entropy(sigma):
    return np.log(sigma * np.sqrt(2.0 * np.pi)) + np.euler_gamma - 0.5


def speed_range_max(s, p_hi=0.995):
    return float(np.quantile(s, p_hi))


def equal_delta_speed_edges(s_max, n_bins):
    return np.linspace(0.0, s_max, n_bins + 1)


def equal_volume_speed_edges(s_max, n_bins):
    u_edges = np.linspace(0.0, s_max**3, n_bins + 1)
    return u_edges ** (1.0 / 3.0)

Entropy of a Gaussian Velocity Histogram

Histograms as Riemann Sums

Approximating a density by its value at the centre of each bin is a piecewise-constant (midpoint-rule) approximation to the integral. We call the corresponding sums Riemann sums. Writing \(p_i\approx f(x_i)\Delta\) inside the entropy gives \[\begin{align} H = -\sum_i f_i\Delta\log f_i - \log\Delta\sum_i f_i\Delta \approx h - \log\Delta, \end{align}\] with both sums converging to integrals as \(\Delta\to 0\).

If we sample isotropic three-dimensional velocities and histogram one Cartesian component with equal-width bins. The discrete Shannon entropy of the occupancy is \[\begin{align} H = -\sum_i p_i\log p_i, \qquad p_i = n_i/N. \end{align}\] Note that empty bins don’t contribute to the entropy. For a smooth density and a fine equal-width partition, \(p_i\approx f(x_i)\Delta\) and \[\begin{align} H \approx h - \log \Delta, \qquad h = \tfrac12\log(2\pi e\sigma^2). \end{align}\] The figure below reports both numbers. Note that they agree up to sampling and tail truncation. From this we can see that the differential entropy, \(h\), is the \(\Delta\)-independent remainder. It is not the entropy of the histogram.

Figure: Equal-width histogram of one Cartesian velocity component. Discrete \(H\) matches \(h-\log\Delta\); \(h\) itself is the remainder after the bin-width term is subtracted.

Pushed Bins Keep the Same Entropy

Discrete Shannon entropy depends only on the probability masses of a partition. Keep the cells and relabel the axis: the masses \(\{p_i\}\) are unchanged, so \(H\) is unchanged, while a density histogram \(p_i/\Delta_i\) changes because the widths change.

The Maxwell speed map \(s=|\mathbf{v}|\) is not itself 1–1 — equal Cartesian cubes that sit on one spherical shell are a grouping, and grouping drops entropy. The figure avoids that. Partition speed by equal Cartesian volume, i.e. equal-width bins in \(u=s^3\). The map \(u\mapsto s=u^{1/3}\) is strictly increasing on \(s\ge 0\). Pushing the bin edges stretches them by \[\begin{align} \Delta s_i \approx \frac{1}{3s_i^2}\,\Delta u. \end{align}\] Drawn as probability bars the occupancy is invariant. Drawn as a density against \(s\) the bars change shape. The two pictures share one \(H\).

Figure: Equal-volume shells in 3D velocity, written two ways. Left: probability masses. Right: the same masses as a density on the speed axis. The bins change shape; \(H\) does not.

The Continuous Limit Drops \(\log\Delta\)

Refine the equal-width partition of a Gaussian component. For each bin width \(\Delta\), \[\begin{align} H(X^\Delta) = -\sum_i p_i\log p_i \approx h - \log\Delta. \end{align}\] Plotting \(H\) against \(-\log\Delta\) gives a line of slope one, offset by the constant \(h=\tfrac12\log(2\pi e\sigma^2)\). The discrete entropy diverges as the bins shrink because each mass \(p_i\approx f\Delta\) contributes an extra \(-\log\Delta\). Differential entropy is defined to be the \(\Delta\)-independent remainder. The continuous functional therefore ignores a term that is driving to \(+\infty\). Differential entropy \(h\) can be negative because densities can exceed \(1\) (mass concentrated below the unit scale). Dropping \(-\log\Delta\) permits that; it is not the cause. And \(h\) is not a code-length bound.

Figure: Discrete entropy of an equal-width Gaussian histogram against \(-\log\Delta\). The points track \(h-\log\Delta\); \(h\) is the finite part. The continuous limit discards a term that diverges.

Entropy of the Maxwell Speed Law

The same isotropic Gaussian sample has speed \(s=|\mathbf{v}|\) with density \[\begin{align} f(s) = 4\pi s^2 \bigl(2\pi\sigma^2\bigr)^{-3/2} e^{-s^2/2\sigma^2} = \sqrt{\frac{2}{\pi}}\,\frac{s^2}{\sigma^3}\,e^{-s^2/2\sigma^2}, \qquad s\ge 0. \end{align}\] That is the Maxwell speed law. Its differential entropy is \[\begin{align} h(s)=\log\bigl(\sigma\sqrt{2\pi}\bigr)+\gamma-\tfrac12, \end{align}\] where \(\gamma\) is Euler’s constant. A single Cartesian component has \(h(v_x)=\tfrac12\log(2\pi e\sigma^2)\). For \(\sigma=1\) these are approximately \(0.996\) and \(1.419\) nats. The like-for-like comparison is three-dimensional velocity against speed: \(h(\mathbf{v})=3h(v_x)\approx 4.257\) nats, and \[\begin{align} h(\mathbf{v})=h(s)+\log 4\pi+\mathbb{E}[\log s^2]. \end{align}\] The \(\mathbb{E}[\log s^2]\) term is the Jacobian. The \(\log 4\pi\) term is the entropy of direction, which the speed map throws away — a grouping, not a change of measure. An equal-\(\Delta s\) histogram of the sampled speeds tracks \(h(s)-\log\Delta s\), just as the Gaussian histogram tracked \(h(v_x)-\log\Delta v\).

Figure: Equal-width histogram of speed from the same 3D Gaussian sample. Discrete \(H\) tracks \(h(s)-\log\Delta\). The Maxwell \(h(s)\) is not the Gaussian \(h(v_x)\).

Two Discretisations of the Same Speeds

These are different collections of masses \(\{p_i\}\), so they have different discrete entropies. Uniform bins in \(s\) are Lebesgue measure on the speed axis. Equal-volume bins are Lebesgue measure on \(\mathbb{R}^3\), expressed in the speed coordinate. Choosing a discretisation is choosing a measure.

Figure: The same speed samples, two partitions, two values of discrete \(H\). Left: equal 3D volume, pushed to the speed axis. Right: equal \(\Delta s\). The discretisation is the measure.

Bin Widths Are a Radon-Nikodym Derivative

For a partition that is not equal-width, the bin-width correction is no longer a single \(\log\Delta\). Probability conservation in cell \(i\) still reads \(p_i \approx f\,\Delta_i\), so \[\begin{align} H = -\sum_i p_i\log p_i \approx h - \sum_i p_i\log\Delta_i, \qquad h \approx H + \sum_i p_i\log\Delta_i. \end{align}\] Take the equal-volume cells of the previous snippets, written once in \(u=s^3\) and once in \(s\). The masses \(\{p_i\}\) (hence \(H\)) are the same. The two differential entropies differ by the average log-ratio of widths, \[\begin{align} h(s)-h(u) \approx \sum_i p_i\log\frac{\Delta s_i}{\Delta u_i} \to \mathbb{E}\bigl[\log\bigl|{\mathrm{d}s}/{\mathrm{d}u}\bigr|\bigr]. \end{align}\] That ratio \(\Delta s_i/\Delta u_i\) is called the discrete Radon-Nikodym derivative of Lebesgue-on-speed with respect to Lebesgue-on-volume. In the continuum limit it becomes the Jacobian \(1/(3s^2)\). Probability conservation \(f_s(s)\,{\mathrm{d}}s = f_u(u)\,{\mathrm{d}}u\) is the same statement.

The Maxwell speed density relative to three-dimensional Lebesgue is the shell factor \(4\pi s^2\). Equivalently, the pushforward of Lebesgue measure on \(\mathbb{R}^3\) has density \(4\pi s^2\) in \(s\): \[\begin{align} f_s(s)\,{\mathrm{d}}s = f(\mathbf{v})\,{\mathrm{d}}^3v, \qquad \frac^3v}s} = 4\pi s^2. \end{align}\] The Gaussian \(h(v_x)\) and the Maxwell \(h(s)\) disagree because they are different coordinates and because each \(h\) is taken against a different undeclared Lebesgue measure.

Figure: Width ratio of the pushed equal-volume cells, against the Jacobian \(1/(3s^2)\). The average log-ratio converts \(h(u)\) into \(h(s)\). That ratio is the discrete Radon-Nikodym derivative.

Jaynes: Promote the Bins to a Measure

\[ h_m(f)=-\int f\log\frac{f}{m}\,{\mathrm{d}}x=-\text{KL}\left( f\,\|\,m \right) \]

Differential entropy \(h=-\int f\log f\,{\mathrm{d}}x\) pretends the coordinate measure is part of the data. The histogram arithmetic showed that it is not: \(H\) is a functional of masses, and the passage to a density subtracts \(\sum_i p_i\log\Delta_i\), whose continuum limit is the log of a reference density \(m\). Jaynes called \(m\) the limiting density of discrete points — the binning measure, promoted to a density (Jaynes, 1963).

Once \(m\) is written down, \[\begin{align} h_m(f) = -\int f\log\frac{f}{m}\,{\mathrm{d}}x = -\text{KL}\left( f\,\|\,m \right). \end{align}\] Taking \(m\) uniform (Lebesgue) in the Cartesian component \(v_x\) recovers the ordinary Gaussian remainder \(h(v_x)\). Taking \(m\) uniform in three-dimensional velocity, and writing it in the speed coordinate, gives \(m(s)\propto s^2\) — the shell volume that had been hiding in the Jacobian. The two Maxwell–Boltzmann entropies disagreed because each \(h\) used a different undeclared \(m\). Relative entropy makes the choice explicit. The Jacobian cancels inside \(\text{KL}\left( f\,\|\,m \right)\) under a common invertible map; the discrete KL has no divergent \(-\log\Delta\) term. The map \(\mathbf{v}\mapsto s=|\mathbf{v}|\) is not invertible. Equality of KL in the speed coordinate with KL in 3D velocity holds here because both \(f\) and Lebesgue on \(\mathbb{R}^3\) are isotropic, so both lose the same direction information. In general the data-processing inequality gives \(\le\), not \(=\).

This names \(m\). Maximising \(h_m\) at fixed mean energy — Jaynes’ maximum entropy principle — is a later step.

Maxwell’s Demon, Compartments, Two Ledgers

Maxwell’s Demon

Maxwell’s demon is a thought experiment described by James Clerk Maxwell in his book, Theory of Heat (Maxwell, 1871) on page 308. He first described the being in an 1867 letter to Tait; Thomson named it a “demon” in 1874.

But if we conceive a being whose faculties are so sharpened that he can follow every molecule in its course, such a being, whose attributes are still as essentially finite as our own, would be able to do what is at present impossible to us. For we have seen that the molecules in a vessel full of air at uniform temperature are moving with velocities by no means uniform, though the mean velocity of any great number of them, arbitrarily selected, is almost exactly uniform. Now let us suppose that such a vessel is divided into two portions, A and B, by a division in which there is a small hole, and that a being, who can see the individual molecules, opens and closes this hole, so as to allow only the swifter molecules to pass from A to B, and only the slower ones to pass from B to A. He will thus, without expenditure of work, raise the temperature of B and lower that of A, in contradiction to the second law of thermodynamics.

James Clerk Maxwell in Theory of Heat (Maxwell, 1871) page 308

He goes on to say:

This is only one of the instances in which conclusions which we have drawn from our experience of bodies consisting of an immense number of molecules may be found not to be applicable to the more delicate observations and experiments which we may suppose made by one who can perceive and handle the individual molecules which we deal with only in large masses

Figure: Maxwell’s demon was designed to highlight the statistical nature of the second law of thermodynamics.

Velocity-bin entropy:

New BallPauseSkip 1000sHistogram

Figure: Maxwell’s Demon. The demon decides balls are either cold (blue) or hot (red) according to their velocity. Balls are allowed to pass the green membrane from right to left only if they are cold, and from left to right only if they are hot. The displayed entropy is the Shannon entropy of the velocity histogram (a coarse-grained proxy, not full thermodynamic entropy).

Maxwell’s demon allows us to connect thermodynamics with information theory (see e.g. Hosoya et al. (2015);Hosoya et al. (2011);Bub (2001);Brillouin (1951);Szilard (1929)). Landauer (1961) described a fundamental connection between information erasure and energy consumption.

Detailed Balance Across the Membrane

In the simulation the membrane does not move. A ball on the left is allowed through only when its speed exceeds \(v_\star\); a ball on the right is allowed through only when its speed is at most \(v_\star\). Between crossings, elastic collisions redistribute energy, so a ball that is cold now can become hot later and then cross. The question is not whether sorting begins — it does — but where the net particle and energy flows settle.

Take equal chamber areas and a 2D ideal gas (the billiard world). On each side the velocity density is Maxwellian, \[ f(\mathbf{v}\mid\beta)=\frac{\beta}{2\pi}\exp\!\left(-\tfrac{1}{2}\beta v^2\right),\qquad \beta=\frac{1}{k_B T}, \] with mean kinetic energy \(k_B T\) per ball (two quadratic degrees of freedom). The one-way effusion flux through a unit aperture is \(\Phi_{\mathrm{tot}}=n/\sqrt{2\pi\beta}\). Those formulae assume a small aperture relative to the mean free path (Knudsen). In the billiard simulation with large balls that may not hold. Restricting to the hot or cold set gives \[\begin{align} \Phi^{\mathrm{hot}}(n,\beta) &= \frac{n}{\pi}\left[v_\star e^{-\beta v_\star^2/2}+\int_{v_\star}^\infty e^{-\beta v^2/2}\,\mathrm{d}v\right],\\ \Phi^{\mathrm{cold}}(n,\beta) &= \Phi_{\mathrm{tot}}(n,\beta)-\Phi^{\mathrm{hot}}(n,\beta). \end{align}\] Particle-number detailed balance across the membrane is \[ \Phi^{\mathrm{hot}}(n_L,\beta_L)=\Phi^{\mathrm{cold}}(n_R,\beta_R). \] Total ball number \(N=N_L+N_R\) and total energy \(E=N_L k_B T_L+N_R k_B T_R\) are conserved. For the simulation parameters (\(v_\star=3\), mean kinetic energy \(12.5\) in code units, equal populations \(N_L=N_R=N/2\) with \(N=39\)) this fixes the temperatures uniquely. The equal-\(N\) slice is a cut, not the whole story: a search over \(N_L/N\) (with \(T_L<T_R\)) finds no Maxwellian pair that balances both number and energy flux.

Figure: Left: temperatures on each side of the selective membrane from particle-flux detailed balance at equal populations (simulation energy budget). Right: entropy of the two-Maxwellian state minus the single-temperature equilibrium. The red point is the simulation threshold \(v_\star=3\).

With \(k_B=1\), equal populations, \(N=39\), \(v_\star=3\) and total energy matching the simulation’s mean kinetic energy \(12.5\), particle-flux detailed balance gives \(T_L\approx 1.94\) and \(T_R\approx 23.1\). Those temperatures are the equal-\(N\) slice. The ideal-gas Shannon entropy of a 2D Maxwellian chamber (position Stirling term plus velocity differential entropy) \[ S(N,\beta)=N\left[2+\log\frac{A}{N}-\log\frac{\beta}{2\pi}\right] \] falls by about \(24\) nats relative to the single-temperature state at the same \(N\) and \(E\) — about \(-0.63\) nats per ball. That drop is the thermodynamic content of the sorting you see on the canvas: hot accumulates on the right, cold on the left, and the joint distribution over \((\mathrm{side},\mathbf{v})\) is more concentrated than the unconstrained Maxwellian.

The compartments, kinetic layers, and two-ledger closure are this course’s construction, not a result in Leff and Rex. Worksheet 3 builds on them.

A Maxwellian is symmetric in a way the membrane is not. Conditioning on \(v>v_\star\) selects a high-energy set; conditioning on \(v\le v_\star\) selects a low-energy set. When the two number fluxes match, the energy fluxes generally do not. The true non-equilibrium steady state therefore cannot be a pair of perfect Maxwellians: the left hot tail is depleted by leakage, the right cold body is depleted by leakage the other way, and collisions continually refill those sets. The local-temperature calculation remains the right first cut — it predicts a cold left, a hot right, and a lower gas entropy — but the precise \((T_L,T_R)\) pair is an approximation to a non-Maxwellian steady state.

The selective membrane is Maxwell’s demon written as a dynamical rule. The detailed-balance calculation shows that the rule really does drive the gas toward a lower-entropy steady state. Restoring Clausius requires putting the membrane — or the memory that controls an equivalent trap door — on the thermodynamic ledger. Szilard (1929) put the cost in measurement (Szilard, 1929). Landauer (1961) priced erasure but did not apply it to the demon (Landauer, 1961). Bennett (1982) made the erasure resolution and argued that measurement can be reversible — not one continuous story (Bennett, 1982). Earman and Norton (1998–99) is one of the dissenting books. The simulation’s “velocity-bin entropy” is a coarse proxy for the same story: as sorting proceeds, the displayed histogram entropy typically drops relative to the unsorted gas.

The Demon Sorts Groups

Maxwell’s sorting rule partitions velocity space into coarse sets — “hot enough to cross” versus “cold enough to cross” — and applies different gates to each set. That is exactly this lecture’s grouping: a deterministic label \(G=g(\mathbf{v})\) plus residual uncertainty inside each label. The membrane calculation assumed each chamber stays close to a Maxwellian at its own \(T\) — local equilibrium, timescale separation between collisions and crossings. When that fails, the true steady state is no longer a pair of perfect Maxwellians (tails deplete). Information engines and demons play on the tension between moving probability among groups (\(H(w)\)) and the within-group bath term \(\sum w_i H(p_i)\) that thermodynamics prices.

The compartments, kinetic layers, and two-ledger closure are this course’s construction, not a result in Leff and Rex. Worksheet 3 builds on them. I will not mark you wrong for disagreeing with the bookkeeping; I will mark you wrong for treating it as a textbook theorem.

Kemeny–Snell lumpability is the precise condition under which a partition of a Markov chain induces another Markov chain on the groups. When it fails, the macro description needs memory of how the system entered the group. Name the issue; do not develop projection-operator formalism here.

Worksheet 3 is the coarse-graining of Maxwell: which description is legal for each compartment, and where the missing entropy is allowed to sit. Szilard, Landauer, Ellis, and the information-engine scale argument wait until that worksheet needs them.

Machine Review

A language-model review of the pre-delivery notes for the L172 Week 4 session (3 November 2026). It is not assessed. It works from the written notes only.

The grouping identity, the die arithmetic, the histogram limit, the Maxwell speed entropy and the 2D effusion fluxes check numerically. Twelve balls in three weighings is MacKay’s Exercise 4.1 and is sufficient. The two-ledger argument is now labelled as the course’s own construction.

Still open

• Likely overrun. The plan gives 25 minutes to questions, MacKay and the die, and 20 minutes to Gaussian–Maxwell through Radon–Nikodym and Jaynes. Weeks 2 and 3 both overran with less. Protect grouping and Maxwell’s sentence; stop MacKay after two-versus-three if the clock is tight (already in the speakernotes). Do not move Radon–Nikodym / Jaynes to Week 5 until after a timing rehearsal.
• Prerequisites. The course says no physics is assumed. This lecture uses Radon–Nikodym derivatives, Knudsen effusion, Kemeny–Snell lumpability and local equilibrium. Each is named. Say which are examinable and which are for colour.
• Kemeny–Snell. Named in the lumpability pointer, not in the bibliography.

Learning objectives

“Define This Week” now has four bullets: grouping, the missing measure, compartments and two ledgers, Worksheet 3. A measurable form of the grouping objective is: compute \(H(G)\) and \(H(X\mid G)\) for a given partition and interpret the residual as information discarded.

Remaining fixes

1. After a timing rehearsal, decide whether Radon–Nikodym and Jaynes open Week 5 (where MaxEnt needs \(m\)) rather than sitting in this slot.
2. Mark which of Radon–Nikodym, Knudsen, lumpability and local equilibrium are examinable.
3. Add Kemeny and Snell to the bibliography if the lumpability pointer stays.

Sources

• Lecture 4
• Lecture 3
• Lecture 2

Further Reading

• Exercise 4.1 and the discussion on pp. 69–70 of MacKay (2003)

• limiting density of discrete points of Jaynes (1963)

Thanks!

For more information on these subjects and more you might want to check the following resources.

• company: Trent AI
• book: The Atomic Human
• twitter: @lawrennd
• podcast: The Talking Machines
• newspaper: Guardian Profile Page
• blog: http://inverseprobability.com

References