Week 4: Grouping and Maxwell’s Demon

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

Worksheet 2 is due and the session starts with Quiz 2. Then the Week 3 overflow: questions, MacKay’s balance, die grouping. Gaussian versus Maxwell–Boltzmann makes the missing measure visible. Maxwell’s demon as compartments and two ledgers; the resolution families are Worksheet 3.

Worksheet 2 is due at the start of this session. Quiz 2 occupies the opening block from 10:05 — Weeks 2–3 as delivered (ensemble, the heat thread through Joule, Helmholtz on the board, Shannon \(H\) versus Boltzmann \(S\)). After the quiz: questions–MacKay–die, Gaussian–Maxwell missing measure, then the demon as compartments and two ledgers. The question is posed here and left open; Worksheet 3 is released. Shannon axioms, HMC, \(Z\), KL, Schottky and finite time stay parked; Szilard, Landauer and Ellis wait for Worksheet 3.

Define This Week

  • Questions; MacKay’s balance; die grouping \(H=H(w)+\sum_i w_i H(p_i)\)
  • Gaussian vs Maxwell: variable, grouping, and measure \(m\)
  • The demon: compartments and two ledgers
  • Worksheet 3: where can the entropy sit?

This Session

Time plan (as delivered 3 November 2026). Clocks from the Panopto SRT (recording 00:00 \(\approx\) 10:00). The published slot is 10:05–11:55 (110 minutes). Quiz 2 is paper, as Quiz 1 was. As taught: MacKay and the die filled the first half (board-first); the missing measure — including the velocity-space shell answer to Harry’s question — took most of the second half; the demon compartments and two ledgers were a short close for Worksheet 3.

ClockBlock
10:05–10:22Quiz 2 (paper; quiet ran long); Worksheet 2 collected
10:22–10:53Questions, MacKay (board), die grouping on the board
10:53–11:03Break
11:03–11:47Anthropomorphic bridge; histogram; shell geometry; Radon–Nikodym / Jaynes \(m\)
11:47–11:55Compartment model; two ledgers; release Worksheet 3

After This Lecture

Worksheet 2 was due at the start of this lecture, before Quiz 2. Worksheet 3 is released today; due 17 November (start of lecture 6), the same morning as Quiz 3.

Quiz 2

Invigilated paper test, no notes, no network, no LLMs. Ten questions on Weeks 2–3 as delivered: Bernoulli / ensemble, the heat thread (Black through Joule and Clausius), Helmholtz \(F=U-TS\) and \(\log Z\), Shannon \(H\) versus Boltzmann \(S\). Not the information game, not MacKay’s balance, not Maxwell’s demon — those start after the quiz.

Class test timer (Quiz 2): 10:05

Information, energy and intelligence course notebook setup

[edit]

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util
cmd = install_command('pods')
%system {cmd}
cmd = install_command('notutils')
%system {cmd}
cmd = install_command('mlai')
%system {cmd}

As Taught

[edit]

As taught on 3 November the information game, MacKay’s balance and the die grouping identity filled the hour before the break, mostly on the board. After the break the missing measure was the main taught material: anthropomorphic entropy as the bridge, histogram Riemann sums, the two-dimensional velocity-space shell (grid versus concentric rings) as the geometric answer to why the speed histogram peaks away from zero, then Radon–Nikodym and Jaynes’ \(m\). Maxwell’s demon was a short close: selective membrane on the canvas, a compartmentalised model (left bulk, right bulk, kinetic layer, optional gate), and the fork between number flux and energy flux. Worksheet 3 starts from that bookkeeping. Szilard, Landauer and Ellis stay parked.

Questions, a Balance, and a Die

[edit]

The Information Game

[edit]

Before writing a formula for entropy, play a game. One player thinks of an object drawn from a known finite list. The other may ask only yes/no questions. How many questions do you need in the worst case? That count is already a measure of information: it is how much uncertainty you must remove to name the object.

With a list of size \(N\) and equal prior probability on each item, an ideal strategy halves the search space at every step. After \(k\) answers you can distinguish at most \(2^k\) hypotheses, so you need \[ k \ge \lceil\log_2 N\rceil \] questions in the worst case. Twenty questions reach a million possibilities: \(2^{20} = 1\,048\,576\). A fair yes/no answer therefore carries one bit of information. That is Shannon’s unit before Shannon’s formula: information as the number of balanced binary distinctions required to resolve uncertainty.

If the prior is not uniform, the best question is no longer a literal half-split of the count of remaining objects. It is a half-split of their probability. The Shannon entropy \(H = -\sum_i p_i\log_2 p_i\) is the information budget, not the question count. The optimal expected number of yes/no questions \(L\) satisfies \(H\le L<H+1\); equality \(L=H\) holds only when every \(p_i\) is a power of \(1/2\). The lecture’s own die is the exhibit: \(H=\log_2 6\approx 2.585\) bits, but a Huffman strategy needs \(8/3\approx 2.667\) questions on average. We will meet the formula next; the game already shows what the number is counting.

MacKay’s Balance and Balls

[edit]

David MacKay opens the compression half of Information Theory, Inference, and Learning Algorithms with a weighing puzzle, Exercise 4.1 (MacKay, 2003). The information-game habit is to ask how we should proceed — what we must name — and then to try the instrument as if it had two readings, like a yes/no question. Only after that sequence do we notice that the balance has a third reading.

Figure: A two-pan balance and twelve identical balls.

You are given twelve balls, identical in appearance. Exactly one differs in weight from the others; you do not know whether that odd ball is heavier or lighter. A two-pan balance compares two equal-sized groups. Design a strategy that finds the odd ball and the direction of the defect.

Resist jumping to a particular grouping. Count first: how many states of the world. Then try the instrument as a two-outcome device, because that is the game we already know. Keep what we put on the pans separate from the information budget those pans can return.

Under a uniform prior over those twenty-four possibilities, the Shannon uncertainty we must remove is \[ H_{\mathrm{start}} = \log_2 24 \approx 4.58 \text{ bits}. \] At the end we need a single named hypothesis: \(H_{\mathrm{end}} = 0\). The deficit to close is therefore about \(4.58\) bits. That difference — start minus end — is the information the experiment must supply. We have not yet said what the balance returns.

Try Two Outcomes

A two-outcome instrument, used optimally, returns at most \(\log_2 2 = 1\) bit per measurement. The information budget then says \[ 2^4 = 16 < 24 \le 32 = 2^5, \] so five uses are not ruled out on paper and four cannot suffice. Whether a physical balance achieves that one bit is a different question: it depends on what we put on the pans.

\[ \log_2 24 \approx 4.58,\qquad 2^4 = 16 < 24 \le 32 = 2^5 \]

Write two columns if you can: left, the information budget; right, the pans. The left column does not know about equal group sizes, leftover balls, or borrowing a ball of known mass. Those constraints live on the right. The rest of this block is the right-hand column, scored afterwards by the left.

The move 6 vs 6 is the binary-search instinct. If we already knew the odd ball was heavier, it would be a perfect split of twelve balls. We do not know that. A down-left reading means “one of the left six is heavy, or one of the right six is light”: twelve of the original twenty-four hypotheses remain. One bit toward the 24, and no ball has been eliminated. Mechanism and maths have already come apart: the pans moved, the pile of balls did not shrink.

The information gain can be perfect for the two-outcome model — an even split of 24 — while the mechanism fails to name a set of six suspect balls. What we carry forward is six maybe-heavy and six maybe-light. If we now pretend it was heavier and keep only the down pan, we have thrown away half of the remaining hypotheses by assumption, not by measurement.

This is the sequence people often initially sketch on the board: 6 vs 6, then 3 vs 3. It is the right mechanism for a different puzzle — twelve balls, one heavier. Hold that sequence, because this is where the mechanism starts departing from our assumptions.

Firstly, what are the possible outcomes for this 3 vs 3? What if we choose the pile that does not contain the odd ball?

OK, so that has revealed a flaw in our reasoning: a balance actually has three outcomes.

Three Readings

A two-pan balance has a ternary alphabet. The reading “level” is not an annoyance to be designed out; it is a symbol that can carry information. The first move of the two-outcome story, 6 vs 6, sets the probability of that symbol to zero: with exactly one odd ball among the twelve, the pans cannot balance. We spent a mechanism choice to turn a three-outcome device into a two-outcome device, and then had to borrow a ball of known mass so that the three-ball endgame could still use the pans.

\[ \log_2 3 \approx 1.58,\qquad 3^2 = 9 < 24 \le 27 = 3^3 \]

A binary question yields at most one bit. If the three readings are equiprobable, one weighing contributes \[ \log_2 3 \approx 1.585 \text{ bits}. \] After \(k\) weighings the strategy tree has at most \(3^k\) leaves, so it can name at most \(3^k\) hypotheses. Compare budgets: \[ \begin{align*} 3^2 &= 9 < 24,\\ 3^3 &= 27 \ge 24. \end{align*} \] Two weighings cannot suffice no matter how clever the groupings. Three weighings can name the 24 hypotheses: after using 24 of the \(3^3=27\) leaves, three paths are left over. Those leftovers are almost a thirteenth ball. The information bounds are \[ \frac{3^n-3}{2}=12 \quad\text{(ball and direction)}, \qquad \frac{3^n-1}{2}=13 \quad\text{(ball alone)}. \] With \(n=3\), twenty-six directed hypotheses fit under twenty-seven leaves, so the information budget allows thirteen balls if you only need to name which one is odd. What the mechanism cannot always deliver is the direction for that thirteenth ball: if it is only ever left off the pans, it is never compared. The 26 fit under 27 on the information side; the mechanism cannot separate two of them. In bits, \[ 3\log_2 3 \approx 4.75 > \log_2 24 \approx 4.58. \] The gap is small. That is why the twelve-ball puzzle feels tight. It is the same comparison we already made with \(\log_2 2\), with a larger per-use budget.

The first weighing that realises the ternary budget puts four against four and leaves four off. Level then means the odd ball is in the four left off; a tip couples a group of four with a direction, as 6 vs 6 did, but now the third symbol is alive. We do not give the rest of the tree. The design principle is the one already present in the information game: choose each measurement so that its outcomes are as nearly equiprobable as possible under the current posterior. That maximises expected information gain per weighing and is what makes the \(3^3 \ge 24\) budget achievable in practice. Adding a known-normal ball, or arranging a weighing that can never balance, spends mechanism capacity that the information budget was counting on.

We stop at the existence argument. Constructing the adaptive tree — which balls on which pan at each step — is MacKay’s exercise, and solving it is the best way to feel what “informative experiment” means. What the lecture owns is the split between ledgers: \(\log_2 24\) versus \(\log_2 2\) or \(\log_2 3\) is the information account; 6 vs 6, 4 vs 4, and padding with a known ball are the mechanical account. The two are not the same, and the known-ball tax is the exhibit.

Coarse-Graining a Die

[edit]

Take a fair six-sided die, \(X\in\{1,\ldots,6\}\) with \(p(x)=1/6\), so \[\begin{align} H(X)=\log_2 6 = 1 + \log_2 3 \approx 2.585 \text{ bits}. \end{align}\] Ask questions in two stages: first learn which group the face falls in, then resolve the face inside the group.

\[ H(G_{\mathrm{parity}})=1 \text{ bit}, \qquad H(X\mid G_{\mathrm{parity}})=\log_2 3 \approx 1.585 \text{ bits} \]

Let \(G_{\mathrm{parity}}=0\) on even faces and \(1\) on odd. Then \(w=(1/2,1/2)\), so \(H(G_{\mathrm{parity}})=1\). Conditional on the group, three faces remain, each with probability \(1/3\), so \(H(X\mid G_{\mathrm{parity}})=\log_2 3\). The stages add: \[\begin{align} H(G_{\mathrm{parity}})+H(X\mid G_{\mathrm{parity}}) = 1+\log_2 3 = \log_2 6 = H(X). \end{align}\]

\[ w=\Bigl(\tfrac16,\tfrac56\Bigr), \qquad H(G_{\mathrm{six}})=h_2\Bigl(\tfrac16\Bigr)\approx 0.65 \text{ bits} \]

Let \(G_{\mathrm{six}}=\mathbf{1}_{\{X=6\}}\). The group weights are unbalanced, \(w=(1/6,5/6)\), so the binary entropy of the first answer is \[\begin{align} H(G_{\mathrm{six}}) = h_2\Bigl(\tfrac16\Bigr) = -\tfrac16\log_2\tfrac16-\tfrac56\log_2\tfrac56 \approx 0.650 \text{ bits}. \end{align}\] That is not one bit. After a “yes” the face is known (\(H=0\)). After a “no” five faces remain (\(H=\log_2 5\)). Averaging, \[\begin{align} H(X\mid G_{\mathrm{six}}) = \tfrac16\cdot 0 + \tfrac56\log_2 5 \approx 1.935 \text{ bits}. \end{align}\] Again \(H(G_{\mathrm{six}})+H(X\mid G_{\mathrm{six}})=H(X)\). Note that “\(\ge 4\)?” on a fair die is also a balanced cut (\(\{1,2,3\}\) vs \(\{4,5,6\}\)), so it is one bit — the unbalanced example has to be chosen carefully.

Nothing about the cube stops you asking “odd?” after “low?” The question can be asked; it cannot be balanced (\(h_2(1/3)\approx 0.918\) bits). A Huffman strategy on the six faces needs \(8/3\approx 2.667\) questions on average against \(H=\log_2 6\approx 2.585\). The balance has a fixed ternary alphabet: each use returns one of three readings. The die has six faces that are not a power of two, so yes/no questions cannot hit \(H\) exactly.

Partitions That Do Not Overlap

[edit]

Write \(d\) for the fine outcome (the face of the die) and number the coarse cells by \(i\). Inside cell \(i\) the conditional \(p(d\mid i)\) is not a free distribution on the whole alphabet: once you know the partition, the other faces have probability zero. That is why those conditionals are better read as weights on a restricted set. The modelling choice, when we coarse-grain, is to divide the faces into cells that are not overlapping. Typically we do not want probability to leak across cells in ways we are not modelling.

\[ p(d)=\sum_i p(d\mid i)\,p(i) \]

The marginal is still \[\begin{align} p(d)=\sum_i p(d\mid i)\,p(i). \end{align}\] Entropy of the system is \(-\sum_d p(d)\log p(d)\). Substituting the mixture form puts a sum inside the logarithm (usually a headache), whereas the log of a product separates. Non-overlap is what licenses the next step. For every face \(d\) there is only one valid partition: if \(d\) is one or two, only \(i=1\) is on; if \(d\) is three through six, only \(i=2\) is on. Every other term in the sum is zero. Those zeros drop out, the log factors, \[\begin{align} \log\bigl(p(d\mid i)\,p(i)\bigr) = \log p(d\mid i)+\log p(i), \end{align}\] and the entropy splits into the entropy of the coarse-graining label plus a weight times the entropy inside each partition (the form \(H(w)+\sum_i w_i H(p_i)\)). This only works because the partitions are on or off.

What would normally be called a mixture model (a mixture of Gaussians, or a mixture of experts) lets components overlap. Then several \(i\) can carry mass for the same \(d\), the sum inside the log does not collapse, and you cannot drop terms. The grouping identity we are writing is the exclusive case: partitions on or off. When you break the world down that way you lose information (you throw the within-partition uncertainty away) and then operate at the coarser grade. In the notes’ standing notation the face is \(X\), the coarse label is \(G=g(X)\), the cell weights are \(w_i=p(i)\), and \(p_i\) is the distribution inside cell \(i\); the same objects as \(d\) and \(i\) above. The overlapping case \(H(I\mid X)>0\), and its mutual-information form, wait for week 5.

Grouping Composition

[edit]

\[ H(X)=H(w)+\sum_i w_i\, H(p_i) \]

Partition the outcomes of \(X\) into groups labelled by \(i\), with group probabilities \(w_i\) and within-group distributions \(p_i\). Then \[\begin{align} H(X)=H(w)+\sum_i w_i\, H(p_i). \end{align}\] The die examples are the case check: \(H(w)\) is the entropy of the coarse question; \(\sum_i w_i H(p_i)\) is what remains once the group is known. If \(G=g(X)\) is the group label, this identity is exactly the chain rule \[\begin{align} H(X)=H(G)+H(X\mid G), \end{align}\] because \(G\) is a function of \(X\) (so \(H(G\mid X)=0\)) and \(H(X\mid G)=\sum_i w_i H(p_i)\).

In the reverse direction, observing \(X\) fixes \(G\) with no leftover uncertainty: \(H(G\mid X)=0\). The price of the coarse question is then exactly \(H(G)\). After you know the group, the remaining work is \(H(X\mid G)\) (which of the fine outcomes inside that group). Coarse-graining keeps \(H(G)\) as your description and discards \(H(X\mid G)\): \[\begin{align} \text{information lost by coarse-graining} = H(X\mid G) = \sum_i w_i\, H(p_i). \end{align}\] The exclusive case is special because \(H(G\mid X)=0\). When components overlap, that conditional entropy is positive and the clean split \(H=H(w)+\sum_i w_i H(p_i)\) fails. Week 5 writes the leftover term in full and names it with mutual information.

The die paid \(\log_2 6\) whichever cut came first. That invariance under regrouping is not an accident of base-two logarithms; it is part of what characterises Shannon entropy among candidate uncertainty measures (Shannon’s third requirement in 1948, (Shannon, 1948); (Khinchin, 1957)). Categorical accounts are also explicit about this (Baez et al., 2011).

Grouping Is a Choice of Abstraction

[edit]

The idealised information game fails when we turn to the mechanism. But we do have some control. We can coarse-grain, and reduce the search space. For example, the low/high grouping is a choice of abstraction. We keep a two-valued group label and throw away which of the three faces occurred. Coarse graining in thermodynamics can be seen similarly. We cannot track Laplace’s curves for every molecule, instead we keep macroscopic numbers. Those macroscopic numbers split into extensive quantities that scale with the system (e.g. \(N\), \(V\), energy, entropy) and intensive quantities that do not (e.g. temperature, pressure). The grouping property constrains the form of \(H\); it is not that the form of \(H\) creates the grouping. Accounting for information works through the groupings: the macrostate is the group label \(G\), its entropy is \(H(w)\), and the unresolved motion inside it is \(\sum_i w_i H(p_i)\).

Gaussian, Maxwell–Boltzmann, and the Missing Measure

[edit]

Entropy depends on which degrees of freedom you count. Jaynes, following Wigner, called that the anthropomorphic character of entropy: a given physical system corresponds to many thermodynamic systems, according to which variables you choose to control (Jaynes, 1965). Steam-table entropy omits degrees of freedom that the tables do not treat (nuclear spins are the classic unused reservoir). Two observers who neglect different parts of the uncertainty can still share a consistent account of what they do measure, because the same functional form of \(H\) sits underneath. That is the bridge into Harry’s question. Each Cartesian velocity component peaks at zero; the speed histogram of the three-dimensional system does not. The disagreement is not only a change of the underlying measure: it is a change of variable, a grouping of directions, and a change of measure.

import numpy as np

rng = np.random.default_rng(0)
n_velocities = 10000
velocity_sigma = 1.0
n_histogram_bins = 40
velocities = rng.normal(0.0, velocity_sigma, size=(n_velocities, 3))
speed = np.linalg.norm(velocities, axis=1)
velocity_x = velocities[:, 0]
speed_1d = np.abs(velocity_x)


def discrete_entropy(counts):
    n = counts.sum()
    p = counts[counts > 0] / float(n)
    return -np.sum(p * np.log(p))


def discrete_entropy_from_masses(p):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]
    return -np.sum(p * np.log(p))


def bin_midpoints(edges):
    return 0.5 * (edges[:-1] + edges[1:])


def riemann_masses(pdf_at_mids, widths):
    # Midpoint masses p_i ≈ f(x_i) Δ_i, renormalised on the truncated support.
    mass = np.asarray(pdf_at_mids, dtype=float) * np.asarray(widths, dtype=float)
    total = mass.sum()
    if total <= 0.0:
        return mass
    return mass / total


def gaussian_pdf(x, sigma):
    return np.exp(-0.5 * x**2 / sigma**2) / np.sqrt(2.0 * np.pi * sigma**2)


def gaussian_differential_entropy(sigma):
    return 0.5 * np.log(2.0 * np.pi * np.e * sigma**2)


def half_normal_pdf(s, sigma):
    # Folded |N(0, sigma^2)| on s >= 0; mode remains at zero.
    return (
        np.sqrt(2.0 / (np.pi * sigma**2))
        * np.exp(-s**2 / (2.0 * sigma**2))
    )


def maxwell_speed_pdf(s, sigma):
    return (
        np.sqrt(2.0 / np.pi)
        * (s**2 / sigma**3)
        * np.exp(-s**2 / (2.0 * sigma**2))
    )


def maxwell_speed_differential_entropy(sigma):
    return np.log(sigma * np.sqrt(2.0 * np.pi)) + np.euler_gamma - 0.5


def speed_range_max(s, p_hi=0.995):
    return float(np.quantile(s, p_hi))


def equal_delta_speed_edges(s_max, n_bins):
    return np.linspace(0.0, s_max, n_bins + 1)


def equal_volume_speed_edges(s_max, n_bins):
    u_edges = np.linspace(0.0, s_max**3, n_bins + 1)
    return u_edges ** (1.0 / 3.0)

Entropy of a Gaussian Velocity Histogram

[edit]

Histograms as Riemann Sums

Approximating a density by its value at the centre of each bin is a piecewise-constant (midpoint-rule) approximation to the integral. We call the corresponding sums Riemann sums. Writing \(p_i\approx f(x_i)\Delta\equiv f_i \Delta\) inside the entropy gives \[\begin{align} H = -\sum_i f_i\Delta\log f_i - \log\Delta\sum_i f_i\Delta \approx h - \log\Delta, \end{align}\] with both sums converging to integrals as \(\Delta\to 0\).

Take one Cartesian component of an isotropic three-dimensional velocity and partition it with equal-width bins. The discrete Shannon entropy of the midpoint masses is \[\begin{align} H = -\sum_i p_i\log p_i, \qquad p_i\approx f(x_i)\Delta. \end{align}\] (Empirical occupancy \(n_i/N\) estimates the same masses.) For a smooth density and a fine equal-width partition, \[\begin{align} H \approx h - \log \Delta, \qquad h = \tfrac12\log(2\pi e\sigma^2). \end{align}\] The figure below is the piecewise-constant midpoint approximation of \(f\), not a sample histogram. Both numbers agree up to tail truncation. The differential entropy \(h\) is the \(\Delta\)-independent remainder. It is not the entropy of the partition.

Figure: Equal-width midpoint Riemann approximation of a Gaussian velocity component. Discrete \(H\) matches \(h-\log\Delta\); \(h\) itself is the remainder after the bin-width term is subtracted.

One Dimension: Velocity Versus Modulus

[edit]

Take one Cartesian component \(v\) of the isotropic sample (or any one-dimensional Gaussian). Its density peaks at the origin. The modulus \(s=|v|\) folds the two tails onto \([0,\infty)\). Probability conservation gives the half-normal \[\begin{align} f(s) = \sqrt{\frac{2}{\pi\sigma^2}}\, e^{-s^2/2\sigma^2}, \qquad s\ge 0. \end{align}\] That density is strictly decreasing: the mode is still at \(s=0\). A nonzero most-probable speed needs the shell volume of two or three dimensions, not the absolute-value map alone.

Figure: One dimension. Left: a Gaussian velocity component peaks at zero. Right: the same sample folded to \(|v|\) is half-normal — the mode is still at zero.

Velocity Space: Grid and Speed Rings

[edit]

Harry’s question — why does the speed histogram peak away from zero when each Cartesian component peaks at zero? — is answered by a change of variables in velocity space, not by folding alone. Draw the plane of two components \((v_x,v_y)\). An isotropic Gaussian is a circular cloud centred at the origin. Partition that plane with an equal-\(\Delta v\) grid: every square has the same area. Now draw concentric circles of radius \(s=|\mathbf{v}|\). The ring between \(s\) and \(s+\Delta s\) contains more and more grid cells as \(s\) grows, because its circumference is \(2\pi s\). Counting samples by which ring they fall in therefore weights larger speeds more heavily than the Cartesian grid does. That circumferential factor is the two-dimensional shell volume. In three dimensions the same geometry becomes a spherical shell of area \(4\pi s^2\), which is the \(s^2\) in the Maxwell speed law. Absolute value alone (the one-dimensional fold) never moves the mode; the shell volume does.

Figure: Two Cartesian velocity components. The pale grid is equal-\(\Delta v\) area. The red circles are equal-\(\Delta s\) shells: circumference grows with \(s\), so a speed histogram weights larger speeds more heavily. Three dimensions replaces rings by spherical shells of area \(4\pi s^2\).

Three Dimensions: Component Versus Speed

[edit]

Keep the same isotropic three-dimensional sample. One Cartesian marginal is still Gaussian and peaks at the origin. Speed \(s=|\mathbf{v}|\) has density \[\begin{align} f(s) = \sqrt{\frac{2}{\pi}}\,\frac{s^2}{\sigma^3}\,e^{-s^2/2\sigma^2}, \qquad s\ge 0. \end{align}\] The factor \(s^2\) is the surface area of a sphere of radius \(s\) in velocity space — the three-dimensional upgrade of the circumferential \(s\) that the concentric rings made visible in two dimensions. Maximising \(f(s)\) gives the most probable speed \[\begin{align} s_{\mathrm{mp}} = \sigma\sqrt{2}. \end{align}\] Unlike the one-dimensional modulus, the mode is not at zero. The peak moves because of the change of variables in three dimensions, not because speed is nonnegative.

Figure: Three dimensions. Left: one Cartesian component still peaks at zero. Right: speed peaks at \(\sigma\sqrt{2}\) because of the \(s^2\) shell volume — not because of taking a modulus.

Pushed Bins Keep the Same Entropy

[edit]

Discrete Shannon entropy depends only on the probability masses of a partition. Keep the cells and relabel the axis: the masses \(\{p_i\}\) are unchanged, so \(H\) is unchanged, while a density histogram \(p_i/\Delta_i\) changes because the widths change.

The Maxwell speed map \(s=|\mathbf{v}|\) is not itself 1–1 — equal Cartesian cubes that sit on one spherical shell are a grouping, and grouping drops entropy. This figure does not rebin: the equal-\(u\) cells are still spherical shells (directions remain grouped). What it avoids is changing the partition. Partition speed by equal Cartesian volume, i.e. equal-width bins in \(u=s^3\). The map \(u\mapsto s=u^{1/3}\) is strictly increasing on \(s\ge 0\). Pushing the bin edges stretches them by \[\begin{align} \Delta s_i \approx \frac{1}{3s_i^2}\,\Delta u. \end{align}\] Drawn as probability bars the occupancy is invariant. Drawn as a density against \(s\) the bars change shape. The two pictures share one \(H\).

Figure: Equal-volume shells in 3D velocity, written two ways. Left: midpoint masses \(p_i\). Right: the same partition as a piecewise-constant density on the speed axis. The bins change shape; \(H\) does not.

The Continuous Limit Drops \(\log\Delta\)

[edit]

Refine the equal-width partition of a Gaussian component. For each bin width \(\Delta\), \[\begin{align} H(X^\Delta) = -\sum_i p_i\log p_i \approx h - \log\Delta. \end{align}\] Plotting \(H\) against \(-\log\Delta\) gives a line of slope one, offset by the constant \(h=\tfrac12\log(2\pi e\sigma^2)\). The discrete entropy diverges as the bins shrink because each mass \(p_i\approx f\Delta\) contributes an extra \(-\log\Delta\). Differential entropy is defined to be the \(\Delta\)-independent remainder. The continuous functional therefore ignores a term that is driving to \(+\infty\). Differential entropy \(h\) can be negative because densities can exceed \(1\) (mass concentrated below the unit scale). Dropping \(-\log\Delta\) permits that; it is not the cause. And \(h\) is not a code-length bound.

Figure: Discrete entropy of an equal-width Gaussian histogram against \(-\log\Delta\). The points track \(h-\log\Delta\); \(h\) is the finite part. The continuous limit discards a term that diverges.

Entropy of the Maxwell Speed Law

[edit]

The same isotropic Gaussian law has speed \(s=|\mathbf{v}|\) with density \[\begin{align} f(s) = 4\pi s^2 \bigl(2\pi\sigma^2\bigr)^{-3/2} e^{-s^2/2\sigma^2} = \sqrt{\frac{2}{\pi}}\,\frac{s^2}{\sigma^3}\,e^{-s^2/2\sigma^2}, \qquad s\ge 0. \end{align}\] That is the Maxwell speed law. Its differential entropy is \[\begin{align} h(s)=\log\bigl(\sigma\sqrt{2\pi}\bigr)+\gamma-\tfrac12, \end{align}\] where \(\gamma\) is Euler’s constant. A single Cartesian component has \(h(v_x)=\tfrac12\log(2\pi e\sigma^2)\). For \(\sigma=1\) these are approximately \(0.996\) and \(1.419\) nats — but that comparison mixes a 1D marginal with a 3D-derived quantity. The like-for-like comparison is three-dimensional velocity against speed: \(h(\mathbf{v})=3h(v_x)\approx 4.257\) nats, and \[\begin{align} h(\mathbf{v})=h(s)+\log 4\pi+\mathbb{E}[\log s^2]. \end{align}\] Three things have changed at once. The variable is now \(s\) rather than \(\mathbf{v}\) (or \(v_x\)). The \(\log 4\pi\) term is the entropy of direction, which the speed map throws away — a grouping. The \(\mathbb{E}[\log s^2]\) term is the Jacobian — a change of measure. Jaynes’ reference density \(m\) is responsible for only that last piece. An equal-\(\Delta s\) midpoint Riemann partition of \(f(s)\) tracks \(h(s)-\log\Delta s\), just as the Gaussian partition tracked \(h(v_x)-\log\Delta v\).

Figure: Equal-width midpoint Riemann approximation of Maxwell speed. Discrete \(H\) tracks \(h(s)-\log\Delta\). The Maxwell \(h(s)\) is not the Gaussian \(h(v_x)\).

Two Discretisations of the Same Speeds

[edit]

The previous figure kept one partition and only relabelled the axis, so \(H\) could not change. Now we rebin. Take the same Maxwell density and the same number of cells, and compare two partitions: (i) the pushed equal-volume partition (equal width in \(u=s^3\)); (ii) a uniform partition of \([0,s_{\max}]\) (equal \(\Delta s\)). These are different collections of midpoint masses \(\{p_i\}\), so they have different discrete entropies. Uniform bins in \(s\) are Lebesgue measure on the speed axis. Equal-volume bins are Lebesgue measure on \(\mathbb{R}^3\), expressed in the speed coordinate. Choosing a discretisation is choosing a measure.

Figure: The same Maxwell density, two midpoint Riemann partitions, two values of discrete \(H\). Left: equal 3D volume, pushed to the speed axis. Right: equal \(\Delta s\). The discretisation is the measure.

Bin Widths Are a Radon-Nikodym Derivative

[edit]

For a partition that is not equal-width, the bin-width correction is no longer a single \(\log\Delta\). Probability conservation in cell \(i\) still reads \(p_i \approx f\,\Delta_i\), so \[\begin{align} H = -\sum_i p_i\log p_i \approx h - \sum_i p_i\log\Delta_i, \qquad h \approx H + \sum_i p_i\log\Delta_i. \end{align}\] Take the equal-volume cells of the previous snippets, written once in \(u=s^3\) and once in \(s\). The masses \(\{p_i\}\) (hence \(H\)) are the same. The two differential entropies differ by the average log-ratio of widths, \[\begin{align} h(s)-h(u) \approx \sum_i p_i\log\frac{\Delta s_i}{\Delta u_i} \to \mathbb{E}\bigl[\log\bigl|{\mathrm{d}s}/{\mathrm{d}u}\bigr|\bigr]. \end{align}\] That ratio \(\Delta s_i/\Delta u_i\) is called the discrete Radon-Nikodym derivative of Lebesgue-on-speed with respect to Lebesgue-on-volume. In the continuum limit it becomes the Jacobian \(1/(3s^2)\). Probability conservation \(f_s(s)\,\mathrm{d}s = f_u(u)\,\mathrm{d}u\) is the same statement.

The pushforward of Lebesgue measure on \(\mathbb{R}^3\) onto the speed coordinate has density \(4\pi s^2\) with respect to \(\mathrm{d}s\). Writing \(f\) for the isotropic density on velocity space, \[\begin{align} f_s(s)\,\mathrm{d}s = f(\mathbf{v})\,4\pi s^2\,\mathrm{d}s. \end{align}\] The Gaussian \(h(v_x)\) and the Maxwell \(h(s)\) disagree for three reasons at once: different variable, grouping of directions into shells, and a different undeclared reference measure. Only the last of these is Jaynes’ \(m\).

Figure: Width ratio of the pushed equal-volume cells, against the Jacobian \(1/(3s^2)\). The average log-ratio converts \(h(u)\) into \(h(s)\). That ratio is the discrete Radon-Nikodym derivative.

Radon-Nikodym Derivative

[edit]

The Form of the Radon-Nikodym Derivative

\[ \mu(A)=\int_A \frac{\mathrm{d}\mu}{\mathrm{d}\nu}\,\mathrm{d}\nu \]

Absolute continuity is the continuum name for what the shared partition already enforced: every cell that carries mass under one description carries mass under the other. When a measure \(\mu\) is absolutely continuous with respect to a measure \(\nu\) on the same space, there exists a nonnegative density - the Radon-Nikodym derivative - such that \[\begin{align} \mu(A) = \int_A \frac{\mathrm{d}\mu}{\mathrm{d}\nu}\,\mathrm{d}\nu \end{align}\] for every measurable \(A\).

Now let’s focus on the densities that we’ve been considering. Let \(F\) have density \(f\) against Lebesgue measure \(\lambda\), and let \(M\) be Jaynes’ reference measure, with density \(m\) against the same \(\lambda\). Provided \(m(x)=0\) only where \(f(x)=0\) (so \(F\ll M\)), we can write \[\begin{align} F(A) &= \int_A f\,\mathrm{d}x \nonumber\\ &= \int_A \frac{f}{m}\, m\,\mathrm{d}x \nonumber\\ &= \int_A \frac{f}{m}\,\mathrm{d}M, \end{align}\] The middle step is just the statement that \(M\) assigns mass \(m(x)\,\mathrm{d}x\) to the sliver \([x,x+\mathrm{d}x]\), so \(\mathrm{d}M = m\,\mathrm{d}x\) (just as \(\mathrm{d}F = f\,\mathrm{d}x\)). In Jaynes’ picture \(m(x)\,\mathrm{d}x\) is the fraction of reference points in the sliver, so \(\int_A g\,\mathrm{d}M\) is the average of \(g\) over the points in \(A\), and \(f/m\) is the mass of \(F\) carried per reference point. By uniqueness of the Radon–Nikodym derivative \[\begin{align} \frac{\mathrm{d}F}{\mathrm{d}M} = \frac{f}{m}. \end{align}\] This is the chain rule \(\frac{\mathrm{d}F}{\mathrm{d}M} = \frac{\mathrm{d}F/\mathrm{d}\lambda}{\mathrm{d}M/\mathrm{d}\lambda}\). Under a change of variables both \(f\) and \(m\) pick up the same Jacobian, which cancels in the ratio, so \(f/m\) is coordinate-free even though neither density is. Relative entropy is the expectation under \(F\) of the log of that ratio, \[\begin{align} \text{KL}\left( f\,\|\,m \right) = \int f\log\frac{\mathrm{d}F}{\mathrm{d}M}\,\mathrm{d}x = \int f\log\frac{f}{m}\,\mathrm{d}x, \end{align}\] and it is finite only when the absolute-continuity condition holds.

The bin-width ratio \(\Delta s_i/\Delta u_i\) of the previous snippet is the same statement on a finite partition: it is the discrete Radon-Nikodym derivative of Lebesgue-on-speed with respect to Lebesgue-on-volume. In the continuum that ratio becomes the Jacobian. Naming \(\mathrm{d}F/\mathrm{d}M=f/m\) makes the next step - declaring \(m\) - a change of what one divides by, not a change of vocabulary.

Jaynes: Promote the Bins to a Measure

[edit]

\[ h_m(f)=-\int f\log\frac{f}{m}\,{\mathrm{d}}x=-\text{KL}\left( f\,\|\,m \right) \]

Differential entropy \(h=-\int f\log f\,{\mathrm{d}}x\) pretends the coordinate measure is part of the data. The histogram arithmetic showed that it is not: \(H\) is a functional of masses, and the passage to a density subtracts \(\sum_i p_i\log\Delta_i\). Jaynes called \(m\) the limiting density of discrete points — the binning measure, promoted to a density (Jaynes, 1963).

Concretely, Jaynes places \(N\) points so that the number falling in \([a,b]\) is approximately \(N\int_a^b m\,\mathrm{d}x\). Each point then owns a cell of width \(\Delta_i\approx 1/(N m(x_i))\) and reference mass \(1/N\): \(m\) says where the cells are fine and where they are coarse, and every cell is equally likely a priori. For a Gaussian \(m\) the points are its quantiles, crowded in the centre and sparse in the tails. With \(p_i\approx f(x_i)\Delta_i\), \[\begin{align} H_N = -\sum_i p_i\log p_i \approx \log N - \int f\log\frac{f}{m}\,\mathrm{d}x, \end{align}\] so the divergence is a pure count of cells, \(\log N\), and what remains is \(h_m\). When \(f=m\) every cell gets mass \(1/N\), \(H_N=\log N\) and \(h_m=0\), its maximum: \(h_m\) measures how far \(f\) is from spreading evenly over the cells that \(m\) declares.

Once \(m\) is written down, \[\begin{align} h_m(f) = -\int f\log\frac{f}{m}\,{\mathrm{d}}x = -\text{KL}\left( f\,\|\,m \right). \end{align}\] Taking \(m\) uniform (Lebesgue) in the Cartesian component \(v_x\) recovers the ordinary Gaussian remainder \(h(v_x)\) (an improper \(m\); \(h_m\) is then defined up to an additive constant). Taking \(m\) uniform in three-dimensional velocity, and writing it in the speed coordinate, gives \(m(s)\propto s^2\) — also improper — the shell volume that the concentric rings made geometric and that had been hiding in the Jacobian. The map \(\mathbf{v}\mapsto s=|\mathbf{v}|\) is not invertible. Equality of KL in the speed coordinate with KL in 3D velocity holds here because the ratio \(f/m\) depends on \(\mathbf{v}\) only through \(s\), so \(s\) is sufficient and the data-processing inequality holds with equality. In general the data-processing inequality gives \(\le\), not \(=\).

The two Maxwell–Boltzmann entropies disagreed because each \(h\) used a different undeclared \(m\). Relative entropy makes the choice explicit. The Jacobian cancels inside \(\text{KL}\left( f\,\|\,m \right)\) under a common invertible map; the discrete KL has no divergent \(-\log\Delta\) term.

Going continuous buys analytic machinery; it does not remove the need to declare \(m\). Which binning — which degrees of freedom — you treat as “equally likely a priori” remains an anthropomorphic choice in Jaynes’ sense. This names \(m\). Maximising \(h_m\) at fixed mean energy — Jaynes’ maximum entropy principle — is the next lecture.

Maxwell’s Demon, Compartments, Two Ledgers

[edit]

Maxwell’s Demon

[edit]

Maxwell’s demon is a thought experiment described by James Clerk Maxwell in his book, Theory of Heat (Maxwell, 1871) on page 308. He first described the being in an 1867 letter to Tait; Thomson named it a “demon” in 1874.

But if we conceive a being whose faculties are so sharpened that he can follow every molecule in its course, such a being, whose attributes are still as essentially finite as our own, would be able to do what is at present impossible to us. For we have seen that the molecules in a vessel full of air at uniform temperature are moving with velocities by no means uniform, though the mean velocity of any great number of them, arbitrarily selected, is almost exactly uniform. Now let us suppose that such a vessel is divided into two portions, A and B, by a division in which there is a small hole, and that a being, who can see the individual molecules, opens and closes this hole, so as to allow only the swifter molecules to pass from A to B, and only the slower ones to pass from B to A. He will thus, without expenditure of work, raise the temperature of B and lower that of A, in contradiction to the second law of thermodynamics.

James Clerk Maxwell in Theory of Heat (Maxwell, 1871) page 308

He goes on to say:

This is only one of the instances in which conclusions which we have drawn from our experience of bodies consisting of an immense number of molecules may be found not to be applicable to the more delicate observations and experiments which we may suppose made by one who can perceive and handle the individual molecules which we deal with only in large masses

Figure: Maxwell’s demon was designed to highlight the statistical nature of the second law of thermodynamics.

Velocity-bin entropy:

Figure: Maxwell’s Demon. The demon decides balls are either cold (blue) or hot (red) according to their velocity. Balls are allowed to pass the green membrane from right to left only if they are cold, and from left to right only if they are hot. The displayed entropy is the Shannon entropy of the velocity histogram (a coarse-grained proxy, not full thermodynamic entropy).

Maxwell’s demon allows us to connect thermodynamics with information theory (see e.g. Hosoya et al. (2015);Hosoya et al. (2011);Bub (2001);Brillouin (1951);Szilard (1929)). Landauer (1961) described a fundamental connection between information erasure and energy consumption.

A Compartment Model of the Demon

[edit]

Maxwell’s sentence is a selective membrane. The useful first cut for Worksheet 3 is to compartmentalise that picture. Treat each bulk (left of the gate, and right of the gate) as a region that is close to local equilibrium at its own temperature, provided one stays a mean free path or more away from the membrane. Between those bulks sits a kinetic layer where the velocity rule continually removes hot particles one way and cold particles the other. That layer is not a Maxwellian. Counting the membrane itself as a fourth compartment is optional: it is the place where the microscopic gate rule lives. Compartmentalisation here is coarse-graining (assigning a legal description to each region). The transition across the gate is the flux one then tries to write — stationarity of number flux, not detailed balance of each microscopic reverse pair. The two ledgers that follow ask whether particle flux and energy flux can both close under those descriptions.

Figure: Three compartments for Worksheet 3: cold left bulk, hot right bulk, and a non-equilibrium kinetic layer at the membrane. The green gate is an optional fourth compartment — the microscopic velocity rule. Number flux and energy flux are the two ledgers that must both close.

Flux Balance Across the Membrane

[edit]

In the simulation the membrane does not move. A ball on the left is allowed through only when its speed exceeds \(v_\star\); a ball on the right is allowed through only when its speed is at most \(v_\star\). Between crossings, elastic collisions redistribute energy, so a ball that is cold now can become hot later and then cross. The question is not whether sorting begins — it does — but where the net particle and energy flows settle. The compartment schematic above is the bookkeeping: each bulk hopes to be locally Maxwellian; the kinetic layer at the gate is where that hope fails. Setting \(\Phi^{\mathrm{hot}}(n_L,\beta_L)=\Phi^{\mathrm{cold}}(n_R,\beta_R)\) is flux balance (stationarity of number flux), not detailed balance: detailed balance would require each microscopic transition to be balanced by its reverse, and the velocity-selective gate is exactly what breaks that.

Take equal chamber areas and a 2D ideal gas (the billiard world). On each side the velocity density is Maxwellian, \[ f(\mathbf{v}\mid\beta)=\frac{\beta}{2\pi}\exp\!\left(-\tfrac{1}{2}\beta v^2\right),\qquad \beta=\frac{1}{k_B T}, \] with mean kinetic energy \(k_B T\) per ball (two quadratic degrees of freedom). The one-way effusion flux through a unit aperture is \(\Phi_{\mathrm{tot}}=n/\sqrt{2\pi\beta}\). Those formulae assume a small aperture relative to the mean free path (Knudsen). In the billiard simulation with large balls that may not hold. Restricting to the hot or cold set gives \[\begin{align} \Phi^{\mathrm{hot}}(n,\beta) &= \frac{n}{\pi}\left[v_\star e^{-\beta v_\star^2/2}+\int_{v_\star}^\infty e^{-\beta v^2/2}\,\mathrm{d}v\right],\\ \Phi^{\mathrm{cold}}(n,\beta) &= \Phi_{\mathrm{tot}}(n,\beta)-\Phi^{\mathrm{hot}}(n,\beta). \end{align}\] Particle-number flux balance (stationarity of the number ledger) across the membrane is \[ \Phi^{\mathrm{hot}}(n_L,\beta_L)=\Phi^{\mathrm{cold}}(n_R,\beta_R). \] Total ball number \(N=N_L+N_R\) and total energy \(E=N_L k_B T_L+N_R k_B T_R\) are conserved. For the simulation parameters (\(v_\star=3\), mean kinetic energy \(12.5\) in code units, equal populations \(N_L=N_R=N/2\) with even \(N=40\)) this fixes the temperatures uniquely. Continuum bookkeeping; \(N\) need not match the discrete canvas. The equal-\(N\) slice is a cut, not the whole story: a search over \(N_L/N\) (with \(T_L<T_R\)) finds no Maxwellian pair that balances both number and energy flux.

Figure: Left: temperatures on each side of the selective membrane from particle-flux balance at equal populations (simulation energy budget). Right: entropy of the two-Maxwellian state minus the single-temperature equilibrium. The red point is the simulation threshold \(v_\star=3\).

With \(k_B=1\), equal populations, \(N=40\), \(v_\star=3\) and total energy matching the simulation’s mean kinetic energy \(12.5\), particle-number flux balance gives \(T_L\approx 1.94\) and \(T_R\approx 23.1\). Those temperatures are the equal-\(N\) slice. The ideal-gas Shannon entropy of a 2D Maxwellian chamber (position Stirling term plus velocity differential entropy) \[ S(N,\beta)=N\left[2+\log\frac{A}{N}-\log\frac{\beta}{2\pi}\right] \] falls by about \(25\) nats relative to the single-temperature state at the same \(N\) and \(E\) — about \(-0.63\) nats per ball. That drop is the thermodynamic content of the sorting you see on the canvas: hot accumulates on the right, cold on the left, and the joint distribution over \((\mathrm{side},\mathbf{v})\) is more concentrated than the unconstrained Maxwellian.

The compartments, kinetic layers, and two-ledger closure are this course’s construction, not a result in Leff and Rex (Leff and Rex, 2003). Worksheet 3 builds on them.

A Maxwellian is symmetric in a way the membrane is not. Conditioning on \(v>v_\star\) selects a high-energy set; conditioning on \(v\le v_\star\) selects a low-energy set. When the two number fluxes match, the energy fluxes generally do not. Equal number of crossings is not equal energy of crossings: the hot direction carries more energy per ball. That is the flaw in treating number balance alone as a stationary state. The true non-equilibrium steady state therefore cannot be a pair of perfect Maxwellians: the left hot tail is depleted by leakage, the right cold body is depleted by leakage the other way, and collisions continually refill those sets. The local-temperature calculation remains the right first cut — it predicts a cold left, a hot right, and a lower gas entropy — but the precise \((T_L,T_R)\) pair is an approximation to a non-Maxwellian steady state.

The canvas can look settled for a second reason that is an artefact, not thermodynamics: a discrete-time billiard with finite resolution can stall, so fewer balls appear to move even when the continuous idealisation would keep exchanging. Treat apparent stationarity in the simulation as a prompt for the two-ledger question, not as proof that both ledgers have closed.

The selective membrane is Maxwell’s demon written as a dynamical rule. The flux-balance calculation shows that the rule really does drive the gas toward a lower-entropy steady state. Restoring Clausius requires putting the membrane — or the memory that controls an equivalent trap door — on the thermodynamic ledger. Szilard (1929) put the cost in measurement (Szilard, 1929). Landauer (1961) priced erasure but did not apply it to the demon (Landauer, 1961). Bennett (1982) made the erasure resolution and argued that measurement can be reversible — not one continuous story (Bennett, 1982). Earman and Norton (1998–99) dissent in a two-part article (Earman and Norton, 1999, 1998). The simulation’s “velocity-bin entropy” is a coarse proxy for the same story: as sorting proceeds, the displayed histogram entropy typically drops relative to the unsorted gas.

The Demon Sorts Groups

[edit]

Maxwell’s sorting rule partitions velocity space into coarse sets — “hot enough to cross” versus “cold enough to cross” — and applies different gates to each set. That is also this lecture’s grouping: a deterministic label \(G=g(\mathbf{v})\) plus residual uncertainty inside each label. The membrane calculation assumed each chamber stays close to a Maxwellian at its own \(T\) — local equilibrium, timescale separation between collisions and crossings. When that fails, the true steady state is no longer a pair of perfect Maxwellians (tails deplete). Information engines and demons play on the tension between moving probability among groups (\(H(w)\)) and the within-group bath term \(\sum w_i H(p_i)\) that thermodynamics prices.

Compartmentalisation is the coarse-graining move: assign a legal description to each bulk and to the kinetic layer. The transition at the gate is then a question of number flux and energy flux — not a single fluent story about “the demon.” The compartments, kinetic layers, and two-ledger closure are this course’s construction, not a result in Leff and Rex (Leff and Rex, 2003). Worksheet 3 builds on them. I will not mark you wrong for disagreeing with the bookkeeping; I will mark you wrong for treating it as a textbook theorem.

Kemeny–Snell lumpability is the precise condition under which a partition of a Markov chain induces another Markov chain on the groups (Kemeny and Snell, 1960). When it fails, the macro description needs memory of how the system entered the group. We don’t develop the projection-operator formalism.

Worksheet 3 is the coarse-graining of Maxwell: which description is legal for each compartment, and where the missing entropy is allowed to sit. Szilard, Landauer, Ellis, and the information-engine scale argument wait until that worksheet needs them.

Machine Review

A language-model review of the as-delivered L172 Week 4 notes (session 3 November 2026; review refreshed after delivery). It is not assessed. It works from the written notes and the delivered timing.

The grouping identity, the die arithmetic, the Huffman \(8/3\), the Maxwell speed mode and entropy, the decomposition \(h(\mathbf{v})=h(s)+\log 4\pi+\mathbb{E}[\log s^2]\), and the 2D effusion fluxes check numerically. At equal \(N\) with \(v_\star=3\), the continuum flux-balance slice gives \(T_L\approx 1.94\), \(T_R\approx 23.1\), \(\Delta S\approx -0.63\) nats per ball; the energy flux runs about \(2.5\times\) hotter left-to-right, and that ratio stays above about \(2\) for any \(N_L/N\). Twelve balls in three weighings is MacKay’s Exercise 4.1. Thirteen balls can name the odd ball alone; the mechanism cannot always recover direction. The two-ledger argument remains this course’s own construction.

Still open

  • Prerequisites. The course says no physics is assumed. This lecture uses Radon–Nikodym derivatives, Knudsen effusion, Kemeny–Snell lumpability and local equilibrium. Each is named. Mark which are examinable and which are for colour.
  • Figure density. The missing-measure half still has several similar histogram figures (pushed bins, two discretisations, bin-width RN). A single two-panel figure plus the width-ratio plot would do.

Settled since delivery

  • Timing: questions–MacKay–die ran about 31 minutes; missing measure about 44; demon close about 8. The pre-delivery 25/20 plan overran as predicted; Radon–Nikodym and Jaynes stayed in Week 4 and fed Week 5.
  • Terminology: membrane number-flux stationarity is now called flux balance, not detailed balance.
  • Thirteen-ball claim: corrected to the \((3^n-3)/2\) versus \((3^n-1)/2\) ledger split.
  • Bibliography: Earman–Norton (two articles), Leff–Rex, and Kemeny–Snell are cited.

Learning objectives

“Define This Week” has four bullets: grouping, the missing measure, compartments and two ledgers, Worksheet 3. A measurable form of the grouping objective is: compute \(H(G)\) and \(H(X\mid G)\) for a given partition and interpret the residual as information discarded. The measure objective should state the three-way split — variable, grouping, measure — with Jaynes’ \(m\) responsible for only the last.

Sources

Further Reading

  • Exercise 4.1 and the discussion on pp. 69–70 of MacKay (2003)

  • limiting density of discrete points of Jaynes (1963)

Thanks!

For more information on these subjects and more you might want to check the following resources.

References

Baez, J.C., Fritz, T., Leinster, T., 2011. A characterization of entropy in terms of information loss. Entropy 13, 1945–1957. https://doi.org/10.3390/e13111945
Bennett, C.H., 1982. The thermodynamics of computation—a review. International Journal of Theoretical Physics 21, 905–940.
Brillouin, L., 1951. Maxwell’s demon cannot operate: Information and entropy. I. Journal of Applied Physics 22, 334–337. https://doi.org/10.1063/1.1699951
Bub, J., 2001. Maxwell’s demon and the thermodynamics of computation. Studies in History and Philosophy of Science Part B: Modern Physics 32, 569–579. https://doi.org/10.1016/S1355-2198(01)00023-5
Earman, J., Norton, J.D., 1999. Exorcist XIV: The wrath of Maxwell’s demon. Part II. From Szilard to Landauer and beyond. Studies in History and Philosophy of Modern Physics 30, 1–40. https://doi.org/10.1016/S1355-2198(98)00026-4
Earman, J., Norton, J.D., 1998. Exorcist XIV: The wrath of Maxwell’s demon. Part I. From Maxwell to Szilard. Studies in History and Philosophy of Modern Physics 29, 435–471. https://doi.org/10.1016/S1355-2198(98)00023-9
Hosoya, A., Maruyama, K., Shikano, Y., 2015. Operational derivation of Boltzmann distribution with Maxwell’s demon model. Scientific Reports 5, 17011. https://doi.org/10.1038/srep17011
Hosoya, A., Maruyama, K., Shikano, Y., 2011. Maxwell’s demon and data compression. Phys. Rev. E 84, 061117. https://doi.org/10.1103/PhysRevE.84.061117
Jaynes, E.T., 1965. Gibbs vs Boltzmann entropies. American Journal of Physics 33, 391–398. https://doi.org/10.1119/1.1971557
Jaynes, E.T., 1963. Information theory and statistical mechanics, in: Ford, K.W. (Ed.), Brandeis University Summer Institute Lectures in Theoretical Physics, Vol. 3: Statistical Physics. W. A. Benjamin, Inc., New York, pp. 181–218.
Kemeny, J.G., Snell, J.L., 1960. Finite Markov chains. Van Nostrand, Princeton.
Khinchin, A.I., 1957. Mathematical foundations of information theory. Dover, New York.
Landauer, R., 1961. Irreversibility and heat generation in the computing process. IBM Journal of Research and Development 5, 183–191. https://doi.org/10.1147/rd.53.0183
Leff, H.S., Rex, A.F. (Eds.), 2003. Maxwell’s demon 2: Entropy, classical and quantum information, computing. Institute of Physics, Bristol.
MacKay, D.J.C., 2003. Information theory, inference and learning algorithms. Cambridge University Press, Cambridge, U.K.
Maxwell, J.C., 1871. Theory of heat. Longmans, Green; Co, London.
Shannon, C.E., 1948. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
Szilard, L., 1929. Über die Entropieverminderung in einem thermodynamischen System bei Eingriffen intelligenter Wesen. Zeitschrift für Physik 53, 840–856. https://doi.org/10.1007/BF01341281