Week 5: Maximum Entropy and the Exponential Family
[jupyter][google colab][reveal][edit]
Abstract:
Quiz 2 feedback, then KL via overlapping coarse-graining and Jaynes’ reference measure. Soft partitions introduce mutual information. After the break: Jaynes’ maximum entropy under constraints, Lagrange on the board, the die, and the discrete MaxEnt proof that recovers Boltzmann. The exponential family is introduced at the end (\(\theta=-\lambda\)); developed in Week 6. Continuous MaxEnt stays unread overflow. The exponential-family tour (\(\psi\) as a cumulant generating function; canonical, Gaussian, Bernoulli) is taught live in Week 6. Thermodynamic length is dropped.
No class test today. Quiz 2 was last week (start of lecture 4). Worksheet 2 was due last week. As delivered 10 November 2026: open with a short generative-AI policy aside (MIT report), then Quiz 2 debrief, then KL (overlap, Radon–Nikodym / Jaynes \(m\), mutual information, properties). Break before MaxEnt. After the break: MaxEnt problem, Lagrange on the board, Laplace / sunrise, Jaynes die, discrete MaxEnt proof. Exponential family introduced at the end (\(\theta=-\lambda\)); developed in Week 6. Continuous MaxEnt stayed unread through Week 6. The exponential-family development, including \(\log Z\) as a cumulant generating function, is the Week 6 opening after Quiz 3. Thermodynamic length is not that development.
This Session
Time plan (as delivered 10 November 2026). Clocks from the Whisper SRT on the Panopto capture (00:00 ≈ 10:00). The slot is 10:05–11:55 (110 minutes, not 120). Break was called before MaxEnt, not mid-MaxEnt. \(\log Z\) as CGF / full exponential-family tour were named at the end and developed in Week 6. Continuous MaxEnt stayed unread through Week 6. Thermodynamic length is dropped.
| Clock | Block |
|---|---|
| 10:02–10:06 | Generative-AI policy aside (MIT report; course page) |
| 10:06–10:09 | This Session / plan |
| 10:09–10:22 | Quiz 2 debrief (means \(8.2\) / \(6.3\); \(\log Z\); Clausius; pressure; Carnot mill) |
| 10:22–10:54 | KL arc: overlap \(\to\) MI; RN / relative entropy; KL properties |
| 10:54–11:05 | Break |
| 11:05–11:40 | MaxEnt: problem; Lagrange board; Laplace sunrise; Jaynes die |
| 11:40–11:58 | Discrete MaxEnt proof \(\to\) Boltzmann / \(Z\); exponential family introduced |
| 11:58–12:00 | Close: quiz and worksheet next week |
How This Course Uses Generative AI
Spoken at the open of 10 November 2026, before Quiz 2 feedback. Prompted by an MIT report on AI and education that asks instructors to say plainly how they use generative AI and to give students a policy with the reason attached. The report’s only fault, as spoken, is that it was written in the wrong Cambridge.
The course page under the course being such an AI lists where generative AI is used to build materials. The emails with comments are auto-populated from marks Neil has already written with the script in front of him — the human stays in the loop; the tool removes the mechanical retyping. That is the same spirit as the LLM worksheets: explore around a contested narrative, then defend what you understand in the room.
Homework from Weeks 3–4
No class test today. Feedback on Quiz 2 first; Worksheet 2 follows in a later pass of this include. Marks do not appear on these slides, and names do not either. The numbers below are cohort figures from the marked scripts. As delivered 10 November 2026 the debrief ran about thirteen minutes, with Carnot-as-waterwheel colour and a short contested-narrative aside before the KL arc.
Quiz 2
The quiz tested ten items on Weeks 2–3 as delivered. It looked at the thread we followed on the history of heat and then Helmholtz’s decomposition into the free energy (\(U\), \(S\), \(\log Z\)). It considered Shannon entropy \(H\) versus physical entropy, \(S\), and the notion of an ensemble, and intensive / extensive variables. The first matching question and the units blank were mostly correctly done. The points tended to be lost on multi-select items.
Named matching (Black, Carnot, Rumford, Joule), the Joule blank on \(k_BTH\), hurricane / grid coarse-graining, and \(S=k_BH\) with units were mostly got when attempted. The two ideas worth looking at again are what \(\log Z\) is, and what Clausius was doing in the heat thread (not naming uncertainty). Pressure as intensive was the third recurring miss: pressure and temperature are weather-map fields (isobars); molecule count and heat energy are extensive — that is why you can draw fields of \(p\) and \(T\).
There were some confusions where people suggested that \(\log Z=U\) and \(\log Z=H\). The Zustandssumme etymology was often known on its own; the available-energy reading was the part that dropped. As a reminder inner energy is composed of an entropy term and an available-energy term in \(\log Z\). Equivalently \(F=U-TS=-k_BT\log Z\). Physical entropy versus Shannon: Boltzmann’s constant carries joules per kelvin; \(\log Z\) is dimensionless; physicists often omit \(k_B\). We’ll see some related ideas when we look at MaxEnt today.
We haven’t provided a good intuition why the partition function is available energy.
Clausius did not coin entropy to name uncertainty. He was dealing with the thermodynamic bookkeeping that arose from combining Joule and Carnot. Entropy was not yet a preamble for Shannon. Uncertainty-as-entropy sits in the statistical tradition (Boltzmann, Gibbs, Wiener) and in Shannon’s \(H\). The quiz was focussed on the engineering story of the heat thread, which sets context for why it wasn’t seen as uncertainty until the rise of statistical mechanics. The Carnot miss \(Q_\mathrm{h}/T_\mathrm{h}=Q_\mathrm{c}/T_\mathrm{c}\) is the reversible-engine balance: entropy entering the hot reservoir equals entropy leaving the cold one.
We can see Carnot’s cycle as analogous to a waterwheel. Entropy enters the bucket at high \(T\), drops through the engine, and leaves at low \(T\); \(T\times S\) is an energy in joules. A reversible engine spills no water. Water that misses the wheel bucket does no work. (Note that kinetic waterwheels (no height drop) remind us that more is going on in a water wheel.
Reminder: you are not being marked on the narrative, it provides structure for you to explore with the LLM worksheets.
As Taught
As taught on 10 November the Quiz 2 debrief ran long with Carnot-as-waterwheel descriptions. Then the KL arc filled the first half: exclusive coarse-graining revisited, soft leftover \(H(G\mid X)\), mutual information as that leftover and as a KL to the product of marginals, Radon–Nikodym / relative entropy, KL properties. The break was called before MaxEnt. After the break: the constrained problem, Lagrange on the board (quadratic subject to \(x+y=1\)), Laplace’s sunrise / insufficient reason into Jaynes, the die at mean 4.5, and the discrete MaxEnt stationarity proof that recovers \(p_i\propto e^{-\sum\lambda_k f_k}/Z\). The exponential family and natural parameters \(\theta=-\lambda\) were introduced in the last minute as next week’s object. The exponential-family tour is live in Week 6 (17 November), after Quiz 3: \(\psi\) as CGF, base measure, Fisher / natural gradient. Continuous MaxEnt was not reached in Week 6 either and remains unread overflow. Thermodynamic length is dropped.
Information, energy and intelligence course notebook setup
We install some bespoke code for creating and saving plots as well as loading data sets.
import importlib.utilcmd = install_command('pods')%system {cmd}cmd = install_command('notutils')%system {cmd}cmd = install_command('mlai')%system {cmd}KL: Overlap, Measure, Mutual Information
Week 4 stopped at exclusive partitions. Soft mixtures leave leftover uncertainty about the label. That leftover is mutual information, and mutual information is a KL. Before MaxEnt we need that language, and we need to recognise that Week 4’s undeclared reference \(m\) was already a KL.
When Coarse Graining Overlaps
Week 4 derived \[\begin{align} H(X)=H(w)+\sum_g w_g\, H(p_g) \end{align}\] under the exclusive assumption \(G=g(x)\), where \(g(x)\) is a function that maps each point \(x\) to precisely one group. This means that we have \(H(G\mid X=x)=0\), i.e. if we know \(x\) we know \(g\).
What if we consider a soft labelling, such that each point \(x\) is mapped to a different group \(g\) with some probability? Keep the cell weights as before, with a soft label \(g\in G\), joint \(p(x,g)\), prior cell weights \(w_g=p(g)\), and \(p_g(x)=p(x\mid g)\). The chain rule on the joint gives \[\begin{align} H(X,G)=H(G)+H(X\mid G)=H(X)+H(G\mid X). \end{align}\] Expand \(H(X\mid G)=\sum_g w_g H(p_g)\) and take the marginal entropy of the coarse label as \(H(G)=H(w)\) to obtain \[\begin{align} H(X)=H(w)+\sum_g w_g\, H(p_g)-H(G\mid X). \end{align}\] The new term \(H(G\mid X)\) is zero when the partition is exclusive: given \(x\) you already know the group. Soft mixtures (mixture of Gaussians, mixture of experts) leave \(H(G\mid X)>0\): an observation \(x\) can still be associated with more than one group. That leftover uncertainty about the label is what mutual information names next.
\[ I(X;G)=H(G)-H(G\mid X)=H(X)-H(X\mid G) \]
Mutual information is the pairwise reading of that leftover term from the chain rule, \[\begin{align} I(X;I) = H(G)-H(G\mid X) = H(X)-H(X\mid G) = H(X)+H(G)-H(X,G). \end{align}\] Substitute into the overlapping identity: \[\begin{align} H(X) = H(X\mid G)+I(X;G) = \sum_g w_g\, H(p_g)+I(X;G), \end{align}\] and equivalently \[\begin{align} H(X) = H(w)+\sum_g w_g\, H(p_g)-H(G\mid X), \end{align}\] because \(I(X;G)=H(w)-H(G\mid X)\). Exclusive coarse-graining is the special case where \(H(G\mid X)=0\), so mutual information is maximised, \(I(X;G)=H(w)\), and the week-4 split is recovered. Overlapping coarse-graining keeps a positive \(H(G\mid X)\): even given \(X=x\) you are not confident which group \(g\) it is allocated to. The relationship is no longer a function \(g(x)\); it is a stochastic map \(p(g\mid x)\). For Shannon that leftover, under the right conditions, is channel capacity — the proof waits.
\[ I(X;G)=\text{KL}\left( p(x,g)\,\|\,p(x)\,p(g) \right) \]
Writing \(I(X;G)=\sum_{x,g} p(x,g)\log\frac{p(x,g)}{p(x)p(g)}\) is a KL-divergence, \(\text{KL}\left( p(x,i)\,\|\,p(x)p(i) \right)\). Independence is when the divergence is zero, so \(I=0\). Exclusive deterministic labelling saturates the KL divergence, the joint distribution is as different from the independent as possible (the mapping from \(x\) to \(g\) is deterministic) and therefore \(I=H(G)\).
Relative Entropy Is KL
Radon-Nikodym Derivative
The Form of the Radon-Nikodym Derivative
\[ \mu(A)=\int_A \frac{\mathrm{d}\mu}{\mathrm{d}\nu}\,\mathrm{d}\nu \]
Absolute continuity is the continuum name for what the shared partition already enforced: every cell that carries mass under one description carries mass under the other. When a measure \(\mu\) is absolutely continuous with respect to a measure \(\nu\) on the same space, there exists a nonnegative density - the Radon-Nikodym derivative - such that \[\begin{align} \mu(A) = \int_A \frac{\mathrm{d}\mu}{\mathrm{d}\nu}\,\mathrm{d}\nu \end{align}\] for every measurable \(A\).
Now let’s focus on the densities that we’ve been considering. Let \(F\) have density \(f\) against Lebesgue measure \(\lambda\), and let \(M\) be Jaynes’ reference measure, with density \(m\) against the same \(\lambda\). Provided \(m(x)=0\) only where \(f(x)=0\) (so \(F\ll M\)), we can write \[\begin{align} F(A) &= \int_A f\,\mathrm{d}x \nonumber\\ &= \int_A \frac{f}{m}\, m\,\mathrm{d}x \nonumber\\ &= \int_A \frac{f}{m}\,\mathrm{d}M, \end{align}\] The middle step is just the statement that \(M\) assigns mass \(m(x)\,\mathrm{d}x\) to the sliver \([x,x+\mathrm{d}x]\), so \(\mathrm{d}M = m\,\mathrm{d}x\) (just as \(\mathrm{d}F = f\,\mathrm{d}x\)). In Jaynes’ picture \(m(x)\,\mathrm{d}x\) is the fraction of reference points in the sliver, so \(\int_A g\,\mathrm{d}M\) is the average of \(g\) over the points in \(A\), and \(f/m\) is the mass of \(F\) carried per reference point. By uniqueness of the Radon–Nikodym derivative \[\begin{align} \frac{\mathrm{d}F}{\mathrm{d}M} = \frac{f}{m}. \end{align}\] This is the chain rule \(\frac{\mathrm{d}F}{\mathrm{d}M} = \frac{\mathrm{d}F/\mathrm{d}\lambda}{\mathrm{d}M/\mathrm{d}\lambda}\). Under a change of variables both \(f\) and \(m\) pick up the same Jacobian, which cancels in the ratio, so \(f/m\) is coordinate-free even though neither density is. Relative entropy is the expectation under \(F\) of the log of that ratio, \[\begin{align} \text{KL}\left( f\,\|\,m \right) = \int f\log\frac{\mathrm{d}F}{\mathrm{d}M}\,\mathrm{d}x = \int f\log\frac{f}{m}\,\mathrm{d}x, \end{align}\] and it is finite only when the absolute-continuity condition holds.
The bin-width ratio \(\Delta s_i/\Delta u_i\) of the previous snippet is the same statement on a finite partition: it is the discrete Radon-Nikodym derivative of Lebesgue-on-speed with respect to Lebesgue-on-volume. In the continuum that ratio becomes the Jacobian. Naming \(\mathrm{d}F/\mathrm{d}M=f/m\) makes the next step - declaring \(m\) - a change of what one divides by, not a change of vocabulary.
Jaynes: Promote the Bins to a Measure
\[ h_m(f)=-\int f\log\frac{f}{m}\,{\mathrm{d}}x=-\text{KL}\left( f\,\|\,m \right) \]
Differential entropy \(h=-\int f\log f\,{\mathrm{d}}x\) pretends the coordinate measure is part of the data. The histogram arithmetic showed that it is not: \(H\) is a functional of masses, and the passage to a density subtracts \(\sum_i p_i\log\Delta_i\). Jaynes called \(m\) the limiting density of discrete points — the binning measure, promoted to a density (Jaynes, 1963).
Concretely, Jaynes places \(N\) points so that the number falling in \([a,b]\) is approximately \(N\int_a^b m\,\mathrm{d}x\). Each point then owns a cell of width \(\Delta_i\approx 1/(N m(x_i))\) and reference mass \(1/N\): \(m\) says where the cells are fine and where they are coarse, and every cell is equally likely a priori. For a Gaussian \(m\) the points are its quantiles, crowded in the centre and sparse in the tails. With \(p_i\approx f(x_i)\Delta_i\), \[\begin{align} H_N = -\sum_i p_i\log p_i \approx \log N - \int f\log\frac{f}{m}\,\mathrm{d}x, \end{align}\] so the divergence is a pure count of cells, \(\log N\), and what remains is \(h_m\). When \(f=m\) every cell gets mass \(1/N\), \(H_N=\log N\) and \(h_m=0\), its maximum: \(h_m\) measures how far \(f\) is from spreading evenly over the cells that \(m\) declares.
Once \(m\) is written down, \[\begin{align} h_m(f) = -\int f\log\frac{f}{m}\,{\mathrm{d}}x = -\text{KL}\left( f\,\|\,m \right). \end{align}\] Taking \(m\) uniform (Lebesgue) in the Cartesian component \(v_x\) recovers the ordinary Gaussian remainder \(h(v_x)\) (an improper \(m\); \(h_m\) is then defined up to an additive constant). Taking \(m\) uniform in three-dimensional velocity, and writing it in the speed coordinate, gives \(m(s)\propto s^2\) — also improper — the shell volume that the concentric rings made geometric and that had been hiding in the Jacobian. The map \(\mathbf{v}\mapsto s=|\mathbf{v}|\) is not invertible. Equality of KL in the speed coordinate with KL in 3D velocity holds here because the ratio \(f/m\) depends on \(\mathbf{v}\) only through \(s\), so \(s\) is sufficient and the data-processing inequality holds with equality. In general the data-processing inequality gives \(\le\), not \(=\).
The two Maxwell–Boltzmann entropies disagreed because each \(h\) used a different undeclared \(m\). Relative entropy makes the choice explicit. The Jacobian cancels inside \(\text{KL}\left( f\,\|\,m \right)\) under a common invertible map; the discrete KL has no divergent \(-\log\Delta\) term.
Going continuous buys analytic machinery; it does not remove the need to declare \(m\). Which binning — which degrees of freedom — you treat as “equally likely a priori” remains an anthropomorphic choice in Jaynes’ sense. This names \(m\). Maximising \(h_m\) at fixed mean energy — Jaynes’ maximum entropy principle — is the next lecture.
Discrete Shannon Entropy Is Bounded
KL divergence \(\text{KL}\left( \cdot\,\|\,\cdot \right)\) measures information lost when \(q\) is used instead of \(p\). It is non-negative in both discrete and continuous settings (with common support). Unlike Shannon entropy, it is defined relative to a reference. Overlapping coarse-graining already wrote mutual information as \(\text{KL}\left( p(x,g)\,\|\,p(x)p(g) \right)\). MaxEnt and \(m\)-projections minimise \(\text{KL}\left( \cdot\,\|\,r \right)\) to a reference; that continuous reading is the unread Week 5 / Week 6 follow-on when the discrete proof fills the hour.
For \(\mathcal{N}(0,\sigma^2)\), differential entropy is \(\frac{1}{2}\log(2\pi e\sigma^2)\) in nats. As \(\sigma\to 0\) it diverges negatively. Discrete Shannon entropy stays in \([0,\log n]\). Thermodynamic \(S=kH\) in week 1 used the discrete sum form on Boltzmann probabilities. Continuous MaxEnt still works because constraints fix scale; comparing to a reference uses KL, which remains non-negative.
Figure: Left: binary Shannon entropy is bounded. Right: Gaussian differential entropy can be negative.
Maximum Entropy
If there were no constraints beyond normalisation, Shannon entropy on a finite alphabet is maximised by the uniform distribution. Thermodynamics adds constraints: mean energy fixed, or mean and variance, or a list of moments. The maximum entropy principle asks for the distribution of largest \(H\) among those that meet the constraints. That is today’s discrete calculation (Jaynes’ die), done after the break on 10 November. Equivalently, one may minimise \(\text{KL}\left( p\,\|\,m \right)\) to a declared reference measure \(m\) under the same constraints; Continuous MaxEnt stays unread overflow (not reached in Week 6 either).
Lagrange Multipliers
Many problems in physics, engineering and machine learning ask us to optimise a function subject to constraints. For example, free energy minimisation, or maximising entropy while matching observed averages.1 In machine learning, principal component analysis can be formulated as finding directions in data of maximum variance. Each direction is constrained to be orthogonal to the previous one. Lagrange multipliers turn these constrained problems into unconstrained ones.
One Equality Constraint
If we want to find a stationary point of a function, \(f(\mathbf{x})\), and we know that another function \(g(\mathbf{x}) = c\) is constrained such that \(g(\mathbf{x}) = c\) we introduce a Lagrange multiplier, often denoted by \(\lambda\) and form the Lagrangian, \[ \mathscr{L}(\mathbf{x}, \lambda) = f(\mathbf{x}) + \lambda\bigl(g(\mathbf{x}) - c\bigr). \] We are optimising with respect to \(\mathbf{x}\) and the gradient of the Lagrangian is given by \[ \nabla_{\mathbf{x}} \mathscr{L} = \nabla f + \lambda \nabla g, \] but we now also have the gradient of the Lagrangian with respect to the Lagrange multiplier, \[ \frac{\partial \mathscr{L}}{\partial \lambda} = g(\mathbf{x}) - c. \] Think about what happens when we set both gradients to zero (as we would do if we were looking for a stationary point of an unconstrained function). The first equation is now \[ \nabla f + \lambda \nabla g = \mathbf{0} \] which tells us that \(\nabla f\) is parallel to \(\nabla g\), and therefore the gradient of the objective, \(\nabla f\), is perpendicular to the constraint surface.
Figure: At stationarity, \(\nabla f + \lambda\nabla g = \mathbf{0}\): the objective gradient is balanced by the constraint gradient scaled by \(\lambda\) (here \(\lambda = -\frac{1}{2}\) in the worked example).
The second condition, \[ g(\mathbf{x}) - c = 0, \] just enforces the constraint itself (\(g(\mathbf{x}) = c\)).
Picture climbing a hill (maximise \(f\)) while staying on a path defined by \(g(\mathbf{x}) = c\). At the highest point reachable on the path, you cannot move along the path and still increase \(f\). That means \(\nabla f\) points off the path, normal to the constraint surface. The multiplier \(\lambda\) measures how strongly the constraint must pull against the unconstrained gradient.
Several Equality Constraints
With multiple constraints \(g_k(\mathbf{x}) = c_k\) for \(k = 1, \ldots, m\), introduce one multiplier \(\lambda_k\) for each constraint: \[ \mathscr{L}(\mathbf{x}, \boldsymbol{\lambda}) = f(\mathbf{x}) + \sum_{k=1}^m \lambda_k \bigl(g_k(\mathbf{x}) - c_k\bigr). \] Stationarity gives \[\begin{align} \nabla f + \sum_{k=1}^m \lambda_k \nabla g_k &= 0, \\ g_k(\mathbf{x}) &= c_k \quad k = 1, \ldots, m. \end{align}\] Each multiplier enforces one constraint; together they specify how far the objective gradient is rotated onto the intersection of the constraint surfaces.
A Worked Example
The method is clearer on a small problem before we apply it to probabilities. Minimise \[ f(x,y) = \frac{1}{2}(x^2 + y^2) \] subject to the linear constraint \(x + y = 1\). Geometrically, we seek the point on the line closest to the origin; physically, this is the same pattern as minimising a quadratic energy or loss while a conserved quantity remains fixed. The Lagrangian is \[ \mathscr{L}(x,y,\lambda) = \frac{1}{2}(x^2 + y^2) + \lambda(x + y - 1). \] Stationarity gives \(x + \lambda = 0\), \(y + \lambda = 0\), and \(x + y = 1\), hence \(x = y = \frac{1}{2}\) and \(\lambda = -\frac{1}{2}\). The gradient of \(f\) at the solution is \((1,1)\), parallel to the constraint normal \((1,1)\) as required.
The same template appears in machine learning: minimise a squared error or regulariser \(\frac{1}{2}\|\mathbf{w}\|^2\) subject to a linear prediction constraint \(\mathbf{w}^\top \mathbf{x} = y\). The multiplier \(\lambda\) then plays the role of a Lagrange multiplier in ridge regression and support-vector margins. We do not need that full story here; the point is that one constraint adds one multiplier and reduces the problem to solving linear stationarity conditions.
Principle of Insufficient Reason
Pierre-Simon Laplace

Figure: Pierre-Simon Laplace 1749-1827.
From Laplace’s perspective we use probability to deal with our ignorance about the data, the physical lawas and the computation. This is Laplace’s gremlin. But how do we estimate probabilities? How do we assign \(p\) when we know almost nothing?
Laplace’s principle of insufficient reason suggestd that if we have mutually exclusive outcomes and nothing distinguishes one from the other then we should asign them equal probability, i.e. \(p_i = \tfrac{1}{n}\).
The Philosophical Essay on Probabilities (Laplace, 1814) argued that probability is relative in part to our ignorance and in part to our knowledge. So when we know that only one of \(n\) mutually exclusive and exhaustive outcomes must occur we don’t have any reason to prefer one over the other. This suggests that we should assign \[\begin{align} p_i = \frac{1}{n}, \qquad i = 1,\ldots,n. \end{align}\] This known as Laplace’s principle of insufficient reason or the principle of indifference. It’s an assignment athat is ‘least-commital’ but consistent with the information we have. In this case the information we have is the different possibilities (the \(n\) outcomes) and the need for normalisation.
Example: Will the sun rise tomorrow?
As an example Laplace considered what the probability that the sun rises tomorrow is. To our knowledge, the sun has risen every day. So is the probability 100%? The example comes from Philosophical Essay on Probabilities (Laplace, 1814).
Figure: Each update is belief \(\times\) likelihood \(\rightarrow\) posterior; a reset frame then promotes the posterior to the new prior (likelihood cleared). Early steps use one more sunrise (\(\propto\theta\)); the jump to \(n=20\) uses the likelihood for \(17\) further rises (\(\propto\theta^{17}\)).
The animation peels probability mass toward \(\theta=1\). Quantify that with the differential entropy of the belief, \[\begin{align} H[p] = -\int_0^1 p(\theta)\log p(\theta)\,\mathrm{d}\theta. \end{align}\] Under insufficient reason the prior is uniform on \([0,1]\), so \(p(\theta)=1\) and \[\begin{align} H_0 = -\int_0^1 1\cdot\log 1\,\mathrm{d}\theta = 0. \end{align}\] After \(n\) sunrises the normalised belief is \(p_n(\theta)=(n+1)\theta^{n}\). Substitute and use \(\int_0^1\theta^{n}\,\mathrm{d}\theta=1/(n+1)\) together with \(\int_0^1\theta^{n}\log\theta\,\mathrm{d}\theta=-1/(n+1)^2\): \[\begin{align} H_n &= -\int_0^1 (n+1)\theta^{n}\bigl(\log(n+1)+n\log\theta\bigr)\,\mathrm{d}\theta = \frac{n}{n+1}-\log(n+1). \end{align}\] Each sunrise reduces this entropy (the number becomes more negative as the density sharpens). The information gained on day \(n\) is the drop \[\begin{align} I_n = H_{n-1}-H_n = \log\!\left(1+\frac{1}{n}\right)-\frac{1}{n(n+1)}. \end{align}\] The first day buys \(I_1=\log 2-\tfrac12\approx 0.193\) nats; the second buys slightly more (\(I_2\approx 0.239\)); thereafter \(I_n\) falls, asymptotically like \(1/(2n^2)\). After Laplace’s \(n\approx 1.8\times 10^6\) days, \(H_n\approx 1-\log n\) is large and negative, and each further day adds almost no information about \(\theta\).
Figure: Differential entropy \(H_n\) of \(p(\theta)\) after \(n\) sunrises, and the information \(I_n\) gained on day \(n\). The prior sits at \(H_0=0\); each success drives \(H_n\) downward.
A second, coarser reading is the binary entropy of tomorrow’s predictive probability \(p_n=(n+1)/(n+2)\): \[\begin{align} h(p_n)=-p_n\log p_n-(1-p_n)\log(1-p_n), \end{align}\] which starts at \(h(1/2)=\log 2\) and falls toward \(0\) as \(p_n\to 1\). That tracks uncertainty about the next day, whereas \(H_n\) tracks uncertainty about the latent rate \(\theta\).
Laplace applies insufficient reason not only to discrete outcomes, but to an unknown daily rate \(\theta\in[0,1]\). Knowing nothing about \(\theta\), he places a uniform prior \(p(\theta)=1\) on the unit interval. Every recorded day is a success: the sun rose. After \(n\) independent rises the likelihood is \(\theta^n\), so \[\begin{align} p(\theta\mid n\text{ rises}) \propto \theta^{n}, \end{align}\] and the probability of one further rise is the posterior mean \[\begin{align} P(\text{rises tomorrow}\mid n\text{ rises}) = \int_0^1 \theta\, p(\theta\mid n\text{ rises})\,\mathrm{d}\theta = \frac{n+1}{n+2}. \end{align}\] This is Laplace’s rule of succession (the same “\(+1\), \(+2\)” that later appears as Laplace smoothing). In modern notation the posterior is \(\mathrm{Beta}(n+1,1)\); after each new sunrise it becomes \(\mathrm{Beta}(n+2,1)\), more sharply peaked at \(\theta=1\), while the predictive odds move from \(1{:}1\) toward certainty without ever quite reaching it.
Laplace did not have the Beta distribution by name, and he did not need the Laplace approximation for this result. In the 1774 memoir on inverse probability he evaluates the normalising integrals exactly — what we now write as \(\int_0^1 \theta^{n}\,\mathrm{d}\theta = 1/(n+1)\) — via what he called Euler’s series. The Beta language is modern packaging of that calculation. The Laplace approximation (a local Gaussian expansion of an integrand about its mode) is a different tool, also named after him; it is not how the rule of succession was obtained.
In the Essay he takes the oldest historical epoch as five thousand years, or \(n=1{,}826{,}213\) days of recorded sunrise, and concludes that — on this information alone — the odds on tomorrow’s sunrise are \(1{,}826{,}214\) to one. He then adds, in the next sentence, that for anyone who recognises the astronomical regularity of days and seasons the probability is incomparably greater. The sunrise calculation is a demonstration of the rule under deliberate ignorance of mechanism, not Laplace’s estimate of whether the sun will rise.
So the preliminary example already shows the pattern we need: insufficient reason supplies the prior of ignorance; each success updates it and peels a little more mass away from small \(\theta\); the predictive probability is not the raw frequency \(n/n=1\).
The principle of insufficient reason, and the rule of succession built from it, answer the case where our information is a list of possibilities and (optionally) raw counts. As soon as we know more — a mean face value on a die, a mean energy, a variance — equal weights, or a uniform prior on a single rate, may fail to encode that information. Jaynes’ maximum entropy principle extends Laplace’s idea: among all distributions that match the stated constraints, choose the one that maximises Shannon entropy \(H(p)=-\sum_i p_i\log p_i\). That choice is maximally noncommittal with respect to everything else.
With only the normalisation constraint \(\sum_i p_i=1\), MaxEnt recovers Laplace’s uniform assignment \(p_i=1/n\). With an additional mean-energy constraint, the same calculation yields Boltzmann weights. With mean and variance, it yields a Gaussian. The exponential family is therefore the family of distributions that generalise insufficient reason from “we know nothing” (or “we know only these counts”) to “we know these moments and nothing more.”
Critiques of bare indifference (Bertrand’s paradox and related reparameterisation puzzles) motivate the need for an explicit state space and explicit constraints — which is exactly what the MaxEnt formalism supplies. We will not pursue those paradoxes here; the operational fix is to state what is held fixed before maximising \(H\). Laplace’s own sunrise caveat makes the same point in plain language: once you know the mechanism, you no longer pretend to know only the counts.

Figure: Ed Jaynes who developed the maximum entropy principle
Maximum Entropy Motivation
Ed Jaynes (Jaynes, 1957) proposed a foundation for statistical mechanics based on information theory. He recast the problem of assigning probabilities as inference with incomplete information: where do the distributions come from, and why \(e^{-\beta E}\)?
A central problem in statistical mechanics is assigning initial probabilities when our knowledge is incomplete. The canonical example is if we know only the average energy of a system, what probability distribution should we use? Jaynes argued that we should use the distribution that maximises entropy subject to the constraints of our knowledge. The formalism is more general than energy: any expectation constraint; for energy the Lagrange multiplier is inverse temperature \(\beta\). Physicists’ skepticism (``we already knew that law’’) and ML skepticism both sit here — MaxEnt as a crossover that re-derives physics and underwrites the exponential family.
Jaynes illustrated the approach with a simple example. If a die has been tossed many times, with an average result of 4.5 rather than the expected 3.5 for a fair die, what probability assignment \(P_n\) (\(n=1,2,\ldots,6\)) should we make for the next toss? The same Lagrange move recovers Boltzmann weights under a mean-energy constraint and the Gaussian under mean and variance.
We need to satisfy two constraints \[\begin{align} \sum_{n=1}^6 P_n &= 1 \\ \sum_{n=1}^6 n P_n &= 4.5 \end{align}\]
Many distributions could satisfy these constraints, but which one makes the fewest unwarranted assumptions? Jaynes argued that we should choose the distribution that is maximally noncommittal with respect to missing information - the one that maximises the entropy, \[\begin{align} S_I = -\sum_{i} p_i \log p_i \end{align}\] This principle leads to the exponential family of distributions, which in statistical mechanics gives us the canonical ensemble and other familiar distributions.
Die Roll Simulation
This simulation illustrates the maximum entropy principle through Jaynes’ dice example (Jaynes, 1957). A fair die has expected outcome 3.5; the Jaynes example asks: if we know only that the average outcome is 4.5, what probability distribution \(P_n\) over the six faces should we assign?
The answer is the maximum-entropy distribution subject to the constraint \(\sum_{n=1}^6 n P_n = 4.5\), which belongs to the exponential family: \[\begin{align} P_n = \frac{e^{\lambda n}}{Z(\lambda)}, \qquad Z(\lambda) = \sum_{n=1}^6 e^{\lambda n} \end{align}\] where \(\lambda > 0\) is chosen so the mean constraint is satisfied. This avoids any unwarranted assumption beyond the available data.
Rolls: 0
Sample mean: —
H(p): —
Outcome weights (auto-normalised to probabilities)
Figure: Interactive die-roll simulation. Click the die or press Roll to sample from the configured distribution. The histogram shows empirical relative frequencies (coloured bars) overlaid on the theoretical probabilities (dashed outlines). Use the sliders to set arbitrary outcome weights, or click a preset to load the uniform distribution (mean 3.5), the Jaynes maximum-entropy distribution (mean 4.5), the simple 50/50 distribution (faces 4 and 5 equally, mean 4.5 with minimal entropy), or a low-biased distribution (mean 2).
Maximum entropy forbids assuming more structure than the constraints; probability is the recipe. One mean-energy constraint gives Boltzmann weights; mean and variance give the Gaussian. The Lagrange multiplier on energy is coldness \(\beta\).
Figure: Jaynes’ die: MaxEnt subject to mean 4.5 versus the uniform distribution.
Derivation of the Boltzmann Distribution
For a given mean energy \(U\), among all distributions with that \(U\), pick the one with largest entropy. Week 2 introduced the Gibbs–Boltzmann occupation; this is the MaxEnt derivation of it.
Maximum entropy subject to normalisation and fixed mean energy gives \(p_i = e^{-\beta E_i}/Z\). The Lagrange multiplier on the energy constraint is coldness \(\beta\) from weeks 1–2. Equilibrium only: saturated entropy under an energy constraint does not give non-equilibrium dynamics. Mean and variance instead of energy recover the Gaussian — the ideal-gas-in-the-room reading of the same Lagrange move.
Figure: Occupation of a two-state system as coldness increases. At low \(\beta\) both states are populated; at high \(\beta\) the ground state dominates.
# Live check: boltzmann([0, 1], 1.0) -> (0.731, 0.269)
boltzmann([0, 1], 1.0)MaxEnt Under Constraints: the Proof
Fix a finite outcome set \(\{x_1,\ldots,x_n\}\) and unknown probabilities \(p_i>0\). The maximum entropy principle asks for the \(p_i\) that maximise Shannon entropy \[\begin{align} H(p) = -\sum_{i=1}^n p_i \log p_i \end{align}\] subject to normalisation and \(m\) moment constraints \[\begin{align} \sum_{i=1}^n p_i = 1, \qquad \sum_{i=1}^n p_i f_k(x_i) = \langle f_k\rangle \quad (k=1,\ldots,m). \end{align}\] Introduce Lagrange multipliers \(\lambda_0,\lambda_1,\ldots,\lambda_m\) and form \[\begin{align} \mathscr{L}(p,\lambda) = \sum_i p_i \log p_i + \lambda_0\Bigl(\sum_i p_i - 1\Bigr) + \sum_{k=1}^m \lambda_k\Bigl(\sum_i p_i f_k(x_i) - \langle f_k\rangle\Bigr). \end{align}\] At an interior maximum, \(\partial \mathscr{L}/\partial p_i = 0\) gives \[\begin{align} \log p_i + 1 + \lambda_0 + \sum_{k=1}^m \lambda_k f_k(x_i) = 0, \end{align}\] so \[\begin{align} p_i = \exp\Bigl(-1-\lambda_0\Bigr)\, \exp\Bigl(-\sum_{k=1}^m \lambda_k f_k(x_i)\Bigr). \end{align}\] Absorbing \(\exp(-1-\lambda_0)\) into the normalisation constant, \[\begin{align} p_i = \frac{\exp\bigl(-\sum_{k=1}^m \lambda_k f_k(x_i)\bigr)} {Z(\lambda_1,\ldots,\lambda_m)}, \qquad Z = \sum_{i=1}^n \exp\Bigl(-\sum_{k=1}^m \lambda_k f_k(x_i)\Bigr). \end{align}\] The multipliers are fixed by substituting this \(p\) back into the constraints. The maximum entropy is \[\begin{align} H_{\max} = \log Z + \sum_{k=1}^m \lambda_k \langle f_k\rangle. \end{align}\] This is the same calculation as Jaynes’ die (one linear constraint on the face value) and, with \(f_1(x_i)=E_i\), the canonical ensemble.
Minimise KL to a Measure
Write continuous MaxEnt as minimising KL (relative entropy) to a reference measure \(m\), \[\begin{align} \text{KL}\left( f\,\|\,m \right)=\int f\log\frac{f}{m}\,dx = D(f\|m), \end{align}\] or equivalently maximising \(h_m(f)=-D(f\|m)\), subject to mean kinetic energy \(\langle E\rangle=\langle\tfrac12 m_{\mathrm{mass}}|\mathbf{v}|^2\rangle\). The solution is \[\begin{align} f(\mathbf{v})\propto m(\mathbf{v})\,e^{-\beta E(\mathbf{v})}. \end{align}\] If \(m\) is Lebesgue on a Cartesian component \(v_x\), the MaxEnt density is Gaussian. If \(m\) is Lebesgue on three-dimensional velocity space, then in the speed coordinate the reference density is proportional to the shell volume \(v^2\), and MaxEnt recovers the Maxwell–Boltzmann speed distribution. The \(v^2\) factor is the density of states. Declaring \(m\) makes the choice of “equally likely a priori” explicit instead of hiding it in the coordinate chart.
\[ f\propto m(\cdot)\,e^{-\beta E} \qquad \begin{cases} m\sim\mathrm{Lebesgue}(v_x) & \Rightarrow\text{Gaussian component}\\ m\sim v^2\,dv & \Rightarrow\text{Maxwell--Boltzmann} \end{cases} \]
As taught on 10 November the continuous MaxEnt / density-of-states reading was not reached after the discrete proof. It was not reached in Week 6 either and remains unread overflow.
The General Maximum-Entropy Formalism
For a more general case, suppose a quantity \(x\) can take values \((x_1, x_2, \ldots, x_n)\) and we know the average values of several functions \(f_k(x)\). The problem is to find the probability assignment \(p_i = p(x_i)\) that satisfies \[\begin{align} \sum_{i=1}^n p_i &= 1 \\ \sum_{i=1}^n p_i f_k(x_i) &= \langle f_k(x) \rangle = F_k \quad k=1,2,\ldots,m \end{align}\] and maximises the entropy \(S_I = -\sum_{i=1}^n p_i \log p_i\).
Using Lagrange multipliers, the solution is the generalised canonical distribution, \[\begin{align} p_i = \frac{\exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i))}{Z(\lambda_1,\ldots,\lambda_m)} \end{align}\] where \(Z(\lambda_1,\ldots,\lambda_m)\) is the partition function, \[\begin{align} Z(\lambda_1,\ldots,\lambda_m) = \sum_{i=1}^n \exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i)) \end{align}\] The Lagrange multipliers \(\lambda_k\) are determined by the constraints, \[\begin{align} \langle f_k \rangle = -\frac{\partial}{\partial \lambda_k}\log Z(\lambda_1,\ldots,\lambda_m) \quad k=1,2,\ldots,m. \end{align}\] The maximum attainable entropy is \[\begin{align} (S_I)_{max} = \log Z + \sum_{k=1}^m \lambda_k \langle f_k \rangle. \end{align}\]
\[ p_i = \frac{\exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i))}{Z(\lambda_1,\ldots,\lambda_m)} \] \[ Z(\ldots) = \sum_{i=1}^n \exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i)) \] \[ \langle f_k \rangle = -\frac{\partial}{\partial \lambda_k}\log Z(\lambda_1,\ldots,\lambda_m) \quad k=1,2,\ldots,m. \]
The Exponential Family
As taught on 10 November the exponential family was introduced in the last minute — natural parameters \(\theta=-\lambda\), fit the \(\lambda\)s by maximum likelihood — and not developed. The live version is Week 6, after Quiz 3: \(\theta=-\lambda\), canonical / Gaussian / Bernoulli, and \(\psi\) as a cumulant generating function.
The constrained MaxEnt distribution is not merely like an exponential family — on a finite (or countable) state space it is one. Take sufficient statistics \(T_k(x)=f_k(x)\). Then the MaxEnt assignment is \[\begin{align} p(x\mid\boldsymbol{\theta}) = \exp\bigl(\boldsymbol{\theta}\cdot T(x) - \psi(\boldsymbol{\theta})\bigr)\, h_0(x), \end{align}\] with \(h_0(x)=\mathbf{1}[x\in\mathcal{X}]\) the density of counting measure \(\#\) on the allowed outcomes (Week 6 writes \(h_0\) rather than \(h\), to avoid colliding with differential entropy and the regulator map) and \[\begin{align} \psi(\boldsymbol{\theta}) = \log \sum_x \exp\bigl(\boldsymbol{\theta}\cdot T(x)\bigr). \end{align}\] The Lagrange multipliers from the proof are the negative natural parameters: \[\begin{align} \theta_k = -\lambda_k. \end{align}\] We use the minus sign so that high-energy states are down-weighted when \(\theta_1=-\beta<0\) in the canonical ensemble. The Quiz 2 debrief read \(\log Z\) as available energy / \(F=-k_BT\log Z\). The same \(Z\) is also a cumulant generating function.
\(\log Z\) as Cumulant Generating Function
The Partition Function
The partition function \(Z(\beta)=\sum_i e^{-\beta E_i}\) is a generating function: \(U=-\partial_\beta\log Z\), \(F=-\beta^{-1}\log Z\), \(S=\beta(U-F)\). The canonical ensemble is an equilibrium construction. Thermodynamic length is not part of this calculation.
Figure: Thermodynamic quantities from \(Z(\beta)\) for the two-state system used as a running example.
import numpy as np
def partition(energies, beta):
return np.sum(np.exp(-beta * np.asarray(energies)))
def thermo_from_Z(beta, energies):
e = np.asarray(energies, dtype=float)
Z = partition(e, beta)
p = np.exp(-beta * e) / Z
U = np.sum(p * e)
F = -np.log(Z) / beta
S = beta * (U - F)
return U, S, FThe two-state plots recover \(U=-\partial_\beta\log Z\) and \(F=-\beta^{-1}\log Z\). In exponential-family coordinates the same object is \(\psi(\boldsymbol{\theta})\): \(\eta=\nabla\psi\) are the mean parameters conjugate to \(\theta\), and \(\nabla^2\psi\) is the covariance of the sufficient statistics. Week 6 teaches that cumulant reading and stops there.
Exponential Families
The exponential family is a class of probability distributions that plays a central role in information theory and statistical mechanics. For our purposes, these distributions have particularly elegant properties when we work with their natural parameters.
Definition
A probability distribution belongs to the exponential family if it can be written in the form \[
p(\mathbf{x} | \boldsymbol{\theta}) = \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x}) - \mathcal{A}(\boldsymbol{\theta})\right) h(\mathbf{x}),
\] where - \(\boldsymbol{\theta} \in \mathbb{R}^d\) are the natural parameters - \(T(\mathbf{x})\) are the sufficient statistics
- \(\mathcal{A}(\boldsymbol{\theta})\) is the log partition function (ensures normalization) - \(h(\mathbf{x})\) is the base measure (often constant)
The natural parameters are called “natural” because they appear linearly in the exponent. This parametrization has special geometric properties that make information-theoretic calculations particularly clean. As we’ll see, working with these connections makes the connections between probability theory and geometry particularly clean.
The Log Partition Function as Cumulant Generator
The setup of the exponential family ensures that the log partition function operates as a cumulant generating function. We denote it by \(\mathcal{A}(\boldsymbol{\theta})\) and it ensures the distribution is normalised. \[ \mathcal{A}(\boldsymbol{\theta}) = \log \int \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x})\right) h(\mathbf{x}) \, d\mathbf{x}. \] But \(\mathcal{A}\)’s role as cumulant generating function also means that we can compute the distributions cumulants through taking its derivatives.
\[ \mathcal{A}(\boldsymbol{\theta}) = \log \int \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x})\right) h(\mathbf{x}) \, d\mathbf{x} \]}
The derivatives of \(\mathcal{A}\) with respect to \(\boldsymbol{\theta}\) give us the cumulants as follows. - First derivative (first cumulant): \(\nabla \mathcal{A}(\boldsymbol{\theta}) = \mathbb{E}[T(\mathbf{x})]\) (the mean) - Second derivative (second cumulant): \(\nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}[T(\mathbf{x})]\) (the covariance) - Higher derivatives: Third and higher cumulants (skewness, kurtosis, etc.)
Connection to Fisher Information
For an exponential family, the second derivative of the log partition function has a special meaning. It is the Fisher information matrix: \[ G(\boldsymbol{\theta}) = \nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}_{\boldsymbol{\theta}}[T(\mathbf{x})]. \] R.A. Fisher introduced this quantity in the context of statistical estimation. Through the Cramér-Rao bound, Fisher information tells us how accurately we can estimate parameters from data, higher Fisher information means better estimation is possible.
The Fisher information is both the Hessian of the log partition function and the covariance of the sufficient statistics. This dual role arises from the log partition function being the cumulant generating function. As we’ll see in the next section, the Fisher information matrix also defines the geometry of the information space.
Entropy in Natural Coordinates
The joint entropy of an exponential family distribution has a straightforward form in natural coordinates, \[ H(\boldsymbol{\theta}) = -\mathbb{E}_{\boldsymbol{\theta}}[\log p(\mathbf{x}|\boldsymbol{\theta})] = \mathcal{A}(\boldsymbol{\theta}) - \boldsymbol{\theta}^\top \nabla \mathcal{A}(\boldsymbol{\theta}) - \mathbb{E}[\log h(\mathbf{x})]. \] When the base measure \(h(\mathbf{x})\) is constant (or when we measure entropy relative to it), this simplifies to \[ H(\boldsymbol{\theta}) = \mathcal{A}(\boldsymbol{\theta}) - \boldsymbol{\theta}^\top \boldsymbol{\mu}, \] where \(\boldsymbol{\mu} = \nabla \mathcal{A}(\boldsymbol{\theta})\) is the mean of the sufficient statistics.
Now we see the utilty of representing through natural parameers, when we take the gradient of entropy with respect to the natural parameters, many terms cancel, \[ \nabla_{\boldsymbol{\theta}} H = \nabla \mathcal{A}(\boldsymbol{\theta}) - \left(\boldsymbol{\theta}^\top \nabla^2 \mathcal{A} + \nabla \mathcal{A}\right) = -\boldsymbol{\theta}^\top \nabla^2 \mathcal{A} = -G(\boldsymbol{\theta})\boldsymbol{\theta}. \] This simple form for the entropy gradient equals negative Fisher information times natural parameters.
Example: Multivariate Gaussian
Let’s make this concrete with the multivariate Gaussian distribution. A Gaussian with mean \(\boldsymbol{\mu}\) and covariance \(\Sigma\) can be written as: \[ p(\mathbf{x}) = \frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}} \exp\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})\right). \] To put this in exponential family form, we need to identify the natural parameters and sufficient statistics.
Gaussian in Exponential Family Form
After expanding the quadratic form and collecting terms, the Gaussian becomes: \[ p(\mathbf{x}) = \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x}) - \mathcal{A}(\boldsymbol{\theta})\right) \cdot \frac{1}{(2\pi)^{d/2}}, \] where: - Natural parameters: \(\boldsymbol{\theta} = \Sigma^{-1}\boldsymbol{\mu}\) (precision-weighted mean) - Sufficient statistics: \(T(\mathbf{x}) = \mathbf{x}\) (the data itself) - Log partition function: \(\mathcal{A}(\boldsymbol{\theta}) = \frac{1}{2}\boldsymbol{\theta}^\top\Sigma\boldsymbol{\theta} + \frac{1}{2}\log|\Sigma|\)
For the Gaussian, we can compute the Fisher information matrix directly: \[ G(\boldsymbol{\theta}) = \nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \Sigma. \] As expected, because the log partition is the covariance matrix. Equivalently, \(G^{-1} = \Sigma^{-1}\) is the precision matrix.
This reveals the intuition behind Fisher’s original concept: - Low variance (high precision) → high Fisher information → parameters easier to estimate (tighter Cramér-Rao bound) - High variance (low precision) → low Fisher information → parameters harder to estimate (looser bound)
Example: Categorical Distribution
Consider a categorical distribution over \(K\) outcomes with probabilities \(\{\pi_1, \ldots, \pi_K\}\) where \(\sum_k \pi_k = 1\). We represent an outcome as a one-hot vector \(\mathbf{x} \in \{0,1\}^K\) where exactly one component is 1.
Categorical in Exponential Family Form
The categorical distribution is: \[ p(\mathbf{x}) = \prod_{k=1}^K \pi_k^{x_k} = \exp\left(\sum_{k=1}^K x_k \log \pi_k\right). \] In exponential family form (using \(K-1\) natural parameters due to the sum-to-one constraint): - Natural parameters: \(\boldsymbol{\theta} = (\log \pi_1, \ldots, \log \pi_{K-1})\) (log probabilities) - Sufficient statistics: \(T(\mathbf{x}) = (x_1, \ldots, x_{K-1})\) (one-hot encoding) - Log partition function: \(\mathcal{A}(\boldsymbol{\theta}) = \log\left(\sum_{k=1}^{K-1} e^{\theta_k} + 1\right)\) (log-sum-exp)
The Fisher information matrix for the categorical distribution is: \[ G_{ij}(\boldsymbol{\theta}) = \begin{cases} \pi_i(1-\pi_i) & \text{if } i=j \\ -\pi_i\pi_j & \text{if } i \neq j \end{cases} \] This matrix encodes how “information” is distributed across categories. When one category dominates (\(\pi_i \approx 1\)), the Fisher information about other categories becomes small—it’s hard to learn about rare events!
Why Exponential Families Matter
Exponential families are not just mathematically convenient, they emerge naturally from underlying principles:
Maximum Entropy: Given constraints on the moments of a distribution, the maximum entropy distribution is in the exponential family (Jaynes, 1957).
Information Geometry: The natural parameters define a coordinate system on the manifold of probability distributions where the Fisher information acts as a Riemannian metric (Amari, 2016).
Sufficient Statistics: The sufficient statistics \(T(\mathbf{x})\) capture all the information in the data needed to estimate \(\boldsymbol{\theta}\), nothing is lost by summarizing data through \(T\).
Computational Tractability: Many operations (computing gradients, conjugate priors, message passing) have closed-form solutions.
For the inaccessible game, exponential families provide the setting where information dynamics have their cleanest mathematical form.
The definition \(p(x\mid\boldsymbol{\theta})=\exp(\boldsymbol{\theta}\cdot T(x)-\psi(\boldsymbol{\theta}))\) is the notation for what we have just derived. Canonical, Gaussian, and Bernoulli belong because each is MaxEnt for its moments. Softmax is the same calculation with a feature map. The two-spin example is the first sufficient statistic that is a product.
Three Perspectives (Brief)
Clausius gives the macroscopic second law and names entropy. Shannon treats \(H\) as a code bound; Szilard and Landauer tie one bit to \(k_BT\ln 2\) of work and erasure; Boltzmann counts macrostates at equilibrium; Jaynes treats \(p\) as least-committal inference under constraint. All agree: \(H\) forbids, \(p\) prescribes. Continuous MaxEnt remains unread overflow.
Define This Week
After This Lecture
No class test today. As closed on 10 November: quiz and worksheet next week. Quiz 3 is 17 November at the start of lecture 6, on Weeks 4–5 as delivered (grouping, soft partitions / MI, KL, MaxEnt, exponential family introduced). After the quiz, Week 6 develops the exponential family (\(\psi\) as CGF; canonical, Gaussian, Bernoulli) and Fisher / natural gradient. As taught, Shannon capacity, DPI, and full GRT diagrams were not reached on 17 November (GRT was a W4 pointer only). Thermodynamic length is not on that path.
Further Reading
limiting density of discrete points of Jaynes (1963)
Chapter 12 of Cover and Thomas (1991)
the whole paper of Jaynes (1957)
Chapter 22 of MacKay (2003)
Chapters 11–12 of Jaynes (2003)
Thanks!
For more information on these subjects and more you might want to check the following resources.
- company: Trent AI
- book: The Atomic Human
- twitter: @lawrennd
- podcast: The Talking Machines
- newspaper: Guardian Profile Page
- blog: http://inverseprobability.com
References
Although because total energy is given by free energy plus temperature-scaled entropy, these two optimisations are two sides of the same coin.↩︎