Maximum Entropy and the Exponential Family
Neil D. Lawrence
2026-11-10
FW26, William Gates Building
This Session
Quiz 2 feedback (cohort figures; \(\log Z\) ; Clausius; intensive pressure)
Overlap: \(H(G\mid X)\) ; name mutual information; MI is a KL
Jaynes \(m\) and Radon–Nikodym: relative entropy is KL
Break, then MaxEnt: Lagrange; die; discrete proof \(\to\) Boltzmann / \(Z\)
Exponential family introduced at the end (\(\theta=-\lambda\) ); developed in Week 6
How This Course Uses Generative AI
Generative AI: Policy in Plain Words
Background tools are fine. In the room you sit the quizzes without a computer.
Use generative AI as much as you like when preparing
In class: no laptop for the quiz; the authentication is you
Marking is Neil; the marking framework was built with generative AI so feedback is faster
Keep this short. It is a policy aside, not the week’s content. Then into Quiz 2 feedback.
Quiz 2 Is Marked
Fifteen papers. A few minutes on matters arising, then MaxEnt.
Quiz 1: mean \(8.2/10\)
Quiz 2: mean \(6.3/10\)
Quiz-only block for now. Add Worksheet 2 habits when that section is written. Keep names off the slides. Worksheets are marking well; quizzes differentiate more.
Quiz 2: Where Marks Were Lost
Mean \(6.3/10\) , down from Quiz 1’s \(8.2\) . Two ideas were responsible for most of the reduction.
What \(\log Z\) is (about three in five)
What Clausius kept and dropped in the story of heat (about two in three)
What values are intensive: e.g. pressure as an intensive variable was a common miss.
What \(\log Z\) Is
From the occupation: take \(\log\) , take the expectation. \(U\) splits.
\(p_i = e^{-\beta E_i}/Z\) ; \(Z\) sums the states
One piece of the split is entropy; the other sits in \(\log Z\)
Up to \(-k_BT\) , that second piece is available energy (not \(U\) , and not \(H\) )
If the room needs the one-liner: \(F=-k_BT\log Z\) . Prefer the split story over listing wrong options. Do not derive MaxEnt here (that is the rest of the hour).
Clausius Did Not Name Uncertainty
Clausius is bookkeeping after Joule and Carnot (not a Shannon preamble).
Kept the mechanical equivalent of heat and Carnot’s limit; dropped conserved caloric
Work can become heat entirely; heat cannot all become work
For a reversible engine, \(Q_\mathrm{h}/T_\mathrm{h}=Q_\mathrm{c}/T_\mathrm{c}\)
Carnot as a Waterwheel
Entropy drops like water. \(T\times S\) is in joules. Spill nothing if reversible.
Carnot: caloric conserved \(\approx\) entropy conserved through the cycle
Joule / Rumford: heat is inexhaustible; work can become heat entirely
Clausius keeps both: heat cannot all become work
Bridge: Worksheet 2 pressed \(H\) versus \(S\) ; this item pressed the physical thread that \(S\) came from. Contested narrative next, then into the KL arc.
A Contested Narrative
The heat story is a narrative to explore around, not a creed to recite.
Worksheets: use an LLM to press the contested edges
Marks are on the quizzes and the understanding you defend in the room
As Taught
KL before the break. MaxEnt after. Exponential family mentioned at the close.
Soft partitions \(\to\) mutual information; MI is a KL
Relative entropy to Jaynes’ measure \(m\) also expressed as a KL
After the break: Lagrange multipliers; Jaynes’ die;
End: discrete MaxEnt proof \(\to\) Boltzmann / \(Z\)
Continuous MaxEnt stays unread; \(\log Z\) as CGF: next week
Do not force continuous MaxEnt if the discrete proof is still live. Name \(\theta=-\lambda\) and stop. Week 6 develops the family and \(\psi\) as a cumulant generating function, then Fisher geometry. Shannon / DPI / full GRT stayed unread on 17 November. Do not teach thermodynamic length.
When Coarse Graining Overlaps
Revisit week 4’s exclusive case. Soft mixtures leave leftover uncertainty about the label.
Exclusive: \(G=g(x)\) forced \(H(G\mid X)=0\) and \(H(X)=H(w)+\sum_g w_g H(p_g)\)
Soft label \(G=g(x)\) with joint \(p(x,g)\) (several \(g\) can share the same \(x\) )
Chain rule always: \(H(X,G)=H(G)+H(X\mid G)=H(X)+H(G\mid X)\)
Rearrange: \(H(X)=H(w)+\sum_g w_g H(p_g)-H(G\mid X)\)
Week 4: the log-sum collapsed only because non-overlap forced every face into one cell. Keep \(H(G\mid X)\) : leftover uncertainty about which component, given the face. That leftover is mutual information next.
Relative Entropy Is KL
Week 4’s undeclared \(m\) was already a KL.
Densities \(f\) and \(m\) : \({\mathrm{d}}F/{\mathrm{d}}m=f/m\)
Relative entropy \(\int f\log(f/m)\,dx\) is \(\text{KL}\left( f\,\|\,m \right)\)
Jaynes’ limiting density of states is that reference \(m\)
Week 4 wrote the ratio and the Jacobian. Relative entropy to \(m\) is KL; undeclared \(m\) was the missing measure. Mutual information was already written as a KL on the previous slides.
Radon-Nikodym Derivative
Recall RN derivative between measures.
Discrete Shannon Entropy Is Bounded
Week 3 derived Shannon entropy for discrete \(p=(p_1,\ldots,p_n)\) .
\(0 \le H(p) \le \log n\) on \(n\) outcomes
Maximum at uniform; zero on a delta
Code interpretation: average length cannot beat \(H\)
KL Divergence
Comparing two distributions needs a functional that is always sensible.
\(\text{KL}\left( p\,\|\,q \right) = \sum_i p_i \log(p_i/q_i)\) discrete; \(\int p\log(p/q)\,dx\) continuous
Always \(\text{KL}\left( p\,\|\,q \right) \ge 0\) ; zero iff \(p = q\) (same support)
Extra surprise when \(q\) stands in for \(p\) — not a metric, but a directed cost
Week 5 KL arc, step (4). Use \(\text{KL}\left( p\,\|\,q \right)\) throughout. Physics synonym: relative entropy. Asymmetry: \(\text{KL}\left( p\,\|\,q \right)\neq\text{KL}\left( q\,\|\,p \right)\) — not a metric. MaxEnt after the break maximises \(H\) or, equivalently, minimises \(\text{KL}\left( \cdot\,\|\,m \right)\) . Week 7: \(m\) -projection. Optional colour: GANs / adversarial training pick a direction of the KL; tell the truth and the whole truth, not nothing but the truth.
Differential Entropy Is a Different Object
Gaussian channel uses \(-\int p\log p\) — differential entropy.
Can be negative ; not bounded below
Not a code-length bound — compare distributions with KL.
Left: binary Shannon entropy is bounded. Right: Gaussian differential entropy can be negative.
Maximum Entropy
Entropy tends to increase. Energy is constrained. Which \(p\) ?
Second law: unconstrained, \(H\) grows
Fixed mean energy (or other moments): not every \(p\) is allowed
MaxEnt: among distributions that meet the constraints, take the largest \(H\)
State the problem before Lagrange. Not ``entropy is maximised in nature’’ as metaphysics: given what we know (the constraints), which distribution assumes least else?
Lagrange Multipliers
Constrained optimisation appears throughout physics, engineering and machine learning.
Lagrange multipliers convert constraints into terms in an auxiliary function
At the optimum, the objective gradient is normal to the constraint surface
One Equality Constraint
Single constraint: \[
\mathscr{L}(\mathbf{x}, \lambda) = f(\mathbf{x}) + \lambda\left(g(\mathbf{x}) - c\right)
\]
Stationarity: \[
\nabla f + \lambda \nabla g = 0, \qquad g(\mathbf{x}) = c
\]
Stationarity: parallel gradients
Geometric picture
At a constrained optimum, \(\nabla f\) is normal to the constraint surface
If \(\nabla f\) had a component tangent to the surface, we could improve \(f\) without leaving it
\(\lambda\) is the strength of the constraint force
Several Equality Constraints
Several constraints: \[
\mathscr{L} = f(\mathbf{x}) + \sum_k \lambda_k \bigl(g_k(\mathbf{x}) - c_k\bigr)
\]
Stationarity: \[
\nabla f + \sum_k \lambda_k \nabla g_k = 0, \qquad g_k(\mathbf{x}) = c_k
\]
A Worked Example
Example: minimise \(\frac{1}{2}(x^2+y^2)\) subject to \(x+y=1\)
\[
\mathscr{L} = \frac{1}{2}(x^2+y^2) + \lambda(x+y-1)
\]
Solution: \(x = y = \frac{1}{2}\) , \(\lambda = -\frac{1}{2}\)
Closest point on a line; same pattern as constrained energy or \(\|\mathbf{w}\|^2\) with a linear constraint
Principle of Insufficient Reason
Pierre-Simon Laplace 1749-1827.
Principle of Insufficient Reason
How to assign probability?
\(n\) mutually exclusive, exhaustive outcomes
Nothing distinguishes one from another
Assign equal probability: \(p_i = 1/n\)
Example: Will the sun rise tomorrow?
Unknown daily rate \(\theta\in[0,1]\) ; insufficient reason \(\Rightarrow\) uniform prior on \(\theta\)
Every day is a success: after \(n\) rises, belief \(\propto\theta^{n}\)
Predictive: \(P(\text{rises tomorrow})=\dfrac{n+1}{n+2}\)
Sunrise for \(n=1{,}826{,}213\) days: odds \(1{,}826{,}214\) to one
Laplace’s caveat: astronomy makes the true odds incomparably greater
Board the short derivation. Emphasise: (i) the example is genuine; (ii) Laplace applies insufficient reason to \(\theta\) , not to tomorrow’s binary outcome alone; (iii) every observation is a success — the sun always rises in the data; (iv) he immediately rejects treating the numerical answer as the scientific probability of sunrise.
Example: Will the sun rise tomorrow?
Entropy Day by Day
Track differential entropy of the belief \(p(\theta)\) as sunrises arrive.
Prior: \(p(\theta)=1\) on \([0,1]\) \(\Rightarrow\) \(H_0=0\)
After \(n\) rises: \(p(\theta)=(n+1)\theta^{n}\)
\(H_n=\dfrac{n}{n+1}-\log(n+1)\) (nats)
Day \(n\) information gain: \(I_n=H_{n-1}-H_n=\log\!\left(1+\dfrac{1}{n}\right)-\dfrac{1}{n(n+1)}\)
Board the one-line integral for \(H_n\) . Emphasise: entropy of the belief about \(\theta\) falls (becomes more negative); that fall is the information gained. Uniform\([0,1]\) has \(H=0\) by convention of differential entropy — not ``zero uncertainty’’ in the Shannon discrete sense.
From Insufficient Reason to MaxEnt
Insufficient reason: uniform when we know only the outcomes (or only \(\theta\in[0,1]\) )
Rule of succession: same idea, after observing counts
Jaynes: maximise \(H(p)\) subject to whatever we do know
No extra constraints \(\Rightarrow\) MaxEnt recovers \(p_i = 1/n\)
Mean energy (or other moments) \(\Rightarrow\) exponential family
The punchline: MaxEnt is Laplace’s principle with constraints. Uniform / rule of succession are the empty-moment cases; Boltzmann and the Gaussian arrive when we add moment constraints.
Edwin Jaynes
Ed Jaynes who developed the maximum entropy principle
Maximum Entropy Motivation
Jaynes (1957) : Statistical mechanics as inference with incomplete information
Maximum entropy principle: maximise uncertainty given constraints
Avoids unwarranted assumptions beyond available data
Dice Example
Dice example: Average result 4.5 instead of 3.5
Constraints:
\(\sum_{n=1}^6 P_n = 1\) (normalization)
\(\sum_{n=1}^6 nP_n = 4.5\) (observed average)
Interactive die-roll simulation. Click the die or press Roll to sample from the configured distribution. The histogram shows empirical relative frequencies (coloured bars) overlaid on the theoretical probabilities (dashed outlines). Use the sliders to set arbitrary outcome weights, or click a preset to load the uniform distribution (mean 3.5), the Jaynes maximum-entropy distribution (mean 4.5), the simple 50/50 distribution (faces 4 and 5 equally, mean 4.5 with minimal entropy), or a low-biased distribution (mean 2).
MaxEnt Recovers Boltzmann and Gaussian
Jaynes’ die (mean 4.5, not 3.5) is the running example. The same Lagrange move recovers physics and statistics.
One constraint: mean energy \(\Rightarrow\) \(p_i \propto e^{-\beta E_i}\)
Two constraints: mean and variance \(\Rightarrow\) Gaussian
Lagrange multiplier on energy is coldness \(\beta\) from week 1
LO5. After the die demo, the canonical ensemble and the Gaussian are the same MaxEnt pattern. Worksheet 2 implements the die.
Jaynes’ die: MaxEnt subject to mean 4.5 versus the uniform distribution.
Derivation of the Boltzmann Distribution
For fixed mean energy \(U\) , maximum entropy recovers Boltzmann weights.
Constraints: \(\sum_i p_i = 1\) and \(\sum_i p_i E_i = U\)
MaxEnt: \(p_i \propto e^{-\beta E_i}\) with coldness \(\beta = 1/k_B T\)
Normalise: \(Z(\beta)=\sum_i e^{-\beta E_i}\) , so \(p_i = e^{-\beta E_i}/Z\)
LO5. This is the derivation postponed from week 2. Board the Lagrange multipliers; do not treat the formula as new.
Occupation of a two-state system as coldness increases. At low \(\beta\) both states are populated; at high \(\beta\) the ground state dominates.
MaxEnt Under Constraints: the Proof
Constrained MaxEnt on a discrete support gives an exponential family.
Discrete \(x_i\) ; maximise \(H(p)=-\sum_i p_i\log p_i\)
Constraints: \(\sum_i p_i=1\) and \(\sum_i p_i f_k(x_i)=\langle f_k\rangle\) for \(k=1,\ldots,m\)
Lagrangian: \(\mathscr{L}=\sum_i p_i\log p_i + \lambda_0\bigl(\sum_i p_i-1\bigr)+\sum_k \lambda_k\bigl(\sum_i p_i f_k(x_i)-\langle f_k\rangle\bigr)\)
Stationarity: \(\partial\mathscr{L}/\partial p_i=0\) \(\Rightarrow\) \(p_i\propto e^{-\sum_k \lambda_k f_k(x_i)}\)
The stationarity condition is \(\partial L/\partial p_i=0\) . Sign: Jaynes subtracts the constraints so the exponential comes out with a minus. Class preferred \(\mu\) over \(\lambda_0\) for the normalisation multiplier. One energy constraint \(\Rightarrow\lambda=\beta\) . As taught: board derivation; name the exponential family and stop if the clock is gone.
Minimise KL to a Measure
Equivalently minimise \(\text{KL}\left( f\,\|\,m \right)\) under a kinetic constraint. The answer depends on \(m\) .
Same constraint: \(\langle\tfrac12 m_{\mathrm{mass}}v^2\rangle\) fixed
Minimise \(\text{KL}\left( f\,\|\,m \right)\) (or maximise \(-D(f\|m)\) )
\(m\) Lebesgue on component \(v_x\) \(\Rightarrow\) Gaussian
\(m\) Lebesgue on 3D velocity (\(m(v)\propto v^2\) in speed) \(\Rightarrow\) Maxwell–Boltzmann
Continuous MaxEnt example after the discrete die. Relative entropy to \(m\) is the KL we named earlier. The \(v^2\) that looked like a Jacobian is the density of states — the reference measure in the speed coordinate. Same principle; declared prior.
Same Principle, Declared Prior
\[
f\propto m(\cdot)\,e^{-\beta E}
\qquad
\begin{cases}
m\sim\mathrm{Lebesgue}(v_x) & \Rightarrow\text{Gaussian component}\\
m\sim v^2\,dv & \Rightarrow\text{Maxwell--Boltzmann}
\end{cases}
\]
Exponential family always carries a base measure
For the ideal gas, that base measure is the density of states
Coordinate dependence of \(h\) was a symptom; \(\text{KL}\left( f\,\|\,m \right)\) is the cure
Same principle, continuous reading: minimise \(\text{KL}\left( f\,\|\,m \right)\) at fixed mean kinetic energy. Lebesgue on \(v_x\) gives a Gaussian component; Lebesgue on \(\mathbb{R}^3\) (shell \(v^2\) in speed) gives Maxwell–Boltzmann. The \(v^2\) is the density of states. As taught 10 November: not walked as a dedicated block.
The Exponential Family
Write \(T_k(x_i)=f_k(x_i)\) : sufficient statistics fixed by the constraints
MaxEnt solution: \(p_i=\dfrac{e^{-\sum_k \lambda_k T_k(x_i)}}{Z}\) on the discrete support
Exponential family: \(p(x\mid\boldsymbol{\theta})\propto \exp\bigl(\boldsymbol{\theta}\!\cdot\! T(x)-\psi(\boldsymbol{\theta})\bigr)\)
Identification: \(\boldsymbol{\theta} = -\boldsymbol{\lambda}\) (natural parameters \(=\) minus Lagrange multipliers)
Same object: \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\)
LO6 punchline. As taught 10 November: name \(\theta=-\lambda\) and that fitting the \(\lambda\) s is maximum likelihood. The CGF and the three members are Week 6. Physics sign: Boltzmann uses \(p_i\propto e^{-\beta E_i}\) so \(\theta_1=-\beta\) when \(T=E\) .
\(\log Z\) as Cumulant Generating Function
Debrief: \(F=-k_BT\log Z\) (available energy)
MaxEnt / exponential family: \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\)
Mean parameters: \(\boldsymbol{\eta}=\nabla\psi=\langle T\rangle\)
Hessian: \(\nabla^2 \psi=\mathrm{Cov}[T]\) (covariance; not thermodynamic length)
Unread on 10 November. Week 6 teaches \(\psi\) as a cumulant generating function and stops at the covariance. Do not derive the Legendre transform, and do not follow the include into Crooks or thermodynamic length.
\(Z\) as Generating Function
The canonical ensemble from the bath: fix \(\beta\) and let the small system fluctuate.
\(Z(\beta) = \sum_i e^{-\beta E_i}\)
\(U = -\partial_\beta \log Z\) , \(F = -\beta^{-1}\log Z\) , \(S = \beta(U-F)\)
Equilibrium identities on the \(\beta\) -manifold
LO3. Work the two-state example on the board. These are equilibrium identities. Do not open finite-time dissipation or thermodynamic length from this slide.
Thermodynamic quantities from \(Z(\beta)\) for the two-state system used as a running example.
\(U = -\partial_\beta \log Z\)
\(F = -\beta^{-1}\log Z\)
Bath justifies the canonical ensemble
Exponential Families
Exponential families: cornerstone of information theory
Natural parameters give us clean geometry
Connects probability, entropy, and Fisher information
Definition
\[
p(\mathbf{x} | \boldsymbol{\theta}) = \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x}) - \mathcal{A}(\boldsymbol{\theta})\right) h(\mathbf{x})
\] * \(\boldsymbol{\theta}\) : natural parameters * \(T(\mathbf{x})\) : sufficient statistics * \(\mathcal{A}(\boldsymbol{\theta})\) : log partition function * \(h(\mathbf{x})\) : base measure
Why “Natural” Parameters?
Natural parameters appear linearly in exponent
Special geometric properties
Clean connection to Fisher information
Simplifies entropy calculations
The Log Partition Function as Cumulant Generator
\[
\mathcal{A}(\boldsymbol{\theta}) = \log \int \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x})\right) h(\mathbf{x}) \, d\mathbf{x}
\] }
Cumulants from Derivatives
\(\nabla \mathcal{A}(\boldsymbol{\theta}) = \mathbb{E}[T(\mathbf{x})]\) (mean)
\(\nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}[T(\mathbf{x})]\) (covariance)
Higher derivatives give higher cumulants
Entropy in Natural Coordinates
\[
H(\boldsymbol{\theta}) = \mathcal{A}(\boldsymbol{\theta}) - \boldsymbol{\theta}^\top \boldsymbol{\mu}
\] where \(\boldsymbol{\mu} = \nabla \mathcal{A}(\boldsymbol{\theta})\)
Why Natural Parameters?
\[
\nabla_{\boldsymbol{\theta}} H = -G(\boldsymbol{\theta})\boldsymbol{\theta}
\] * Key result for information dynamics * Terms involving \(\nabla \mathcal{A}\) cancel * Fisher information connects geometry to dynamics
Example: Multivariate Gaussian
Example: Categorical Distribution
Why Exponential Families Matter
Why exponential families matter: * Emerge from maximum entropy principles * Define information geometry * Sufficient statistics capture all information * Computationally tractable * Clean form for information dynamics
LO6. Two-level system = Bernoulli with \(\theta=-\beta\varepsilon\) . Softmax and two-spin are the same theorem with richer \(T(x)\) ; name them if the clock is tight. Flag matrix exponential family for week 8.
Three Perspectives (Brief)
Same \(H\) ; different operational assumptions about what probability is over.
Carnot/Clausius: engines and the second law
Shannon: a code
Szilard/Landauer: a bit in memory
Boltzmann: a macrostate; Jaynes: a state of knowledge
All layers: \(H\) forbids, \(p\) prescribes
LO7 in brief. As taught 10 November: not reached. Full colour waits for Worksheet 4 / week 8. Intended answer to ``how is entropy understood today?’’ until week 8.
Define This Week
What is KL divergence (discrete and continuous)?
What is mutual information, and why is it a KL?
What is the maximum entropy principle?
What is the exponential family, and why \(\theta=-\lambda\) ?
Why is \(\log Z\) a cumulant generating function? (developed in Week 6)
After This Lecture
Quiz 3: 17 November, first ten minutes
Worksheet due at the start of lecture 6