Maximum Entropy and the Exponential Family

Neil D. Lawrence

FW26, William Gates Building

This Session

  • Quiz 2 feedback (cohort figures; \(\log Z\); Clausius; intensive pressure)
  • Overlap: \(H(G\mid X)\); name mutual information; MI is a KL
  • Jaynes \(m\) and Radon–Nikodym: relative entropy is KL
  • Break, then MaxEnt: Lagrange; die; discrete proof \(\to\) Boltzmann / \(Z\)
  • Exponential family introduced at the end (\(\theta=-\lambda\)); developed in Week 6

How This Course Uses Generative AI

Generative AI: Policy in Plain Words

Background tools are fine. In the room you sit the quizzes without a computer.

  • Use generative AI as much as you like when preparing
  • In class: no laptop for the quiz; the authentication is you
  • Marking is Neil; the marking framework was built with generative AI so feedback is faster

Homework from Weeks 3–4

Quiz 2 Is Marked

Fifteen papers. A few minutes on matters arising, then MaxEnt.

  • Quiz 1: mean \(8.2/10\)
  • Quiz 2: mean \(6.3/10\)

Quiz 2

Quiz 2: Where Marks Were Lost

Mean \(6.3/10\), down from Quiz 1’s \(8.2\). Two ideas were responsible for most of the reduction.

  • What \(\log Z\) is (about three in five)
  • What Clausius kept and dropped in the story of heat (about two in three)
  • What values are intensive: e.g. pressure as an intensive variable was a common miss.

What \(\log Z\) Is

From the occupation: take \(\log\), take the expectation. \(U\) splits.

  • \(p_i = e^{-\beta E_i}/Z\); \(Z\) sums the states
  • One piece of the split is entropy; the other sits in \(\log Z\)
  • Up to \(-k_BT\), that second piece is available energy (not \(U\), and not \(H\))

Clausius Did Not Name Uncertainty

Clausius is bookkeeping after Joule and Carnot (not a Shannon preamble).

  • Kept the mechanical equivalent of heat and Carnot’s limit; dropped conserved caloric
  • Work can become heat entirely; heat cannot all become work
  • For a reversible engine, \(Q_\mathrm{h}/T_\mathrm{h}=Q_\mathrm{c}/T_\mathrm{c}\)

Carnot as a Waterwheel

Entropy drops like water. \(T\times S\) is in joules. Spill nothing if reversible.

  • Carnot: caloric conserved \(\approx\) entropy conserved through the cycle
  • Joule / Rumford: heat is inexhaustible; work can become heat entirely
  • Clausius keeps both: heat cannot all become work

A Contested Narrative

The heat story is a narrative to explore around, not a creed to recite.

  • Worksheets: use an LLM to press the contested edges
  • Marks are on the quizzes and the understanding you defend in the room

As Taught

KL before the break. MaxEnt after. Exponential family mentioned at the close.

  • Soft partitions \(\to\) mutual information; MI is a KL
  • Relative entropy to Jaynes’ measure \(m\) also expressed as a KL
  • After the break: Lagrange multipliers; Jaynes’ die;
  • End: discrete MaxEnt proof \(\to\) Boltzmann / \(Z\)
  • Continuous MaxEnt stays unread; \(\log Z\) as CGF: next week

KL: Overlap, Measure, Mutual Information

When Coarse Graining Overlaps

Revisit week 4’s exclusive case. Soft mixtures leave leftover uncertainty about the label.

  • Exclusive: \(G=g(x)\) forced \(H(G\mid X)=0\) and \(H(X)=H(w)+\sum_g w_g H(p_g)\)
  • Soft label \(G=g(x)\) with joint \(p(x,g)\) (several \(g\) can share the same \(x\))
  • Chain rule always: \(H(X,G)=H(G)+H(X\mid G)=H(X)+H(G\mid X)\)
  • Rearrange: \(H(X)=H(w)+\sum_g w_g H(p_g)-H(G\mid X)\)

Mutual Information

We assess this with mutual information.

\[ I(X;G)=H(G)-H(G\mid X)=H(X)-H(X\mid G) \]

  • Exclusive: \(H(G\mid X)=0\) \(\Rightarrow\) \(I(X;G)=H(G)=H(w)\)
  • Overlap: \(0<I(X;G)<H(w)\) (the coarse label is no longer free given \(X\))
  • Same algebra: \(H(X)=H(X\mid G)+I(X;G)\)

Mutual information is a KL divergence

This \(I\) is KL between the joint and the product of marginals.

\[ I(X;G)=\text{KL}\left( p(x,g)\,\|\,p(x)\,p(g) \right) \]

  • Write \(I=\sum_{x,g} p(x,g)\log\dfrac{p(x,g)}{p(x)p(g)}\)
  • Independence: \(I=0\); exclusive labelling: \(I=H(G)\)
  • Soft partitions are bookkeeping; MaxEnt still maximises \(H\) under constraints

Relative Entropy Is KL

Week 4’s undeclared \(m\) was already a KL.

  • Densities \(f\) and \(m\): \({\mathrm{d}}F/{\mathrm{d}}m=f/m\)
  • Relative entropy \(\int f\log(f/m)\,dx\) is \(\text{KL}\left( f\,\|\,m \right)\)
  • Jaynes’ limiting density of states is that reference \(m\)

Radon-Nikodym Derivative

Recall RN derivative between measures.

The Form of the Radon-Nikodym Derivative

If \(\mu\) is absolutely continuous with respect to \(\nu\), there is a density \(\mathrm{d}\mu/\mathrm{d}\nu\).

\[ \mu(A)=\int_A \frac{\mathrm{d}\mu}{\mathrm{d}\nu}\,\mathrm{d}\nu \]

  • \(M\) has density \(m\): \(\mathrm{d}F/\mathrm{d}M=f/m\)
  • That ratio is what sits inside \(\text{KL}\left( f\,\|\,m \right)\)
  • The width ratio \(\Delta s_i/\Delta u_i\) was the same object on a partition

Jaynes: Promote the Bins to a Measure

The well-defined continuum object is relative entropy against a declared \(m\).

\[ h_m(f)=-\int f\log\frac{f}{m}\,{\mathrm{d}}x=-\text{KL}\left( f\,\|\,m \right) \]

  • \(m\) is Jaynes’ limiting density of discrete points — the binning, written as a density
  • \(m\) = density of bins: Gaussian \(m\) means fine bins in the centre
  • Uniform \(m\) in \(v_x\) recovers ordinary \(h(v_x)\)
  • Uniform \(m\) in 3D velocity is \(m(s)\propto s^2\) in speed

Discrete Shannon Entropy Is Bounded

Week 3 derived Shannon entropy for discrete \(p=(p_1,\ldots,p_n)\).

  • \(0 \le H(p) \le \log n\) on \(n\) outcomes
  • Maximum at uniform; zero on a delta
  • Code interpretation: average length cannot beat \(H\)

KL Divergence

Comparing two distributions needs a functional that is always sensible.

  • \(\text{KL}\left( p\,\|\,q \right) = \sum_i p_i \log(p_i/q_i)\) discrete; \(\int p\log(p/q)\,dx\) continuous
  • Always \(\text{KL}\left( p\,\|\,q \right) \ge 0\); zero iff \(p = q\) (same support)
  • Extra surprise when \(q\) stands in for \(p\) — not a metric, but a directed cost

Differential Entropy Is a Different Object

Gaussian channel uses \(-\int p\log p\) — differential entropy.

  • Can be negative; not bounded below
  • Not a code-length bound — compare distributions with KL.

Maximum Entropy

Entropy tends to increase. Energy is constrained. Which \(p\)?

  • Second law: unconstrained, \(H\) grows
  • Fixed mean energy (or other moments): not every \(p\) is allowed
  • MaxEnt: among distributions that meet the constraints, take the largest \(H\)

Lagrange Multipliers

  • Constrained optimisation appears throughout physics, engineering and machine learning.
  • Lagrange multipliers convert constraints into terms in an auxiliary function
  • At the optimum, the objective gradient is normal to the constraint surface

One Equality Constraint

Single constraint: \[ \mathscr{L}(\mathbf{x}, \lambda) = f(\mathbf{x}) + \lambda\left(g(\mathbf{x}) - c\right) \]

Stationarity: \[ \nabla f + \lambda \nabla g = 0, \qquad g(\mathbf{x}) = c \]

Stationarity: parallel gradients

Geometric picture

  • At a constrained optimum, \(\nabla f\) is normal to the constraint surface
  • If \(\nabla f\) had a component tangent to the surface, we could improve \(f\) without leaving it
  • \(\lambda\) is the strength of the constraint force

Several Equality Constraints

Several constraints: \[ \mathscr{L} = f(\mathbf{x}) + \sum_k \lambda_k \bigl(g_k(\mathbf{x}) - c_k\bigr) \]

Stationarity: \[ \nabla f + \sum_k \lambda_k \nabla g_k = 0, \qquad g_k(\mathbf{x}) = c_k \]

A Worked Example

Example: minimise \(\frac{1}{2}(x^2+y^2)\) subject to \(x+y=1\)

\[ \mathscr{L} = \frac{1}{2}(x^2+y^2) + \lambda(x+y-1) \]

Solution: \(x = y = \frac{1}{2}\), \(\lambda = -\frac{1}{2}\)

Closest point on a line; same pattern as constrained energy or \(\|\mathbf{w}\|^2\) with a linear constraint

Principle of Insufficient Reason

Principle of Insufficient Reason

  • How to assign probability?
  • \(n\) mutually exclusive, exhaustive outcomes
  • Nothing distinguishes one from another
  • Assign equal probability: \(p_i = 1/n\)

Example: Will the sun rise tomorrow?

  • Unknown daily rate \(\theta\in[0,1]\); insufficient reason \(\Rightarrow\) uniform prior on \(\theta\)
  • Every day is a success: after \(n\) rises, belief \(\propto\theta^{n}\)
  • Predictive: \(P(\text{rises tomorrow})=\dfrac{n+1}{n+2}\)
  • Sunrise for \(n=1{,}826{,}213\) days: odds \(1{,}826{,}214\) to one
  • Laplace’s caveat: astronomy makes the true odds incomparably greater

Example: Will the sun rise tomorrow?

Entropy Day by Day

Track differential entropy of the belief \(p(\theta)\) as sunrises arrive.

  • Prior: \(p(\theta)=1\) on \([0,1]\) \(\Rightarrow\) \(H_0=0\)
  • After \(n\) rises: \(p(\theta)=(n+1)\theta^{n}\)
  • \(H_n=\dfrac{n}{n+1}-\log(n+1)\) (nats)
  • Day \(n\) information gain: \(I_n=H_{n-1}-H_n=\log\!\left(1+\dfrac{1}{n}\right)-\dfrac{1}{n(n+1)}\)

From Insufficient Reason to MaxEnt

  • Insufficient reason: uniform when we know only the outcomes (or only \(\theta\in[0,1]\))
  • Rule of succession: same idea, after observing counts
  • Jaynes: maximise \(H(p)\) subject to whatever we do know
  • No extra constraints \(\Rightarrow\) MaxEnt recovers \(p_i = 1/n\)
  • Mean energy (or other moments) \(\Rightarrow\) exponential family

Edwin Jaynes

Maximum Entropy Motivation

  • Jaynes (1957): Statistical mechanics as inference with incomplete information
  • Maximum entropy principle: maximise uncertainty given constraints
  • Avoids unwarranted assumptions beyond available data

Dice Example

  • Dice example: Average result 4.5 instead of 3.5
  • Constraints:
    • \(\sum_{n=1}^6 P_n = 1\) (normalization)
    • \(\sum_{n=1}^6 nP_n = 4.5\) (observed average)

Die Roll Simulation

click die or button to roll

Rolls: 0
Sample mean: —
H(p): —


Outcome weights (auto-normalised to probabilities)

MaxEnt Recovers Boltzmann and Gaussian

Jaynes’ die (mean 4.5, not 3.5) is the running example. The same Lagrange move recovers physics and statistics.

  • One constraint: mean energy \(\Rightarrow\) \(p_i \propto e^{-\beta E_i}\)
  • Two constraints: mean and variance \(\Rightarrow\) Gaussian
  • Lagrange multiplier on energy is coldness \(\beta\) from week 1

Derivation of the Boltzmann Distribution

For fixed mean energy \(U\), maximum entropy recovers Boltzmann weights.

  • Constraints: \(\sum_i p_i = 1\) and \(\sum_i p_i E_i = U\)
  • MaxEnt: \(p_i \propto e^{-\beta E_i}\) with coldness \(\beta = 1/k_B T\)
  • Normalise: \(Z(\beta)=\sum_i e^{-\beta E_i}\), so \(p_i = e^{-\beta E_i}/Z\)

MaxEnt Under Constraints: the Proof

Constrained MaxEnt on a discrete support gives an exponential family.

  • Discrete \(x_i\); maximise \(H(p)=-\sum_i p_i\log p_i\)
  • Constraints: \(\sum_i p_i=1\) and \(\sum_i p_i f_k(x_i)=\langle f_k\rangle\) for \(k=1,\ldots,m\)
  • Lagrangian: \(\mathscr{L}=\sum_i p_i\log p_i + \lambda_0\bigl(\sum_i p_i-1\bigr)+\sum_k \lambda_k\bigl(\sum_i p_i f_k(x_i)-\langle f_k\rangle\bigr)\)
  • Stationarity: \(\partial\mathscr{L}/\partial p_i=0\) \(\Rightarrow\) \(p_i\propto e^{-\sum_k \lambda_k f_k(x_i)}\)

Minimise KL to a Measure

Equivalently minimise \(\text{KL}\left( f\,\|\,m \right)\) under a kinetic constraint. The answer depends on \(m\).

  • Same constraint: \(\langle\tfrac12 m_{\mathrm{mass}}v^2\rangle\) fixed
  • Minimise \(\text{KL}\left( f\,\|\,m \right)\) (or maximise \(-D(f\|m)\))
  • \(m\) Lebesgue on component \(v_x\) \(\Rightarrow\) Gaussian
  • \(m\) Lebesgue on 3D velocity (\(m(v)\propto v^2\) in speed) \(\Rightarrow\) Maxwell–Boltzmann

Same Principle, Declared Prior

\[ f\propto m(\cdot)\,e^{-\beta E} \qquad \begin{cases} m\sim\mathrm{Lebesgue}(v_x) & \Rightarrow\text{Gaussian component}\\ m\sim v^2\,dv & \Rightarrow\text{Maxwell--Boltzmann} \end{cases} \]

  • Exponential family always carries a base measure
  • For the ideal gas, that base measure is the density of states
  • Coordinate dependence of \(h\) was a symptom; \(\text{KL}\left( f\,\|\,m \right)\) is the cure

The General Maximum-Entropy Formalism

\[ p_i = \frac{\exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i))}{Z(\lambda_1,\ldots,\lambda_m)} \] \[ Z(\ldots) = \sum_{i=1}^n \exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i)) \] \[ \langle f_k \rangle = -\frac{\partial}{\partial \lambda_k}\log Z(\lambda_1,\ldots,\lambda_m) \quad k=1,2,\ldots,m. \]

The Exponential Family

  • Write \(T_k(x_i)=f_k(x_i)\): sufficient statistics fixed by the constraints
  • MaxEnt solution: \(p_i=\dfrac{e^{-\sum_k \lambda_k T_k(x_i)}}{Z}\) on the discrete support
  • Exponential family: \(p(x\mid\boldsymbol{\theta})\propto \exp\bigl(\boldsymbol{\theta}\!\cdot\! T(x)-\psi(\boldsymbol{\theta})\bigr)\)
  • Identification: \(\boldsymbol{\theta} = -\boldsymbol{\lambda}\) (natural parameters \(=\) minus Lagrange multipliers)
  • Same object: \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\)

\(\log Z\) as Cumulant Generating Function

  • Debrief: \(F=-k_BT\log Z\) (available energy)
  • MaxEnt / exponential family: \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\)
  • Mean parameters: \(\boldsymbol{\eta}=\nabla\psi=\langle T\rangle\)
  • Hessian: \(\nabla^2 \psi=\mathrm{Cov}[T]\) (covariance; not thermodynamic length)

The Partition Function

\(Z\) as Generating Function

The canonical ensemble from the bath: fix \(\beta\) and let the small system fluctuate.

  • \(Z(\beta) = \sum_i e^{-\beta E_i}\)
  • \(U = -\partial_\beta \log Z\), \(F = -\beta^{-1}\log Z\), \(S = \beta(U-F)\)
  • Equilibrium identities on the \(\beta\)-manifold

  • \(U = -\partial_\beta \log Z\)
  • \(F = -\beta^{-1}\log Z\)
  • Bath justifies the canonical ensemble

Exponential Families

  • Exponential families: cornerstone of information theory
  • Natural parameters give us clean geometry
  • Connects probability, entropy, and Fisher information

Definition

\[ p(\mathbf{x} | \boldsymbol{\theta}) = \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x}) - \mathcal{A}(\boldsymbol{\theta})\right) h(\mathbf{x}) \] * \(\boldsymbol{\theta}\): natural parameters * \(T(\mathbf{x})\): sufficient statistics * \(\mathcal{A}(\boldsymbol{\theta})\): log partition function * \(h(\mathbf{x})\): base measure

Why “Natural” Parameters?

  • Natural parameters appear linearly in exponent
  • Special geometric properties
  • Clean connection to Fisher information
  • Simplifies entropy calculations

The Log Partition Function as Cumulant Generator

\[ \mathcal{A}(\boldsymbol{\theta}) = \log \int \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x})\right) h(\mathbf{x}) \, d\mathbf{x} \]}

Cumulants from Derivatives

  • \(\nabla \mathcal{A}(\boldsymbol{\theta}) = \mathbb{E}[T(\mathbf{x})]\) (mean)
  • \(\nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}[T(\mathbf{x})]\) (covariance)
  • Higher derivatives give higher cumulants

Connection to Fisher Information

\[ G(\boldsymbol{\theta}) = \nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}_{\boldsymbol{\theta}}[T(\mathbf{x})] \] * R.A. Fisher: bounds estimation accuracy (Cramér-Rao) * Fisher information = Hessian of \(\mathcal{A}\) * Fisher information = Covariance of sufficient statistics * Defines geometry of information space * More details in next section

Entropy in Natural Coordinates

\[ H(\boldsymbol{\theta}) = \mathcal{A}(\boldsymbol{\theta}) - \boldsymbol{\theta}^\top \boldsymbol{\mu} \] where \(\boldsymbol{\mu} = \nabla \mathcal{A}(\boldsymbol{\theta})\)

Why Natural Parameters?

\[ \nabla_{\boldsymbol{\theta}} H = -G(\boldsymbol{\theta})\boldsymbol{\theta} \] * Key result for information dynamics * Terms involving \(\nabla \mathcal{A}\) cancel * Fisher information connects geometry to dynamics

Example: Multivariate Gaussian

Gaussian in Exponential Family Form

Gaussian as exponential family: * \(\boldsymbol{\theta} = \Sigma^{-1}\boldsymbol{\mu}\) (natural parameters) * \(T(\mathbf{x}) = \mathbf{x}\) (sufficient statistics) * \(\mathcal{A}(\boldsymbol{\theta}) = \frac{1}{2}\boldsymbol{\theta}^\top\Sigma\boldsymbol{\theta} + \frac{1}{2}\log|\Sigma|\)

Fisher Information for Gaussian

\[ G(\boldsymbol{\theta}) = \Sigma \] * Fisher information = covariance * \(G^{-1} = \Sigma^{-1}\) = precision * High variance → low Fisher info → harder estimation (looser Cramér-Rao) * Low variance → high Fisher info → easier estimation (tighter Cramér-Rao)

Example: Categorical Distribution

Categorical in Exponential Family Form

Categorical distribution: * \(\boldsymbol{\theta} = (\log \pi_1, \ldots, \log \pi_{K-1})\) * \(T(\mathbf{x}) = (x_1, \ldots, x_{K-1})\) (one-hot) * \(\mathcal{A}(\boldsymbol{\theta}) = \log\left(\sum_{k=1}^{K-1} e^{\theta_k} + 1\right)\)

Fisher Information for Categorical

\[ G_{ij} = \begin{cases} \pi_i(1-\pi_i) & i=j \\ -\pi_i\pi_j & i \neq j \end{cases} \] * Rare categories → low Fisher info * Uniform distribution → balanced information

Why Exponential Families Matter

Why exponential families matter: * Emerge from maximum entropy principles * Define information geometry * Sufficient statistics capture all information * Computationally tractable * Clean form for information dynamics

Three Perspectives (Brief)

Same \(H\); different operational assumptions about what probability is over.

  • Carnot/Clausius: engines and the second law
  • Shannon: a code
  • Szilard/Landauer: a bit in memory
  • Boltzmann: a macrostate; Jaynes: a state of knowledge
  • All layers: \(H\) forbids, \(p\) prescribes

Define This Week

  • What is KL divergence (discrete and continuous)?
  • What is mutual information, and why is it a KL?
  • What is the maximum entropy principle?
  • What is the exponential family, and why \(\theta=-\lambda\)?
  • Why is \(\log Z\) a cumulant generating function? (developed in Week 6)

After This Lecture

  • Quiz 3: 17 November, first ten minutes
  • Worksheet due at the start of lecture 6

Further Reading

  • limiting density of discrete points of Jaynes (1963)

  • Chapter 12 of Cover and Thomas (1991)

  • the whole paper of Jaynes (1957)

  • Chapter 22 of MacKay (2003)

  • Chapters 11–12 of Jaynes (2003)

Thanks!

References

Cover, T.M., Thomas, J.A., 1991. Elements of information theory. Wiley, New York.
Jaynes, E.T., 2003. Probability theory: The logic of science. Cambridge University Press, Cambridge, UK.
Jaynes, E.T., 1963. Information theory and statistical mechanics, in: Ford, K.W. (Ed.), Brandeis University Summer Institute Lectures in Theoretical Physics, Vol. 3: Statistical Physics. W. A. Benjamin, Inc., New York, pp. 181–218.
Jaynes, E.T., 1957. Information theory and statistical mechanics. Physical Review 106, 620–630. https://doi.org/10.1103/PhysRev.106.620
MacKay, D.J.C., 2003. Information theory, inference and learning algorithms. Cambridge University Press, Cambridge, U.K.