Maximum Entropy and the Exponential Family

This Session

Information, entropy and intelligence course notebook setup

Maximum Entropy

Lagrange Multipliers

• Constrained optimisation appears throughout physics, engineering and machine learning.
• Lagrange multipliers convert constraints into terms in an auxiliary function
• At the optimum, the objective gradient is normal to the constraint surface

One Equality Constraint

Single constraint: \[ \mathscr{L}(\mathbf{x}, \lambda) = f(\mathbf{x}) + \lambda\left(g(\mathbf{x}) - c\right) \]

Stationarity: \[ \nabla f + \lambda \nabla g = 0, \qquad g(\mathbf{x}) = c \]

Stationarity: parallel gradients

Geometric picture

• At a constrained optimum, \(\nabla f\) is normal to the constraint surface
• If \(\nabla f\) had a component tangent to the surface, we could improve \(f\) without leaving it
• \(\lambda\) is the strength of the constraint force

Several Equality Constraints

Several constraints: \[ \mathscr{L} = f(\mathbf{x}) + \sum_k \lambda_k \bigl(g_k(\mathbf{x}) - c_k\bigr) \]

Stationarity: \[ \nabla f + \sum_k \lambda_k \nabla g_k = 0, \qquad g_k(\mathbf{x}) = c_k \]

A Worked Example

Example: minimise \(\frac{1}{2}(x^2+y^2)\) subject to \(x+y=1\)

\[ \mathscr{L} = \frac{1}{2}(x^2+y^2) + \lambda(x+y-1) \]

Solution: \(x = y = \frac{1}{2}\), \(\lambda = -\frac{1}{2}\)

Closest point on a line; same pattern as constrained energy or \(\|\mathbf{w}\|^2\) with a linear constraint

Laplace and the Principle of Insufficient Reason

Week 1: Laplace suggested that we treat ignorance with probability. He applied this idea in his estimates of probabilities. He asked himself the question how should we assign \(p\) when we know almost nothing?

This is Laplace’s example from: Philosophical Essay on Probabilities (Laplace, 1814).

Track differential entropy of the belief \(p(\theta)\) as sunrises arrive.

Jaynes and Maximum Entropy

Maximum Entropy Motivation

• Jaynes (1957): Statistical mechanics as inference with incomplete information
• Maximum entropy principle: maximise uncertainty given constraints
• Avoids unwarranted assumptions beyond available data

Dice Example

Die Roll Simulation

Jaynes’ die (mean 4.5, not 3.5) is the running example. The same Lagrange move recovers physics and statistics.

Derivation of the Boltzmann Distribution

For fixed mean energy \(U\), maximum entropy recovers Boltzmann weights. ::: incremental

• Constraints: \(\sum_i p_i = 1\) and \(\sum_i p_i E_i = U\)
• MaxEnt: \(p_i \propto e^{-\beta E_i}\) with coldness \(\beta = 1/k_B T\)
• Normalise: \(Z(\beta)=\sum_i e^{-\beta E_i}\), so \(p_i = e^{-\beta E_i}/Z\)

:::

MaxEnt Under Constraints: the Proof

This is the week’s main theorem: constrained MaxEnt \(\Rightarrow\) exponential family.

The General Maximum-Entropy Formalism

\[ p_i = \frac{\exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i))}{Z(\lambda_1,\ldots,\lambda_m)} \] \[ Z(\ldots) = \sum_{i=1}^n \exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i)) \] \[ \langle f_k \rangle = -\frac{\partial}{\partial \lambda_k}\log Z(\lambda_1,\ldots,\lambda_m) \quad k=1,2,\ldots,m. \]

The Exponential Family

Exponential Families

• Exponential families: cornerstone of information theory
• Natural parameters give us clean geometry
• Connects probability, entropy, and Fisher information

Definition

\[ p(\mathbf{x} | \boldsymbol{\theta}) = \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x}) - \mathcal{A}(\boldsymbol{\theta})\right) h(\mathbf{x}) \] * \(\boldsymbol{\theta}\): natural parameters * \(T(\mathbf{x})\): sufficient statistics * \(\mathcal{A}(\boldsymbol{\theta})\): log partition function * \(h(\mathbf{x})\): base measure

Why “Natural” Parameters?

• Natural parameters appear linearly in exponent
• Special geometric properties
• Clean connection to Fisher information
• Simplifies entropy calculations

The Log Partition Function as Cumulant Generator

\[ \mathcal{A}(\boldsymbol{\theta}) = \log \int \exp\left(\boldsymbol{\theta}^\top T(\mathbf{x})\right) h(\mathbf{x}) \, d\mathbf{x} \]}

Cumulants from Derivatives

• \(\nabla \mathcal{A}(\boldsymbol{\theta}) = \mathbb{E}[T(\mathbf{x})]\) (mean)
• \(\nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}[T(\mathbf{x})]\) (covariance)
• Higher derivatives give higher cumulants

Connection to Fisher Information

\[ G(\boldsymbol{\theta}) = \nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}_{\boldsymbol{\theta}}[T(\mathbf{x})] \] * R.A. Fisher: bounds estimation accuracy (Cramér-Rao) * Fisher information = Hessian of \(\mathcal{A}\) * Fisher information = Covariance of sufficient statistics * Defines geometry of information space * More details in next section

Entropy in Natural Coordinates

\[ H(\boldsymbol{\theta}) = \mathcal{A}(\boldsymbol{\theta}) - \boldsymbol{\theta}^\top \boldsymbol{\mu} \] where \(\boldsymbol{\mu} = \nabla \mathcal{A}(\boldsymbol{\theta})\)

Why Natural Parameters?

\[ \nabla_{\boldsymbol{\theta}} H = -G(\boldsymbol{\theta})\boldsymbol{\theta} \] * Key result for information dynamics * Terms involving \(\nabla \mathcal{A}\) cancel * Fisher information connects geometry to dynamics

Example: Multivariate Gaussian

Gaussian in Exponential Family Form

Gaussian as exponential family: * \(\boldsymbol{\theta} = \Sigma^{-1}\boldsymbol{\mu}\) (natural parameters) * \(T(\mathbf{x}) = \mathbf{x}\) (sufficient statistics) * \(\mathcal{A}(\boldsymbol{\theta}) = \frac{1}{2}\boldsymbol{\theta}^\top\Sigma\boldsymbol{\theta} + \frac{1}{2}\log|\Sigma|\)

Fisher Information for Gaussian

\[ G(\boldsymbol{\theta}) = \Sigma \] * Fisher information = covariance * \(G^{-1} = \Sigma^{-1}\) = precision * High variance → low Fisher info → harder estimation (looser Cramér-Rao) * Low variance → high Fisher info → easier estimation (tighter Cramér-Rao)

Example: Categorical Distribution

Categorical in Exponential Family Form

Categorical distribution: * \(\boldsymbol{\theta} = (\log \pi_1, \ldots, \log \pi_{K-1})\) * \(T(\mathbf{x}) = (x_1, \ldots, x_{K-1})\) (one-hot) * \(\mathcal{A}(\boldsymbol{\theta}) = \log\left(\sum_{k=1}^{K-1} e^{\theta_k} + 1\right)\)

Fisher Information for Categorical

\[ G_{ij} = \begin{cases} \pi_i(1-\pi_i) & i=j \\ -\pi_i\pi_j & i \neq j \end{cases} \] * Rare categories → low Fisher info * Uniform distribution → balanced information

Why Exponential Families Matter

Why exponential families matter: * Emerge from maximum entropy principles * Define information geometry * Sufficient statistics capture all information * Computationally tractable * Clean form for information dynamics

Looking Ahead

Next steps: * Natural parameters → coordinate system * Fisher information → information topography
* Entropy gradients → drive dynamics * \(\nabla H = -G(\boldsymbol{\theta})\boldsymbol{\theta}\) → key to everything

Softmax is MaxEnt with Features

Features, Not Just the Outcome

\[ p(y\mid x)=\frac{\exp(\theta\cdot f(x,y))}{\sum_{y'}\exp(\theta\cdot f(x,y'))} \] * Constraint: expected features match the data * Same multipliers as the die

Maximum Likelihood is the Same Fit

Two Spins, One Coupling

\[ p(s)\propto\exp(h_1 s_1+h_2 s_2+J s_1 s_2) \] * Constraints: \(\langle s_1\rangle\), \(\langle s_2\rangle\), \(\langle s_1 s_2\rangle\) * \(J\) is the multiplier on a correlation

The Same Hamiltonian, Two Readings

The Legendre Transform

Swap the Independent Variable

\[ y = \phi'(x),\qquad \phi^*(y) = xy - \phi(x) \] * Convex \(\phi\): the slope is a coordinate * \(\phi^*\) is the same information, different argument * The pair inverts: \(x = (\phi^*)'(y)\)

Helmholtz Was the First Example

Helmholtz Versus Gibbs

Entropy as Conjugate of the Log-Partition

\[ \boldsymbol{\eta} = \nabla A(\boldsymbol{\theta}),\qquad H = A(\boldsymbol{\theta}) - \boldsymbol{\theta}\cdot\boldsymbol{\eta} \] * \(A\) is convex: Hessian \(=\) Fisher \(=\) covariance * \(A^*(\boldsymbol{\eta}) = \boldsymbol{\theta}\cdot\boldsymbol{\eta} - A = -H\) * Same move as \(F = U - TS\)

Bernoulli: Check the Arithmetic

Bernoulli: \(\theta = \mathrm{logit}(p)\), \(A = \log(1+e^{\theta})\), \(\eta = p\) \[ H = A - \theta p = -p\log p -(1-p)\log(1-p) \] * If the subtraction misses \(H\), the signs are wrong

Two Charts, Named for Later

Three Perspectives

Same \(H\); three operational assumptions about what probability is over.

Define This Week

Named, Not Yet Answered

After This Lecture