Exponential Family, Fisher, and Overviews

Neil D. Lawrence

FW26, William Gates Building

This Session

  • Quiz 3 (ten minutes); collect Worksheet 3; release Worksheet 4
  • Exponential family: \(\theta=-\lambda\); Week 1 families in \((\theta,\psi)\) form
  • \(\psi(\boldsymbol{\theta})=\log Z\); entropy \(H=\psi-\theta\cdot\eta\) (when \(h_0=1\))
  • Fisher from the score: \(G=\mathrm{Cov}[T]=\nabla^2\psi\); \(I\neq H\)
  • Shannon: \(R\le C\); DPI; Good Regulator overview (\(D,S,A,Z\))

As Taught

Exp family and Fisher before the close. Shannon and DPI wait.

  • MaxEnt re-derived: \(\theta=-\lambda\), \(\psi=\log Z\), \(T\) as sufficient stats
  • CGF: \(\nabla\psi=\langle T\rangle\); Hessian is Fisher / \(\mathrm{Cov}[T]\)
  • After the break: base measure (Poisson \(1/k!\), binomial multiplicity)
  • Bernoulli logit / softplus; Gaussian natural params; \(H=\psi-\theta\cdot\eta\)
  • Natural gradient of entropy is \(-\theta\); GRT only as a W4 pointer

Quiz 3

  • Paper; no notes, no network, no LLMs
  • Ten minutes
  • Worksheet 3 in as you sit

Exponential Family

Exponential Family: \(\theta=-\lambda\)

Week 5’s discrete MaxEnt solution is an exponential family.

  • Sufficient statistics: \(T_k(x_i)=f_k(x_i)\), fixed by the constraints
  • MaxEnt: \(p_i=\exp\bigl(-\sum_k \lambda_k T_k(x_i)\bigr)/Z\)
  • Family: \(p(x\mid\boldsymbol{\theta})\propto \exp\bigl(\boldsymbol{\theta}\!\cdot\! T(x)-\psi(\boldsymbol{\theta})\bigr)\)
  • Identification: \(\boldsymbol{\theta}=-\boldsymbol{\lambda}\)

Counting Measure Is a Prior

Weeks 4–5: every density is \({\mathrm{d}}P/{\mathrm{d}}\mu\). On a finite alphabet the unnoticed \(\mu\) is counting measure.

  • Counting \(\#\): each allowed label weight 1; \(\int f\,{\mathrm{d}}\#=\sum_x f(x)\)
  • Shannon: \(H(P)=\log N-\mathrm{KL}(P\|u)\) with \(u=\#/N\)
  • Exponential family: \(p=\exp(\theta\cdot T-\psi)\,h_0\), today \(h_0=\mathbf{1}[x\in\mathcal{X}]\)
  • Same role as Jaynes’ \(m\); not a new ``measure of uncertainty’’

Three Members, One Family

Canonical, Gaussian, and Bernoulli are the same theorem with different \(T(x)\).

  • Canonical: one energy constraint, \(T=E\), \(\theta=-\beta\)
  • Gaussian: mean and variance constraints
  • Bernoulli / two-level: \(T\in\{0,1\}\), \(\theta=-\beta\varepsilon\)

Common Distributions as Exponential Families

Week 1 Families, Exponential-Family Form

Each named distribution from the probability review is an exponential family — and each carries a base measure \(h_0\).

  • Form: \(p=\exp\bigl(\boldsymbol{\theta}\!\cdot\! T-\psi\bigr)\,h_0\)
  • Two stories for \(h_0\): counting prior (\(h_0=1\)) versus combinatorial / continuum measure
  • Bernoulli, categorical: plain counting on the labels
  • Binomial, Poisson, multinomial: result is a macroscopic of a microscopic sum
  • Gaussian: Lebesgue (continuous counterpart of #)

Bernoulli and Binomial

Same \(\theta\); different \(h_0\). That is the measure story in one pair.

  • Bernoulli: \(\theta=\mathrm{logit}(\pi)\), \(\psi=\log(1+e^{\theta})\), \(h_0=1\) (counting)
  • Binomial: same \(\theta\); \(\psi=n\log(1+e^{\theta})\); \(h_0=\binom{n}{k}\) (multiplicity)

Bernoulli as Exponential Family

Bernoulli: Natural Parameter and \(\psi\)

\[ P(x\mid\theta)=\exp\bigl(x\,\theta-\psi(\theta)\bigr)\,h_0(x),\qquad x\in\{0,1\} \]

  • \(\theta=\log\dfrac{\pi}{1-\pi}\),\(T(x)=x\)
  • \(\psi(\theta)=\log(1+e^{\theta})\),\(h_0(x)=1\) (counting on \(\{0,1\}\))

Binomial as Exponential Family

Binomial: Natural Parameter and \(\psi\)

\[ P(k\mid\theta)=\exp\bigl(k\,\theta-\psi(\theta)\bigr)\,h_0(k),\qquad k=0,\ldots,n \]

  • \(\theta=\log\dfrac{\pi}{1-\pi}\),\(T(k)=k\)
  • \(\psi(\theta)=n\log(1+e^{\theta})\),\(h_0(k)=\binom{n}{k}\)

Poisson and Multinomial

Poisson makes the non-constant \(h_0\) unavoidable: without \(1/k!\) you do not get Poisson.

  • Poisson: \(\theta=\log\lambda\), \(\psi=e^{\theta}\), \(h_0=1/k!\)
  • Categorical: \(h_0=1\) (counting again); multinomial: \(h_0=n!/\prod_j x_j!\)

Poisson as Exponential Family

Poisson: Why \(h_0\) Is Not Optional

\[ P(k\mid\theta)=\exp\bigl(k\,\theta-\psi(\theta)\bigr)\,h_0(k),\qquad k\in\mathbb{N}_0 \]

  • \(\theta=\log\lambda\),\(T(k)=k\),\(\psi(\theta)=e^{\theta}\)
  • \(h_0(k)=1/k!\) — without it, mean-count MaxEnt is geometric, not Poisson

Multinomial as Exponential Family

Multinomial / Categorical: Natural Parameters and \(\psi\)

\[ P(\mathbf{x}\mid\boldsymbol{\theta})=\exp\bigl(\boldsymbol{\theta}\!\cdot\! T(\mathbf{x})-\psi(\boldsymbol{\theta})\bigr)\,h_0(\mathbf{x}) \]

  • \(\theta_j=\log\dfrac{\pi_j}{\pi_K}\),\(T=(x_1,\ldots,x_{K-1})\)
  • \(\psi(\boldsymbol{\theta})=n\log\bigl(1+\sum_{j=1}^{K-1}e^{\theta_j}\bigr)\)
  • Categorical: \(n=1\), \(h_0=1\); multinomial: \(h_0=\dfrac{n!}{x_1!\cdots x_K!}\)

Gaussian

Continuous close: \(h_0\) is Lebesgue — the continuum counterpart of \(\#\).

  • \(T=(y,y^2)\); \(\theta_1=\mu/\sigma^2\), \(\theta_2=-1/(2\sigma^2)\)
  • \(h_0\): Lebesgue (not counting; not \(1/k!\))
  • \(\psi\) is quadratic in \(\theta_1\) and logarithmic in \(\theta_2\)

Gaussian as Exponential Family

Gaussian: Natural Parameters and \(\psi\)

\[ p(y\mid\boldsymbol{\theta})=\exp\bigl(\theta_1 y+\theta_2 y^2-\psi(\boldsymbol{\theta})\bigr)\,h_0(y) \]

  • \(\theta_1=\dfrac{\mu}{\sigma^2}\),\(\theta_2=-\dfrac{1}{2\sigma^2}\),\(T(y)=(y,y^2)\)
  • \(h_0\): Lebesgue (continuum counterpart of counting \(\#\))
  • \(\psi(\theta_1,\theta_2)=-\dfrac{\theta_1^2}{4\theta_2}-\dfrac12\log(-2\theta_2)-\dfrac12\log(2\pi)\)

\(\psi(\theta)\) as Cumulant Generating Function

\(\psi(\theta)=\log Z\) Generates Cumulants

  • Week 3: \(F=-k_BT\log Z\) (available energy)
  • Same object: \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\)
  • Mean parameters: \(\boldsymbol{\eta}=\nabla\psi=\langle T\rangle\)
  • Next: entropy from \(\log p\), then Fisher from the score

Entropy of an Exponential Family

Entropy: the Lovely Form

\[ \log p(x\mid\boldsymbol{\theta})=\boldsymbol{\theta}\cdot T(x)-\psi(\boldsymbol{\theta}) \]

  • Take \(-\mathbb{E}[\log p]\): \(H=\psi-\boldsymbol{\theta}\cdot\boldsymbol{\eta}\) when \(h_0=1\)
  • with \(\boldsymbol{\eta}=\nabla\psi=\langle T\rangle\)
  • In general: \(H=\psi-\boldsymbol{\theta}\cdot\boldsymbol{\eta}-\mathbb{E}[\log h_0(X)]\)

Fisher from the Same \(\log p\)

Differentiate once; take a second moment.

  • Score: \(\nabla_{\boldsymbol{\theta}}\log p = T-\boldsymbol{\eta}\)
  • \(\mathbb{E}[\mathrm{score}]=0\); \(G=\mathbb{E}[\mathrm{score}\,\mathrm{score}^\top]=\mathrm{Cov}[T]\)
  • But \(\boldsymbol{\eta}=\nabla\psi\), so \(G=\nabla^2\psi\)

###{Fisher Metric from the Exponential Family}

Fisher’s Information \(\neq\) Shannon’s Entropy

Same word; different mathematical definition.

  • Shannon \(H\): uncertainty in an outcome — \(H=\psi-\theta\cdot\eta\) (when \(h_0=1\))
  • Fisher \(I(\theta)\): sensitivity of \(\log p\) to \(\theta\) — \(I=\mathrm{Cov}[T]=\nabla^2\psi\)
  • \(H\) is a state function of \(p\); \(I\) is a property of a family \(p(\cdot\mid\theta)\)

Same Matrix, Three Names

These are the same matrix.

  • \(\mathrm{Cov}[T]\) from cumulants
  • Fisher \(G\) from the score \(T-\eta\)
  • \(\nabla^2\psi\) from differentiating \(\eta=\nabla\psi\)
  • Cramér–Rao: \(\mathrm{cov}(\hat{\boldsymbol{\theta}})\succeq G^{-1}\) (stated, not proved)

Information Distance

\[ \mathrm{d}s^2 = \mathrm{d}\boldsymbol{\theta}^\top G(\boldsymbol{\theta})\,\mathrm{d}\boldsymbol{\theta} \]

  • \(G\) is a Riemannian metric on the manifold of distributions
  • Large \(G\): nearby \(\theta\) are easy to tell apart from data
  • Small \(G\): the parameter is almost invisible in samples

Dual Charts \((\theta,\eta)\)

Exponential families are dually flat under two coordinate systems.

  • Natural parameters \(\boldsymbol{\theta}\) (e-coordinates)
  • Mean parameters \(\boldsymbol{\eta}=\nabla \psi=\langle T\rangle\) (m-coordinates)
  • Hessians of the dual potentials are inverse metrics
  • Details in the notes

Shannon’s Communication Theorem

Shannon’s Noisy Channel Coding Theorem

A channel is a conditional \(p(y\mid x)\). Rate cannot beat capacity.

  • Mutual information \(I(X;Y)\) is the channel currency
  • Capacity \(C = \max_{p(x)} I(X;Y)\)
  • No-go: reliable communication requires \(R \le C\)
  • Prescription: the input \(p(x)\) that achieves \(C\)

Scaffolding, Not Outcomes

Chain Rule, Mutual Information, Capacity

Four results we will need later — stated, not proved today.

  • Chain rule: \(H(X,Y) = H(X) + H(Y|X)\)
  • Mutual information: \(I(X;Y) = H(X)-H(X|Y) = H(X)+H(Y)-H(X,Y)\)
  • Capacity: no rate above \(C\); achieving \(C\) requires the capacity-achieving \(p(x)\)
  • Data processing: if \(X\to Y\to Z\) is Markov, \(I(X;Z)\le I(X;Y)\)

Data-Processing Inequality

Data-Processing Inequality

If \(X\to Y\to Z\) is Markov, processing cannot create information about \(X\).

  • Markov: \(p(z\mid x,y)=p(z\mid y)\)
  • Statement: \(I(X;Z)\le I(X;Y)\)
  • Soft partitions: leftover \(H(G\mid X)\) already spent information
  • Full proof: notes overflow (Cover–Thomas 2.8.1)
  • No-go: processing cannot create \(I\)
  • Week 7: state Shannon \(R\le C\) and DPI if unread; GRT overview; goal measure

Good Regulator

Good Regulator Overview

Notation: \(D\), \(S\), \(A\), \(Z\)

The CGF was \(\psi(\boldsymbol{\theta})\). After the break, \(A\) is action.

  • \(D\): disturbance
  • \(S\): system state
  • \(A\): action (the regulator’s response; Conant–Ashby often write \(R\))
  • \(Z\): reguland / outcome (Ashby’s essential variable)

Two Setups

  • Cause-controlled: act on \(D\) (or on a model of \(D\))
  • Error-controlled: act on the state \(S\) that \(D\) caused
  • DPI: \(I(A;D)\le I(S;D)\) when \(A\) only sees \(S\)

What ``Good’’ Means Today

Success is low entropy in the reguland — with two caveats.

  • Objective sketch: \(\min_{\pi(a\mid s)} H(Z)\)
  • Slogan: every good regulator must be a model of the system
  • Weak reading of ``model’’: \(H(A\mid S)=0\), i.e. \(A=h(S)\)
  • Full theorem and caveats: Week 8 review

Requisite Variety

Quantity bound, not the structure theorem.

  • \(H(Z)\ge H(D)-I(D;A)\)
  • Useless variety: large \(H(A)\) with small \(I(D;A)\)
  • Ashby: how much regulatory information is required
  • GRT (Week 8): how that information may be structured

Define This Week

  • Why is the exponential family the MaxEnt family, and why \(\theta=-\lambda\)?
  • Why is \(H=\psi-\theta\cdot\eta\) when \(h_0=1\) (and what changes for Poisson), and why does that make \(G=\nabla^2\psi=\mathrm{Cov}[T]\)?
  • How does Fisher’s \(I(\theta)\) differ from Shannon’s \(H\)?
  • What does \(R\le C\) forbid, and what does DPI state?
  • Name \(D\), \(S\), \(A\), \(Z\) — Week 7 develops the overview

After This Lecture

  • Worksheet 4 out (dialogue; Exchange 1 before GRT); due 1 December before Quiz 4
  • Week 7: GRT overview; goal measure; Shannon / DPI statement
  • Week 8: Quiz 4; limits synthesis; no-barber \(\to\) quantum (unassessed)

Further Reading

  • Fisher metric; dual flatness of Amari (2016)

  • noisy channel coding / capacity of Shannon (1948)

  • Chapter 7 of Cover and Thomas (1991)

  • Chapters 8–10 of MacKay (2003)

  • the Good Regulator Theorem of Conant and Ashby (1970)

  • requisite variety; Shannon form of Ashby (1956)

Thanks!

References

Amari, S., 2016. Information geometry and its applications, Applied mathematical sciences. Springer, Tokyo. https://doi.org/10.1007/978-4-431-55978-8
Ashby, W.R., 1956. An introduction to cybernetics. Chapman & Hall, London.
Conant, R.C., Ashby, W.R., 1970. Every good regulator of a system must be a model of that system. International Journal of Systems Science 1, 89–97.
Cover, T.M., Thomas, J.A., 1991. Elements of information theory. Wiley, New York.
MacKay, D.J.C., 2003. Information theory, inference and learning algorithms. Cambridge University Press, Cambridge, U.K.
Shannon, C.E., 1948. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x