FW26, William Gates Building
Exp family and Fisher before the close. Shannon and DPI wait.
Week 5’s discrete MaxEnt solution is an exponential family.
Weeks 4–5: every density is \({\mathrm{d}}P/{\mathrm{d}}\mu\). On a finite alphabet the unnoticed \(\mu\) is counting measure.
Canonical, Gaussian, and Bernoulli are the same theorem with different \(T(x)\).
Each named distribution from the probability review is an exponential family — and each carries a base measure \(h_0\).
Same \(\theta\); different \(h_0\). That is the measure story in one pair.
\[ P(x\mid\theta)=\exp\bigl(x\,\theta-\psi(\theta)\bigr)\,h_0(x),\qquad x\in\{0,1\} \]
\[ P(k\mid\theta)=\exp\bigl(k\,\theta-\psi(\theta)\bigr)\,h_0(k),\qquad k=0,\ldots,n \]
Poisson makes the non-constant \(h_0\) unavoidable: without \(1/k!\) you do not get Poisson.
\[ P(k\mid\theta)=\exp\bigl(k\,\theta-\psi(\theta)\bigr)\,h_0(k),\qquad k\in\mathbb{N}_0 \]
\[ P(\mathbf{x}\mid\boldsymbol{\theta})=\exp\bigl(\boldsymbol{\theta}\!\cdot\! T(\mathbf{x})-\psi(\boldsymbol{\theta})\bigr)\,h_0(\mathbf{x}) \]
Continuous close: \(h_0\) is Lebesgue — the continuum counterpart of \(\#\).
\[ p(y\mid\boldsymbol{\theta})=\exp\bigl(\theta_1 y+\theta_2 y^2-\psi(\boldsymbol{\theta})\bigr)\,h_0(y) \]
\[ \log p(x\mid\boldsymbol{\theta})=\boldsymbol{\theta}\cdot T(x)-\psi(\boldsymbol{\theta}) \]
Differentiate once; take a second moment.
###{Fisher Metric from the Exponential Family}
Same word; different mathematical definition.
These are the same matrix.
\[ \mathrm{d}s^2 = \mathrm{d}\boldsymbol{\theta}^\top G(\boldsymbol{\theta})\,\mathrm{d}\boldsymbol{\theta} \]
Exponential families are dually flat under two coordinate systems.
A channel is a conditional \(p(y\mid x)\). Rate cannot beat capacity.
Four results we will need later — stated, not proved today.
If \(X\to Y\to Z\) is Markov, processing cannot create information about \(X\).
The CGF was \(\psi(\boldsymbol{\theta})\). After the break, \(A\) is action.
Success is low entropy in the reguland — with two caveats.
Quantity bound, not the structure theorem.
Fisher metric; dual flatness of Amari (2016)
noisy channel coding / capacity of Shannon (1948)
Chapter 7 of Cover and Thomas (1991)
Chapters 8–10 of MacKay (2003)
the Good Regulator Theorem of Conant and Ashby (1970)
requisite variety; Shannon form of Ashby (1956)