Week 6: Exponential Family, Fisher, and Overviews

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

After Quiz 3, develop the exponential family introduced at the end of Week 5: natural parameters θ=-λ, the canonical ensemble, the Gaussian, and the Bernoulli as one MaxEnt family, and ψ(θ)=log Z(-θ) as a cumulant generating function. The Hessian ∇²ψ is the Fisher information metric on that family: Fisher’s I(θ) is not Shannon’s H, and ds² = dθᵀ G dθ measures distinguishability. Dual charts are introduced briefly. Then Shannon’s noisy-channel coding theorem (R≤C), the data-processing inequality as a statement, and a Good Regulator overview only: setup diagrams with disturbance D, system S, action A (not R), reguland Z; the slogan and its two caveats. Full Conant–Ashby development is the Week 8 review.

Quiz 3 opens the session — Weeks 4–5 as delivered (grouping, demon compartments and two ledgers, soft partitions / MI / KL, MaxEnt and the exponential family introduced). Collect Worksheet 3 as they sit (due 10:00). Release Worksheet 4 early: it is another Skepticus dialogue; Exchange 1 asks what the Good Regulator Theorem is and how it relates to modern notions of general intelligence, begun before that theorem is taught. After the quiz, develop the exponential family: \(\theta=-\lambda\), canonical / Gaussian / Bernoulli, \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\) as a cumulant generating function, and the Hessian of \(\psi\) as Fisher. Shannon capacity, DPI, and a full Good Regulator overview stay in the file; as taught on 17 November they were not reached (GRT was a two-minute W4 pointer only). W4 due 1 December (start of lecture 8).

This Session

Time plan (as delivered 17 November 2026). Clocks from the Whisper SRT on the Panopto capture (00:00 ≈ 10:00). The slot is 10:05–11:55 (110 minutes, not 120). Quiz 3 quiet ran to ~10:32. Shannon capacity and DPI were not walked. Path length of \(G\) is not today’s object. Full GRT diagrams wait for Week 7.

ClockBlock
10:05–10:32Quiz 3 (paper); collect Worksheet 3; release Worksheet 4
10:32–10:34Maxwell-speed / high-d ring aside
10:34–11:00Exp family: \(\theta=-\lambda\); \(\psi=\log Z\) as CGF; cumulants
11:00–11:05MaxEnt property Q&A; break called
11:05–11:15Break
11:15–11:42Base measure (Poisson / binomial); Bernoulli; Gaussian natural params
11:42–11:56\(H=\psi-\theta\cdot\eta\); Fisher / natural gradient; NN deltas
11:56–11:58GRT pointer for Worksheet 4 only

As Taught

As taught on 17 November Quiz 3 ran long (quiet until ~10:32 on the Panopto clock; Whisper filled the gap with thank-you loops). A short Maxwell-speed / high-dimensional ring aside opened the spoken half. The first half was a live MaxEnt re-derivation into exponential-family form (\(\psi\), \(\theta=-\lambda\), \(T\)), then \(\psi=\log Z\) as a cumulant generating function: first derivative \(\langle T\rangle\), second \(\mathrm{Var}[T]\), with skew / kurtosis / Cauchy (MacKay lighthouse) as colour. The break was called after the CGF overview and a MaxEnt-property Q&A. After the break: counting versus non-counting base measures (Poisson \(1/k!\), binomial \(\binom{n}{k}\), coin-toss limit), Bernoulli logit / sigmoid / softplus CGF and the ML ``deltas’’ moral, Gaussian natural parameters \(\mu/\sigma^2\) and \(-1/(2\sigma^2)\) with a live completion of the square, then entropy \(H=\psi-\theta\cdot\eta\) and \(\nabla_\theta H=-G\theta\) (natural gradient). Shannon capacity and DPI were not walked. The Good Regulator was a two-minute close for Worksheet 4 (minimise entropy of \(Z\); Ashby; body temperature) — no \(D,S,A,Z\) diagrams. Continuous MaxEnt remains unread overflow from Week 5.

Quiz 3

Invigilated paper test, no notes, no network, no LLMs. Ten questions on Weeks 4–5 as delivered: exclusive grouping \(H=H(w)+\sum_i w_i H(p_i)\), soft partitions \(H(G\mid X)\) / \(I(X;G)\) / KL, the missing measure, Maxwell’s demon as compartments and two ledgers, MaxEnt (die / discrete proof), and the exponential family introduced (\(\theta=-\lambda\)). Not CGF, not continuous MaxEnt, not Fisher, not Legendre, not Crooks, not GRT.

Class test timer (Quiz 3): 10:05

Information, energy and intelligence course notebook setup

[edit]

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util
cmd = install_command('pods')
%system {cmd}
cmd = install_command('notutils')
%system {cmd}
cmd = install_command('mlai')
%system {cmd}

Exponential Family

[edit]

On a finite state space the constrained MaxEnt distribution is an exponential family. With sufficient statistics \(T_k(x)=f_k(x)\) and base density \(h_0\) of the counting measure \(\#\) on the allowed outcomes, \[\begin{align} p(x\mid\boldsymbol{\theta}) &= \exp\bigl(\boldsymbol{\theta}\cdot T(x) - \psi(\boldsymbol{\theta})\bigr)\, h_0(x),\\ \psi(\boldsymbol{\theta}) &= \log \sum_x \exp\bigl(\boldsymbol{\theta}\cdot T(x)\bigr), \end{align}\] where \(h_0(x)=\mathbf{1}[x\in\mathcal{X}]\) (one on allowed labels, zero elsewhere). The Lagrange multipliers from the Week 5 proof are the negative natural parameters, \(\theta_k=-\lambda_k\). The minus sign keeps high-energy states down-weighted when \(\theta_1=-\beta<0\).

A probability is always a density with respect to some reference measure \(\mu\): the Radon–Nikodym derivative \({\mathrm{d}}P/{\mathrm{d}}\mu\). Every entropy-like quantity in the course is the same expression, \[\begin{align} H_\mu(P)=-\int\log\fracP}\mu}\,{\mathrm{d}}P, \end{align}\] with a different \(\mu\) each time. Lecture 4 already said that choosing a discretisation is choosing a measure; the counting measure \(\#\) is the case so natural that nobody notices the choice has been made.

Separate two uses of the word Lebesgue. The Lebesgue integral \(\int f\,{\mathrm{d}}\mu\) is a single integration machine that works for any measure \(\mu\): a sum is a Lebesgue integral against \(\#\), and the ordinary calculus integral is a Lebesgue integral against Lebesgue measure \(\lambda\). Lebesgue measure \(\lambda\) itself is one particular choice of \(\mu\): length, area, or volume, so \(\lambda([a,b])=b-a\). Integrating against \(\#\) gives a sum, \(\int f\,{\mathrm{d}}\#=\sum_x f(x)\), and the density of \(P\) with respect to \(\#\) is the probability mass function \(p(x)=P(\{x\})\). Shannon entropy is exactly \(H_\#(P)=-\sum_x p(x)\log p(x)\).

Counting measure is a prior: every outcome gets weight 1. On a finite set of \(N\) outcomes write \(u=\#/N\) for the uniform distribution. Then \[\begin{align} H(P)=\log N-\mathrm{KL}(P\|u). \end{align}\] So Shannon entropy is a negative KL divergence against the uniform that \(\#\) defines. Three immediate consequences: (i) \(H\ge 0\), because each atom has counting measure 1 so \(p(x)\le 1\); (ii) \(H\) is unchanged by any relabelling of the outcomes (\(\#\) is invariant under bijections); (iii) on a countably infinite set \(\#\) has infinite total mass, there is no uniform \(u\), and entropy need not be finite.

The same template covers the continuous and relative cases. Lebesgue measure \(\lambda\) is the continuum counterpart: differential entropy is \(H_\lambda\), can be negative, and changes under nonlinear reparametrisation because \(\lambda\) does. Two distinct effects meet here and should stay separate. Under a change of variable the Jacobian \(|{\mathrm{d}}y/{\mathrm{d}}x|\) appears — that is Lecture 4’s density-of-states factor (for example \(v^2\) in the speed coordinate). Separately, putting counting measure on a grid of spacing \(\Delta\) and rescaling by \(\Delta\) converges to \(\lambda\) as \(\Delta\to 0\); a binned Shannon entropy then carries an additive \(-\log\Delta\), which is the log of that rescaling factor between \(\#\) and \(\lambda\), not a Jacobian. Why not just use counting measure on \(\mathbb{R}\)? Because \(\#\) and \(\lambda\) are mutually singular: \(\lambda(\{x\})=0\) for every point, while a discrete law puts positive mass on points, so neither measure admits a density with respect to the other. A mixed law — daily rainfall exactly zero with probability \(0.3\), otherwise continuous — has a density only with respect to something like \(\delta_0+\lambda\).

KL divergence uses a probability \(Q\) as reference, \[\begin{align} \mathrm{KL}(P\|Q)=\int\log\fracP}Q}\,{\mathrm{d}}P, \end{align}\] provided \(P\) is absolutely continuous with respect to \(Q\) (\(P\ll Q\)). If \(P\) puts mass anywhere \(Q\) does not, the Radon–Nikodym derivative fails and \(\mathrm{KL}(P\|Q)=+\infty\). When both densities are written against a common base \(\mu\), the ratio of densities cancels \(\mu\), which is why KL (and mutual information) has no \(-\log\Delta\) divergence and is unchanged when the same invertible map is applied to both arguments.

Maximum entropy relative to \(\mu\) gives \({\mathrm{d}}P/{\mathrm{d}}\mu=\exp(\theta\cdot T-\psi)\). The factor \(h_0\) in the exponential family is the density of that base measure when sums or integrals are written against a default chart. Today \(\nu=\#\) on the allowed set and \(h_0={\mathrm{d}}\nu/{\mathrm{d}}\#=\mathbf{1}[x\in\mathcal{X}]\), so \(\psi\) is a log-sum. Even in discrete cases the base need not be plain counting: Poisson uses \(h_0(x)=1/x!\) on \(\mathbb{N}\), binomial uses \(\binom{n}{x}\) — combinatorial multiplicity as the discrete counterpart of the \(v^2\) density of states. We write \(h_0\) rather than \(h\) to avoid colliding with differential entropy and with the regulator map \(A=h(S)\) later in the module.

Extension: where does \(h_0\) come from? Often \(x\) is a coarse summary of a finer description. For example, \(k\) = number of successes in \(n\) binary trials. Put the uniform (counting) measure on the micro states and push it forward through the summary: \(h_0(k)=\#\{\text{micro states with summary }k\}=\binom{n}{k}\). Because the tilt \(\exp(\theta k-\psi(\theta))\) depends only on \(k\), the distribution stays uniform within each group of micro states. So \(h_0\) is the density of states, Boltzmann’s \(W\), and \(\mathbb{E}[\log h_0(K)]\) is the entropy lost by recording only \(k\). (For the Poisson, the micro states are unordered event times in \([0,1]\), and \(h_0(k)=1/k!\) is a volume rather than a count.) Coarse-graining by \(T\) is a deterministic channel, so \(H(K)\le H_{\mathrm{micro}}\) — a small DPI-style fact that the Data-Processing Inequality later makes precise.

Each of these is MaxEnt for its moment constraints, so each belongs to the exponential family. The canonical ensemble is the mean-energy case already recovered in Week 5. A Gaussian is MaxEnt for a fixed mean and variance. A two-level system with energies \(0\) and \(\varepsilon\) is a Bernoulli with natural parameter \(\theta=-\beta\varepsilon\). The distribution bank that follows is where both measure morals land: Bernoulli / categorical use plain counting (\(h_0=1\)); binomial / Poisson / multinomial use combinatorial \(h_0\) as discrete density of states.

Common Distributions as Exponential Families

[edit]

Week 1 introduced five families without writing them in natural coordinates: Bernoulli, binomial, Poisson, multinomial/categorical, and Gaussian. Each is MaxEnt for its moment constraints relative to an appropriate base measure \(h_0\), so each belongs to the exponential family \[ p(x\mid\boldsymbol{\theta})=\exp\bigl(\boldsymbol{\theta}\cdot T(x)-\psi(\boldsymbol{\theta})\bigr)\,h_0(x). \] The slides below record, for each family, the natural parameter(s) \(\boldsymbol{\theta}\), the sufficient statistic \(T\), the cumulant generating function \(\psi\), and — the point of this block — which \(h_0\) is in force. Two emphases run through the tour. First, when \(h_0=1\) (Bernoulli; categorical \(n=1\)), the base is counting measure: every allowed label has prior weight one, the same unnoticed prior as \(H=\log N-\mathrm{KL}(P\|u)\). Second, when \(h_0\) is non-constant (binomial \(\binom{n}{k}\), Poisson \(1/k!\), multinomial \(n!/\prod_j x_j!\)), that factor is the discrete density of states: same moment constraints relative to plain \(\#\) would give a different family. Gaussian uses Lebesgue — the continuum counterpart of \(\#\).

Bernoulli as Exponential Family

[edit]

A Bernoulli(\(\pi\)) trial with \(P(X=1)=\pi\) is already in exponential-family form once we take the logit as the natural parameter, \[ \theta=\log\frac{\pi}{1-\pi},\qquad T(x)=x,\qquad \psi(\theta)=\log(1+e^{\theta}). \] The base density is \(h_0(x)=1\): counting measure on the two labels. That is the unnoticed prior from the counting-measure slide — each of \(\{0,1\}\) has weight one, so Shannon \(H\) on a Bernoulli is \(\log 2-\mathrm{KL}(P\|u)\) with \(u\) uniform on those two outcomes. Recovering the mean parameter is the logistic map: \(\pi=\sigma(\theta)=e^{\theta}/(1+e^{\theta})\), and \(\psi'(\theta)=\pi\). The two-level thermal system with energies \(0\) and \(\varepsilon\) is the same family with \(\theta=-\beta\varepsilon\).

Binomial as Exponential Family

[edit]

Binomial\((n,\pi)\) is \(n\) i.i.d. Bernoulli trials counted. The natural parameter is the same logit as the Bernoulli, \[ \theta=\log\frac{\pi}{1-\pi},\qquad T(k)=k, \] but the cumulant generating function scales with the number of trials, \[ \psi(\theta)=n\log(1+e^{\theta}), \] and the base measure carries the combinatorial multiplicity \(h_0(k)=\binom{n}{k}\). That multiplicity is the discrete density of states — the same role \(v^2\) played for Maxwell–Boltzmann. MaxEnt under a mean-count constraint relative to plain counting measure does not recover the binomial; MaxEnt relative to \(\binom{n}{k}\,{\mathrm{d}}\#\) does. Same \(\theta\) as Bernoulli; different prior weight on the count \(k\). Mean and variance follow by differentiation: \(\psi'(\theta)=n\pi\), \(\psi''(\theta)=n\pi(1-\pi)\).

Poisson as Exponential Family

[edit]

Poisson(\(\lambda\)) has probability mass \[ P(k)=\frac{\lambda^k e^{-\lambda}}{k!}. \] In exponential-family coordinates the rate becomes a natural parameter through \(\theta=\log\lambda\), the count is the sufficient statistic \(T(k)=k\), and the cumulant generating function is simply \[ \psi(\theta)=e^{\theta}=\lambda. \] The factorial in the denominator is the base measure \(h_0(k)=1/k!\) with respect to counting measure on \(\mathbb{N}_0\). This is the sharpest discrete illustration of the Weeks 4–5 story: the exponential tilt \(\exp(\theta\,T-\psi)\) is only half the model; the base \(h_0\) is the other half. MaxEnt under a fixed mean count relative to plain counting measure (\(h_0=1\)) yields a geometric distribution. MaxEnt relative to \(h_0=1/k!\) yields Poisson. Same constraint, different measure, different family — the discrete counterpart of Lebesgue versus \(v^2\mathrm{d}v\). Differentiating recovers the familiar moments: \(\psi'(\theta)=\lambda\) and \(\psi''(\theta)=\lambda\) (mean equals variance).

Multinomial as Exponential Family

[edit]

A categorical distribution on \(K\) exclusive labels is a multinomial with a single trial (\(n=1\)). Write the outcome as a one-hot vector \(\mathbf{x}\) with \(\sum_{j=1}^K x_j=1\). Because of the simplex constraint we use \(K-1\) free natural parameters, \[ \theta_j=\log\frac{\pi_j}{\pi_K},\qquad j=1,\ldots,K-1, \] with sufficient statistics \(T(\mathbf{x})=(x_1,\ldots,x_{K-1})\) and cumulant generating function \[ \psi(\boldsymbol{\theta})=\log\Bigl(1+\sum_{j=1}^{K-1}e^{\theta_j}\Bigr) \] (the log-sum-exp that recovers \(\pi_K=1/\bigl(1+\sum_j e^{\theta_j}\bigr)\)). For \(n\) independent trials the counts \(\mathbf{x}\) are multinomial: the same \(\boldsymbol{\theta}\) and \(T\), but \[ \psi(\boldsymbol{\theta})=n\log\Bigl(1+\sum_{j=1}^{K-1}e^{\theta_j}\Bigr) \] and base measure \(h_0(\mathbf{x})=n!/(x_1!\cdots x_K!)\). Softmax regression is this family with features in place of the raw one-hot.

Gaussian as Exponential Family

[edit]

The univariate Gaussian \(\mathcal{N}(\mu,\sigma^2)\) is MaxEnt under fixed mean and variance, so it belongs to a two-parameter exponential family. Expanding the quadratic energy gives natural parameters \[ \theta_1=\frac{\mu}{\sigma^2},\qquad \theta_2=-\frac{1}{2\sigma^2}, \] sufficient statistics \(T(y)=(y,y^2)\), and cumulant generating function \[ \psi(\theta_1,\theta_2)=-\frac{\theta_1^2}{4\theta_2}-\frac12\log(-2\theta_2)-\frac12\log(2\pi). \] The base measure is Lebesgue — the continuous counterpart of counting measure \(\#\) — and the \((2\pi)\) factor sits in \(\psi\) (equivalently in a constant piece of \(h_0\)). Differentiating recovers the moments: \(\nabla\psi=(\mu,\mu^2+\sigma^2)\). With known variance the family collapses to one parameter, \(\theta=\mu/\sigma^2\), \(T(y)=y\), and \(\psi(\theta)=\tfrac12\sigma^2\theta^2\) (up to an additive constant independent of \(\theta\)). The multivariate case replaces \(\theta_1\) by \(\Sigma^{-1}\boldsymbol{\mu}\) and \(\theta_2\) by \(-\tfrac12\Sigma^{-1}\) acting on \(yy^\top\).

\(\psi(\theta)\) as Cumulant Generating Function

The Quiz 2 debrief already read \(\log Z\) as available energy, \(F=-k_BT\log Z\). In exponential-family coordinates the same function is \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\). Its first derivative is the mean of \(T\). Differentiating again will give \(\mathrm{Cov}[T]\), but we derive it after the entropy form.

Entropy of an Exponential Family

[edit]

Once \(p\) is written as an exponential family, Shannon entropy collapses to an algebraic identity in \(\psi\) and \(\boldsymbol{\eta}\). That identity is the bridge to Fisher information: the same \(\log p\) whose expectation is \(-H\) has a score whose second moment is \(\nabla^2\psi\).

Write \(\log p(x\mid\boldsymbol{\theta})=\boldsymbol{\theta}\cdot T(x)-\psi(\boldsymbol{\theta})+\log h_0(x)\). When the base is counting measure on the allowed set (\(h_0=1\)), Shannon entropy collapses to \[\begin{align} H(\boldsymbol{\theta}) &=-\mathbb{E}_{\boldsymbol{\theta}}[\log p(X\mid\boldsymbol{\theta})] =\psi(\boldsymbol{\theta})-\boldsymbol{\theta}\cdot\mathbb{E}_{\boldsymbol{\theta}}[T(X)] =\psi(\boldsymbol{\theta})-\boldsymbol{\theta}\cdot\boldsymbol{\eta}, \end{align}\] with \(\boldsymbol{\eta}=\nabla\psi\). In general the base survives as \[\begin{align} H(\boldsymbol{\theta})=\psi(\boldsymbol{\theta})-\boldsymbol{\theta}\cdot\boldsymbol{\eta}-\mathbb{E}_{\boldsymbol{\theta}}[\log h_0(X)], \end{align}\] so the short form \(H=\psi-\theta\cdot\eta\) is the \(h_0=1\) special case (Bernoulli, categorical, Jaynes’ die), not a licence to drop \(1/k!\) from a Poisson. Equivalently, when \(h_0=1\), \(H=-\psi^*(\boldsymbol{\eta})\) where \(\psi^*\) is the Legendre conjugate of \(\psi\). That is the form that makes the next differentiation step cheap.

The coarse-graining picture explains the difference. Every micro state has probability \(\exp(\theta k-\psi)\), so the micro entropy is \(H_{\mathrm{micro}}=\psi-\theta\cdot\eta\). The macro entropy of \(K\) (relative to counting measure on \(k\)) is \(H(K)=\psi-\theta\cdot\eta-\mathbb{E}[\log h_0(K)]\). The chain rule relates them: \[\begin{align} H_{\mathrm{micro}}=H(K)+\underbrace{H(\mathrm{micro}\mid K)}_{\mathbb{E}[\log h_0(K)]}, \end{align}\] because given \(K=k\) the distribution is uniform over \(h_0(k)\) micro states. So \(\mathbb{E}[\log h_0]\) is the entropy lost by coarse-graining — Boltzmann’s \(\log W\), averaged over macrostates. For the Poisson, \(h_0=1/k!\) is a volume rather than a count, \(\log h_0=-\log k!\) is negative, and \(H(\mathrm{micro}\mid K)\) is a differential entropy: the short formula is then not a Shannon entropy of anything discrete, and plugging \(h_0=1\) into it gives the wrong answer.

Differentiate \(\log p=\boldsymbol{\theta}\cdot T-\psi\) under the expectation: \[ \nabla_{\boldsymbol{\theta}}\log p(x\mid\boldsymbol{\theta})=T(x)-\nabla\psi(\boldsymbol{\theta})=T(x)-\boldsymbol{\eta}. \] The score has mean zero. Fisher’s information matrix is the second moment of the score, \[ G_{ij}(\boldsymbol{\theta})=\mathbb{E}_{\boldsymbol{\theta}}[\partial_i\log p\,\partial_j\log p]=\mathrm{Cov}_{\boldsymbol{\theta}}[T]_{ij}. \] Differentiating \(\boldsymbol{\eta}=\nabla\psi\) once more yields the same matrix as a Hessian: \(G=\nabla^2\psi\). So the cumulant-generating property is the Fisher metric on an exponential family — which follows from the entropy calculation.

###{Fisher Metric from the Exponential Family}

Fisher (1925) called the expected squared score information about a parameter. For a scalar parameter, \[ I(\theta)=\mathbb{E}_\theta\!\left[\left(\frac{\partial}{\partial\theta}\log p(x\mid\theta)\right)^2\right]. \] On an exponential family that is exactly \(\nabla^2\psi=\mathrm{Cov}[T]\), as derived above. Shannon’s \(H\) measures uncertainty in a random variable; Fisher’s \(I\) measures how much a sample tells you about \(\theta\). Conflating the two words is the first mistake to block — and the exponential family is where both appear from one \(\log p\).

Three readings of one matrix: covariance of the sufficient statistics, Fisher’s expected squared score, and the Hessian of the cumulant generating function. Fréchet–Rao–Cramér identifiability (\(\mathrm{cov}(\hat{\boldsymbol{\theta}})\succeq G^{-1}\)) is the estimation reading. That matrix is the Fisher information metric on the statistical manifold.

The Fisher matrix defines infinitesimal information distance. Larger \(G\) means a given parameter step is more distinguishable. Integrating \(\mathrm{d}s\) along a path of equilibria would give Crooks’ thermodynamic length; that development is dropped from the module. Today the metric itself is the object.

Amari’s dual flatness: exponential families admit affine coordinates in \(\boldsymbol{\theta}\) and in \(\boldsymbol{\eta}=\nabla\psi\), with dual potentials whose Hessians are inverse Fisher metrics. The Legendre dual is what they already computed for entropy: \(\psi^*(\eta)=\theta\cdot\eta-\psi=-H\) when \(h_0=1\) (otherwise subtract \(\mathbb{E}[\log h_0]\) from \(H\)).

Shannon’s Communication Theorem

Shannon’s communication theorem (noisy-channel coding) is the information-theoretic twin of Carnot: it forbids reliable rates above capacity and does not construct a code. Capacity \(C=\max_{p(x)}I(X;Y)\) is the no-go number; the capacity-achieving input distribution is the prescription. Week 5 already made mutual information first-class via soft partitions; today that \(I\) is the rate currency on a channel.

Scaffolding, Not Outcomes

[edit]

The chain rule \(H(X,Y)=H(X)+H(Y|X)\) is the algebraic source of multi-information. Mutual information \(I(X;Y)=H(X)-H(X|Y)\) is the pairwise case; we do not yet treat \(n>2\). Channel capacity \(C\) is a no-go on rate; the capacity-achieving input distribution is the prescription. The data-processing inequality is the no-go on \(I\): processing cannot create information. Cover and Thomas Theorem 2.8.1 is the reading; the proof waits for week 8, when \(I\) is first-class.

Figure: Mutual information for a binary symmetric channel; capacity is achieved at uniform input.

Data-Processing Inequality

The data-processing inequality is the no-go on \(I\): further processing of \(Y\) cannot increase information about \(X\). Week 5’s soft-partition leftover \(H(G\mid X)\) is the same grammar — a stochastic map spends mutual information. As taught on 17 November this statement was not walked; Week 7 recovers it as statement/moral. The board proof stays notes overflow / enrichment, not assessed.

Good Regulator

Good Regulator Overview

[edit]

Overview only. Full Conant–Ashby development — concavity argument, existence of a deterministic optimum, and the two caveats in depth — is the Week 8 review. Today’s job is the setup, the notation, and the slogan and its two caveats. Diagrams follow Joery’s oneshot figure in the good-regulator notes: disturbance \(D\), system state \(S\), action \(A\), reguland \(Z\).

Conant and Ashby (Conant and Ashby, 1970) often label the regulator \(R\). We write \(A\) for the action so the policy \(\pi(a\mid s)\) is first-class. The reguland — the variable whose entropy successful regulation minimises — is \(Z\). The system state is \(S\); the disturbance is \(D\). The cumulant generating function earlier today was \(\psi(\boldsymbol{\theta})=\log Z(-\boldsymbol{\theta})\); that \(Z\) was the partition function, not the reguland — different objects, same letter in the physics tradition.

Figure: Cause-controlled regulation (left) acts on the disturbance \(D\); error-controlled regulation (right) acts on the state \(S\) that \(D\) caused.

A disturbance \(D\) causes events in a system \(S\); the regulator counters with an action \(A\); the pair produces a reguland \(Z=\phi(S,A)\) (Conant and Ashby, 1970). Cause-controlled regulation acts on \(D\) (or on an inferred model of \(D\)). Error-controlled regulation acts on the state \(S\) caused by \(D\), so by the data-processing inequality \(I(A;D)\le I(S;D)\). Both diagrams are shown above; the existence proof that an \(H(Z)\)-optimal policy may be taken deterministic waits for Week 8.

Successful regulation, for Conant and Ashby, minimises the entropy of the reguland: \(\min_\pi H(Z)\). Among optima there exists a deterministic policy \(A=h(S)\), equivalently \(H(A\mid S)=0\). That is the sense in which the regulator is a model'' --- deliberately weak. Do not collapse the slogan into\(A\) contains a rich model of \(S\)’’: a constant action also has \(H(A\mid S)=0\). Do not confuse low \(H(Z)\) with a desirable outcome. Requisite variety in Shannon form will use the same letters: residual uncertainty satisfies \(H(Z)\ge H(D)-I(D;A)\) when \(A\) is the regulatory response. Week 8 reviews the theorem properly and applies it to capability claims.

Ashby’s law of requisite variety in Shannon form (Ashby, 1956), written in today’s letters: residual uncertainty in the reguland is at least the disturbance entropy minus the information the action carries about the disturbance. Requisite variety asks how much relevant \(I\) is required; the Good Regulator Theorem asks how that \(I\) may be organised. Full development of both, with the concavity argument, is Week 8.

Define This Week

After This Lecture

Quiz 3 opened the session (quiet until ~10:32), before the exponential family and Fisher geometry were developed. Worksheet 4 was released so Exchange 1 could begin before any GRT material (due 1 December, before Quiz 4). It is a Skepticus dialogue on GRT and general intelligence, not the parked notebook brief. As taught, Shannon capacity and DPI were not walked; the Good Regulator was a two-minute close (entropy of \(Z\); Ashby), not the \(D,S,A,Z\) diagrams. Week 7 (Good Regulator, Goals, and Channels) takes the assessed spine: unmarked W3 review; GRT overview; goal measure via Radon–Nikodym; Shannon \(R\le C\) and DPI as statement/moral. Multi-information depth, DPI board proof, and IB stay notes overflow. Week 8 after Quiz 4 / W4 is unassessed: short classical limits synthesis, then protected no-barber \(\to\) von Neumann / quantum motivation, then course close.

Further Reading

  • Fisher metric; dual flatness of Amari (2016)

  • noisy channel coding / capacity of Shannon (1948)

  • Chapter 7 of Cover and Thomas (1991)

  • Chapters 8–10 of MacKay (2003)

  • the Good Regulator Theorem of Conant and Ashby (1970)

  • requisite variety; Shannon form of Ashby (1956)

Thanks!

For more information on these subjects and more you might want to check the following resources.

References

Amari, S., 2016. Information geometry and its applications, Applied mathematical sciences. Springer, Tokyo. https://doi.org/10.1007/978-4-431-55978-8
Ashby, W.R., 1956. An introduction to cybernetics. Chapman & Hall, London.
Conant, R.C., Ashby, W.R., 1970. Every good regulator of a system must be a model of that system. International Journal of Systems Science 1, 89–97.
Cover, T.M., Thomas, J.A., 1991. Elements of information theory. Wiley, New York.
MacKay, D.J.C., 2003. Information theory, inference and learning algorithms. Cambridge University Press, Cambridge, U.K.
Shannon, C.E., 1948. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x