Week 6: Fisher Metric and Thermodynamic Length

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

Fisher’s notion of information (sensitivity of the score, not Shannon uncertainty), the Fréchet–Rao–Cramér reading as identifiability, and the Fisher matrix as a Riemannian metric. A short bridge from Crooks’ fluctuation theorem (1999) and Jarzynski to the near-equilibrium bound ⟨W_ex⟩ ≥ ℒ²/τ; thermodynamic length (Crooks 2007) is then the Fisher–Rao length of a path of equilibrium states. Students should be able to define thermodynamic length today; the Schottky peak as a Fisher peak is named, not interpreted.

No class test today. Clarify what Fisher meant by information, then geometry. Bridge from Crooks (1999)/Jarzynski to the near-equilibrium bound, then define thermodynamic length (Crooks 2007). Worksheet 3 is released; due 24 November (start of lecture 7). Quiz 3 is then.

This Session

Time plan (120 minutes)

Information, entropy and intelligence course notebook setup

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util

cmd = install_command('pods')

%system {cmd}

cmd = install_command('mlai')

%system {cmd}

What Did Fisher Mean by Information?

Week 5 left us with a Hessian: for an exponential family, \[ G(\boldsymbol{\theta})=\nabla^2\mathcal{A}(\boldsymbol{\theta})=\mathrm{Cov}_{\boldsymbol{\theta}}[T(\mathbf{x})]. \] Today we ask what that object is. Fisher (1925) called the expected squared score information about a parameter. For a scalar parameter \(\theta\), \[ I(\theta)=\mathbb{E}_\theta\!\left[\left(\frac{\partial}{\partial\theta}\log p(x\mid\theta)\right)^2\right]. \] That is not Shannon entropy. Shannon’s \(H\) measures uncertainty in a random variable; Fisher’s \(I\) measures how much a sample tells you about \(\theta\) — the sensitivity of the log-likelihood to the parameter. Conflating the two words is the first mistake to block today.

Identifiability: Fréchet, Rao, Cramér

Independently of Fisher’s estimation programme, Fréchet, Rao, and Cramér arrived at the same matrix as a bound on how well parameters can be identified. The Cramér–Rao inequality \[ \mathrm{cov}(\hat{\boldsymbol{\theta}})\succeq G^{-1}(\boldsymbol{\theta}) \] says: where Fisher information is large, unbiased estimators can be precise; where it vanishes, the parameter is not identifiable from data. The geometric reading follows: \(G\) is a metric of distinguishability on the manifold of distributions.

For the canonical two-state system of week 2, the natural parameter is inverse temperature \(\beta\). Then \(G(\beta)\) is proportional to the heat capacity \(C\). Schottky’s anomaly — the peak of \(C\) when both states are populated — is therefore a peak of Fisher information: maximal thermal response is maximal distinguishability of nearby \(\beta\). Name that identity today. The purely entropic reading (what that peak means for information and intelligence) waits for week 8. Do not call \(G\) ``the rate of entropy production’‘; that role belongs to path costs such as Crooks’ \(\langle W_{\mathrm{ex}}\rangle\), defined later this lecture.

What is a Riemannian geometry?

KL Divergence and Two Entropies

Fisher Information as Geometry

We already have three readings of the same matrix: Fisher’s expected squared score, Fréchet–Rao–Cramér identifiability, and — for exponential families — the Hessian \[ G(\boldsymbol{\theta}) = \nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}_{\boldsymbol{\theta}}[T(\mathbf{x})]. \] We now develop the geometric interpretation: that matrix defines a metric on the space of probability distributions.

The Statistical Manifold

Consider the space of all probability distributions in an exponential family, parametrized by \(\boldsymbol{\theta}\). This space forms a manifold — a smooth, curved space where each point represents a different distribution.

The Fisher information matrix \(G(\boldsymbol{\theta})\) acts as a Riemannian metric on this manifold. Think of measuring distances on a curved surface like a sphere: you need a metric to tell you how far apart two nearby points are. The Fisher information provides exactly this for the space of probability distributions, telling us how to measure “statistical distance” between distributions.

The Fisher information defines the information distance between nearby distributions. If we move from parameters \(\boldsymbol{\theta}\) to \(\boldsymbol{\theta} + \text{d}\boldsymbol{\theta}\), the infinitesimal distance in information space is \[ \text{d}s^2 = \text{d}\boldsymbol{\theta}^\top G(\boldsymbol{\theta}) \text{d}\boldsymbol{\theta} \] where the Fisher information playing the role of the metric. Larger Fisher information means a given parameter change corresponds to a larger “information distance,” the distributions are more distinguishable.

Connection to Statistical Estimation

The geometric picture restates the Fréchet–Rao–Cramér bound from the open of the lecture. For any unbiased estimator \(\hat{\boldsymbol{\theta}}\), \[ \text{cov}(\hat{\boldsymbol{\theta}}) \succeq G^{-1}(\boldsymbol{\theta}), \] where \(\succeq\) denotes that the left side minus the right side is positive semidefinite. Higher Fisher information (stronger metric) means tighter estimation; \(G^{-1}\) is the error ellipsoid.

The Fisher information plays two distinct but related roles:

1. As a metric: It defines information distance, telling us how “far apart” distributions are.

2. In gradient flow: Recall from the exponential family definitions that that \(\nabla H = -G(\boldsymbol{\theta})\boldsymbol{\theta}\). This means entropy gradient ascent in exponential families involves the Fisher information, \[ \dot{\boldsymbol{\theta}} = \nabla H = -G(\boldsymbol{\theta})\boldsymbol{\theta}. \]

The appearance in the gradient comes from the specific structure of exponential families (where \(G = \nabla^2 \mathcal{A}\)). Together, they determine how the system flows through information space, with the geometry guiding the dynamics.

Examples Revisited

For the Gaussian distribution, we saw that \(G(\boldsymbol{\theta}) = \Sigma\). This means: - The information metric is the covariance matrix - The inverse \(G^{-1} = \Sigma^{-1}\) is the precision matrix

Geometrically, the information ellipsoid has the same shape as the probability ellipsoid. This direct connection between the Fisher information and covariance is special to Gaussians (and arises because we’re working in natural parameters \(\boldsymbol{\theta} = \Sigma^{-1}\boldsymbol{\mu}\)).

For a categorical distribution with \(K\) outcomes, the Fisher information has a special structure. Using the natural parameters \(\theta_k = \log \pi_k\), the Fisher information is \[ G_{ij}(\boldsymbol{\theta}) = \delta_{ij}\pi_i - \pi_i\pi_j = \begin{cases} \pi_i(1 - \pi_i) & i = j \\ -\pi_i\pi_j & i \neq j \end{cases} \]

This metric defines the probability simplex geometry. Distributions near the center of the simplex (all \(\pi_k \approx 1/K\)) have different local geometry than those near the corners (one \(\pi_k \approx 1\)). The Fisher metric captures this intrinsic curvature.

Information Geometry: The Big Picture

The Fisher information matrix is a foundational element of information geometry, a field that studies probability distributions using differential geometric tools. Key insights:

1. **mari’s Dually Flat Structure*: Exponential families have a special property. They are “dually flat” under two different coordinate systems (natural parameters \(\boldsymbol{\theta}\) and expectation parameters \(\boldsymbol{\mu}\)). The Fisher metric connects these.

2. Geodesics: The shortest path between two distributions (in the information geometry sense) is a geodesic. For exponential families, geodesics have elegant forms that will connect to our least action principles.

3. Curvature: The curvature of the statistical manifold (measured by the Riemann curvature tensor derived from \(G\)) tells us about the intrinsic structure of the family. Exponential families have zero curvature in a certain sense—they are “flat” manifolds.

These geometric properties will be essential when we study constrained information dynamics and emergence.

The Fisher matrix \(g_{ij}=\mathbb{E}[\partial_i\log p\,\partial_j\log p]\) defines a Riemannian metric on the statistical manifold. On exponential families, e-flat (\(\theta\)) and m-flat (\(\eta\)) charts are dual; Hessians of \(A\) and \(A^*\) are inverse metrics. The Pythagorean theorem for KL holds on dual flats.

Figure: Fisher components for \(\mathcal{N}(0,\sigma^2)\) blow up as \(\sigma^2\to 0\).

From Fluctuation Theorem to Length

\[ \frac{P(W)}{P_R(-W)} = e^{\beta(W-\Delta F)} \]

Crooks’ fluctuation theorem (Crooks, 1999) relates the work distribution of a forward protocol to that of the time-reversed protocol. For a system driven between two equilibrium endpoints at inverse temperature \(\beta\), \[ \frac{P(W)}{P_R(-W)} = e^{\beta(W-\Delta F)}, \] where \(\Delta F\) is the free-energy difference of those endpoints. This is an exact statement about trajectories, not yet a length on the equilibrium manifold.

Integrating the fluctuation theorem recovers Jarzynski’s equality (Jarzynski, 1997), \[ \langle e^{-\beta W}\rangle = e^{-\beta\Delta F}. \] Jensen’s inequality on the convex map \(x\mapsto e^{-\beta x}\) then gives \(\langle W\rangle \ge \Delta F\): the second law for isothermal work, as an average over trajectories. Define excess work \(W_{\mathrm{ex}}=W-\Delta F\). Then \(\langle W_{\mathrm{ex}}\rangle\ge 0\), with equality only for reversible protocols.

The fluctuation theorem holds arbitrarily far from equilibrium. Near equilibrium, for a slow protocol \(\lambda(t)\) of duration \(\tau\) that stays close to the instantaneous equilibrium state, linear response expands the mean excess work. The quadratic form that appears is the Fisher information metric \(\mathcal{I}(\lambda)\) on the manifold of equilibrium states. Defining thermodynamic length \[ \mathcal{L} = \int_0^\tau \sqrt{\dot\lambda^\top \mathcal{I}(\lambda)\,\dot\lambda}\,dt \] yields the bound \(\langle W_{\mathrm{ex}}\rangle \ge \mathcal{L}^2/\tau\) (Crooks, 2007). Geodesics of \(\mathcal{I}\) are therefore the leading-order minimum-dissipation protocols — the geometric consequence of the fluctuation relation, not a separate postulate.

(Welling et al., 2026) Chapter 17 derives the fluctuation theorem and Jarzynski’s equality, then stops: it does not define thermodynamic length, does not put a Fisher–Rao metric on equilibrium states, and does not cite Crooks (2007). Fisher information appears later in GAIST (Section 22.6) as the osmotic term \(\nabla\log\rho\) in the Schrödinger-bridge action — a different object from the Fisher matrix \(g_{ij}\) on the board today. Read GAIST Ch.~17 for the trajectory-level story; read Crooks (2007) for the geometric consequence we define next.

\[ \mathcal{L} = \int_0^\tau \sqrt{\dot\lambda^\top \mathcal{I}(\lambda)\,\dot\lambda}\,dt \]

Crooks (2007) packages the near-equilibrium expansion as Fisher–Rao length on the equilibrium manifold (Crooks, 2007). In linear response, \(\langle W_{\mathrm{ex}}\rangle\ge\mathcal{L}^2/\tau\). The metric is the prescription for measuring a change of state; geodesics minimise the leading dissipative cost. The exact distributional statement underneath is Crooks (1999) / Jarzynski, above.

Figure: Straight-line path in \((\mu,\sigma^2)\) whose Fisher–Rao length Worksheet 3 computes.

Define This Week

Interpret later: Schottky’s entropic reading; optimal trajectories and intelligence (week 8). Natural gradient as descent in the same metric is week 7.

After This Lecture

Worksheet 3: Fisher matrix for a Gaussian; vanilla versus natural gradient; Crooks length of the straight-line path from \((0,1)\) to \((2,4)\). Due 24 November. Quiz 3 is 24 November at the start of lecture 7.

Further Reading

• Chapters 1–2 of Amari (2016)

• fluctuation theorem (statement) of Crooks (1999)

• equality; second-law corollary of Jarzynski (1997)

• Chapter 17 (fluctuation theorem; stops before length) of Welling et al. (2026)

• the whole paper of Crooks (2007)

Thanks!

For more information on these subjects and more you might want to check the following resources.

• company: Trent AI
• book: The Atomic Human
• twitter: @lawrennd
• podcast: The Talking Machines
• newspaper: Guardian Profile Page
• blog: http://inverseprobability.com

References