Fisher Metric and Thermodynamic Length

Neil D. Lawrence

FW26, William Gates Building

This Session

  • Fisher’s information \(\neq\) Shannon’s \(H\)
  • Fréchet–Rao–Cramér: identifiability
  • Fisher metric; dual flatness
  • Fluctuation theorem \(\to\) \(\langle W\rangle\ge\Delta F\) \(\to\) length bound
  • Thermodynamic length: define, do not interpret

Information, energy and intelligence course notebook setup

What Did Fisher Mean by Information?

Fisher’s information \(\neq\) Shannon’s entropy * Shannon \(H\): uncertainty in an outcome * Fisher \(I(\theta)\): sensitivity of \(\log p\) to \(\theta\) * Same word; different operational reading

Score and Sensitivity

\[ I(\theta)=\mathbb{E}\!\left[(\partial_\theta\log p)^2\right] \] * Large \(I\): data distinguish nearby \(\theta\) well * Small \(I\): parameter almost invisible in samples

Identifiability: Fréchet, Rao, Cramér

  • Same \(G\): Fisher (estimation) and Fréchet–Rao–Cramér (identifiability)
  • \(\mathrm{cov}(\hat\theta)\succeq G^{-1}\): high \(G\) \(\Rightarrow\) tight bound
  • Metric reading: \(G\) measures how distinguishable nearby distributions are

What is a Riemannian geometry?

Euclidean vs Riemannian Distance

In flat space the ruler is the same everywhere. On a curved surface it is not.

  • Euclidean: \(\mathrm{d}s^2 = \mathrm{d}\mathbf{x}^\top \mathbf{I}\, \mathrm{d}\mathbf{x}\) (constant metric)
  • Riemannian: \(\mathrm{d}s^2 = \mathrm{d}\mathbf{x}^\top g(\mathbf{x})\, \mathrm{d}\mathbf{x}\) (position-dependent)
  • Geodesic: shortest path; curves in non-flat space

Geodesics and Length

\[ \ell(\gamma) = \int_0^1 \sqrt{\dot\gamma^\top g(\gamma)\,\dot\gamma}\;\mathrm{d}t \] * Geodesic minimises \(\ell\) * Flat space: straight lines; sphere: great circles * Statistical manifold: \(g = G\) (Fisher) — connects distributions

KL Divergence and Two Entropies

KL Divergence: Forward and Reverse

\(D_{\mathrm{KL}}\) measures how different two distributions are — but it is asymmetric.

  • \(D_{\mathrm{KL}}(p\|q) = \sum_x p(x)\log\frac{p(x)}{q(x)} \ge 0\), zero iff \(p=q\)
  • Forward (\(p\|q\)): mass-covering — \(q\) must not miss mass of \(p\)
  • Reverse (\(q\|p\)): mode-seeking — \(q\) concentrates on a mode of \(p\)
  • Second-order Taylor: \(D_{\mathrm{KL}}(p_{\boldsymbol{\theta}+\mathrm{d}\boldsymbol{\theta}}\|p_{\boldsymbol{\theta}}) = \tfrac{1}{2}\,\mathrm{d}\boldsymbol{\theta}^\top G(\boldsymbol{\theta})\,\mathrm{d}\boldsymbol{\theta} + O(\|\mathrm{d}\boldsymbol{\theta}\|^3)\)

KL Curvature = Fisher Metric

\[ D_{\mathrm{KL}}(p_{\boldsymbol{\theta}+\mathrm{d}\boldsymbol{\theta}}\|p_{\boldsymbol{\theta}}) = \tfrac{1}{2}\,\mathrm{d}\boldsymbol{\theta}^\top G(\boldsymbol{\theta})\,\mathrm{d}\boldsymbol{\theta} + O(\|\mathrm{d}\boldsymbol{\theta}\|^3) \] * \(G\) = curvature of KL at coincidence * Two KLs, one local metric * Pythagorean theorem holds on dually flat submanifolds

Fisher Information as Geometry

Same \(G\), three origins: \[ G(\boldsymbol{\theta}) = \nabla^2 \mathcal{A}(\boldsymbol{\theta}) = \mathrm{Cov}_{\boldsymbol{\theta}}[T(\mathbf{x})] \] * Fisher / CR / Hessian — now as Riemannian metric

The Statistical Manifold

Statistical Manifold: * Each point \(\boldsymbol{\theta}\) = a probability distribution * Space of all distributions = curved manifold * Fisher information = metric (ruler) on this space * Measures “closeness” between distributions

Information Distance

\[ \text{d}s^2 = \text{d}\boldsymbol{\theta}^\top G(\boldsymbol{\theta}) \text{d}\boldsymbol{\theta} \] * Measures information distance between distributions * Larger \(G\) = distributions more distinguishable * Smaller \(G\) = distributions harder to tell apart

Connection to Statistical Estimation

Cramér–Rao (restated geometrically): \[ \text{cov}(\hat{\boldsymbol{\theta}}) \succeq G^{-1}(\boldsymbol{\theta}) \] * \(G^{-1}\) = error ellipsoid * High \(G\) → tight estimation; low \(G\) → loose

Why This Matters for Dynamics

Two Roles of Fisher Information: 1. Metric → defines distances between distributions 2. In gradient → \(\nabla H = -G(\boldsymbol{\theta})\boldsymbol{\theta}\)

\[ \dot{\boldsymbol{\theta}} = \nabla H = -G(\boldsymbol{\theta})\boldsymbol{\theta} \]

Examples Revisited

Gaussian: Geometry of Covariance

Gaussian: \(G(\boldsymbol{\theta}) = \Sigma\) * Information metric = covariance * \(G^{-1} = \Sigma^{-1}\) = precision
* Information ellipsoid = probability ellipsoid * Special to Gaussians in natural parameters

Categorical: Simplex Geometry

Categorical: \[ G_{ij} = \delta_{ij}\pi_i - \pi_i\pi_j \] * Defines probability simplex geometry * Center of simplex: balanced information * Corners: concentrated information * Metric captures curvature

Information Geometry: The Big Picture

Information Geometry: * Fisher metric → Riemannian geometry * Exponential families → dually flat structure * Geodesics → shortest paths between distributions * Zero curvature → special “flat” structure * Key for constrained dynamics later

The Fisher Metric

A statistical manifold: each point is a distribution \(p(x\mid\theta)\).

  • Fisher matrix: \(g_{ij} = \mathbb{E}[\partial_i\log p\,\partial_j\log p]\)
  • Exponential families: e-flat (\(\theta\)) and m-flat (\(\eta\)) charts
  • Pythagorean theorem for KL on dual flats

From Fluctuation Theorem to Length

Before length: the exact nonequilibrium relation that the second law and the \(\mathcal{L}^2/\tau\) bound sit inside.

\[ \frac{P(W)}{P_R(-W)} = e^{\beta(W-\Delta F)} \]

  • Forward protocol: work distribution \(P(W)\)
  • Reverse protocol: \(P_R(-W)\)
  • \(\Delta F\): equilibrium free-energy difference between endpoints
  • Integrate Crooks \(\Rightarrow\) Jarzynski: \(\langle e^{-\beta W}\rangle = e^{-\beta\Delta F}\)
  • Jensen’s inequality: \(\langle W\rangle \ge \Delta F\)
  • Excess work \(W_{\mathrm{ex}} = W - \Delta F\) is nonnegative on average
  • Far from equilibrium: Crooks/Jarzynski (exact, distributional)
  • Near equilibrium, slow driving: expand \(\langle W_{\mathrm{ex}}\rangle\) in linear response
  • Leading cost is Fisher–Rao length of the protocol: \(\langle W_{\mathrm{ex}}\rangle \ge \mathcal{L}^2/\tau\)
  • Crooks (1999): \(P(W)/P_R(-W)=e^{\beta(W-\Delta F)}\) — exact
  • Jarzynski: \(\langle e^{-\beta W}\rangle=e^{-\beta\Delta F}\) \(\Rightarrow\) \(\langle W\rangle\ge\Delta F\)
  • Crooks (2007): near equilibrium, \(\langle W_{\mathrm{ex}}\rangle\ge\mathcal{L}^2/\tau\) with Fisher–Rao \(\mathcal{L}\)

Thermodynamic Length (Crooks 2007)

For a slow protocol \(\lambda(t)\) on the equilibrium manifold, define length with the Fisher metric.

\[ \mathcal{L} = \int_0^\tau \sqrt{\dot\lambda^\top \mathcal{I}(\lambda)\,\dot\lambda}\,dt \]

  • No-go: \(\langle W_{\mathrm{ex}}\rangle \ge \mathcal{L}^2/\tau\) in linear response
  • Prescription: measure change of state with the Fisher metric
  • Intelligence question: week 8
  • No-go: \(\langle W_{\mathrm{ex}}\rangle \ge \mathcal{L}^2/\tau\)
  • Prescription: length is measured with the Fisher metric
  • Intelligence question: week 8

Define This Week

  • Fisher’s \(I(\theta)\) versus Shannon’s \(H\)
  • Identifiability: Fréchet–Rao–Cramér / Cramér–Rao
  • Schottky peak as Fisher peak (named only)
  • Fisher metric as a Riemannian metric; dual charts
  • Crooks (1999) \(\to\) Jarzynski \(\to\) \(\langle W\rangle\ge\Delta F\)
  • What is thermodynamic length?

After This Lecture

  • Worksheet 3 released; due 24 November
  • LLM exercise: thermodynamic length, two sides

Further Reading

  • Chapters 1–2 of Amari (2016)

  • fluctuation theorem (statement) of Crooks (1999)

  • equality; second-law corollary of Jarzynski (1997)

  • Chapter 17 (fluctuation theorem; stops before length) of Welling et al. (2026)

  • the whole paper of Crooks (2007)

Thanks!

References

Amari, S., 2016. Information geometry and its applications, Applied mathematical sciences. Springer, Tokyo. https://doi.org/10.1007/978-4-431-55978-8
Crooks, G.E., 2007. Measuring thermodynamic length. Phys. Rev. Lett. 99, 100602. https://doi.org/10.1103/PhysRevLett.99.100602
Crooks, G.E., 1999. Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences. Phys. Rev. E 60, 2721–2726. https://doi.org/10.1103/PhysRevE.60.2721
Jarzynski, C., 1997. Nonequilibrium equality for free energy differences. Physical Review Letters 78, 2690–2693. https://doi.org/10.1103/PhysRevLett.78.2690
Welling, M., Lu, S., Holdijk, L., 2026. Generative AI and stochastic thermodynamics: A tale of free energies. Cambridge University Press, Cambridge, U.K.