[jupyter][google colab][edit]

Abstract

Compute the Fisher information matrix of the Gaussian family, compare vanilla and natural gradient, and estimate the thermodynamic length of a path between two Gaussians. Then explain dual flatness and catch what an LLM gets wrong about the natural gradient.

Worksheet 3

Released after lecture 4 (3 November). Due by 10:00 at the start of lecture 6 (17 November). Estimated three hours. This worksheet is 15% of the module mark.

Builds on lectures 4–5. The Fisher matrix and natural-gradient trajectories press the MaxEnt / Landauer material into geometry (gaussian-fisher-eigen, natural-gradient-paths, crooks-path-sketch). Part A3 integrates Fisher–Rao length along the straight-line path \((0,1)\to(2,4)\).

Submit candidatenumber_worksheet3.ipynb and candidatenumber_worksheet3.md on Moodle. LLMs are permitted. Use the Lecture 1 habit: curiosity first, then skeptical probes. Authenticity is checked by the paired in-class quiz on a new parameterised family.

Lecture 5 defined Crooks’ thermodynamic length as the Fisher–Rao length of a path of equilibrium states. You will compute that length for a straight-line path in ((,^2)). You do not yet need to say what this has to do with intelligence – that is lecture 8.

Information, energy and intelligence course notebook setup

[edit]

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util
cmd = install_command('pods')
%system {cmd}
cmd = install_command('notutils')
%system {cmd}
cmd = install_command('mlai')
%system {cmd}
import numpy as np
import matplotlib.pyplot as plt

Part A – Code

[edit]

The Gaussian family ((,^2)) has two useful coordinate systems. Mean parameters ((,^2)) are the familiar ones. Natural parameters (=(/2,;-1/(22))) are those in which (p(x)=(T(x)-A())) is linear.

Question 1

In a markdown cell, derive the natural parameters () from the standard Gaussian log-density and identify the sufficient statistics (T(x)=(x,x^2)) and the log-partition function (A()). Then derive the Fisher matrix (g_{ij}=[_ip_jp]) in the mean parameterisation ((,^2)). Implement fisher_gaussian(mu, sigma2) (analytic is preferred; automatic differentiation or finite differences are acceptable). Plot the eigenvalues of (g) against (^2) at (). In what regime is the geometry most curved, and why is (^2) a singularity?

20 marks

Question 2

Draw 500 samples from ((2,4)). Fit a Gaussian by maximising the log-likelihood with (i) vanilla gradient ascent (+) and (ii) natural gradient ascent (+,g^{-1}()). Start both from ((_0,^2_0)=(0,1)), 200 steps, (). Plot the trajectories on the ((,^2)) plane and the loss curves. Which converges faster, and why does the natural gradient path look more direct?

20 marks

Question 3

Thermodynamic length (Crooks). Using the Fisher matrix from the first question, estimate the length of the straight-line path in mean parameters from (_0=(0,1)) to (_1=(2,4)), \[ \mathcal{L}=\int_0^1\sqrt{\dot\theta(t)^\top g(\theta(t))\,\dot\theta(t)}\,dt. \] A simple Riemann sum is enough. Report () and the implied dissipation bound (^2/) for (). Optional: try a qualitatively different path (for example, change (^2) first, then ()) and say which is shorter. Do not interpret this as a claim about intelligence; that is lecture 8.

15 marks

Part B – Reflection

Question 4

Dually flat geometry (200 words). In your own words, explain what it means for the exponential family to be dually flat. Why does this make computations that would otherwise require a full Riemannian treatment tractable? Use the Pythagorean theorem for KL as your central example.

20 marks

Question 5

LLM perspective (200 words). Ask an LLM: “Why is natural gradient descent better than ordinary gradient descent for statistical models?” Identify one thing it explained well and one thing it got wrong or glossed over, and correct it. Include the key response as an appendix, not counted in the word limit.

25 marks

Marking

  • 60–74: Fisher matrix implemented; both gradient methods run; reflection notes that natural gradient removes parameterisation dependence.
  • 75–79: dual flatness is precise, not just “easier”; trajectories annotated; thermodynamic length computed.
  • 80–89: a second path is compared, or the optional Pythagorean check on the Jaynes die from Worksheet 2 is done; reflection makes a connection not in the lectures.
  • 90–100: original experiment – for example, a non-exponential family where the Pythagorean identity fails, or a geodesic that is measurably shorter than the straight line.

Optional extension

On the categorical family over ({1,,6}), show that the MaxEnt die from Worksheet 2 is the m-projection of the uniform distribution onto the mean-constraint hyperplane by checking ((p|r)=(p|q)+(q|r)) numerically, where (q) is MaxEnt and (r) is uniform.

Submission

Upload both files to Moodle by 10:00 on Tuesday 17 November (start of lecture 6).