Projection, Natural Gradient, and Optimal Protocols

Neil D. Lawrence

FW26, William Gates Building

This Session

  • MaxEnt as m-projection
  • Natural gradient: \(F^{-1}\nabla L\)
  • Geodesics = minimum-dissipation protocols

Information, entropy and intelligence course notebook setup

MaxEnt as Projection

Start from a reference \(r\) — often uniform. Impose moment constraints \(\mathbb{E}[T(X)]=\eta\).

  • MaxEnt distribution \(q\) minimises \(\mathrm{KL}(q\|r)\) subject to constraints
  • On an exponential family this is the \(m\)-projection onto the constraint surface
  • Constraint in \(\eta\); exponential family is a straight line in \(\theta\)
  • Uniform, then constrain
  • m-projection onto the constraint surface

Natural Gradient

Vanilla gradient ascent depends on how you parameterise \(\theta\). Natural gradient does not.

  • Steepest ascent in Fisher metric: \(\theta \leftarrow \theta + \eta\, F^{-1}\nabla L\)
  • Same \(F\) as Crooks’ \(\mathcal{I}\)
  • Worksheet 3 compares trajectories on the Gaussian plane
  • Vanilla gradient depends on coordinates
  • Natural gradient does not
  • Same \(F\) as Crooks’ \(\mathcal{I}\)

Geodesics as Optimal Protocols

Last week’s no-go: \(\langle W_{\mathrm{ex}}\rangle\ge\mathcal{L}^2/\tau\). This week’s prescription: travel by the geodesic.

  • Geodesic = shortest Fisher–Rao path between equilibria
  • Natural gradient = local steepest descent in the same metric
  • Wasserstein and Schrödinger bridges: week 8
  • No-go: \(\mathcal{L}^2/\tau\) (last week)
  • Prescription: the geodesic; \(F^{-1}\nabla L\)
  • Other geometries: week 8

Define This Week

  • MaxEnt as a projection
  • Why natural gradient is the correct gradient
  • The exponential family, now geometrically

Named, Not Yet Answered

  • Alternating \(m\)-projection onto two constraint sets? (week 8)

This Week’s Pair

After This Lecture

  • Quiz 3: 24 November, first ten minutes
  • LLM: is the natural gradient a bound or a recipe?

Further Reading

  • Sections 22.3–22.4 (Wasserstein geodesics, not Fisher–Rao) of Welling et al. (2026)

  • Chapters 3–4 of Amari (2016)

Thanks!

References

Amari, S., 2016. Information geometry and its applications, Applied mathematical sciences. Springer, Tokyo. https://doi.org/10.1007/978-4-431-55978-8
Welling, M., Lu, S., Holdijk, L., 2026. Generative AI and stochastic thermodynamics: A tale of free energies. Cambridge University Press, Cambridge, U.K.