Multi-Information and von Neumann Entropy

Neil D. Lawrence

FW26, William Gates Building

This Session

  • Quiz 4 (ten minutes)
  • Same marginals, different joints; then \(I+H=C\)
  • DPI and the information bottleneck
  • Why \(I=C\) forces von Neumann entropy

Quiz 4

  • Device required; Moodle open
  • No notes, no network, no LLMs
  • Ten minutes

Multi-Information

Sampling Two Dimensional Variables

Correlation

  • Correlation is when two variables are dependent

Sampling Two Dimensional Variables

Independent Gaussians

Correlated Gaussians

Anticorrelated Gaussians

The Classical Observer

The Classical Observer - Correlated

The Classical Observer - Anti-correlated

Mutual information is the pairwise case. Multi-information (Watanabe) generalises to \(n\) variables.

  • \(I = \sum_i h_i - H \ge 0\)
  • With fixed marginal entropies \(C = \sum_i h_i\), conservation \(I + H = C\)
  • \(I\) = stored correlation; \(H\) = free uncertainty

Data-Processing Inequality

Data-Processing Inequality

If \(X\to Y\to Z\) is a Markov chain, processing cannot create information about \(X\).

  • Markov: \(p(z\mid x,y)=p(z\mid y)\), equivalently \(I(X;Z\mid Y)=0\)
  • \(I(X;Y,Z)=I(X;Y)+I(X;Z\mid Y)=I(X;Y)\)
  • \(I(X;Y,Z)=I(X;Z)+I(X;Y\mid Z)\ge I(X;Z)\)
  • Therefore \(I(X;Z)\le I(X;Y)\)

KL Decay and Grouping

Under Markov dynamics, \(\text{KL}\left( p_t\,\|\,\pi \right)\) falls — coarse-graining in time.

  • Physics often says ``relative entropy’’; same object as week-3 / week-6 KL
  • Data-processing for distributions: each step cannot increase divergence to equilibrium
  • Same grouping split as the die, now for approach to \(\pi\):

\[ \text{KL}\left( p\,\|\,\pi \right)=\text{KL}\left( w\,\|\,w_\pi \right)+\sum_i w_i\, \text{KL}\left( p_i\,\|\,\pi_i \right) \]

Arrow of Time from the Same Split

How much of equilibration is between groups, how much within?

  • Between: \(\text{KL}\left( w\,\|\,w_\pi \right)\) — redistribution of macro probability
  • Within: \(\sum_i w_i \text{KL}\left( p_i\,\|\,\pi_i \right)\) — relaxation to local \(\pi_i\)
  • Demon / Crooks / transport: all move mass among groups while baths handle the rest

Information Bottleneck

Information Bottleneck

Least-committal \(T\) given a relevance constraint — MaxEnt, now on mutual information.

  • Markov: \(Y \leftrightarrow X \leftrightarrow T\)
  • Minimise \(I(X;T)\) subject to \(I(T;Y)\ge R\)
  • Lagrangian: \(I(X;T)-\beta I(T;Y)\)
  • DPI bound: \(I(T;Y)\le I(X;Y)\)

The Inaccessible Game

The Inaccessible Game

  • System isolated from observation
  • External observer cannot extract information
  • Internal state is inaccessible
  • Zero-player game with information-theoretic rules

Why “Inaccessible?”

Why “Inaccessible?”

From BLF axioms: Info gained = \(H(p) - H(q)\)

Our axiom: \(\sum h_i = C\) (constant)

\[\Delta(\sum h_i) = 0 \Rightarrow \text{observer learns nothing!}\]

What Makes It a Game?

Game Characteristics:

  • Zero-player game
  • State = probability distribution \(p(\mathbf{x}|\boldsymbol{\theta})\)
  • Rule = maximize entropy production
  • Constraint = \(\sum h_i = C\)
  • Dynamics = emerge from information geometry

Connection to Physical Reality

Physical Connections:

  • GENERIC structure emerges
  • Energy ↔︎ Entropy equivalence
  • Landauer’s principle derivable
  • Bridge between information and physics

Von Neumann Entropy

The Origin Paradox: Shannon vs von Neumann

Origin target:

  • \(I=C\)
  • \(H=0\)

Classical obstruction: \(H(X|Y)\ge 0\)

Quantum resolution: \(\mathsf{H}(A|B)\) can be negative

Why the classical origin is impossible

von Neumann entropy changes the rules

von Neumann entropy:

\[\textsf{H}(\rho)=-\text{tr}(\rho\log\rho)\]

Conditional:

\[\textsf{H}(A|B)=\textsf{H}(AB)-\textsf{H}(B) \text{ (can be < 0)}\]

Worked toy example: Bell state

Bell state:

  • \(\textsf{H}(AB)=0\)
  • \(\textsf{H}(A)=\textsf{H}(B)=\log 2\)
  • \(\textsf{H}(A|B)=-\log 2\)

Connection back to TIG

The Matrix Exponential Family

\[ \rho(\boldsymbol{\theta}) = \exp\!\left(\sum_k \theta_k F_k - \psi(\boldsymbol{\theta})\,\mathbf{I}\right) \]

  • \(\boldsymbol{\theta}\): natural parameters
  • \(\psi(\boldsymbol{\theta}) = \log\,\mathrm{tr}\exp\!\left(\sum_k\theta_k F_k\right)\): cumulant generating function
  • \(G(\boldsymbol{\theta}) = \nabla^2\psi(\boldsymbol{\theta})\): BKM metric (quantum Fisher information)

Faithful States

  • Implies faithful states (full rank \(\rho\))
  • Pure states are on boundary of family
  • BKM Metric is divergent

Maximum Entropy and Density Matrices

  • Jaynes recognized density matrices as quantum extension
  • Same maximum entropy principle applies
  • Von Neumann entropy: \(S = -\text{Tr}(\rho \log \rho)\)
  • Provides unified approach to classical and quantum systems

Classically, \(I=C\) forces \(H=0\) — a delta joint. Quantum mechanics breaks that pattern.

  • \(S(\rho) = -\mathrm{Tr}(\rho\log\rho)\)
  • Bell state: \(S(\rho)=0\) but marginals have \(S>0\)
  • Matrix exponential family = quantum MaxEnt

Purely Entropic Readings: First Attempt

Define This Week

  • Multi-information versus mutual information?
  • What is the data processing inequality?
  • What is the information bottleneck?
  • What is von Neumann entropy?
  • What is the matrix exponential family?

After This Lecture

  • Limits on intelligence follow

Further Reading

  • Theorem 2.8.1 of Cover and Thomas (1991)

  • the method; optional of Tishby et al. (1999)

  • https://inverseprobability.com/talks/notes/information-engines.html — Jaynes’ world and unified intelligence perspective of Information Engines seminar notes

  • Chapter 11 of Nielsen and Chuang (2010)

  • the Brandeis lectures of Jaynes (1963)

Thanks!

References

Cover, T.M., Thomas, J.A., 1991. Elements of information theory. Wiley, New York.
Jaynes, E.T., 1963. Information theory and statistical mechanics, in: Ford, K.W. (Ed.), Brandeis University Summer Institute Lectures in Theoretical Physics, Vol. 3: Statistical Physics. W. A. Benjamin, Inc., New York, pp. 181–218.
Nielsen, M.A., Chuang, I.L., 2010. Quantum computation and quantum information, 10th Anniversary. ed. Cambridge University Press, Cambridge, UK.
Tishby, N., Pereira, F.C., Bialek, W., 1999. The information bottleneck method, in: Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing. pp. 368–377.