Week 8: Multi-Information and von Neumann Entropy

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

Watanabe’s multi-information and the conservation law \(I+H=C\); the data-processing inequality and the information bottleneck as the no-go/prescription pair on \(I\); then the classical limit \(I=C\) and von Neumann entropy. Material may run long; limits on intelligence absorb the overflow.

Quiz 4 occupies the first ten minutes. Then multi-information, \(I+H=C\), the data-processing inequality, the information bottleneck, and von Neumann entropy. Worksheet 4 is due at the start of this lecture. Limits on intelligence follow and can absorb overflow.

This Session

Time plan (do not cut DPI / IB / von Neumann to fit the hour)

If the slot is still listed as one hour, run long: the second half (limits) can start late or pick up unfinished interpret questions. Schottky / Good Regulator remain notes-only until the limits lecture: preview the Shannon split \(H(E)\ge H(D)-I(D;R)\) versus the existence claim \(H(A\mid S)=0\), but leave the concavity proof, caveats, and IB gloss for the second half. DPI and IB are taught here (LO10).

Quiz 4

Ten MCQs from the multi-information / limits bank. Auto-graded.

Information, energy and intelligence course notebook setup

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util

cmd = install_command('pods')

%system {cmd}

cmd = install_command('notutils')

%system {cmd}

cmd = install_command('mlai')

%system {cmd}

Multi-Information

Figure: Samples from independent Gaussian variables that represent horizontal and vertical velocities when our system is at equilibrium.

Sampling Two Dimensional Variables

Figure: Samples from correlated Gaussian variables that represent vertical and horizontal velocity.

Figure: Two independent Gaussians for the \(x\) and \(y\) velocity of a ball.

Figure: A correlated Gaussian for the \(x\) and \(y\) velocity of a ball. If all balls were correlated in this way, this would imply that the whole box is moving towards the upper right or bottom left.

Figure: An anti-correlated Gaussian for the \(x\) and \(y\) velocity of a ball. If all balls were anti-correlated in this way, this would imply that the whole box is moving towards the upper left or bottom right.

The Classical Observer

Figure: Here the observer is monitoring the movements of the particles. We’ve plotted the velocities alongside the 1 standard deviation contour of their theoretical distribution.

The Classical Observer - Correlated

Figure: Again the observer is monitoring the movements of the particles, but here their motion is correlated (\(\rho=0.95\)).

The Classical Observer - Anti-correlated

Figure: Here the observer is monitoring the movements of the particles, but here their motion is anti-correlated (\(\rho=-0.95\)).

Independent, correlated, and anti-correlated velocities can share the same one-dimensional Gaussians. The marginal entropies \(h_x\) and \(h_y\) do not see the tilt. The joint does. Multi-information is that leftover: \(I=\sum h_i-H\).

Multi-information \(I=\sum_i h_i-H\ge 0\) generalises mutual information. With fixed marginal entropies \(C=\sum_i h_i\), conservation \(I+H=C\) is a no-go: you cannot have both high stored correlation and high free uncertainty without bound. The mechanics analogy is enough: \(I\) like potential (stored correlation), \(H\) like kinetic (free uncertainty). A brief inaccessible-game introduction may follow as colour; do not expand it into a separate outcome.

Figure: Multi-information and joint entropy for uniform-marginal binary pairs; \(I+H=C\) with \(C=2\).

Data-Processing Inequality

Mutual information is the pairwise case of multi-information: \(I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)=H(X)+H(Y)-H(X,Y)\). Week 2 defined it from the chain rule and stated the data-processing inequality. We prove it now, because this week’s object is \(I\).

The data-processing inequality (Cover and Thomas, 1991, Thm 2.8.1) is the no-go on \(I\). If \(X\to Y\to Z\) is Markov, then \(I(X;Z)\le I(X;Y)\). The first expansion of \(I(X;Y,Z)\) uses the chain rule and drops \(I(X;Z\mid Y)\) by the Markov property. The second expansion is at least \(I(X;Z)\) because conditional mutual information is non-negative. Equality holds when \(I(X;Y\mid Z)=0\) — \(Y\) is a sufficient statistic of \(X\) for \(Z\), or \(Z\) already captures everything \(Y\) knew about \(X\).

This is the same grammar as \(I+H=C\). Conservation says you cannot have both high stored correlation and high free uncertainty without bound. Data processing says you cannot increase stored correlation by further processing. A claim that an intelligent system will “just process more” and thereby recover information that a Markov stage has already destroyed is a DPI violation.

Information Bottleneck

The data-processing inequality forbids creating information by processing. It does not say which information to keep. The information bottleneck (Tishby et al., 1999) is the prescription: given an observation \(X\) and a relevance variable \(Y\), choose a representation \(T\) of \(X\) that keeps \(I(T;Y)\) and discards the rest of \(I(X;T)\).

The information bottleneck of Tishby, Pereira and Bialek is a constrained optimisation on mutual information. The encoder \(p(t\mid x)\) is Markov given \(X\), so \(Y \leftrightarrow X \leftrightarrow T\). Data processing then supplies the bound \(I(T;Y)\le I(X;Y)\): no representation of \(X\) can be more informative about \(Y\) than \(X\) itself. The Lagrange multiplier \(\beta\) interpolates. Small \(\beta\) spends almost no \(I\) on the representation. Large \(\beta\) recovers \(T=X\). The self-consistent encoder is an exponential family, \[ p(t\mid x) \propto p(t)\,\exp\bigl(-\beta\,\mathrm{KL}\bigl(p(y\mid x)\,\|\,p(y\mid t)\bigr)\bigr), \] the same Boltzmann form as week 4, now with a KL sufficient statistic. We do not run Blahut–Arimoto. The point is the pair: DPI is the no-go; the bottleneck is how you spend \(I\).

Week 4 asked for the least-committal \(p\) given moments. Here the constraint is a mutual information. Same Lagrange move, different constraint. Later in this session the bottleneck evaluates intelligence claims: a system that keeps only what is relevant for action is making an IB statement. The second half first proves the Good Regulator existence claim \(H(A\mid S)=0\) from concavity of entropy; only then may one offer the control gloss \(\min I(S;A)\) subject to \(H(Z)\le\epsilon\) as an IB-shaped reading — a gloss after the theorem, not the theorem itself. Requisite variety and the Good Regulator remain named tools for LO13; the information bottleneck itself is the LO10 theorem.

Figure: DPI supplies the feasible region. The information bottleneck traces the upper boundary: keep \(I(T;Y)\), spend as little \(I(X;T)\) as possible.

The Inaccessible Game

The Inaccessible Game

We call our framework the “inaccessible game” because the system is isolated from external observation: an outside observer cannot extract or inject information, making the game’s internal state inaccessible.

Like other zero-player games, such as Conway’s Game of Life (Gardner, 1970), the system evolves according to internal rules without external interference. But unlike cellular automata where rules are chosen by design, in our game the rules emerge from an information-theoretic principle.

Why “Inaccessible?”

The game is inaccessible because our fourth axiom—information isolation—ensures that an external observer learns nothing. Recall from Baez-Fritz-Leinster that information gained from observing a process \(f: p \rightarrow q\) is measured by entropy change: \[ F(f) = H(p) - H(q). \]

But if marginal entropies are conserved (\(\sum h_i = C\)), then an external observer sees:

• Before observation: \(\sum h_i = C\)
• After observation: \(\sum h_i = C\)
  
• Information gained: \(\Delta(\sum h_i) = 0\)

The observer learns nothing! The system is informationally isolated from the outside world.

What Makes It a Game?

The “game” has the following characteristics:

Players: None (zero-player game)

State: A probability distribution over variables, parametrized by natural parameters \(\boldsymbol{\theta}\)

Rules: Evolve to maximize entropy production subject to the constraint \(\sum h_i = C\)

Dynamics: Emerge from information geometry (Fisher information) and the constraint structure

The game starts in a maximally correlated state (high multi-information \(I\), low joint entropy \(H\)) and evolves toward states of higher entropy. Since \(I + H = C\), this means the system breaks down correlations (\(I\) decreases) as entropy increases.

Connection to Physical Reality

Why should we care about this abstract game? Because its dynamics exhibit structure that connects to fundamental physics:

1. GENERIC structure: The dynamics decompose into reversible (conservative) and irreversible (dissipative) components
2. Energy-entropy equivalence: In the thermodynamic limit, our marginal entropy constraint becomes equivalent to energy conservation
3. Landauer’s principle: Information erasure necessarily involves the dissipative part of the dynamics

The inaccessible game provides an information-theoretic foundation for thermodynamic principles. It suggests that physical laws might emerge from information-theoretic constraints, rather than information theory being derived from physics.

Optional depth, not lectured today: the two-bin Jaynes-world tour in the information-engines seminar notes. Students who followed natural gradient in week 6 already have the local geometry that game uses.

Von Neumann Entropy

The Origin Paradox: Shannon vs von Neumann

The inaccessible game wants a very specific kind of ``origin’’ state:

• maximal multi-information (\(I=C\)), but
• zero joint entropy (\(H=0\)).

Informally: the system is globally fully specified (no global uncertainty), but every subsystem looks maximally uncertain on its own.

If your modeling desiderata include this configuration—globally pure but locally maximally uncertain—classical probability cannot represent it. The obstruction is not a technicality; it is a structural inequality: classical conditional entropy is nonnegative.

Quantum states (specifically, entangled states) represent this configuration: von Neumann conditional entropy can be negative, and maximally entangled states are exactly the configurations that achieve ``locally maximally uncertain’’ while remaining globally pure.

This does not prove quantum mechanics is necessary for all purposes; it shows that if you want this particular constraint set realised, you need the quantum framework.

Why the classical origin is impossible

Consider two variables \(X,Y\) (the same idea scales). Recall \[ I(X;Y)=H(X)+H(Y)-H(X,Y). \] If \(H(X,Y)=0\) then the joint distribution puts all mass on a single outcome \((x^\star,y^\star)\), which forces \(H(X)=H(Y)=0\) as well. So you cannot have \(H(X,Y)=0\) while keeping \(H(X)+H(Y)\) positive.

Another way to say the same thing is through conditional entropy: \[ H(X|Y)=H(X,Y)-H(Y)\ge 0. \] If \(H(X,Y)=0\) but \(H(Y)>0\), then \(H(X|Y)=-H(Y)<0\), contradicting \(H(X|Y)\ge 0\).

So the origin condition “globally pure but locally uncertain” is forbidden in the commutative/Shannon world.

von Neumann entropy changes the rules

In noncommutative probability, the joint system is described by a density matrix \(\rho_{AB}\) and subsystems by reduced density matrices \(\rho_A=\text{tr}_B(\rho_{AB})\), \(\rho_B=\text{tr}_A(\rho_{AB})\).

The von Neumann entropy is \[ \textsf{H}(\rho)=-\text{tr}(\rho\log\rho). \] We can define conditional entropy by the same formal identity: \[ \textsf{H}(A|B)=\textsf{H}(\rho_{AB})-\textsf{H}(\rho_B). \] But now \(\textsf{H}(A|B)\) can be . That is not a bug: it is exactly how entanglement shows up as “more-than-classical correlation.”

Worked toy example: Bell state

Take two qubits and the Bell state \[ \ket{\Phi^+}=\frac{1}{\sqrt 2}(\ket{00}+\ket{11}), \qquad \rho_{AB}=\ket{\Phi^+}\bra{\Phi^+}. \] This is a joint state, so \[ \textsf{H}(\rho_{AB})=0. \] But each marginal is maximally mixed: \[ \rho_A=\rho_B=\frac{1}{2}I, \qquad \textsf{H}(\rho_A)=\textsf{H}(\rho_B)=\log 2. \] Therefore \[ \textsf{H}(A|B)=\textsf{H}(AB)-\textsf{H}(B)=0-\log 2=-\log 2<0. \] This is exactly the pattern TIG wants at the origin:

• globally pure (\(\textsf{H}(AB)=0\)),
• locally maximally uncertain (\(\textsf{H}(A),\textsf{H}(B)\) maximal).

So the quantum/noncommutative formalism makes the origin without changing the goal — only changing the structural notion of “information loss.”

Connection back to TIG

This is why the paper pivots to von Neumann entropy:

• It preserves the “information as entropy difference” intuition.
• It removes the classical origin obstruction.
• It forces an expectation-first, noncommutative probability framing.

From here, the next question is: once expectations are primitive, what are the reversible transformations that preserve this expectation structure? That leads into the unitary/commutator story.

The Matrix Exponential Family

The matrix exponential family is the non-commutative analogue of the classical exponential family. A density matrix \(\rho\) belongs to the matrix exponential family if it can be written as \[ \rho(\boldsymbol{\theta}) = \exp\!\left(\sum_k \theta_k F_k - \psi(\boldsymbol{\theta})\,\mathbf{I}\right), \] where \(\boldsymbol{\theta} = (\theta_k)\) are the natural parameters, \(\{F_k\}\) are Hermitian operators playing the role of sufficient statistics, and the scalar \[ \psi(\boldsymbol{\theta}) = \log\,\text{tr}\!\exp\!\left(\sum_k \theta_k F_k\right) \] is the quantum log-partition function, which ensures \(\mathrm{tr}(\rho) = 1\).

The gradient of \(\psi\) recovers quantum expectation values: \[ \nabla_{\theta_k}\psi(\boldsymbol{\theta}) = \text{tr}(\rho\, F_k) = \langle F_k\rangle_\rho, \] just as in the classical case the gradient of the cumulant generating function gives the mean of the sufficient statistic. The Hessian of \(\psi\) is the Bogoliubov–Kubo–Mori (BKM) metric, \[ G_{jk}(\boldsymbol{\theta}) = \nabla^2_{\theta_j\theta_k}\psi(\boldsymbol{\theta}), \] the quantum analogue of the Fisher information matrix. When all \(F_k\) commute this reduces to the classical Fisher information. The von Neumann entropy is recovered from \(\psi\) by a Legendre transform, \[ S(\rho) = -\mathrm{tr}(\rho\log\rho) = \psi(\boldsymbol{\theta}) - \boldsymbol{\theta}^\top\nabla\psi(\boldsymbol{\theta}), \] exactly mirroring the classical relation between the log-partition function and the Shannon entropy.

The structural parallel between the classical and matrix cases is exact, with one crucial difference: the non-commutativity of the operators \(F_k\). In the classical case the sufficient statistics \(T_k(\mathbf{x})\) are ordinary functions and commute freely. In the matrix case the Hermitian operators \(F_k\) need not commute, \([F_j, F_k] \neq 0\), and the matrix exponential \(\exp(\sum_k \theta_k F_k)\) does not factorise over the individual terms. This non-commutativity has two consequences. First, the BKM metric differs from its classical counterpart: it involves a symmetrised operator product averaged against the state \(\rho\). Second, a pure state (\(S(\rho) = 0\)) can coexist with strictly positive marginal entropies, because the marginals of an entangled pure state are mixed. The classical exponential family has no analogue of this: a pure joint distribution has zero entropy everywhere.

A density matrix \(\rho\) is faithful if it is strictly positive definite, \(\rho > 0\), meaning every eigenvalue is strictly positive. Every member of the matrix exponential family is faithful: since the matrix exponential \(\exp(A)\) is positive definite for any Hermitian \(A\), the density matrix \(\rho(\boldsymbol{\theta})\) has strictly positive eigenvalues for all finite \(\boldsymbol{\theta}\).

Faithfulness matters for two reasons. First, the BKM metric \(G(\boldsymbol{\theta})\) is only well-defined for faithful states — it involves expressions of the form \(\int_0^1 \text{tr}(\rho^t A \rho^{1-t} B)\,\text{d}t\) that are singular when \(\rho\) has a zero eigenvalue. Second, faithful states are the interior of the set of density matrices; pure states (\(S(\rho) = 0\), rank 1) lie on the boundary. As a sequence of faithful states approaches a pure state, the natural parameters \(\boldsymbol{\theta}\) diverge and the BKM metric degenerates. This is precisely why the LME origin — a pure state — lies at infinite BKM distance from any interior point and is unreachable in finite game time.

Maximum Entropy and Density Matrices

In Jaynes (1957) Jaynes showed how the maximum entropy formalism is applied, in later papers such as Jaynes (1963) he showed how his maximum entropy formalism could be applied to von Neumann entropy of a density matrix.

As Jaynes noted in his 1962 Brandeis lectures: “Assignment of initial probabilities must, in order to be useful, agree with the initial information we have (i.e., the results of measurements of certain parameters). For example, we might know that at time \(t = 0\), a nuclear spin system having total (measured) magnetic moment \(M(0)\), is placed in a magnetic field \(H\), and the problem is to predict the subsequent variation \(M(t)\)… What initial density matrix for the spin system \(\rho(0)\), should we use?”

Jaynes recognized that we should choose the density matrix that maximizes the von Neumann entropy, \[\begin{align} S = -\text{Tr}(\rho \log \rho), \end{align}\] subject to constraints from our measurements, \[\begin{align} \text{Tr}(\rho M_{op}) = M(0), \end{align}\] where \(M_{op}\) is the operator corresponding to total magnetic moment.

The solution is the quantum version of the maximum entropy distribution, \[\begin{align} \rho = \frac{1}{Z}\exp(-\lambda_1 A_1 - \lambda_2 A_2 - \cdots - \lambda_m A_m), \end{align}\] where \(A_i\) are the operators corresponding to measured observables, \(\lambda_i\) are Lagrange multipliers, and \(Z = \text{Tr}[\exp(-\lambda_1 A_1 - \cdots - \lambda_m A_m)]\) is the partition function.

This unifies classical entropies and density matrix entropies under the same information-theoretic principle. It clarifies that quantum states with minimum entropy (pure states) represent maximum information, while mixed states represent incomplete information.

Jaynes further noted that “strictly speaking, all this should be restated in terms of quantum theory using the density matrix formalism. This will introduce the \(N!\) permutation factor, a natural zero for entropy, alteration of numerical values if discreteness of energy levels becomes comparable to \(k_BT\), etc.”

Classically, \(I=C\) forces \(H=0\) — a delta joint. A pure entangled state has von Neumann entropy \(S(\rho)=0\) while marginals remain uncertain. The matrix exponential family is the quantum MaxEnt family; Jaynes applied the same Lagrange move to \(\rho\) in 1963.

import numpy as np

def bell_state():
    v = np.array([1.0, 0.0, 0.0, 1.0]) / np.sqrt(2.0)
    return np.outer(v, v.conj())

def von_neumann(rho):
    w = np.linalg.eigvalsh(rho)
    w = w[w > 1e-12]
    return -np.sum(w * np.log2(w))

def partial_trace_A(rho):
    rho = rho.reshape(2, 2, 2, 2)
    return np.trace(rho, axis1=1, axis2=3)

rho = bell_state()
print('S(rho)=', von_neumann(rho))
print('S(rho_A)=', von_neumann(partial_trace_A(rho)))

Purely Entropic Readings: First Attempt

Schottky and the Good Regulator get a first answer today in notes, not on slides. The second half of this session completes both — with a short proof for the Good Regulator, not only a slogan.

Schottky connects back to lecture 1’s heat-capacity peak and to week 6’s Fisher reading: the peak of thermal response is a peak of distinguishability. The purely entropic interpretation waits for the limits block.

Do not conflate Ashby’s law of requisite variety with the Good Regulator Theorem. Requisite variety asks how much regulatory information is required. In Shannon form, with disturbance \(D\), response \(R\), and essential variable \(E\), \[ H(E)\ge H(D)-I(D;R). \] Useless controller states appear as small \(I(D;R)\). The Good Regulator Theorem (Conant and Ashby, 1970) is an existence claim about structure: among policies that minimise outcome entropy \(H(Z)\) there is a simplest deterministic map \(A=h(S)\), i.e. \(H(A\mid S)=0\). The second half earns that claim from the concavity of Shannon entropy (and, for stochastic channels, extreme points of the policy polytope). Two caveats to keep in mind before any slogan: a constant action also has \(H(A\mid S)=0\); low \(H(Z)\) means predictable, not desirable. An information-bottleneck-shaped reading \(\min I(S;A)\) subject to \(H(Z)\le\epsilon\) may follow as a gloss after the theorem — it is not the theorem itself, and it is not Conant and Ashby’s equation. ## Define This Week

After This Lecture

Worksheet 4 was due at the start of this lecture, before Quiz 4. Limits on intelligence follow in the second hour.

Further Reading

• Theorem 2.8.1 of Cover and Thomas (1991)

• the method; optional of Tishby et al. (1999)

• https://inverseprobability.com/talks/notes/information-engines.html — Jaynes’ world and unified intelligence perspective of Information Engines seminar notes

• Chapter 11 of Nielsen and Chuang (2010)

• the Brandeis lectures of Jaynes (1963)

Thanks!

For more information on these subjects and more you might want to check the following resources.

• company: Trent AI
• book: The Atomic Human
• twitter: @lawrennd
• podcast: The Talking Machines
• newspaper: Guardian Profile Page
• blog: http://inverseprobability.com

References