Week 8: Probability Transport and Limits on Intelligence

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

Second half of the final session: intelligent agency as transport of probability mass. Three geometries of an optimal change of state — Crooks / Fisher–Rao, Wasserstein, and the Schrödinger bridge — and the evaluation of superintelligence claims with Landauer, \(\mathcal{L}^2/\tau\), and \(I+H=C\).

Worksheet 4 is due at the start of this session. Quiz 4 occupies the first ten minutes. Then 110 minutes to close the course. This is the interpret week for thermodynamic length, information and intelligence, and the two purely entropic readings.

This Session

Time plan (60 minutes, second half of final session)

Information, entropy and intelligence course notebook setup

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util

cmd = install_command('pods')

%system {cmd}

cmd = install_command('mlai')

%system {cmd}

Agency as Transport

Erwin Schrödinger

Figure: Erwin Schrödinger, the Austrian physicist, photographed in 1933.

Schrödinger’s Bridge and Optimal Information Transport

Given an initial and a final probability distribution, what is the most likely stochastic process that connects them? This question was posed by Erwin Schrödinger in 1931.

On a discrete grid, Schrödinger interpolation is entropy-regularised optimal transport. Sinkhorn is the discrete algorithm, not a fourth geometry.

Figure: Illustrative \(2\times 2\) coupling — week 4 moments become week 8 marginals.

The entropic constraint in Sinkhorn distances has a direct connection to the first half of this session: Cuturi (2013) shows that constraining \(\mathrm{KL}(P \| rc^T) \le \alpha\) is equivalent to constraining the mutual information \(I(X;Y) \le \alpha\) of the coupling. The identity \(I + H = C\) relates mutual information to capacity; here we see mutual information serving as a budget on how deterministic the transport plan can be.

Three Geometries

Superintelligence at zero dissipation or infinite update rate is perpetual motion: it claims to repeal a no-go. Students must say where the analogy is tight and where it breaks.

geometries = {
    'Crooks/Fisher-Rao': {'minimise': 'dissipation', 'no_go': 'L^2/tau'},
    'Wasserstein': {'minimise': 'ground cost', 'no_go': 'mass conservation'},
    'Schrodinger': {'minimise': 'relative entropy', 'no_go': 'fixed marginals'},
}

GAIST on the Wasserstein and Schrödinger Geometries

Chapter 14 of (Welling et al., 2026) is the discrete-time Schrödinger bridge: minimise \(D_{\mathrm{KL}}[Q\|P]\) subject to fixed endpoint marginals \(q_0 = \mu\) and \(q_{t_f} = \nu\). When the reference \(P\) is uniform this is Jaynes’ maximum caliber — MaxEnt over paths. The algorithm is iterative proportional fitting: Sinkhorn–Knopp on the Gibbs kernel \(K_{ij}=\exp(-C_{ij}/\varepsilon)\). Chapter 22 retells the same story in continuous time: Wasserstein distance via Monge, Kantorovich and Benamou–Brenier (Section 22.2); dissipated work as the OT action (Section 22.3); the Schrödinger bridge as OT plus an osmotic Fisher-information term (Section 22.6).

Section 22.3 states a thermodynamic speed limit \[ \Sigma_{\tau_f} \ge \frac{W_2^2[\rho_0,\rho_{\tau_f}]}{T\tau_f} \] and a finite-time correction to Landauer (Landauer, 1961), following Proesmans, Ehrich and Bechhoefer (2020). That inequality is tight in the Wasserstein geometry. It is not Crooks’ \(\mathcal{L}^2/\tau\) bound. GAIST presents \(W_2^2/(T\tau)\) as the thermodynamic speed limit because the book is working in overdamped dissipative dynamics with a ground-cost interpretation. We are not collapsing the three geometries: Fisher–Rao / Crooks remains the near-equilibrium length on the equilibrium manifold; Wasserstein is minimum ground-cost of moving mass; the Schrödinger bridge is the maximum-entropy interpolation.

Read Chapter 14 and Sections 22.2–22.3 for the Wasserstein and Schrödinger prescriptions. Keep Crooks (2007) for Fisher–Rao. Peyré and Cuturi remain the place to get the OT intuition without the physics: Chapters 1–2 for Wasserstein; §4.2 for Sinkhorn, if you want the iteration. Sinkhorn computes the Schrödinger / entropic coupling. It is not Crooks’ geodesic.

Limits on Intelligence

Information-Theoretic Limits on Intelligence

Just as the second law of thermodynamics places fundamental limits on mechanical engines, no matter how cleverly designed, the idea is that information theory places fundamental limits on information engines, no matter how cleverly implemented.

What Intelligent Systems Must Do

Any intelligent system, whether biological or artificial, must perform certain fundamental operations:

1. Acquire information from its environment (sensing, observation)
2. Store information about the world (memory)
3. Process information to make decisions (computation)
4. Erase information to make room for new data (memory management)
5. Act on the world using the processed information

Each of these operations has information-theoretic costs that cannot be eliminated by clever engineering.

Landauer’s Principle

Landauer’s principle (Landauer, 1961) establishes that erasing one bit of information requires dissipating at least \(k_BT\log 2\) of energy as heat, where \(k_B\) is Boltzmann’s constant and \(T\) is temperature.

This isn’t an engineering limitation, it’s a fundamental consequence of the second law. To reset a bit to a standard state (say, always 0) requires reducing its entropy from 1 bit to 0 bits. That entropy must go somewhere, and it ends up as heat in the environment.

This doesn’t mean AI can’t be powerful or transformative — internal combustion engines transformed the world despite thermodynamic limits. But it does mean there are hard bounds on what’s possible, and claims that ignore these bounds are as unrealistic as promises of perpetual motion.

A Unified View of Intelligence Through Information

The multiple perspectives we’ve explored - entropy games, information engines, least action principles, and Schrödinger’s bridge - provide complementary views of intelligence as optimal information processing. Each framework highlights different aspects of this fundamental process:

1. The Entropy Game shows us that intelligence can be measured by how efficiently a system reduces uncertainty through strategic questioning or observation.

2. Information Engines reveal how intelligence converts information into useful work, subject to thermodynamic constraints.

3. Least Action Principles demonstrate that intelligence follows optimal paths through information space, minimizing cumulative uncertainty.

4. Schrödinger’s Bridge illuminates how intelligence can be viewed as optimal transport of probability distributions, finding the most likely paths between states of knowledge.

These perspectives converge on a unified view: intelligence is fundamentally about optimal information processing. Whether we’re discussing human cognition, artificial intelligence, or biological systems, the capacity to efficiently acquire, process, and utilize information lies at the core of intelligent behavior.

This unified perspective offers promising directions for both theoretical research and practical applications. By understanding intelligence through the lens of information theory and thermodynamics, we may develop more principled approaches to artificial intelligence, gain deeper insights into cognitive processes, and discover fundamental limits on what intelligence can achieve.

Purely Entropic Good Regulator

Conant and Ashby define successful regulation by minimising the entropy of the outcome \(Z\): \(H(Z)\to\min\) (Conant and Ashby, 1970). A regulator is a policy \(\pi(a\mid s)=p(a\mid s)\). This is an existence result that comes from the concavity of Shannon entropy: among optimal regulators there is a simplest one with zero conditional action entropy, \(H(A\mid S)=0\), equivalently \(A=h(S)\). That is the sense in which the regulator is a ``model’’ of the system — a deliberately weak sense, as the caveats below make precise.

Write Ashby’s law of requisite variety in Shannon form. With disturbance \(D\), regulatory response \(R\), and essential/outcome variable \(E\), the usual bound is \[ H(E)\ge H(D)+H(R\mid D)-H(R). \] Mutual information rewrites the last two terms at once: \[ H(E)\ge H(D)-I(D;R). \] Verbally: residual uncertainty after regulation is at least the uncertainty in the disturbance minus the information the regulator has about that disturbance. Having many controller states helps only when those states are correlated with \(D\). Large \(H(R\mid D)\) is large wasted variety. Requisite variety asks how much regulatory information is required. The Good Regulator Theorem asks how that information may be structured.

Fix \(p(s)\) and a channel \(p(z\mid s,a)\). Regulation is the choice of a conditional \(\pi(a\mid s)\). There is no time index and no value function: a single decision. Success is \(\min H(Z)\), not reward maximisation. Define \(A\) to be a model of \(S\) when \(H(A\mid S)=0\), equivalently when there is a function \(h\) with \(A=h(S)\) almost surely. That definition is operational and weak; optimality, not the definition, will force which maps matter.

Assume first a deterministic outcome map \(Z=\psi(S,A)\). Suppose an allegedly optimal policy randomises, at some \(s\) with \(p(s)>0\), between actions \(a_1\) and \(a_2\) that produce different outcomes \(z_1\ne z_2\). Transfer probability \(\alpha\) from \((s,a_1)\) to \((s,a_2)\). Only two probabilities in the marginal of \(Z\) change: \((u,v)\mapsto(u-\alpha,v+\alpha)\). Their contribution to entropy is \[ g(\alpha)=-(u-\alpha)\log(u-\alpha)-(v+\alpha)\log(v+\alpha), \] with \[ g''(\alpha)=-\frac{1}{u-\alpha}-\frac{1}{v+\alpha}<0. \] So \(g\) is strictly concave. An interior mixture cannot be a minimum of \(H(Z)\): move mass in one of the two directions and entropy falls. Contradiction. Therefore, at an optimum, whenever the policy randomises in state \(s\), all actions in the support must produce the same outcome. Equivalently, for deterministic \(\psi\), every optimal policy satisfies \(H(Z\mid S=s)=0\) for every \(s\) with \(p(s)>0\).

If several actions at \(s\) all produce the same \(z^\ast(s)\), choose one of them and discard the unnecessary randomisation. Do this independently for every \(s\) to obtain \(A=h(S)\). The marginal of \(Z\) is unchanged, so \(H(Z)\) is unchanged: the new deterministic regulator remains optimal, and \(H(A\mid S)=0\). That is the entropy form of the Good Regulator Theorem (Conant and Ashby, 1970).

The deterministic-\(\psi\) assumption is not needed for existence. With a genuinely stochastic channel \(p(z\mid s,a)\), the marginal \(p(z)\) is linear in the policy \(\pi\). The space of policies is a product of probability simplices; its extreme points are exactly the deterministic policies. Since \(H(Z)\) is concave in \(p(z)\), \(H(Z)\) is concave as a function of \(\pi\). A concave function on a compact polytope attains a minimum at an extreme point. Hence there always exists a deterministic entropy-minimising policy, and \(H(A\mid S)=0\) survives even when the world is stochastic. What does not survive is the stronger claim \(H(Z\mid S=s)=0\): irreducible environmental noise can leave \(Z\) uncertain.

Calling \(H(A\mid S)=0\) a model is a deliberately weak definition. A regulator that always does the same thing, \(A=a_0\), also has \(H(A\mid S)=0\) and \(I(A;S)=0\). The theorem does not say that every optimal regulator contains lots of information about the system. It says that an optimal regulator can be chosen as a deterministic function of state. Whether that map is informative depends on the regulation problem: optimality, not the definition, rules out useless constants when state-dependent action is required.

Minimising \(H(Z)\) only makes the result predictable. It does not make the result desirable. A controller that reliably produces catastrophe has \(H(Z)=0\). If \(G\) denotes good outcomes, maximising \(\Pr(Z\in G)\) and minimising \(H(Z)\) are different objectives. Either maximise goal achievement first and entropy second, or define \(Z\) as an error variable so that concentration at \(Z=0\) coincides with success. Keep that fence in front of the phrase ``Good Regulator.’’

The information bottleneck asks for a compact representation that preserves relevance. In control language a closely related problem is \[ \min_{p(a\mid s)} I(S;A)\quad\text{subject to}\quad H(Z)\le\epsilon. \] That is a course-compatible reinterpretation of the Good Regulator idea — not Conant and Ashby’s equation, and not a consequence of \(H(A\mid S)=0\) alone. Sufficient statistics for the outcome need the further condition \(I(A;Z)=I(S;Z)\). When \(A=h(S)\) is deterministic, \(I(S;A)=H(A)\), so minimising model complexity is minimising \(H(A)\) while retaining the distinctions in \(S\) that the outcome demands. States that share an action may merge; states that demand different actions must remain distinguishable. That gloss is relevant compression. It is optional colour after the theorem, not the theorem itself.

Chain \(D\to X\to T\to A\). Data processing gives \(I(D;T)\le I(D;X)\) and \(I(D;A)\le I(D;T)\): processing cannot manufacture missing environmental information. Put the tools together for LO13: DPI forbids creating \(I\); requisite variety demands enough relevant \(I\); the Good Regulator supplies an optimal deterministic organisation \(H(A\mid S)=0\); the information bottleneck is how one may spend \(I\) on action-relevant distinctions. Landauer prices acquiring, storing, updating, and erasing whatever model is kept. The course close is unchanged: entropy forbids; probability prescribes. The Good Regulator is an earned theorem inside that fence, not the fence.

Data-processing inequality and the information bottleneck were taught under LO10 earlier in this session. Today they are evaluation tools, not new outcomes: a claim that an intelligent system will just process more is a DPI violation; a system that keeps only what is relevant for action is an IB statement. Requisite variety supplies the quantity bound \(H(E)\ge H(D)-I(D;R)\). The Good Regulator Theorem supplies an existence claim earned from concavity: among \(H(Z)\)-minimisers there is a deterministic policy with \(H(A\mid S)=0\). Caveats: constant \(A=a_0\) also has \(H(A\mid S)=0\); low \(H(Z)\) is predictability, not desirability. The IB-shaped reading \(\min I(S;A)\) subject to \(H(Z)\le\epsilon\) is a gloss after that theorem, not Conant and Ashby’s statement and not the course punchline. Viable-system material is optional. Inaccessible-game material, if present, is colour only. Prescriptions operate inside fences set by physics and embodiment: whatever model is kept is not informationally free once Landauer prices sensing, storage, processing, and erasure. ```{.python}

```{no_gos = [‘Landauer,’ ‘Crooks L^2/tau,’ ‘I+H=C,’ ‘human bandwidth’] prescriptions = [‘Boltzmann/MaxEnt p,’ ‘Crooks geodesic,’ ‘Wasserstein plan,’ ‘Schrodinger bridge’]

Interpret This Week

After This Lecture

Quiz 4 opened the session. No further assessed work after today. Feedback within the 21-day ACS window. The last LLM exercise: ask the model what thermodynamic length has to do with intelligence, then write down whether it offered a no-go, a prescription, or collapsed the two.

Further Reading

• Chapters 1–2; §4.2 (Sinkhorn, optional) of Peyré and Cuturi (2019)

• Chapter 14 and Sections 22.2–22.3 of Welling et al. (2026)

• the whole paper of Crooks (2007)

• §§3–4 (optional) of Cuturi (2013)

• the Good Regulator Theorem of Conant and Ashby (1970)

• requisite variety; Shannon form of Ashby (1956)

• Chapter 1 of Lawrence (2024)

Thanks!

For more information on these subjects and more you might want to check the following resources.

• company: Trent AI
• book: The Atomic Human
• twitter: @lawrennd
• podcast: The Talking Machines
• newspaper: Guardian Profile Page
• blog: http://inverseprobability.com

References