Worksheet 4: Multi-Information, von Neumann Entropy, and Limits on Intelligence

[jupyter][google colab][edit]

Abstract

Compute multi-information and verify (I+H=C), then treat the Bell state as the quantum resolution of (I=C) with non-trivial marginals. The reflection asks you to contrast three geometries of an optimal change of state and to evaluate the perpetual-motion analogy with the course’s formal constraints.

Worksheet 4

{Released after lecture 6 (17 November). Due by 10:00 at the start of lecture 8 (1 December), before Quiz 4. Estimated three hours. This is the capstone worksheet, 15% of the module mark.

Builds on lectures 6–7. Part A1–A2 use the \(I+H=C\) family (multi-information-rho figure). Part A3 treats the Bell state. Part B contrasts three geometries and the perpetual-motion synthesis, and may use the data-processing inequality and the information bottleneck taught under LO10.

Submit candidatenumber_worksheet4.ipynb and candidatenumber_worksheet4.md on Moodle. LLMs are permitted. Use the Lecture 1 habit: curiosity first, then skeptical probes. Authenticity is checked by Quiz 4 at the start of lecture 8, on a new example (three independent coins; a GHZ state rather than a Bell state).

Information, energy and intelligence course notebook setup

[edit]

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util
cmd = install_command('pods')
%system {cmd}
cmd = install_command('notutils')
%system {cmd}
cmd = install_command('mlai')
%system {cmd}
import numpy as np

Part A – Code

[edit]

Question 1

Write multi_information(joint_probs) that takes a joint array (shape (2,2,2) for three binary variables) and returns the marginal entropies (h_1,h_2,h_3), the joint entropy (H), and the multi-information (I=h_i-H). Evaluate it on (a) three independent fair coins, (b) (X_2=X_1) with (X_3) independent, (c) (X_1=X_2=X_3). For (c), verify (I+H=_i h_i).

15 marks

Question 2

A one-parameter family of joints on two binary variables, with uniform marginals for every (): \[ p(1,1)=p(0,0)=\tfrac14(1+\rho),\qquad p(1,0)=p(0,1)=\tfrac14(1-\rho). \] Then (C=2) bits throughout. Plot (I()) and (H()) on the same axes. Confirm (I+H=2) for all (). At what () is (H=0)? What does the joint look like there, and what does (I=C) mean in that case?

15 marks

Question 3

The Bell state (|^+=(|00+|11)) has density matrix (=|++|). Write () as a (4) matrix in the computational basis. Compute the von Neumann entropy (S()=-(_2)) using numpy.linalg.eigvalsh. Trace out the second qubit to get (_A) and compute (S(_A)). In a markdown cell: why is (S()=0) while (S(_A)>0), and what does that say about achieving (I=C) with non-trivial marginals?

20 marks

Part B – Reflection

Question 4

Three geometries (200 words). Shannon abstracted a code as probability over symbols. Write a short argument for why probability transport – moving a distribution from (p) to (q) – is a natural abstraction of agency. Then distinguish, in your own words, three notions of an optimal trajectory that this module has named: Crooks / Fisher–Rao thermodynamic length, Wasserstein optimal transport, and the Schrödinger bridge. What would it mean to collapse them, and why must you not?

25 marks

Question 5

The perpetual motion analogy (300 words). Using at least two formal no-gos from the course (Landauer, (^2/), (I+H=C), the data-processing inequality, human bandwidth), argue why unbounded intelligence – a system that processes information without entropic cost – has the same shape as perpetual motion. Say which of your constraints are no-gos and which of the three geometries (or the information bottleneck) are prescriptions. Where is the analogy tight? Where does it break down? If you used an LLM, include the key responses as an appendix, not counted in the word limit.

25 marks

Marking

  • 60–74: multi-information computed; Bell density matrix written out; perpetual-motion analogy addressed at a surface level.
  • 75–79: (S()=0) with (S(_A)>0) is correctly linked to (I=C); the three geometries are distinguished, not collapsed.
  • 80–89: the perpetual-motion argument is formal (not just an analogy) and uses Landauer or (^2/) or \(I+H=C\) / DPI tightly; Part A2 extended (three variables, or a continuous family).
  • 90–100: original contribution – a case where the analogy fails, and a proposal for what a stronger information-theoretic limit would require.

Submission

Upload both files to Moodle by 10:00 on Tuesday 1 December (start of lecture 8). Quiz 4 follows immediately and will not reuse your examples.