Introduction, Probability Review, and Motivation

Neil D. Lawrence

FW26, William Gates Building

This Session

  • Room FW26; eight Tuesdays from today
  • Motivation, including Maxwell / Landauer as a Week 4 preview
  • Probability → doubt / Socrates → energy seed → Daniel Bernoulli
  • Worksheet 1 released today; entropy continues next week

Time Plan

Clock Block
10:05–10:23 Course mechanics; questions list; course site
10:23–10:32 Perpetual motion and superintelligence
10:32–10:52 Preview: Maxwell’s demon, Landauer, no-go versus mechanism
10:52–11:00 Laplace’s demon and gremlin; Jaynes as a pointer
11:00–11:07 Break
11:07–11:30 Probability review: product/sum/Bayes; named distributions
11:30–11:47 Textbooks aside; Socratic worksheets and LLM habit
11:47–11:58 Energy seed; Boltzmann \(p_i \propto e^{-\beta E_i}\)
11:58–12:00 Daniel Bernoulli pointer; Worksheet 1 / Quiz 1 (past five-to)

Course Mechanics

  • Four quizzes (start of lectures 2, 4, 6, 8)
  • Four worksheets (LLM dialogue / probe + reflection; due weeks 2, 4, 6, 8)
  • Identify with CRSid (filename and form fields; not full name)
  • Questions: GitHub Discussions (new Q&A thread each time)

Questions We Will Return To

List: mlatcl.github.io/iei/questions/

  • Meet the questions today
  • Define later; interpret later still
  • Two of them only make sense after week 8

The Atomic Human

  • A trade book on intelligence, embodiment, and doubt — not a textbook
  • The polemic lives there; this course tries to put mathematics under it
  • Week 2 returns to embodiment as bandwidth

Perpetual Motion and Superintelligence

  • 1925: Motor vehicles (no perpetual motion)
  • 2025: Promises of superintelligence singularity
  • Same fundamental impossibility?

Why Perpetual Motion Failed

  • First kind: energy from nothing — first law
  • Second kind: work from a single bath — Kelvin–Clausius
  • Isolated entropy does not decrease (almost always)

An Equivalent Statement for Intelligence?

Maxwell’s Demon:

  • “Intelligent” entity that violates 2nd law
  • Resolution: Landauer’s principle?
  • Information erasure requires energy

Implication

  • Intelligence has thermodynamic cost
  • Information processing has physical limits

Preview: Maxwell’s Demon

A sorting mind in a gas. Fast molecules one way, slow the other. Temperature splits. Entropy of the gas appears to fall.

  • Partition; frictionless trap door; an “intelligent” entity that measures
  • Temperature sets the variance of each Cartesian velocity component
  • Apparent violation of the second law — that is why the puzzle lasts
  • Preview of Week 4; we do not resolve the ledger today

Preview: Landauer’s Principle

Erasing a bit has a thermodynamic price. Computation, in the reversible ideal, is claimed not to.

  • Erase one bit: at least \(k_B T\ln 2\) of heat (Landauer, 1961)
  • \(k_B \sim 10^{-23}\,\mathrm{J\,K}^{-1}\); at room temperature this is tiny
  • Classical computers dissipate far above the bound; we are not at Landauer yet
  • Why erasure answers the demon — memory, or an infinite tape — is Week 4

Preview: No-Go versus Mechanism

Entropy tells you what will not happen. It does not hand you a machine.

  • Shannon: a distribution, then a prohibition — no algorithm named
  • Physics talks: heat baths, pistons, frictionless doors
  • Second law: entropy is produced somewhere; not when the Earth hits the Sun
  • Week 2 develops this as entropy versus probability

The curve described by a simple molecule of air or vapor is regulated in a manner just as certain as the planetary orbits; the only difference between them is that which comes from our ignorance.

Probability is relative, in part to this ignorance, in part to our knowledge.}

Laplace’s Gremlin

A Philosophical Essay on Probabilities, trans. Truscott and Emory (1902)

Preview: From Insufficient Reason to MaxEnt

  • Laplace assigns probability to ignorance; he does not yet write entropy
  • Insufficient reason: if nothing distinguishes the outcomes, set them equal
  • Jaynes: that assignment is maximum entropy (Jaynes, 2003)
  • Week 5 is the derivation; today it is a pointer

Probability Review

Joint, Marginal, Conditional

  • Joint \(P(x,y)\): both
  • Marginal \(P(x)\): regardless of \(y\)
  • Conditional \(P(x\mid y)\): \(x\) given \(y\)

Product Rule and Sum Rule

  • Product: \(P(x,y)=P(x\mid y)P(y)\)
  • Sum: \(P(y)=\sum_x P(x,y)\)
  • Both are normalising bookkeeping, not modelling assumptions

Bayes’ Rule

\[ P(y\mid x)=\frac{P(x\mid y)P(y)}{P(x)} \]

  • Follows from the product rule and symmetry of the joint
  • Inverts the conditioning — updates a prior given a likelihood
  • \(P(x)=\sum_y P(x\mid y)P(y)\) when \(y\) is discrete

Bayes’ Theorem Example

  • There are two barrels in front of you. Barrel One contains 20 apples and 4 oranges. Barrel Two other contains 4 apples and 8 oranges. You choose a barrel randomly and select a fruit. It is an apple. What is the probability that the barrel was Barrel One?

Bayes’ Rule Example: Answer I

  • We are given: \[\begin{aligned} P(\text{F}=\text{A}|\text{B}=1) = & 20/24 \\ P(\text{F}=\text{A}|\text{B}=2) = & 4/12 \\ P(\text{B}=1) = & 0.5 \\ P(\text{B}=2) = & 0.5 \end{aligned}\]

Bayes’ Rule Example: Answer II

  • We use the sum rule to compute: \[\begin{aligned} P(\text{F}=\text{A}) = & P(\text{F}=\text{A}|\text{B}=1)P(\text{B}=1) \\& + P(\text{F}=\text{A}|\text{B}=2)P(\text{B}=2) \\ = & 20/24\times 0.5 + 4/12 \times 0.5 = 7/12 \end{aligned}\]
  • And Bayes’ rule tells us that: \[\begin{aligned} P(\text{B}=1|\text{F}=\text{A}) = & \frac{P(\text{F} = \text{A}|\text{B}=1)P(\text{B}=1)}{P(\text{F}=\text{A})}\\ = & \frac{20/24 \times 0.5}{7/12} = 5/7 \end{aligned}\]

Common Distributions

Named Distributions You Need

Five families recur across modelling. Know the support, the parameter, and one generative story for each.

Bernoulli and Binomial

  • Bernoulli(\(p\)): single binary trial; \(P(X=1)=p\)
  • Binomial(\(n,p\)): \(n\) i.i.d. Bernoulli trials; count of successes
  • Mean \(np\), variance \(np(1-p)\)

Poisson and Multinomial

  • Poisson(\(\lambda\)): counts in a fixed interval; mean \(=\) variance \(=\lambda\)
  • Multinomial(\(n,\mathbf{p}\)): \(n\) trials into \(K\) categories; generalises the binomial
  • Categories are exclusive; \(\sum_k p_k = 1\)

Poisson Distribution

  • Poisson distribution is used for ‘count data.’ For non-negative integers, \(y\), \[P(y) = \frac{\lambda^y}{y!}\exp(-\lambda)\]
  • Here \(\lambda\) is a rate parameter that can be thought of as the number of arrivals per unit time.
  • Poisson distributions can be used for disease count data. E.g. number of incidence of malaria in a district.

Poisson Distribution

Gaussian

  • \(\mathcal{N}(\mu,\sigma^2)\): continuous density on \(\mathbb{R}\)
  • Fixed by mean and variance; MaxEnt under those constraints
  • Multivariate form: mean vector and covariance matrix

The Gaussian Density

  • Perhaps the most common probability density.

\[\begin{align} p(y| \mu, \sigma^2) & = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y- \mu)^2}{2\sigma^2}\right)\\& \buildrel\triangle\over = \mathscr{N}\left(y|\mu,\sigma^2\right) \end{align}\]

Gaussian Density

Gaussian Density

\[ \mathscr{N}\left(y|\mu,\sigma^2\right) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y-\mu)^2}{2\sigma^2}\right) \]

\(\sigma^2\) is the variance of the density and \(\mu\) is the mean.

Old Books, and the Wikipedia Effect

  • Rogers and Mayhew: engineering thermodynamics, father then son
  • An out-of-print 1961 physics thermodynamics text — often clearer than a modern one
  • Wikipedia about 2006 versus Wikipedia now: intuition edited into correctness
  • Not required reading; a warning about where intuition lives

Doubt

  • Certainty is impossible (Laplace’s gremlin) — doubt is inevitable
  • Probability quantifies uncertainty; this course centres on entropy
  • Bring doubt into your own reasoning: challenge statements (Socratic habit)

Worksheets and LLMs: Can you be Socrates?

Curiosity, Then Skepticism

Dialectical Vertigo

What you are describing, Skepticus, is a chronic but minor ailment of philosophers. It is called dialectical vertigo, and its cure is the immediate application of straightforward argumentation.

The Grasshopper to Skepticus in The Grasshopper: Games, Life and Utopia (Suits, 1978 pg 37)

Interacting with LLMs

  • Curiosity: open a question; let the model give a long answer (like Grasshopper)
  • Skepticism: take a claim and press it — “if that were true, then …” (like Skepticus)
  • Active thought is the point; the submitted transcript will be evidence of the probe

Worksheet Habit

  • Explore with as many models as you like
  • Submit one conversation of exactly ten prompt/answer turns
  • Early turns: open and curious; later turns: skeptical probes
  • Then a short handwritten reflection (print the proforma; scan to PDF)
  • Templates: assessments/handouts/ (dialogue.md + reflection proforma)
  • L172_<CRSid>_worksheet1_dialogue.md and …_reflection.pdf (scan)
  • Quiz 1 (next week) checks foundations you should have met here and in W1

Energy in Machine Learning

Energy as a Score

In ML, an energy is (often) a scalar assigned to a configuration: lower means preferred.

  • Losses are energies over parameters or predictions
  • Energy-based model: \(p(x)\propto e^{-E(x)}\) — same grammar as Boltzmann
  • Minimising \(E\) is the zero-temperature / MAP limit

Energies from Distributions You Already Know

From \(p\) to \(E\): Four Outcomes

Tabulate \(-\log P(y)\) for a four-outcome distribution. Those values are energies.

\(y\) 1 2 3 4
\(P(y)\) 0.3 0.2 0.1 0.4
\(E(y)=-\log P(y)\) 1.204 1.609 2.302 0.916
  • Most probable outcome \(y=4\) has the lowest energy
  • Rarest \(y=3\) has the highest energy
  • Shannon entropy is \(\mathbb{E}[E(y)] = 1.280\) nats — average energy under \(p\)

Bernoulli / Fair Coin

A coin with \(P(H)=p\), \(P(T)=1-p\) is a two-state energy model.

Bernoulli: \(P\) and \(E\)

Bernoulli: Overlay

Bernoulli Energy

  • \(E_0=-\log(1-p)\),\(E_1=-\log p\)
  • Fair coin: degenerate energies \(E_0=E_1=\log 2\)
  • Biased coin: the favoured face is the deeper well

Poisson: Count Energy

A Poisson(\(\lambda\)) is the same map \(E=-\log P\) on the non-negative integers.

Poisson: \(P\) and \(E\)

Poisson: Overlay

Poisson Energy

  • \(E(k)=-\log P(k)=\lambda-k\log\lambda+\log(k!)\) (up to the usual reading of Stirling)
  • Mode of \(P\) \(=\) minimizer of \(E\)
  • Same \(p\propto e^{-E}\) grammar as the Bernoulli two-state model

Gaussian: Quadratic Energy

The univariate Gaussian we wrote earlier is already an energy-based model.

\[ p(y)\propto \exp\!\left(-\frac{(y-\mu)^2}{2\sigma^2}\right) \quad\Longleftrightarrow\quad E(y)=\frac{(y-\mu)^2}{2\sigma^2} \]

Gaussian: \(p\) and \(E\)

Gaussian: Overlay

Gaussian Energy

  • \(E(y)=\frac{(y-\mu)^2}{2\sigma^2}\) up to an additive constant absorbed in \(Z\)
  • Least-squares regression uses the same quadratic energy on residuals
  • Normaliser \((2\pi\sigma^2)^{-1/2}\) is the partition function for this \(E\)

Quadratic Energy (Regression)

\[ E(\mathbf{w})=\sum_n\left(y_n-\mathbf{w}^\top\mathbf{x}_n\right)^2 \]

  • Least squares = energy of the residual
  • Gaussian negative log-likelihood (up to additives and scale)
  • Gauss already linked this quadratic energy to the normal density

Cross-Entropy Energy (Classification)

\[ E=-\sum_c y_c\log\hat{y}_c \]

  • Cross-entropy / log-loss scores the predictive distribution
  • Softmax: \(\hat{y}_c\propto e^{-E_c}\) — logits are negative energies
  • Same arithmetic as \(\mathbb{E}[-\log p]\) (Shannon entropy of a model)

Boltzmann Machine: Pairwise Energy

\[ E(\mathbf{s})=-\sum_i b_i s_i-\sum_{i<j}W_{ij}s_i s_j \]

  • Binary units \(s_i\in\{0,1\}\) or \(\{\pm 1\}\)
  • Biases plus pairwise couplings — classic BM has no triple terms
  • \(p(\mathbf{s})\propto e^{-E(\mathbf{s})/T}\): a learnable Gibbs distribution

Boltzmann Seed

A Prescription to Interrogate

For fixed mean energy \(U\), the maximum-entropy occupation is the Boltzmann distribution — same \(E\) grammar as the ML energies above.

  • \(p_i \propto e^{-\beta E_i}\) with coldness \(\beta = 1/kT\)
  • Normaliser \(Z=\sum_i e^{-\beta E_i}\), so \(p_i = e^{-\beta E_i}/Z\)
  • Week 2: Hydrodynamica / entropy bridge / embodiment; Week 3 clears Gibbs / \(F\) / HMC then Shannon

Next Week: Hydrodynamica

We stop at Daniel Bernoulli. Week 2 reopens Laplace’s molecule, then Hydrodynamica.

  • Why Laplace talks of molecules before atoms are settled
  • Bernoulli’s assumptions; ensemble as coarse-graining
  • Entropy bridge; embodiment (Gibbs / \(F\) / HMC spill to Week 3)

Define This Week

  • Product rule, sum rule, Bayes
  • Bernoulli, binomial, Poisson, multinomial, Gaussian
  • Energy scores: quadratic, cross-entropy, BM pairwise
  • Seed: \(p_i = e^{-\beta E_i}/Z\)

After This Lecture

  • Worksheet 1 released; due 20 October (Shannon naming dialogue + handwritten reflection)
  • Quiz 1 next week: probability and Week 1 energy seeds
  • Then: Hydrodynamica → entropy bridge → embodiment

Further Reading

  • Probability distributions: Section 1.2 of Bishop (2006)

  • Boltzmann machines (optional colour) of Ackley et al. (1985)

  • Hopfield networks (optional colour) of Hopfield (1982)

Thanks!

References

Ackley, D., Hinton, G.E., Sejnowski, T.J., 1985. A learning algorithm for Boltzmann machines. Cognitive Science 9, 147–169.
Bishop, C.M., 2006. Pattern recognition and machine learning. springer.
Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 79, 2554–2558.
Jaynes, E.T., 2003. Probability theory: The logic of science. Cambridge University Press, Cambridge, UK.
Landauer, R., 1961. Irreversibility and heat generation in the computing process. IBM Journal of Research and Development 5, 183–191. https://doi.org/10.1147/rd.53.0183
Lawrence, N.D., 2024. The atomic human: Understanding ourselves in the age of AI. Allen Lane.
Suits, B., 1978. The grasshopper: Games, life and utopia. University of Toronto Press, Toronto.