Week 1: Introduction, Probability Review, and Motivation

[jupyter][google colab][reveal][edit]

Neil D. Lawrence

Abstract:

In this lecture we motivate the course and describe the mechanics of delivery. That will involve a review of Laplace’s demon and Laplace’s gremlin. The aim is to centre the notion of uncertainty in the material. Maxwell’s demon and Landauer’s principle appear as a preview of Week 4; entropy as no-go versus mechanism is a preview of Week 2. We’ll also show how we would like to see you deploy that in your course worksheets, where marks will be given for answers that demonstrate the type of skepticism we see in the Socratic method. We will give a brief review of probability, then relate energy to probability through the way the term energy is used in machine learning (typically a “thermodynamic” or “Gibbs” energy), and introduce the Boltzmann seed. We stop at Daniel Bernoulli as the bridge into kinetic theory; Hydrodynamica, entropy as information, and embodiment continue at the start of Week 2. We will release Worksheet 1 that will need to be submitted before lecture starts in Week 2.

First meeting. FW26. The slot is 10:05–11:55 (110 minutes, not 120); no class test. Worksheet 1 (Socratic dialogue) is released and due by 10:00 at the start of lecture 2 (20 October). Quiz 1 is also at the start of lecture 2: probability foundations and the Week 1 energy / Boltzmann seeds. Hydrodynamica, entropy review, and embodiment open lecture 2 after the quiz. Maxwell / Landauer and the no-go contrast are previews, not the later treatments.

This Session

The course is called Information, Energy and Intelligence because information is the well-defined quantity, intelligence is not, and energy is how the two meet a physical ledger. After the break comes a handwritten reflection on the worksheet habit: we mark the quality of the probe, not the prose. Socrates is previewed in the opening as “I know that I do not know” before the worksheet slides. The Atomic Human comes out as a trade book, not a textbook.

Time plan (as delivered 13 October 2026). Clocks from the Panopto SRT (recording 00:00 ≈ 10:00). The slot is 10:05–11:55 (110 minutes, not 120). Speech from 10:04; this session ran to 12:00. Maxwell / Landauer / no-go are previews of later weeks, spoken in the first hour. Probability, Socrates, and the energy seed follow the break. Daniel Bernoulli is a pointer, not Hydrodynamica.

Course Mechanics

Eight lectures, Tuesdays, this room. Four in-class Moodle quizzes, ten minutes, at the start of lectures 2, 4, 6 and 8. Students need a device. No notes, no network, no LLMs during the quiz. The first quiz is foundations plus this lecture, not two weeks of new material; later quizzes cover the preceding pair of lectures. Four take-home worksheets, each an LLM-centred exercise under the curiosity–skepticism habit from this lecture. Worksheets are released in lectures 1, 2, 4 and 6 and due at the start of lectures 2, 4, 6 and 8 respectively. Identify work by CRSid in filenames and on the handwritten proforma, not by full name. For module questions — clarifications, logistics, conceptual confusion — use GitHub Discussions; start from the pinned Course Q&A — start here post and open a new Q&A thread per question. Personal or confidential matters stay on email or the course office.

Questions We Will Return To

The questions page is published today. Students should meet the whole list. They should not expect to answer most of it. Two stages: define (textbook answer, usually this week or the week the object is introduced) and interpret (the course’s own reading, often week 5 or week 8). The course site is mlatcl.github.io/iei; Reveal.js is the slide engine, not a command to uncover hidden text. Notes for weeks 1–4 have been walked; after that the headlines are mine and the prose may still move. Week 4, Maxwell’s demon, is a motivating idea for the whole course and is previewed today. Thermodynamics is a contested subject even among people who have spent their working lives in it; Daniel Bernoulli is named here as eighteenth-century colour, not as this week’s derivation.

Information, energy and intelligence course notebook setup

We install some bespoke code for creating and saving plots as well as loading data sets.

import importlib.util

cmd = install_command('pods')

%system {cmd}

cmd = install_command('notutils')

%system {cmd}

cmd = install_command('mlai')

%system {cmd}

The Atomic Human

Figure: The Atomic Human (Lawrence, 2024). Not the course text.

I will get the book out once or twice in this first session. It is not the course text. I wrote it to explain, for a general audience, why I find claims of unconstrained superintelligence so badly formed. The course is the attempt to make that argument mathematical. You are not obliged to read it. Chapter 1 is on the reading list as optional colour. We come back to the embodiment argument properly in Week 2.

Perpetual Motion and Superintelligence

Imagine in 1925 a world where the automobile is already transforming society, but big promises are being made for things to come. The stock market is soaring, the 1918 pandemic is forgotten. And every major automobile manufacturer is investing heavily on the promise they will each be the first to produce a car that needs no fuel. A perpetual motion machine.

Well, of course that didn’t happen. But I sometimes wonder if what we’re seeing today 100 years later is the modern equivalent of that. In 2026 billions are being invested in promises of superintelligence and artificial general intelligence that will transform everything.

We know why perpetual motion is impossible: the second law of thermodynamics tells us that entropy always increases. So we can’t have motion without entropy production. No matter how clever the design, you cannot extract energy from nothing, and you cannot create a closed system that does useful work indefinitely without an external energy source.

How might we make an equivalent statement for the bizarre claims around superintelligence? Some inspiration comes from Maxwell’s demon, an “intelligent” entity which operates against the laws of thermodynamics. The inspiration comes because the demon suggests that for the second law to hold there must be a relationship between the demon’s decisions and thermodynamic entropy.

One of the resolutions comes from Landauer’s principle, the notion that erasure of information requires heat dissipation. This suggests there are fundamental information-theoretic constraints on intelligent systems, just as there are thermodynamic constraints on engines.

I’ve no doubt that AI technologies will transform our world just as much as the automobile has. But I also have no doubt that the promise of unconstrained superintelligence is just as silly as the promise of perpetual motion.

Preview: Maxwell’s Demon

Figure: James Clerk Maxwell (1831–1879). The demon is an 1860s thought experiment, not a machine we will build today.

Boltzmann wrote entropy as the Greek letter eta. Capital eta looks like an H, and we have called it \(H\) ever since. The second law is then \(\mathrm{d}H/\mathrm{d}t \ge 0\): entropy of an isolated system does not decrease. Perpetual motion fails because, in practice, organised motion is converted into entropy. The United States Patent Office will not examine perpetual-motion applications; they receive too many.

Why should any of that matter for intelligence? Maxwell’s demon is the 1860s thought experiment that ties a decision to a thermodynamic ledger. An entity sits in a box of ideal gas, at a partition, with a trap door it can open without friction. If a fast molecule approaches from one side it lets it through; if a slow one approaches from the other, likewise. Temperature is the variance of molecular speed — a few hundred metres per second in this room. After enough such decisions one side is hotter than the other. The entropy of the gas has fallen. The second law sits with the speed of light in the physicist’s mind: if you think you have transgressed it, the error is in the thinking.

The demon is also the place where nineteenth-century physics starts to conflate information, entropy, and energy. If information processing has a physical cost, we can begin to ask questions about intelligence that are not just polemic. I am happy to rank an action under a tightly specified goal and a time bound — get these tables out of the room, stacked, in twenty-five minutes — and then compare the energy two methods spent. Thermodynamics already knows the tension: wait for equilibrium and the engine is more efficient; demand the work in a hurry and you pay in dissipation. What I will not do is rank an entity across every criterion by which a human might be judged. That is Bostrom’s superintelligence (Bostrom, 2014), and it is not a scientific ranking.

Preview: Landauer’s Principle

One resolution offered for the demon is Landauer’s principle: deleting information has a minimum energy cost, \(k_B T\ln 2\) per bit, at the temperature of the bath (Landauer, 1961). Boltzmann’s constant is of order \(10^{-23}\) joules per kelvin, so at \(300\,\mathrm{K}\) the bound is not a constraint on any machine we have built. It still states a relationship between information and energy.

The usual further claim is that computation need not cost energy, because it can be made reversible. The computers we actually run are not reversible, which is why data centres draw power. After Feynman’s death, Tony Hey collected the lectures as Feynman Lectures on Computation; Chapter 5 is the place he talks about this. Some of what he says remains contested, including the claim that measurement and computation are the same operation. Early arguments against the demon blamed the measurement. The present consensus in physics is that measurement need not produce entropy if computation is reversible. I am not sure I believe the reversible ideal. We do not have examples of reversible computers.

A student question from the first delivery is the right one, and we will spend Week 4 on it: why is Maxwell’s demon a story about erasure? The sketch — not a proof — is that the demon needs a memory. If that memory scales with the number of particles that pass, either the tape is unbounded (a Turing machine that will not fit in the box) or at some point the demon must erase. Arguments then split. Physicists almost always declare a mechanism: heat baths, frictionless doors, pistons. Counter-arguments then say that mechanism cannot exist — thermal noise, no frictionless door. The second law, like Shannon’s capacity, is not a mechanism argument. It is a no-go. Mixing the two levels is why the demon literature is still noisy. Week 4 is the treatment; today is the hook.

Preview: No-Go versus Mechanism

Shannon’s contribution, for this course, is less the proofs than the refusal to talk about mechanism. He writes down a probability distribution. He does not say which device emitted it. Under those circumstances the properties of entropy make a set of things impossible. That is what a capacity theorem is. The second law has the same shape: entropy must be produced somewhere. It does not tell you when organised motion will stop. Dynamics would. Entropy will not.

That is also why I have come to doubt that probability is enough, even though most of my working life has been probabilistic modelling. Probability insists that we specify what will happen, including the parts of the support we do not care about. Entropy is the language in which we say what will not happen. Week 2 takes that contrast properly, including MacKay, no-go versus prescription, and the common distributions as entropies. Today we only need the distinction, because it is why the demon debate talks past itself: one side is arguing mechanism, the other is arguing a no-go.

Pierre-Simon Laplace

Figure: Pierre-Simon Laplace 1749-1827.

Laplace’s Determism

Figure: Laplace’s determinism book reference.

Famously, Laplace considered the idea of a deterministic Universe, one in which the model is known, or as the below translation refers to it, “an intelligence which could comprehend all the forces by which nature is animated”. He speculates on an “intelligence” that can submit this vast data to analysis and propsoses that such an entity would be able to predict the future.

Given for one instant an intelligence which could comprehend all the forces by which nature is animated and the respective situation of the beings who compose it—an intelligence sufficiently vast to submit these data to analysis—it would embrace in the same formulate the movements of the greatest bodies of the universe and those of the lightest atom; for it, nothing would be uncertain and the future, as the past, would be present in its eyes.

This notion is known as Laplace’s demon or Laplace’s superman.

Figure: Laplace’s determinsim in English translation.

Laplace’s Gremlin

Unfortunately, most analyses of his ideas stop at that point, whereas his real point is that such a notion is unreachable. Not so much superman as strawman. Just three pages later in the “Philosophical Essay on Probabilities” (Laplace, 1814), Laplace goes on to observe:

The curve described by a simple molecule of air or vapor is regulated in a manner just as certain as the planetary orbits; the only difference between them is that which comes from our ignorance.

Probability is relative, in part to this ignorance, in part to our knowledge.

Figure: Laplace’s gremlin in the original French.

Figure: To Laplace, determinism is a strawman. Ignorance of mechanism and data leads to uncertainty which should be dealt with through probability.

In other words, we can never make use of the idealistic deterministic Universe due to our ignorance about the world, Laplace’s suggestion, and focus in this essay is that we turn to probability to deal with this uncertainty. This is also our inspiration for using probability in machine learning. This is the true message of Laplace’s essay, not determinism, but the gremlin of uncertainty that emerges from our ignorance.

The “forces by which nature is animated” is our model, the “situation of beings that compose it” is our data and the “intelligence sufficiently vast enough to submit these data to analysis” is our compute. The fly in the ointment is our ignorance about these aspects. And probability is the tool we use to incorporate this ignorance leading to uncertainty or doubt in our predictions.

Laplace’s Gremlin

The curve described by a simple molecule of air or vapor is regulated in a manner just as certain as the planetary orbits; the only difference between them is that which comes from our ignorance. Probability is relative, in part to this ignorance, in part to our knowledge. We know that of three or greater number of events a single one ought to occur; but nothing induces us to believe that one of them will occur rather than the others. In this state of indecision it is impossible for us to announce their occurrence with certainty. It is, however, probable that one of these events, chosen at will, will not occur because we see several cases equally possible which exclude its occurrence, while only a single one favors it.

— Pierre-Simon Laplace (Laplace, 1814), pg 5

The representation of ignorance through probability is the true message of Laplace, I refer to this message as “Laplace’s gremlin”, because it is the gremlin of uncertainty that interferes with the demon of determinism to mean that our predictions are not deterministic.

Our separation of the uncertainty into the data, the model and the computation give us three domains in which our doubts can creep into our ability to predict. Over the last three lectures we’ve introduced some of the basic tools we can use to unpick this uncertainty. You’ve been introduced to, (or have yow reviewed) Bayes’ rule. The rule, which is a simple consequence of the product rule of probability, is the foundation of how we update our beliefs in the presence of new information.

The real point of Laplace’s essay was that we don’t have access to all the data, we don’t have access to a complete physical understanding, and as the example of the Game of Life shows, even if we did have access to both (as we do for “Conway’s universe”) we still don’t have access to all the compute that we need to make deterministic predictions. There is uncertainty in the system which means we can’t make precise predictions.

Gremlins are imaginary creatures used as an explanation of failure in aircraft, causing crashes. In that sense the Gremlin represents the uncertainty that a pilot felt about what might go wrong in a plane which might be “theoretically sound” but in practice is poorly maintained or exposed to conditions that take it beyond its design criteria. Laplace’s gremlin is all the things that your model, data and ability to compute don’t account for bringing about failures in your ability to predict. Laplace’s gremlin is the uncertainty in the system.

Figure: Gremlins are seen as the cause of a number of challenges in this World War II poster.

Preview: From Insufficient Reason to MaxEnt

Figure: E. T. Jaynes. Maximum entropy is the modern writing of Laplace’s principle of insufficient reason.

Laplace’s gremlin is uncertainty from ignorance — of data, of law, of compute. He had a rule for turning that ignorance into a distribution: the principle of insufficient reason. If \(n\) outcomes are mutually exclusive and nothing distinguishes them, take \(p_i = 1/n\). He never wrote that rule in the language of entropy. Jaynes, who read Laplace hard, did: the principle of maximum entropy is the modern form of insufficient reason (Jaynes, 2003). Laplace stayed with probability. One of the arguments of this course is that entropy lets us make declarations about what will not happen that probability, used alone, does not. Week 5 is where we derive the occupation. Do not do Lagrange multipliers today.

A question from the first delivery: what about uncertainty about the specification itself — the risks you cannot even name? Dennis Lindley argued that you integrate that level out and it propagates. I believed that for a long time. I am less sure it is that simple. Anders Hansen, in mathematics here, has written about the need for an “I do not know” that current LLMs lack. Those are pointers, not this week’s work.

Probability Review

Notation: we often write \(P(x,y)\) for \(P(X=x,Y=y)\). Unlike a generic bivariate function, \(P(x,y)=P(y,x)\).

The product rule relates joint and conditional. The sum rule recovers a marginal by summing out the variable you do not care about. Continuous analogues replace sums by integrals.

Bayes is not a third axiom; it is the product rule rearranged. The barrels example below is a small discrete inversion of the conditioning.

Figure: Two barrels: Barrel One holds 20 apples and 4 oranges; Barrel Two holds 4 apples and 8 oranges. A barrel is chosen uniformly at random and a single fruit is drawn.

Work the numbers on the board if time allows. The same pattern applies to coins, medical tests, or any two-hypothesis discrete update.

Quiz 1 will ask a small discrete Bayes inversion (barrels, coins, two hypotheses) and recognition of the named distributions below.

Common Distributions

These are prerequisites restated, not new theory. Several of them reappear later as maximum-entropy distributions under moment constraints.

A fair coin is Bernoulli(\(1/2\)). A two-state thermal system with fixed energies is Bernoulli once occupation probabilities are written down.

Poisson Distribution

Figure: The Poisson distribution.

Differential entropy of a Gaussian grows with \(\sigma\) and can be negative. For a review, recognise the density and the two parameters.

The Gaussian Density

The Gaussian density is perhaps the most commonly used probability density. It is defined by a mean, \(\mu\), and a variance, \(\sigma^2\). The variance is taken to be the square of the standard deviation, \(\sigma\).

\[\begin{align} p(y| \mu, \sigma^2) & = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y- \mu)^2}{2\sigma^2}\right)\\& \buildrel\triangle\over = \mathscr{N}\left(y|\mu,\sigma^2\right) \end{align}\]

Figure: The Gaussian PDF with \({\mu}=1.7\) and variance \({\sigma}^2=0.0225\). Mean shown as red line. It could represent the heights of a population of students.

Colour, not examinable beyond recognising the families. Football scores are a rough Poisson with rate about one goal per team per match. De Moivre used the Gaussian as a binomial approximation; Laplace used the same quadratic in what we would now call a Laplace approximation, expanding a log posterior about its mode; Gauss credits Laplace when he fits the orbit of Ceres. Heights are a standard story and an invalid support, because a Gaussian puts mass on negative height. The velocities of air, to the extent the gas is ideal, are Gaussian — that is the Maxwell–Boltzmann distribution on the Cartesian components, with temperature setting the variance. The cosmic microwave background is another Gaussian that people have spent a great deal of money testing for non-Gaussianity (Planck).

Old Books, and the Wikipedia Effect

Before the worksheets I want to say something about books. I was a mechanical engineer. The thermodynamics text was Rogers and Mayhew, Engineering Thermodynamics, Work and Heat Transfer. It was my father’s copy from Sheffield in the 1960s, and it was still the assigned book when I did the same degree in the 1990s. Thermodynamics does not move quickly. I did not feel I understood it then, and I do not feel I understand it now. The diagrams of critical points and liquid–vapour mixtures are printed, not drawn in LaTeX, and the volume still smells like a 1960s book.

When I have asked language models for a thermodynamics text they are, unsurprisingly, fluent in books they have absorbed. One recommendation was a physicist’s thermodynamics book, printed in 1961, now out of print. I bought a copy. In places it explains more clearly than a modern textbook. I think of this as the Wikipedia effect. If you lived through 2006, Wikipedia articles then often explained what a thing was and what it was for. Later edits made them more correct, and often unreadable unless you already had the degree. The Gaussian process article is the example I use. Look at the 2006 revision if you want the intuition; look at the current one if you want the statement that will survive a referee. I wonder whether thermodynamics books have followed the same path: we have not understood the intuition much better, but some of the older books still carry it. I am not asking you to read these volumes. If a particular passage would help, I may scan it.

Doubt

What Laplace’s gremlin implies in practice is that we can never be certain about outcome, so doubt is an inevitable consequence of our lack of knowledge. It is what drives us to develop quantitative notions for uncertainty. We have already introduced probability. Next we meet energy as used in machine learning, then entropy — the concept that unifies information and uncertainty, and therefore we argue (perhaps) intelligence.

We will look to apply these ideas to how we approach our learning as well as what we learn. Bringing doubt into your own reasoning is a way in which you can develop your understanding. Learning to express that doubt through challenging a statement. The model we will use finds its origins in Socrates.

Worksheets and LLMs: Can you be Socrates?

The general approach we’d like to take to this course is that of a “community of inquiry” (see Chapter 4, Lipman (2012)). The unusual modern twist on this notion is that the LLMs themselves become part of that community.

Each of us will have different perspectives on what an LLM does and does not provide. You are welcome to bring those perspectives into your work. In particular, for each worksheet, you will be asked to reflect on the LLM responses and the process. Part of that reflection should be specific to the exercise and what you learnt about the subject. But I would like part of that reflection to be general about your understanding of the LLM and what it does and doesn’t provide. For worksheets 2, 3, and 4 a portion of that reflection will be on how you feel your understanding of LLMs as a tool of inquiry has evolved (if it evolved!).

The assessment model is based on is 1. a form of questioning enquiry generally called “the Socratic method” is an informative way of exploring a topic 2. an impression that the current generation of LL models is weak when asked to sustain Socratic dialogue. Answers tend to be expansive and when read carefully can lead to ‘dialectical vertigo’ 3. The new idea we’re testing “Socratic method” can be deployed by reversing the role of Socrates and the student.

So in other words we are asking you to take on the role of Socrates in these conversations.

The general background is an idea that in order to develop your understanding of a subject through interaction with an LLM you need two components to your enquiry: curiousity and skepticism. The curiousity allows you to generate the prompt and the LLM to regurgitate some of its knowled (or perform searches that it summarises). But the skepticism engages with that summary through challenging the conclusions that the LLM has. In the Socratic elenchus that challenge is through pointing out a logical inconsistency that arises (Vlastos (1993)), perhaps through a side implication. For our purpose that challenge may not take exactly that form. But it should push back on the narrative the LLM provides. Generating such push back also requires you to engage with the material the LLM has provided.

For Socrates these are curated dialogues (written by Plato, e.g. Plato (1914)). So its normally the case that his challenges hit home. In your case, that won’t normally be the case. And we don’t expect you to curate your dialogue. What we’d like instead is a period of inquiry that is then summarised by a single dialogue that is played out with one LLM in a short session of 10 prompts and responses.

Classical Socratic practice (elenchus) tests consistency by questioning, not by lecturing. Contemporary seminar pedagogy keeps the same habit: the questioner holds the inquiry. Current LLMs default to exposition and agreement; they rarely sustain adversarial follow-ups without being steered. Assigning the student the Socrates role forces engagement with the subject matter rather than passive acceptance of a fluent summary.

Dialectical Vertigo

In the book, The Grasshopper (Suits, 1978), the eponymous protagonist, Grasshopper, asks his student, Skepticus, to take on this role.

What you are describing, Skepticus, is a chronic but minor ailment of philosophers. It is called dialectical vertigo, and its cure is the immediate application of straightforward argumentation.

The Grasshopper to Skepticus in The Grasshopper: Games, Life and Utopia (Suits, 1978 pg 37)

Worksheets are marked with 5 points for curiosity, 5 for skepticism, and 5 points for the reflection (15% of the module). Worksheet 1 uses a markdown dialogue template (YAML records CRSid, model, and interface) and a separate handwritten reflection proforma. Moodle upload names follow the departmental pattern L172_<CRSid>_worksheet1_dialogue.md and L172_<CRSid>_worksheet1_reflection.pdf. Use your CRSid in the YAML and on the proforma; do not write your full name.

Energy in Machine Learning

An energy in machine learning is a scalar score on a configuration: parameters, predictions, or binary states. Often an energy is equivalent to an objective function or an error function. In this respect lower is preferred. Energy-based models turn that score into a distribution by the same exponential map used in statistical physics, [ p(x)e^{-E(x)}. ] Training normally then amounts to shaping \(E\) so that observed configurations sit in low-energy valleys.

We can reinterpret many machine learning algorithms in this way by simply writing familiar loss functions in the energy language. Note these are not new algorithms, just the energy interpretation.

Energies from Distributions You Already Know

The map \(p\propto e^{-E}\) runs both ways. Given a distribution we already introduced, set \[ E_i = -\log p_i \] (up to an additive constant that cancels in the normalisation). High probability is low energy. The numbers below reuse examples from earlier in the lecture. For each classic family we plot \(P\) (or \(p\)) beside \(E=-\log P\), then overlay the two on twin axes.

Exponentiating recovers the distribution (natural units): \(p(y)=e^{-E(y)}\) already normalises here because we took \(E=-\log p\) exactly. In general one writes \(p_i = e^{-E_i}/Z\) with \(Z=\sum_j e^{-E_j}\). The entropy calculation from the review is therefore also an average energy.

A Bernoulli(\(p\)) on \(\{0,1\}\) has energies \(E(x)=-\log P(x)\) (nats). High probability is low energy. The panels below use a biased coin \(p=0.3\): first \(P\) and \(E\) side by side, then the same two curves overlaid on twin axes so the inverse relationship is visible in one frame.

Figure: Bernoulli(\(p=0.3\)). Left: the distribution. Right: energy \(E=-\log P\) in nats. The favoured outcome sits lower in energy.

Figure: Same Bernoulli: probability (bars, left axis) and energy (markers, right axis). Where \(P\) is high, \(E\) is low.

A fair coin has equal energies on heads and tails; uncertainty is maximal. A biased coin tilts the energy landscape. The same grammar appears as a two-level thermal system: fix energies \(E_0,E_1\) and coldness \(\beta\), then \[ p_1 = \frac{e^{-\beta E_1}}{e^{-\beta E_0}+e^{-\beta E_1}}, \] which is Bernoulli with that success probability. Week 2 returns to this as the Schottky / two-state seed.

A Poisson(\(\lambda\)) on the non-negative integers has \[ P(k)=\frac{\lambda^k e^{-\lambda}}{k!},\qquad E(k)=-\log P(k) \] (nats). The bulk of the mass sits in a shallow energy valley; the tails climb as \(k!\) in the denominator of \(P\) becomes the dominant cost.

Figure: Poisson(\(\lambda=4\)). Left: the count distribution. Right: energy \(E=-\log P\). The mode is the energy minimum.

Figure: Same Poisson: probability (bars) against energy (markers). Tails are high-energy, low-probability states.

The univariate Gaussian already written as \[ p(y)\propto\exp\!\left(-\frac{(y-\mu)^2}{2\sigma^2}\right) \] is an energy-based model with quadratic energy. Taking \(E(y)=-\log p(y)\) (nats) adds only the normaliser \(\tfrac12\log(2\pi\sigma^2)\), so the shape remains a parabola about \(\mu\). The height example \(\mu=1.7\), \(\sigma^2=0.0225\) is reused below.

Figure: Height Gaussian. Left: density. Right: energy \(E=-\log p\) — a parabola about \(\mu=1.7\) (additive constant from the normaliser does not change the shape).

Figure: Same Gaussian: density (left axis) and energy (right axis). The mode of \(p\) is the bottom of the energy well.

Reading the Gaussian density as \(p\propto e^{-E}\) makes the link to regression immediate: minimising \(\sum_n(y_n-\mathbf{w}^\top\mathbf{x}_n)^2\) is minimising a sum of quadratic energies, which is maximum likelihood under independent Gaussian noise. The continuous normaliser plays the role of \(Z\).

Under \(y_n=\mathbf{w}^\top\mathbf{x}_n+\varepsilon_n\) with \(\varepsilon_n\sim\mathscr{N}\left(0,\sigma^2\right)\), the negative log-likelihood is \(\frac{1}{2\sigma^2}E(\mathbf{w})\) plus constants. So ordinary least squares is maximum likelihood for a Gaussian or equivalently, minimising a quadratic energy.

When \(y\) is a one-hot label and \(\hat{y}\) a categorical prediction, cross-entropy is the energy of that pair. Softmax turns a vector of class energies (or logits) into a distribution. The functional \(-\log\hat{y}_c\) is the same one that appears in Shannon entropy \(\mathbb{E}[-\log P(y)]\), applied to a model distribution rather than the data distribution.

Ackley, Hinton and Sejnowski (Ackley et al., 1985) made the couplings \(W_{ij}\) learnable; Hopfield (Hopfield, 1982) used a similar pairwise energy for associative memory with fixed weights. When a Boltzmann or Gibbs occupation is written \(p\propto e^{-\beta E}\), the symbol \(E\) is the same kind of object minimised in regression and classification — a score over configurations, exponentiated into a distribution.

Boltzmann Seed

In machine learning an energy is often just a negative log probability, measured in nats, which is a peculiar unit for something that in physics is measured in joules. Energy-based models are, among other things, a way of not writing the partition function \(Z\). In physics \(Z\) is the whole story. We return to it with free energy; do not compute it today.

import numpy as np

def boltzmann(energies, beta):
    """Boltzmann probabilities $p_i \\propto e^{-\\beta E_i}$."""
    log_w = -beta * np.asarray(energies, dtype=float)
    log_w -= log_w.max()
    w = np.exp(log_w)
    return w / w.sum()

# Live check: boltzmann([0, 1], 1.0) -> about (0.731, 0.269)

Figure: Daniel Bernoulli (1700–1782). Pressure from a gas is given by velocity of gas acting on sides (but constant velocity). Portrait c.1720–1725, Historisches Museum Basel (Wikimedia Commons, CC BY-SA 4.0).

Define This Week

After This Lecture

Worksheet 1: Socratic dialogue on the history of the name entropy — why Shannon used the word; von Neumann anecdote versus Shannon’s Tolman footnote and the Wiener–Gibbs trail. Exactly ten turns (mandated opening pair, then free probes); handwritten reflection on the scanned proforma (separate upload). Use the handouts. Due 20 October, start of lecture 2. Quiz 1 in the first ten minutes of lecture 2 covers today’s probability review and energy / Boltzmann seeds; lecture 2 then presses Laplace’s molecule as anachronism, Bernoulli’s assumptions, and the ensemble as coarse-graining before Hydrodynamica and the entropy bridge. As delivered, a copy of The Atomic Human went to the student who asked the most questions; that is not part of the mark.

Further Reading

• Probability distributions: Section 1.2 of Bishop (2006)

• Boltzmann machines (optional colour) of Ackley et al. (1985)

• Hopfield networks (optional colour) of Hopfield (1982)

Thanks!

For more information on these subjects and more you might want to check the following resources.

• company: Trent AI
• book: The Atomic Human
• twitter: @lawrennd
• podcast: The Talking Machines
• newspaper: Guardian Profile Page
• blog: http://inverseprobability.com

References