Introduction, Probability Review, and Motivation

• Room FW26; eight Tuesdays from today
• How worksheets and LLMs work (you are Socrates)
• Probability and entropy review; then motivation and a Boltzmann seed

• Four Moodle quizzes (start of lectures 2, 5, 7, 8)
• Four worksheets (W1: Socratic dialogue; later: notebook + reflection)
• Marking is anonymous: candidate number, never name or CRSid on the file

• Meet the questions today
• Define later; interpret later still
• Two of them only make sense after week 8

• Entropy: no-go theorems
• Probability: what we should do
• Same \(H\), two jobs

• No-go: what is forbidden
• Prescription: the \(p\) you should adopt
• We will draw this idea out every week

• Joint \(P(x,y)\): both
• Marginal \(P(x)\): regardless of \(y\)
• Conditional \(P(x\mid y)\): \(x\) given \(y\)

• Product: \(P(x,y)=P(x\mid y)P(y)\)
• Sum: \(P(y)=\sum_x P(x,y)\)
• Both are normalising bookkeeping, not modelling assumptions

• Follows from the product rule and symmetry of the joint
• Inverts the conditioning — updates a prior given a likelihood
• \(P(x)=\sum_y P(x\mid y)P(y)\) when \(y\) is discrete

• Bernoulli(\(p\)): single binary trial; \(P(X=1)=p\)
• Binomial(\(n,p\)): \(n\) i.i.d. Bernoulli trials; count of successes
• Mean \(np\), variance \(np(1-p)\)

• Poisson(\(\lambda\)): counts in a fixed interval; mean \(=\) variance \(=\lambda\)
• Multinomial(\(n,\mathbf{p}\)): \(n\) trials into \(K\) categories; generalises the binomial
• Categories are exclusive; \(\sum_k p_k = 1\)

• \(\mathcal{N}(\mu,\sigma^2)\): continuous density on \(\mathbb{R}\)
• Fixed by mean and variance; MaxEnt under those constraints (week 5)
• Multivariate form: mean vector and covariance matrix

• Curiosity: open a question; let the model give a long answer
• Skepticism: take a claim and press it — “if that were true, then …”
• Active thought is the point; the transcript is evidence of the probe

• Explore with as many models as you like
• Submit one conversation of about ten prompt/answer turns
• Early turns: open and curious; later turns: skeptical probes
• Then a short reflection on what you learned

• Template: assessments/handouts/ (dialogue + reflection)
• Filename: candidatenumber_worksheet1_dialogue.md (and reflection)
• Quiz 1 (next week) checks foundations you should have met here and in W1

• Discrete: \(H(p) = -\sum_i p_i \log p_i\)
• Base 2: bits; natural log: nats
• Fair coin: \(H=1\) bit; certain outcome: \(H=0\)

• Not yet Clausius’s thermodynamic entropy — same formula, different job
• Not yet a channel-capacity theorem
• Operational split for this course: entropy often forbids; probability prescribes

• \(H(X,Y)\) joint uncertainty
• \(H(X\mid Y)\) residual uncertainty after observing \(Y\)
• Chain rule: \(H(X,Y)=H(X)+H(Y\mid X)\)

• 1925: Motor vehicles (no perpetual motion)
• 2025: Promises of superintelligence singularity
• Same fundamental impossibility?

\[\frac{\text{d}H}{\text{d}t} \geq 0\]

• Entropy always increases
• No motion without entropy production
• No work without energy input

Maxwell’s Demon:

• “Intelligent” entity that violates 2nd law
• Resolution: Landauer’s principle?
• Information erasure requires energy

• Intelligence has thermodynamic cost
• Information processing has physical limits

• Carnot (1824): no real engine beats a reversible cycle between two baths
• Clausius (1850s): heat cannot flow from cold to hot without work
• Clausius (1865): names entropy — the state’s transformation content

• Macroscopic: Carnot \(\to\) Clausius (second law, entropy named)
• Microscopic: Maxwell, Boltzmann, Gibbs (same \(S\), counted states)
• Information: Shannon, Jaynes (same \(H\), different job)

• Lecture 3: why bits, and \(H=-\sum_i p_i\log p_i\) from axioms
• Same functional form as Boltzmann \(S\); different operational reading
• The intelligence question sharpens in lecture 4 (Landauer, Bauby)

• \(p_i \propto e^{-\beta E_i}\) with coldness \(\beta = 1/kT\)
• Normaliser \(Z=\sum_i e^{-\beta E_i}\), so \(p_i = e^{-\beta E_i}/Z\)
• Week 2: derive, account with free energy \(F=U-TS\), name the bath

• Product rule, sum rule, Bayes
• Bernoulli, binomial, Poisson, multinomial, Gaussian
• \(H=-\sum_i p_i\log p_i\) (bits or nats)
• Seed: \(p_i = e^{-\beta E_i}/Z\)

• Worksheet 1 released; due 20 October (Socratic dialogue)
• Quiz 1 next week: probability, entropy, Week 1 seeds