Read Training Loss, Validation Loss, and Diagnostic Predictions

Derive the corpus's 0.3466-nat uncertainty floor, compare training and independent validation loss, and inspect full next-token probability rows without making claims beyond the toy distribution.

Training loss measures examples that influenced parameter updates. Validation loss measures held-out examples without updating the model. Neither number has meaning until we know the uncertainty in the data.

Calculate the Best Population Loss

The four sequences are equally likely. After <bos>, the next-token distribution is

P(A)=\frac12,\qquad P(X)=\frac14,\qquad P(Y)=\frac14.

Its entropy is

-\frac12\ln\frac12 -\frac14\ln\frac14 -\frac14\ln\frac14 =\frac32\ln2 \approx1.0397.

The prefix <bos> A B occurs in half the sequences. Its next token is equally likely to be C or D, so its population contribution is

\frac12\left(-\ln\frac12\right)=\frac12\ln2.

All other target positions are deterministic. Averaging the two uncertain contributions across four predicted positions gives

\mathcal L_{min}= \frac{(3/2)\ln2+(1/2)\ln2}{4} =\frac12\ln2 \approx0.3466\text{ nats per token}.

A model that represents the true distribution should not drive fresh same-distribution validation loss toward zero.

Read the Verified Run

The frozen 1,000-step CPU run reports:

Metric	Before training	After training
sampled-corpus training loss	2.1898	0.3494
independent validation loss	2.1918	0.3579

The small gap is consistent with finite sampled counts. It does not demonstrate general language ability; training and validation contain only four possible patterns from the same known distribution.

Probabilities Reveal What the Mean Hides

Inspect conditional rows as diagnostics. The verified model assigns most mass:

after <bos>, to A, X, and Y;
after <bos> A B, to C and D;
after <bos> X Y, to X;
after each complete ordinary pattern, to <eos>.

Compare the full probability row with the empirical and theoretical target distribution. A single top token discards whether the model represents a genuine alternative.

Validation Can Answer Only Its Designed Question

This split does not test:

unseen tokens;
longer contexts;
a new grammar;
world knowledge;
coherent open-ended generation;
transfer to another task.

Those claims require different data and evaluation. A clean tiny experiment is valuable because its conclusions are narrow and inspectable.

Q1. Calculate the population floor

Using the derivation above, calculate the minimum average next-token loss in nats per token.

Your answer

Compute it first, then check your number.

Hint

The numerator is

2\ln2

and the denominator is 4.

Solution

2\ln2/4=\ln2/2\approx0.346574

nats per token.

Samples Are Secondary Evidence Here

Chapter 8 develops generation properly. During training, short diagnostic predictions can expose mistakes, but loss, full probability rows, and controlled interventions remain the primary evidence.