Milestone 1 of 7

Freeze the question, data, and configuration

Record the vocabulary, corpus, split, seed, architecture, objective, update settings, and success checks before training.

State the experiment before training changes the evidence. This milestone creates the immutable record against which every later run is compared.

Milestone goal

Write config.json and data.json, load them in Python, and print a human- readable summary. Do not begin model training until the vocabulary, corpus, split rule, dimensions, random state, optimizer, and success checks are visible.

Freeze the vocabulary and corpus

Use this vocabulary in the declared order:

<bos>:0  A:1  B:2  C:3  D:4  X:5  Y:6  <eos>:7

The population contains four equally likely sequences:

<bos> A B C <eos>
<bos> A B D <eos>
<bos> X Y X <eos>
<bos> Y X Y <eos>

Each five-token sequence yields four inputs and four next-token targets. Keep whole sequences inside one row; do not concatenate them and accidentally train across an <eos>–<bos> boundary.

Sample the training and validation rows independently. The reference uses 256 training rows with seed 17 and 1,024 validation rows with seed 29. Save the realized count of each sequence. The seeds define the sampling procedure; the counts show what was actually sampled.

Freeze the model and update configuration

Field	Reference value
vocabulary size	8
context length	4
model width	4
heads	2
blocks	2
MLP width	8
initialization standard deviation	0.1
model seed	7
batch size	32
optimizer	AdamW, zero weight decay
learning rate	0.01
updates	1,000
gradient clip norm	1.0
dtype and device	float32, CPU

Also record framework versions, tied-readout convention, causal-mask convention, loss reduction, batch-sampling seed, checkpoint fields, and the exact commands used to run the project.

Derive reference losses before training

A uniform prediction over eight tokens has cross-entropy

L_{uniform}=\log 8\approx2.0794.

The corpus has two irreducible choices. After <bos>, the next-token distribution is $(0.5,0.25,0.25)$ over A, X, and Y, with entropy 1.5 bits. After <bos> A B, C and D are equally likely, and that context occurs for half the population, contributing another 0.5 bit on average. The other targets are deterministic. Across four target positions:

L_{floor}=\frac{2\text{ bits}}{4} =\frac{\log 2}{2} \approx0.3466\text{ nats per token}.

This is a population reference, not a guarantee that a finite sampled split or particular optimizer lands on it exactly.

Define success before observing results

The project succeeds when:

the decoder passes shape, parameter-count, and causal-mask checks;
one repeated sequence can be overfit;
corpus validation loss falls well below the uniform baseline and approaches the known population floor without tuning on validation;
checkpoint reload reproduces logits and restored state reproduces the next update;
cached and full-prefix logits agree within a recorded tolerance;
one instrumented trace matches the ordinary forward pass;
one intervention is reported without a claim beyond its evidence.

These checks do not establish broad language competence.

Question. Choose a complete frozen record

Which record is sufficient to begin comparing two training runs?

Choose one

Vocabulary and corpus, split rule and realized counts, model dimensions, seeds, optimizer settings, runtime, and success checks.

Only the final training loss and one generated sequence.

Only the model seed; every other field can be inferred from the checkpoint filename.

Select one choice, then check.

Hint

Include enough information to recreate both the inputs and every learned update.

Solution

A complete record includes data and split details, realized counts, model configuration, all random seeds, optimizer and schedule, dtype, device, software versions, and predeclared success checks.

Acceptance gate

Before continuing, load both files in a fresh Python process and assert:

token IDs are unique and cover 0 through 7;
every sequence begins with <bos>, ends with <eos>, and has length 5;
every input and target row has length 4 and integer dtype;
model width is divisible by head count;
context length covers each input row;
training and validation sampling use different, recorded seeds.

Deliverable: the two machine-readable files, their printed summary, the two reference-loss derivations, and a short statement of scope and success.