Build, Train, and Inspect a Tiny Decoder

Build one decoder-only Transformer whose complete computation can be checked. Then train it, generate from it, inspect one prediction, change one internal component, and report what the evidence does and does not support.

This is an investigation, not a race to the lowest loss. A successful project connects the contracts developed across the subject:

If one link cannot be checked, preserve the failure and repair it before moving on. A final token sample cannot compensate for an unverified causal mask or an unreproducible checkpoint.

Project question

Can a 368-parameter causal decoder learn the conditional structure of a fixed toy corpus, reproduce its run, generate under declared rules, and expose one prediction to a controlled intervention?

The question is deliberately narrow. The project does not test broad language ability, factual knowledge, instruction following, or production throughput.

Frozen baseline

The reference model has:

It runs on CPU. Small size is part of the method: every shape, probability, parameter, checkpoint field, and intervention remains inspectable.

Working environment

Use a local Python environment with PyTorch installed. Record the Python and PyTorch versions rather than assuming another machine has the same runtime. The checked reference used CPU, Python 3.14, and PyTorch 2.13.0, but the code does not require that exact Python minor version.

Keep the work in one directory:

tiny-decoder/
  config.json
  data.json
  model.py
  train.py
  generate.py
  inspect.py
  checkpoints/
  outputs/
  report.md

Your names may differ. The important property is that another reader can find the configuration, source, checkpoint, raw outputs, and report without reconstructing them from a notebook's hidden state.

The three downloadable chapter scripts are checked references:

• training and checkpoint script;
• generation and KV-cache script;
• inspection and intervention script.

Attempt each milestone before using a reference to diagnose it. Copying output without reconstructing its contract does not complete the project.

Milestone order

1. freeze the question, corpus, split, seeds, model, and success checks;
2. assemble the decoder and verify shapes, parameters, masking, and logits;
3. overfit one sequence as a narrow training-path diagnostic;
4. train the fixed corpus and preserve validation and checkpoint evidence;
5. prove cached and uncached generation agree before comparing policies;
6. trace and ablate one prediction with a declared metric and replacement;
7. write a report that separates observations, interventions, and conclusions.

The order is a dependency graph. Do not tune the corpus run before the one-sequence diagnostic passes, and do not interpret cached logits before they match the full-prefix reference.

Final artifacts

Submit:

1. a frozen machine-readable configuration and data record;
2. shape, parameter-count, causality, and forward checks;
3. a one-sequence overfit record and GO/NO-GO decision;
4. training and validation evidence plus a restorable checkpoint;
5. a prefix-equivalence table, cache trace, and two declared generation runs;
6. one faithful internal trace and one declared intervention;
7. a concise report with exact rerun commands, alternatives, and limitations.

Definition of done

The project is complete only when all seven milestone artifacts exist and their acceptance checks pass. A training script that produces plausible text but lacks validation, cache equivalence, intervention evidence, or a report is an incomplete project.