Turn a Token Corpus into Training Examples
Convert bounded token sequences into causal contexts and one-position-shifted targets. Define boundary tokens, window rules, and train-validation separation before batching or optimization.
A corpus is the collection of token sequences used to construct training examples. Before batching or optimization, define exactly where each sequence begins, where it ends, and which token each context must predict.
The frozen vocabulary is:
| ID | Token | Role in this corpus |
|---|---|---|
| 0 | <bos> | beginning-of-sequence marker |
| 1–6 | A–Y | ordinary tokens |
| 7 | <eos> | end-of-sequence marker |
For the five-token sequence
<bos> A B C <eos>
the model receives four contexts at once:
| Logits row | Visible prefix | Target |
|---|---|---|
| 0 | <bos> | A |
| 1 | <bos> A | B |
| 2 | <bos> A B | C |
| 3 | <bos> A B C | <eos> |
The input row is [0,1,2,3]; the target row is [1,2,3,7]. The causal mask
ensures that logits row 0 cannot use A, even though all four input positions
are present in one tensor during training.
Longer Streams Need a Window Rule
If a token stream is longer than context length , choose windows explicitly. For tokens , one length- input beginning at is
A stride of 1 creates overlapping windows. A stride of creates fewer, non-overlapping windows. Neither is automatically correct: the choice changes which contexts and targets appear and how often they are counted.
This chapter avoids that additional choice by using independent five-token sequences whose input length is exactly four.
Boundary Tokens Are Part of the Learning Problem
Without <bos>, the model has no explicit row for learning the first ordinary
token. Without <eos>, it is never trained to assign probability to stopping
after the final ordinary token.
These markers are vocabulary tokens, not instructions outside the model. They receive embeddings, logits, and losses like other included tokens.
Keep Validation Examples out of Parameter Updates
Training examples influence gradients and parameters. Validation examples are held aside and used only to measure the current model. If validation targets enter an optimizer step, the reported validation loss is no longer independent evidence about those examples.
The chapter samples training and validation sequences with different random seeds from the same four-pattern distribution. This tests a narrow question: does training recover the same small conditional rule on fresh draws? It does not test generalization to new vocabulary or a new grammar.
Q1. Shift one sequence
For sequence <bos> Y X Y <eos>, write the four input IDs and four target IDs.
Answer it first, then check.
Hint
<bos>=0, X=5, Y=6, and <eos>=7.Solution
[0,6,5,6] and the target is [6,5,6,7].Check the Data before the Model
Print several decoded input-target rows and their source sequence. Confirm the target shift, sequence boundaries, vocabulary range, and train-validation separation before calculating any logits.