Pad contexts and build masks

Padding changes the shape of a row, not its history. A mask records which positions are real.

Goal

Turn preceding contexts into fixed-width integer rows and build masks that agree exactly with the padding.

Inputs

Use the unpadded examples and positive context width C. If a real context is shorter than C, put <pad> tokens on the left. If it is C or longer, keep the last C real tokens. The target remains the ID at the original target position.

For every row, context_mask is 0 for each leading padding position and 1 for each real position. Thus a row has zero or more leading zeros followed by ones. The mask sum equals the recorded real-context length. <pad> is a structural value here; it is not an input token from a source document.

Deliverables

Extend src/examples.py to produce context_ids with shape (n, C), boolean context_mask with shape (n, C), and target_ids with shape (n,). Preserve document ID, target position, context start, and real-context length in the metadata beside these arrays.

Checks

Use history shorter than, equal to, and longer than C. Check exact left padding, the width and dtype of every array, the mask/pad equivalence, the mask-sum invariant, and the target's unchanged position. Reject a mask with a one before a zero, a mask whose sum disagrees with metadata, or an ID that is not the configured <pad> at a zero position.

Check an empty document produces no rows and a valid one-token source document still produces the two rows formed from <bos> token <eos>. Verify all input examples remain unchanged and that no right padding or target leakage appears.

Workspace

Keep padding and mask construction in src/examples.py or a small dedicated array function. Do not group or batch rows yet.

Hints

Review

Select a short context and read its row from left to right. State which values are structural padding, which values came from the source, and why the target is not represented by a mask position.

How to check your work

Checks compare array shapes, dtypes, left-padding, and invariants with the supplied fixture. The supplied fixture does not introduce dynamic padding.