Build the complete aligned example table

The example table is the bridge between source documents and arrays. It must keep enough identity to explain every row later.

Goal

Build the complete padded example table in corpus-manifest order and prove its counts, positions, spans of history, and composite identities.

Inputs

Use every validated encoded document and the configured context width. Process documents in manifest order and target positions in increasing order. For each row save document_id, target_position, context_start, real_context_length, group, tokenizer/corpus identity, padded context, mask, and target.

The target positions are exactly 1..len(ids)-1 for a non-empty document. An empty document stays in the document and group summaries but contributes no row. The composite identity (document_id, target_position) is unique and is enough to recover the unpadded context and target from the source sequence.

Deliverables

Implement the complete ordered table in src/examples.py and save output/examples_metadata.csv plus the aligned array records. Include source document order and the total example count. Add a conservation summary that compares each document's expected and actual row count.

Checks

Check len(ids) - 1 rows for every non-empty document and zero for every empty document. Check increasing target positions, context_start = max(0, t-C), the recorded real-context length, unique identities, and no cross-document contexts. Check that the union of per-document rows equals the complete table without duplicates or omissions.

Use repeated token IDs to prove that identity is positional, not based on token text. Read the table back and confirm the tokenizer and corpus identity have not changed. Test that independently sorting one array is detected as an alignment failure.

Workspace

Keep source-order construction and metadata creation in src/examples.py. Write the table under output/; later milestones will select groups through shared indexes rather than rebuilding rows.

Hints

Review

Choose one first-position row, one row after the context width, and the final <eos> row. Reconstruct each from its document ID and position and compare it with the saved row.

How to check your work

Checks compare order, counts, identities, and metadata fields with the version-matched fixture. The supplied fixture does not shuffle examples.