Project 9

Build a Sequence Dataset

Turn encoded documents into fixed-width preceding contexts and following-token targets while preserving every document, position, group, mask, and remainder.

  • 8 milestones
  • Optional
  • Browser workspace

Project question: Can you turn encoded documents into preceding-context examples, keep every target tied to its document and position, and save one deterministic dataset another program can inspect and replay?

What makes this a project

Language-model data begins before a model. A document is represented by token IDs, and each example asks for one later ID using only the IDs that came before it. In this project you will verify the supplied tokenizer and corpus, construct those examples, pad short contexts, make masks, keep document groups aligned, form honest batches, and save evidence that can be read back.

This is a bounded data-building project. It does not teach probabilities, loss, or training. The group names are supplied operating labels; later subjects explain training, validation, testing, and leakage. Keep the source identity and every transformation visible.

You will produce:

  • a checked sequence configuration and document-group record;
  • context, mask, and target examples with (document_id, target_position) identities;
  • aligned per-group arrays and metadata;
  • batches that keep a smaller final remainder instead of dropping it;
  • selected-example inspection records, figures, and exact plot data;
  • a manifest and a separate replay agreement record; and
  • a concise dataset card whose claims stay within this fixture.

What you should know first

The project follows the Python path through Chapter 13. You should be able to validate structured records, work with integer IDs and NumPy arrays, slice and pad sequences, keep several arrays aligned, save JSON/CSV/NumPy artifacts, and test deterministic programs. TEXT-02 and DATA-02 are useful preparation but optional. The supplied tokenizer and corpus identities provide the boundary if you skipped them.

Supplied inputs

Use the versioned tokenizer configuration and vocabulary, encoded documents in corpus-manifest order, plans/document_groups.csv, plans/sequence_config.json, and the tiny hand-check fixtures. Every document keeps its document_id, corpus identity, and encoded sequence. The input group file assigns each document exactly once to working, checking, or reserved. Do not infer, randomize, or rename those assignments.

Example contract

For an encoded document ids and context width C, each target position t from 1 through len(ids) - 1 creates one row:

real_context = ids[max(0, t - C):t]
context_ids  = left-pad real_context with <pad> to width C
context_mask = 0 for padding and 1 for real tokens
target_id    = ids[t]

The target is never placed in its own context. A context never crosses a document boundary. Record the document ID, target position, context start, real-context length, and group. A non-empty encoded document therefore yields len(ids) - 1 rows; an empty document yields none but remains in accounting.

The saved arrays for each group are aligned:

context_ids   shape (example_count, C), integer
context_mask  shape (example_count, C), boolean
target_ids    shape (example_count,), integer

There are zero or more leading padding positions, followed only by real positions. The mask sum equals the recorded real-context length. <unk> stays visible; it is not guessed or replaced by source text.

The project workspace

project/
  README.md
  data/                              # supplied, read only
  plans/document_groups.csv         # supplied, read only
  plans/sequence_config.json        # supplied starting config
  src/config.py                      # reader implementation
  src/documents.py                  # reader implementation
  src/examples.py                   # reader implementation
  src/groups.py                     # reader implementation
  src/batches.py                    # reader implementation
  src/inspect.py                    # reader implementation
  src/main.py                       # reader implementation
  output/                            # generated artifacts
  tests/public_cases.py             # supplied, read only

Keep document validation, example construction, grouping, batching, and inspection independently inspectable. Maintain one workspace through every milestone.

Project milestones

Work through these in order. Each milestone produces evidence used by the next one, while project completion remains separate from lesson progress.

  1. 1Verify tokenizer, corpus, documents, and configuration

    Check artifact identity, encoded-document order and boundaries, vocabulary IDs, group coverage, positive sizes, empty documents, and the fixed keep-remainder policy.

  2. 2Construct examples for one tiny document

    Build every preceding-context and following-target pair by hand without copying a target into its own context or crossing a document boundary.

  3. 3Pad contexts and build masks

    Left-pad each context to the configured width and verify exact pad positions, boolean masks, and real-context lengths.

  4. 4Build the complete aligned example table

    Preserve composite identity, source order, context boundaries, targets, and tokenizer and corpus identity for every generated example.

  5. 5Apply document groups through shared indexes

    Select aligned arrays and metadata into supplied working, checking, and reserved groups while proving disjointness and complete coverage.

  6. 6Form one deterministic pass of batches

    Batch each group in saved order, keep and identify every smaller final batch, and invent no batch for an empty group.

  7. 7Inspect, save, and replay the dataset

    Decode explicitly selected examples, save arrays and exact plot data, read artifacts back, and replay the dataset in a separate directory.

  8. 8Write and audit the dataset card

    State source identity, example rules, shapes, groups, batches, padding, unknowns, empty documents, replay evidence, and unsupported claims.

Required evidence

Save examples_metadata.csv, per-group NumPy archives, group_summary.csv, batch_manifest.json, selected-example records, example-count and real-context-length figures with exact plot data, sequence_dataset_manifest.json, a replay agreement or mismatch record, and report.md as the dataset card.

Limits

This project does not teach probabilities, count models, loss functions, embeddings, attention, neural networks, optimization, epochs, random shuffling, train/validation/test theory, leakage analysis, packing documents, cross-document contexts, dynamic padding, framework data loaders, distributed sampling, streaming, or performance tuning. It does not claim that this fixture is balanced, independent, sufficient for a useful model, or suitable for every later subject.

Review

The final review asks whether every input document and group is accounted for, every row has a unique composite identity, targets never leak into contexts, padding and masks agree, aligned arrays stay aligned, remainders remain visible, empty groups produce no invented batches, replay is separate, and the card distinguishes observations from claims.