Milestone 4 of 11

Encode document and term records

Write bounded length-prefixed UTF-8 document IDs and lexical term records while retaining byte offsets and exact text identity.

The payload begins with identities and dictionary records. Encode them in a fixed order so a decoder never has to guess which field comes next.

Goal

Write strict length-prefixed UTF-8 document IDs and term records with canonical counts, uniqueness, ordering, and configured length limits.

Inputs

Use the validated source index, document ordinal table, integer codec, and configuration. Every text field is a strict UTF-8 byte sequence preceded by its byte length. The document table is in manifest order. Terms are non-empty, unique, and strictly lexicographically increasing under Python string order.

The payload's ordered prefix is:

document_count
repeat document_count times:
  document_id_byte_length
  document_id_utf8_bytes
term_count
repeat term_count times in increasing term-text order:
  term_byte_length
  term_utf8_bytes
  posting_count
  posting records follow in the postings milestone

Deliverables

Implement the document and dictionary portion of src/payload.py. Produce ordered field metadata for output/payload_layout.jsonl, including field name, source identity, byte offset, encoded length, and decoded value or value type.

Checks

Check empty and zero-term indexes, non-empty unique strict-UTF-8 IDs, exact case, non-ASCII, and underscore terms, lexicographic order, document and term counts, and every configured byte-length bound. Confirm each field's length is the UTF-8 byte count, not the character count.

Reject invalid UTF-8, empty IDs or terms, duplicate IDs or terms, out-of-order IDs or terms, truncated length or text fields, count or length values above configured bounds, and source identity changes. Ensure payload offsets advance exactly and source records remain immutable.

Workspace

Keep strict text encoding, document records, and dictionary records in src/payload.py. Extend payload_layout.jsonl with ordered metadata. Do not write posting gaps or the outer container yet.

Hints

HintCount bytes, not characters

Encode the string strictly first, then measure the resulting byte sequence for its length field.

HintOrder before writing

Validate document and term order before serializing. Sorting during decoding would hide a malformed source payload.

HintOffsets are evidence

Record the offset before each length and byte field. A layout trace makes truncation and a wrong length visible.

Review

Choose a non-ASCII term and compare its character count with its UTF-8 byte count. Why would using the former make the next payload field unreadable?

How to check your work

Checks compare field order, lengths, offsets, UTF-8 bytes, and rejection reasons with the hand-built payload fixtures. The supplied fixture preserves Python string order and never repairs malformed text.