Milestone 5 of 11

Encode complete positional postings

Encode every posting count, document-ordinal gap, position count, and full-token position gap with traceable offsets and groups.

The dictionary becomes an index only when every posting and position is encoded. Keep a trace from each source value to its gap, byte groups, and offset.

Goal

Compose posting counts, document gaps, position counts, and position gaps into the canonical payload while retaining complete source-to-byte trace rows.

Inputs

Use the validated source index, document ordinal table, gap traces, canonical integer codec, strict dictionary records, and all configured bounds. For each term, postings are in corpus-manifest document order. Every posting has a positive position count; every position sequence contains the TEXT-04 full-token positions in strict increasing order.

The posting stream follows each term's posting_count:

document_ordinal_gap
position_count
repeat position_count times:
  full_token_position_gap

Deliverables

Extend src/payload.py to encode complete postings and write the canonical payload bytes. Extend output/payload_layout.jsonl with every posting count, document gap, position count, position gap, byte offset, encoded length, source identity, and encoded groups.

Checks

Check one and several postings, repeated terms across documents, zero and large first ordinals and positions, positive later gaps, term and position counts, total postings and total positions, exact term/document order, and every bound. Decode each trace cumulatively and prove document IDs and full-token positions return exactly. Check every integer uses the canonical shortest code and every offset stays inside the payload.

Reject zero posting or position counts, non-increasing ordinals or positions, ordinal overflow, mismatched term counts, count or length values over limits, and any source mutation. The payload must contain no untracked bytes between fields.

Workspace

Keep posting composition in src/payload.py, using src/gaps.py and src/varint.py. Do not parse the payload, add the outer header, or claim a rebuilt index yet.

Hints

HintCounts have two jobs

Write a count because the decoder needs it, then compare it with the source count. A count that merely permits a loop is not enough evidence.

HintTrace absolute and gap values

For every gap, retain the source absolute value, preceding value, gap, encoded groups, and reconstructed value. This catches an off-by-one at the first item.

HintBound before allocation

Check declared posting and position counts against both configured limits and remaining payload budget before writing or later reading their repetitions.

Review

Take a term whose first posting has ordinal zero and whose second posting has a large ordinal. Trace both document gaps and one position sequence. Which values are absolute, and which are differences?

How to check your work

Checks compare the payload layout and trace with the complete hand-check fixture. The reference records every field and preserves exact TEXT-04 postings.