Define the complete system boundary and distinguish architecture, parameters, data, training, post-training, inference, serving, and product behavior.
plannedLarge Language Models
Develop a language model from data to deployment. Make the corpus, tokenizer, training budget, batches, checkpoints, evaluation, inference runtime, serving boundary, context state, and operational evidence explicit.
Transformers explains the architecture and the calculations inside it. Large Language Models begins with that foundation and studies the surrounding decisions that turn a Transformer into a trained and usable model.
The subject will build one small language model end to end. Each stage records its inputs, outputs, assumptions, checks, and failure modes so that the final system can be understood and reproduced rather than treated as one opaque training script.
Chapters
The twelve planned chapters follow one model from corpus design and tokenization through training, evaluation, inference, serving, observation, and rollback.
Turn raw sources into a versioned corpus through provenance, licensing, extraction, normalization, filtering, deduplication, contamination controls, and held-out evidence.
plannedTrain and inspect a tokenizer while making normalization, vocabulary, special tokens, byte fallback, sequence lengths, and model compatibility explicit.
plannedChoose model size, context length, data volume, batch size, and training budget through measurable constraints rather than parameter count alone.
plannedConvert tokenized documents into packed causal batches without losing boundaries, masks, targets, padding rules, sampling policy, or reproducibility.
plannedRun, inspect, checkpoint, interrupt, resume, and diagnose pretraining while preserving optimizer, scheduler, random, data-order, and configuration state.
plannedEvaluate held-out loss, perplexity, fixed prompts, capability probes, memorization, contamination, uncertainty, and regressions under declared protocols.
plannedPackage weights, architecture, tokenizer, configuration, provenance, checksums, evaluation evidence, and loading tests as one versioned model artifact.
plannedLoad the artifact and implement verified prompt processing, prefill, decoding, cache management, sampling, stopping, batching, and resource accounting.
plannedDesign a small serving boundary with request validation, limits, scheduling, cancellation, streaming, errors, concurrency, and measurable latency.
plannedSeparate model context from application state while handling templates, truncation, retrieval, tools, memory, provenance, and prompt-injection boundaries.
plannedDeploy a versioned model, observe quality and operations, compare releases, protect user data, detect regressions, and execute a tested rollback.
planned