LLM Primer

Language Modeling

Language Modeling

Language Modeling turns Deep Learning toward text: a context is observed, a distribution over possible next tokens is estimated, and that prediction problem becomes the foundation for modern LLMs.

Preface

This subject sits between Deep Learning and Transformers. Deep Learning made tensors, losses, gradients, training loops, embeddings, and PyTorch visible. Language Modeling asks what those tools must do when the data is text and the task is prediction over sequences.

The path starts with text, tokens, and probability before it reaches neural models. Count models, smoothing, evaluation, embeddings, recurrence, gated recurrence, decoding, and attention motivation prepare the reader for the Transformer book.

How to study this path

Keep the prediction visible.

For every chapter, ask the same plain question: given this context, what can come next, and how does the model assign probabilities to those options?

  • Do the hand-counted probability examples before reading the code.
  • Keep text preparation separate from modeling decisions.
  • Track whether a token is a character, word, subword, byte, or special marker.
  • Treat each model limitation as the reason the next chapter exists.
  • Read decoding as a separate choice from the model itself.
  • Use recurrence and gated recurrence to make attention feel necessary rather than mysterious.

Chapters

Move through the path in order.

The Language Modeling chapters are listed in prerequisite order. The path bridges Deep Learning and Transformers: text, tokens, probability, count models, evaluation, embeddings, neural models, recurrence, decoding, and attention motivation.

Chapter 1What Is a Language Model?

Sequences, context, next-token prediction, probability over continuations, and why prediction can learn useful structure.

6 lessons + 5 review pagesChapter 2Text as Data

Corpora, documents, sentences, normalization, punctuation, splits, and leakage in language tasks.

6 lessons + 5 review pagesChapter 3Unicode and Text Normalization

Code points, bytes, grapheme clusters, UTF-8, normalization, case folding, punctuation, and scripts without spaces.

6 lessons + 5 review pagesChapter 4Tokens and Vocabularies

Tokens, vocabularies, unknown tokens, character tokens, word tokens, subword motivation, rare words, and vocabulary size tradeoffs.

6 lessons + 5 review pagesChapter 5Subword Tokenization

Rare words, open vocabulary, BPE, merge operations, WordPiece, unigram tokenization, byte-level BPE, special tokens, padding, and truncation.

6 lessons + 5 review pagesChapter 6Count-Based Language Models

Unigrams, bigrams, trigrams, conditional probability, Markov assumptions, sparse counts, and zero probability.

6 lessons + 5 review pagesChapter 7Smoothing and Backoff

Add-one smoothing, interpolation, backoff, Good-Turing and Kneser-Ney as historical names, rare events, and sparse-data overconfidence.

6 lessons + 5 review pagesChapter 8Evaluating Language Models

Negative log likelihood, cross-entropy, perplexity, bits per token, validation text, memorization, leakage, and evaluation limits.

6 lessons + 5 review pagesChapter 9Embeddings in Language

Embedding matrices, distributional intuition, co-occurrence, word2vec context, learned similarity, nearest neighbors, analogies, polysemy, and static versus contextual representations.

6 lessons + 5 review pagesChapter 10Neural Language Models

Token ids, embedding lookup, next-token training examples, output scores, softmax, cross-entropy, and generalization beyond count tables.

6 lessons + 5 review pagesChapter 11Recurrent Neural Networks

Step-by-step sequence processing, hidden state, shared parameters across time, unrolling, and backpropagation through time.

6 lessons + 5 review pagesChapter 12Long-Range Dependency Problems

Long dependencies, vanishing gradients, exploding gradients, memory bottlenecks, and why attention became attractive.

6 lessons + 5 review pagesChapter 13LSTMs and GRUs

Gates, cell state, hidden state, forget/input/output gates, GRU update/reset gates, controlled information flow, and remaining limits.

6 lessons + 5 review pagesChapter 14Decoding and Sampling

Greedy decoding, temperature, top-k sampling, nucleus sampling as a preview, repetition, uncertainty, and generation behavior.

6 lessons + 5 review pagesChapter 15From Recurrence to Attention

Sequential bottlenecks, fixed hidden-state bottlenecks, direct access to earlier positions, alignment intuition, weighted reading, and long-context motivation.

6 lessons + 5 review pagesChapter 16Language Modeling as a Foundation for LLMs

Pretraining objective, scale, data quality, benchmarks, hallucination as model behavior, and why transformers became the default architecture.

6 lessons + 5 review pages

Progress

Progress is saved locally on this device. Use it as a private reading marker while moving through the path.