Chapter 6
Count-Based Language Models
Unigrams, bigrams, trigrams, conditional probability, Markov assumptions, sparse counts, and zero probability.
What this chapter does
Counts make the language-modeling objective concrete before neural networks hide the table inside parameters.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Count-based language modeling as visible context counts and continuation probabilities.
- 02Unigram Models
Estimating token probabilities from overall frequency, with no context.
- 03Bigram Models
Using one previous token as context for next-token probabilities.
- 04Trigram Models
Using two previous tokens as context and seeing sparsity appear.
- 05Conditional Probability
The count formula behind n-gram next-token estimates.
- 06Markov Assumptions and Sparse Counts
Why fixed context windows are clear but limited by sparse evidence.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What count-based models reveal before smoothing and evaluation.
Summary and Revision NotesA compact review of unigrams, bigrams, trigrams, conditional probability, Markov assumptions, and sparse counts.
ExercisesChapter-level practice for n-gram counts and conditional probabilities.
HintsLow-spoiler nudges for the Chapter 6 exercises.
SolutionsExplained solutions for the Chapter 6 exercises.
Before moving on
- Compute a bigram probability by hand.
- Identify where a count model loses context.
- Sample tiny text from counted probabilities.
Where this leads
- Smoothing and Backoff
- Evaluating Language Models