Exercises
These exercises check the first language-modeling habits: identify the context, count next tokens, and read probabilities as a distribution.
In the sequence small models teach ideas, suppose the model predicts ideas.
How many previous words are in the context?
Compute it first, then check your number.
Hint
Count the words before ideas.
Solution
The target is ideas. The previous words are small, models, and teach.
The context length is 3.
A tiny corpus contains we learn math, we learn code, and we learn math.
How many times does math appear after we learn?
Compute it first, then check your number.
Hint
Look only at continuations after the exact context we learn.
Solution
The continuations after we learn are math, code, and math. The token
math appears 2 times.
Using the same corpus, what is P(code | we learn) as a decimal?
Compute it first, then check your number.
Hint
Use count(code after we learn) / count(any token after we learn).
Solution
There are 3 observations after we learn. The token code appears once.
A model assigns probabilities 0.1, 0.2, 0.3, and 0.4 to four possible
next tokens. What do they add to?
Compute it first, then check your number.
Hint
Add all four numbers.
Solution
The probabilities form a complete distribution over the four options.
A model assigns tea: 0.25, coffee: 0.5, and water: 0.25. Enter the
probability of the most likely token.
Compute it first, then check your number.
Hint
Find the largest probability.
Solution
coffee has probability 0.5, which is larger than 0.25 and 0.25. The most
likely probability is 0.5.
Enter 1 if language modeling mainly marks one grammatical continuation as
correct. Enter 2 if it assigns probabilities over continuations.
Compute it first, then check your number.
Hint
Several continuations can be grammatical at once.
Solution
The correct answer is 2. A language model assigns probabilities over possible
continuations. It is not mainly a grammar checker.
For the context def add(a, b): return, enter 1 if programming structure
matters for useful prediction, or 0 if it does not.
Compute it first, then check your number.
Hint
Ask whether ordinary English word frequency is enough to continue this context.
Solution
Enter 1. The context is code, so useful prediction depends on programming
syntax and conventions.