Sequences and Context
Language arrives in order.
the cat sat
The order matters. The sequence cat the sat contains the same words, but it
does not carry the same structure.
Sequence
A sequence is an ordered list of items.
In language modeling, the items may be:
- characters;
- words;
- subwords;
- bytes;
- special tokens.
This chapter uses simple words first. Later chapters explain why modern models usually use subword or byte-level tokenization.
Context
A context is the part of the sequence the model can see before prediction.
context: the cat
target: sat
The target token is not part of the context. The model must predict it from what came before.
This separation is the first discipline of language modeling. If the answer is allowed into the context, the task is no longer prediction.
Why More Context Can Help
Consider the one-word context:
bank
The next word depends on the surrounding passage. It may be about money, a river, a game, or an aircraft turn. More context can reduce uncertainty:
I deposited cash at the bank
Now finance is more likely than a river bank. The model still has uncertainty, but the context has made some continuations more plausible than others.
In the sequence we learn language models, suppose the model predicts
models. How many previous words are in the context?
Compute it first, then check your number.
Hint
Count only the words before the target word. Do not count the target itself.
Solution
The target word is models. The words before it are we, learn, and
language.
There are 3 previous words in the context.
Before Moving On
A sequence is ordered. A context is the visible prefix used for prediction. The target is what the model must predict, not what it may already know.