Neural Next-Token Prediction

A neural language model receives a context and predicts the next token.

For a tiny example:

context: I like
target: tea

The model produces scores for vocabulary items such as:

tea, code, music, books

Softmax turns those scores into probabilities. Training changes parameters so the observed target receives higher probability.

The model is not counting only the exact context. It is learning a function that maps many contexts to distributions.

Exercise

If one context has one observed next token target during training, how many target tokens are used for that prediction?