Conclusion

Neural language models keep the same prediction problem but change the mechanism. Token ids become embeddings, the neural network produces output scores, softmax turns scores into probabilities, and cross-entropy trains the parameters.

The key advantage over plain counts is shared structure. The model can learn patterns that help beyond exact contexts in the training corpus.

Next we study recurrence, the major pre-attention method for processing sequences.