Summary and Revision Notes

Core ideas

• Neural language models still predict the next token from context.
• Token ids become embedding vectors.
• The output layer produces one score per vocabulary item.
• Softmax turns scores into probabilities.
• Cross-entropy trains the model against observed next tokens.
• Shared parameters allow generalization beyond exact count tables.

Check yourself

• Can you explain why vocabulary size controls output-score count?
• Can you connect cross-entropy training to perplexity evaluation?
• Can you explain why neural models are not just n-gram tables?