Chapter 10

Neural Language Models

Token ids, embedding lookup, next-token training examples, output scores, softmax, cross-entropy, and generalization beyond count tables.

What this chapter does

Neural language models replace count tables with learned parameters while keeping the same next-token prediction question visible.

Lessons

Read these in order.

Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.

01
Introduction
Neural language models as learned next-token probability functions.
02
Neural Next-Token Prediction
The same prediction task, now produced by learned parameters.
03
Input Embeddings and Output Scores
Token embeddings enter the model; output logits score the vocabulary.
04
Softmax Over a Vocabulary
Turning output scores into probabilities across all token choices.
05
Training with Cross-Entropy
Using next-token targets, cross-entropy, gradients, and updates.
06
Generalization Beyond Counts
How shared parameters help beyond exact observed n-gram contexts.

Review and practice

Close the chapter deliberately.

Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.

Conclusion

What neural language models establish before recurrence.

Summary and Revision Notes

A compact review of embeddings, logits, softmax, cross-entropy, and neural generalization.

Exercises

Chapter-level practice for neural language-model shapes and training signals.

Hints

Low-spoiler nudges for the Chapter 10 exercises.

Solutions

Explained solutions for the Chapter 10 exercises.

Before moving on

Trace token ids to logits and loss.
Explain next-token training examples.
Map a tiny neural language model back to the Deep Learning training loop.

Where this leads

Recurrent Neural Networks
Decoding and Sampling