Training with Cross-Entropy

Training compares the model's probability distribution with the observed next token.

If the target token is tea, the loss rewards probability assigned to tea and penalizes probability assigned elsewhere. In practice this is cross-entropy loss over the vocabulary.

Backpropagation computes gradients. The optimizer updates embeddings and other model parameters. After many examples, the model learns patterns that help it predict future tokens.

Important connection

The evaluation loss and the training loss are closely related. Training uses cross-entropy to improve parameters. Evaluation reports cross-entropy or perplexity on held-out text.

Exercise

If a batch has 8 prediction positions, how many target next tokens does it have?