LLM Primer

Introduction

Before neural networks, language models could be built from counts.

A count-based model looks at a corpus, counts what follows what, and turns those counts into probabilities. This is simple enough to compute by hand, and that is why it belongs here.

The goal is not to return to older methods for production. The goal is to make the language-modeling objective visible:

given context -> count continuations -> estimate probabilities

Neural language models still solve a version of this prediction problem. They replace explicit count tables with learned parameters.

n-gram models choose a fixed context windowunigramno previous tokenpredict nextbigramone previous tokenpredict nexttrigramtwo previous tokenspredict nextmore context can help, but it also makes counts sparse
Count models make the context length explicit: fixed windows are clear, but limited.

What this chapter covers

  • unigram models;
  • bigram models;
  • trigram models;
  • conditional probability;
  • Markov assumptions and sparse counts.