Why Tokens Can Become Vectors

Tokens can become vectors because learning can assign useful coordinates to discrete items.

The token id itself is only an address. The embedding vector is learned from use. If two tokens help make similar predictions in similar contexts, training may move their vectors into related regions.

This is the first bridge to language modeling:

text -> tokens -> token ids -> embedding vectors -> neural computation

The model does not need a dictionary definition before training begins. It needs a way to attach trainable numbers to each token, then adjust those numbers from prediction errors.

This idea scales. Modern language models begin by turning token ids into vectors. Later transformer layers refine those vectors using context. But the first step is still an embedding lookup.

Exercise: Token pipeline

Enter 1 for the usual first neural step after token ids: embedding lookup, or 2 for final answer generation.

Your answer

Compute it first, then check your number.

Exercise: Learned meaning

Enter 1 if embeddings are learned from training signal, or 2 if they must be hand-written definitions.

Your answer

Compute it first, then check your number.