Conclusion

Embeddings let discrete token ids enter neural networks as learned vectors. They also create a representation space where similarity and context can be studied.

Static embeddings taught an important lesson: prediction can produce useful geometry. But language needs context. A single vector for bank cannot fully serve both river bank and bank account.

Next we use embeddings inside neural language models.