Distributional Intuition

One old idea in language says that words are partly known by the company they keep.

If two words appear in similar contexts, a model may learn similar vectors for them. For example, tea and coffee may appear near words such as drink, cup, and hot.

This does not mean embeddings understand the world as humans do. It means training can turn context patterns into useful geometry.

Small view

If cat and dog often appear near pet, animal, and feed, their learned vectors may become closer than cat and voltage.

Similarity comes from use, not from a dictionary definition pasted into the model.

Exercise

If two tokens often appear in similar contexts, should their embeddings often become more similar? Answer 1 for yes or 0 for no.