Learned Similarity and Neighbors

Once tokens have vectors, we can compare vectors.

Cosine similarity is often used to ask whether two embedding vectors point in a similar direction. Nearest-neighbor search asks which vectors are closest to a given vector.

This is useful for inspection, but it must be read carefully. A nearest neighbor is not a proof of meaning. It is a clue about how the model's training shaped the representation.

Example

If the nearest neighbors of king include queen, prince, and monarch, that suggests the vector space has captured some related usage.

If the neighbors are strange, that also teaches something: the corpus, tokenizer, training task, or model size may be shaping the geometry in unexpected ways.

Exercise

If a token has 4 nearest neighbors listed, how many neighbor entries are being inspected?