Vectors in Embeddings
An embedding is a vector used as a representation.
A word, token, image, product, document, or user can be mapped to a list of numbers. The model learns that list so useful relationships become computable.
For example, a tiny embedding might look like:
cat = [0.8, 0.1, 0.4]
dog = [0.7, 0.2, 0.5]
car = [0.1, 0.9, 0.2]
These numbers are not hand-written descriptions. In real models, they are learned from data.
The point is not that the first coordinate literally means "animal" or that the second coordinate literally means "vehicle." The point is that the whole vector can participate in useful relationships. Nearby directions, distances, and scores can become useful even when individual coordinates are hard to name.
Why Vectors Help
Once meaning is represented as vectors, we can compute with it.
- Distance can compare how far representations are.
- Cosine similarity can compare direction.
- Dot products can produce scores.
- Matrix multiplication can transform many vectors at once.
This is why the same vector operations keep appearing in language models.
Which vector operation is often used to compare the direction of two embeddings?
Enter 1 for cosine similarity, 2 for Euclidean distance.
Compute it first, then check your number.
HintThink about direction
Distance compares location. Cosine similarity compares direction.
SolutionCosine similarity
Cosine similarity is often used to compare the direction of two embeddings. It uses the dot product and the vector lengths:
A Tiny Similarity Check
Suppose:
cat = [0.8, 0.1]
dog = [0.7, 0.2]
car = [0.1, 0.9]
The cat and dog vectors point in more similar directions than cat and
car. A model can use that kind of geometry to compare representations.
We should still be careful. This tiny picture is only an intuition. Real embeddings may have hundreds or thousands of coordinates, and the useful directions are learned from the training process.
The careful claim is:
embedding geometry can be useful
without being perfectly human-readable
That is a stronger habit than either extreme. We should not treat embeddings as magic, and we should not pretend every coordinate has a simple label.
In the tiny example above, enter 1 if cat is directionally closer to
dog, or 2 if it is directionally closer to car.
Compute it first, then check your number.
HintCompare patterns
cat and dog both have a larger first coordinate and a smaller second
coordinate.
SolutionSimilar coordinate pattern
Enter 1. In this toy example:
cat = [0.8, 0.1]
dog = [0.7, 0.2]
car = [0.1, 0.9]
cat and dog have a similar coordinate pattern. car points in a
different direction.
Embeddings Are Learned Coordinates
An embedding coordinate usually does not have a simple label such as "catness" or "brightness." Some directions may become interpretable, but the vector is a learned representation, not a neat human dictionary.
That distinction matters.
If we assume each coordinate has an obvious human meaning, we will overread the numbers. If we ignore the geometry, we miss the fact that models can organize useful relationships inside vector spaces.
Enter 1 if this statement is too strong:
Every coordinate
in a learned embedding
has a simple human label.
Compute it first, then check your number.
HintRemember learned representation
The model learns coordinates because they are useful, not because each one was named by a person.
SolutionUseful does not mean obvious
Enter 1. Embeddings can be useful without each coordinate having a simple
human-readable label.
Why This Matters Later
Embeddings prepare us for several later ideas.
In neural networks, activations are vectors. In attention, queries and keys are vectors whose dot products become scores. In interpretability, we ask what features, directions, and circuits inside the model are doing.
So the vector chapter is not only background mathematics. It gives us the language for representation.
Enter 1 if dot products can be used as scores between vector
representations.
Compute it first, then check your number.
HintRecall the dot product page
The dot product returns one number from two vectors.
SolutionOne scalar score
Enter 1. A dot product multiplies matching entries and adds them,
producing a scalar. That scalar can be used as a score.
Next, we close the chapter and collect the important ideas.