Embeddings as Parameters

Embedding rows are model parameters.

When a row is selected in a forward pass, the loss can send gradients to that row. The optimizer updates it just like other parameters.

For a tiny row:

e = [0.5, -0.2]
gradient = [0.1, -0.4]
learning_rate = 0.5

e_next = e - learning_rate * gradient

So:

e_next = [0.5, -0.2] - 0.5 * [0.1, -0.4]
       = [0.45, 0.0]

Only selected rows receive gradients from that example. Over many examples, different rows move according to how they are used.

Exercise: Embedding update first coordinate

Let e = [1, 2], gradient [0.4, -0.2], and learning rate 0.5. What is the first coordinate after the update?

Your answer

Compute it first, then check your number.

Exercise: Selected row

Enter 1 if every row must receive a gradient from one lookup, or 2 if the selected row receives the direct gradient from that lookup.

Your answer

Compute it first, then check your number.