Categorical Inputs

Categorical inputs name categories rather than measured quantities.

Examples:

token id
product id
country code
class label
day of week

It is usually wrong to treat arbitrary ids as ordinary numbers. If cat = 4 and dog = 7, that does not mean dog is numerically three units larger than cat.

An embedding avoids that false ordering. The id is used only to select a row. The row's vector is what enters the model.

This distinction matters. The id is an address. The embedding vector is the learned representation.

Exercise: Id meaning

Enter 1 if token id 9 is automatically larger in meaning than token id 3, or 2 if ids are usually just addresses.

Your answer

Compute it first, then check your number.

Exercise: Embedding input

Enter 1 if the raw id vector enters later layers directly, or 2 if the selected embedding vector enters later layers.

Your answer

Compute it first, then check your number.