Hidden State

The hidden state is a vector that carries information forward.

It is not a copy of every previous token. It is a learned summary. That summary may preserve useful information and lose other information.

At step t, the model uses:

previous hidden state
current token representation

to produce:

new hidden state
optional output scores

The hidden state is the bridge from past tokens to future predictions.

Exercise

If an RNN has hidden size 128, how many numbers are in one hidden-state vector?