Fixed Hidden-State Bottleneck

In a recurrent model, the hidden state has fixed size.

That state must carry useful past information forward. For long text, this can compress too much into one vector.

Attention offers a different route: keep representations for positions and let later positions read from them with weights.

Exercise

If hidden size is 512, how many numbers are in the hidden state regardless of sequence length?