Memory Bottlenecks

An RNN hidden state has fixed size.

That fixed-size vector must carry whatever information from the past is useful for the future. In a long sequence, this creates a bottleneck.

Some information is preserved. Some is compressed. Some is overwritten.

If a model must remember a name, a topic, a quote boundary, and a code variable, one hidden state may not preserve all of it well across many steps.

Exercise

If hidden size is fixed at 128 for both short and long sequences, does the state size grow with sequence length? Answer 1 for yes or 0 for no.