Solutions

LM-C12-C-001

Yes. A bigram context sees one previous token, so a 10-token dependency is much longer.

LM-C12-C-002

0.5 * 0.5 * 0.5 = 0.125, and 0.125 * 8 = 1.

LM-C12-C-003

3 * 3 = 9.

LM-C12-C-004

The hidden-state size remains 256 numbers even after 100 time steps.

LM-C12-C-005

Yes. Attention allowed more direct comparison with earlier positions instead of compressing everything through one recurrent state.