Generalization Beyond Counts

A count model needs observed contexts. A neural model can share information through parameters.

If I like tea and I like coffee appear often, a neural model may learn patterns about like, drinks, and nearby contexts. That learned structure can help on contexts that were rare or never seen exactly.

This depends on architecture, data, training, and scale. It is one reason neural language models can move beyond exact n-gram tables.

What generalization means here

Generalization means the model uses learned patterns to assign useful probabilities to examples not memorized as exact training rows.

Exercise

If a neural model can use shared parameters across contexts, can it sometimes handle a context not seen exactly? Answer 1 for yes or 0 for no.