Leakage in Language Tasks

Leakage happens when the model gets access to evidence it should not have.

In language modeling, leakage can be subtle. Examples:

• the answer appears in the prompt
• near-duplicate text appears in both train and test
• future text appears in training for a time-based task
• preprocessing uses statistics computed from all splits

Suppose the test sentence is:

The secret code is blue.

If the exact sentence appears in training, a model may score it well because it has seen it before. That score does not prove general prediction ability.

Leakage is not only a benchmark problem. It can also distort your own learning experiments. A tiny model can look smarter than it is if the split is careless.

Before trusting a language-modeling result, ask what the model was allowed to see.