Data Quality

A language model learns from its training data.

Data quality affects what the model sees often, what it sees rarely, what it copies, and what it never learns. Duplicates can encourage memorization. Low quality text can teach poor patterns. Missing domains can leave gaps.

Data is not an afterthought. It is part of the model.

Practical reading

When evaluating an LLM claim, ask:

• what kind of data shaped the model?
• what was filtered?
• what was duplicated?
• what domains are missing or overrepresented?

Exercise

If a dataset contains exact duplicates of the same document 4 times, how many copies of that document are present?