Hypothesis Tests
A hypothesis test asks whether observed evidence is surprising under a baseline assumption.
The baseline assumption is often called the null hypothesis.
Working Intuition
A test does not prove a model is good.
It asks a narrower question:
Would evidence this extreme be unusual if the null assumption were true?
That question can be useful, but it is easy to overstate.
A small p-value says the observed result would be unusual under the null assumption. It does not say the effect is large, useful, or caused by the model change you wanted to test.
A large p-value is also not proof that nothing is happening. It may mean the experiment did not have enough evidence to show a clear difference.
In ML
Hypothesis tests can help compare systems, but they do not replace practical judgment. Effect size, dataset quality, and deployment risk still matter.
They are most helpful when the experiment is well designed: fixed evaluation data, a clear baseline, a clear metric, and a decision rule chosen before looking at the result.
The phrase "chosen before looking" matters. If we try many metrics, many subgroups, or many variants and only report the one that looks impressive, the test no longer means what it seemed to mean.
Enter 1 if a hypothesis test alone proves a model is useful in production.
Enter 0 otherwise.
Compute it first, then check your number.
Hint
The word "alone" matters.
Solution
Enter 0. A hypothesis test can provide evidence, but it does not by itself prove that a model is useful in production.
What is the usual name for the baseline assumption in a hypothesis test?
Answer it first, then check.
Hint
The opening section names it.
Solution
The baseline assumption is usually called the null hypothesis. The test asks whether the observed evidence would be surprising under that assumption.
If a test finds evidence of a difference, does that automatically mean the difference is large enough to matter in practice?
Answer it first, then check.
Hint
Statistical evidence and practical importance are different questions.
Solution
No. A test can suggest a difference exists, but effect size and deployment context decide whether the difference matters.
Does a large p-value prove that two models are exactly the same?
Answer it first, then check.
Hint
Failure to find clear evidence is not proof of equality.
Solution
No. A large p-value may mean the experiment did not show clear evidence against the null assumption. It does not prove exact equality.
Enter 1 if trying many tests and reporting only the most impressive one can
make the evidence look stronger than it is.
Compute it first, then check your number.
Hint
Think about searching until something looks surprising.
Solution
Enter 1. If many tests are tried and only the most impressive result is
reported, the evidence can be overstated.
Before Moving On
Use tests as one part of evidence, not as a substitute for understanding the experiment.