Conditioning
Conditioning describes how sensitive a problem is to small input changes.
If a tiny input change can cause a large output change, the problem is ill-conditioned.
If small input changes cause small output changes, the problem is well conditioned.
Why This Matters
Numerical error is unavoidable.
In a well-conditioned problem, small errors tend to stay small.
In an ill-conditioned problem, small errors can become large output errors.
This is different from a buggy implementation. A correct algorithm can still struggle when the underlying problem is highly sensitive.
A Small Picture
Suppose an input measurement is off by 0.001.
If the output changes by about 0.002, the problem is not amplifying the error
very much.
If the output changes by 100, the problem is sensitive. Even careful code may
have trouble producing reliable digits unless the formulation changes.
ML Reading
Conditioning appears in optimization, matrix computations, loss surfaces, and training dynamics.
It helps explain why scaling data, normalizing activations, and choosing stable parameterizations can matter.
A poorly scaled optimization problem can make some directions of the loss surface very steep and others very flat. Gradient descent then has to choose between moving too slowly in flat directions and overshooting in steep directions.
This is why conditioning is not just a numerical linear algebra word. It affects how forgiving a training problem is. Bad conditioning can make a correct update rule behave awkwardly because small errors or step-size choices are amplified unevenly.
If a small input error causes a very large output error, is the problem ill-conditioned?
Enter 1 for yes, 0 for no.
Compute it first, then check your number.
Hint
Solution
Yes. Large output sensitivity to small input changes is ill-conditioning. The problem is amplifying small errors, so even careful computation can lose reliable digits.
Does ill-conditioning always mean the code is wrong?
Answer it first, then check.
Hint
Conditioning describes sensitivity of the problem itself.
Solution
No. A problem can be ill-conditioned even when the implementation is correct.
An input error of 0.001 changes output A by 0.002 and output B by 50.
Which output shows stronger sensitivity: A or B?
Answer it first, then check.
Hint
Look for the larger output change from the same input error.
Solution
Output B shows stronger sensitivity because the same tiny input error produced a much larger output change.
Can scaling or normalizing data sometimes improve numerical behavior in optimization?
Answer it first, then check.
Hint
The ML reading connects conditioning to scaling and normalization.
Solution
Yes. Scaling and normalization can make the optimization problem less awkward by reducing extreme differences between directions or feature scales.
Enter 1 if correct code can still produce unreliable results on an
ill-conditioned problem.
Compute it first, then check your number.
Hint
Separate algorithm bugs from problem sensitivity.
Solution
Enter 1. A correct implementation can struggle if the problem strongly
amplifies small input or rounding errors.
Before Moving On
Conditioning is about the problem, not only the code.