Floating-Point Numbers
A floating-point number is a finite computer representation of a real number.
It can store many useful values, but not all real values.
Approximation
Some decimal numbers do not have an exact floating-point representation.
That means a computer may store a nearby value instead.
This is usually fine, but repeated operations can accumulate small errors.
For example, a decimal such as 0.1 may be stored as a nearby binary
floating-point value. The stored value is close enough for many uses, but it is
not the same as exact real arithmetic.
Precision and Range
A floating-point format has two practical limits:
- precision: how many significant digits can be represented
- range: how large or tiny a number can be before overflow or underflow appears
Smaller formats use less memory and can be faster, but they have less precision and often less range.
ML Reading
Deep learning uses many floating-point formats. Larger formats usually give more precision. Smaller formats use less memory and can be faster.
The tradeoff is speed and memory versus numerical accuracy.
This is why numerical code often compares floating-point results with a tolerance instead of exact equality. If two computations are mathematically the same but take different rounded paths, their stored results can differ slightly.
If a computer cannot store a real number exactly, does it usually store a nearby representable value?
Enter 1 for yes, 0 for no.
Compute it first, then check your number.
Hint
Solution
Yes. It stores a nearby representable value. Enter 1. Floating point has only
finitely many stored values, so many real numbers must be rounded to a nearby
one.
Is floating-point arithmetic the same as exact real-number arithmetic?
Answer it first, then check.
Hint
Floating point stores finite approximations.
Solution
No. Floating point approximates many real numbers and uses finite precision. That means two mathematically equivalent computation paths can differ slightly after rounding.
Do smaller floating-point formats usually save memory?
Answer it first, then check.
Hint
Smaller formats store fewer bits per number.
Solution
Yes. Smaller formats use fewer bits per value, so they usually save memory.
Which limit is about how large or tiny a stored number can be: precision or
range?
Answer it first, then check.
Hint
Precision is about significant digits.
Solution
Range is about how large or tiny a number can be before the format fails to represent it well.
Enter 1 if floating-point results are often compared with a tolerance instead
of exact equality.
Compute it first, then check your number.
Hint
Think about two computations that round intermediate values differently.
Solution
Enter 1. Floating-point values can differ by tiny rounding errors, so code
often checks whether values are close enough instead of exactly equal.
Before Moving On
Floating point is useful approximation, not exact real arithmetic.