FP32, FP16, and BF16

FP32, FP16, and BF16 are common names for floating-point formats.

In practice:

FP32 is a common stable default.
FP16 uses less memory but has narrower numerical range.
BF16 uses less memory and keeps a range closer to FP32, but has less precision.

This is a tradeoff, not a ranking.

FP16 can be fast and memory-efficient, but it can overflow or underflow more easily. BF16 is often attractive for deep learning because range matters for activations and gradients.

For this chapter, you do not need bit layouts. You need the practical question:

Is this dtype giving the model enough range and precision for this computation?

DL-C16-T02-001Exercise: Stable default

Enter 1 for FP32 or 2 for FP16: which is usually the more stable default when speed and memory are not the main concern?

Your answer

Compute it first, then check your number.

DL-C16-T02-002Exercise: Tradeoff

Enter 1 if dtype choice is a tradeoff, or 2 if one dtype is always best for every computation.

Your answer

Compute it first, then check your number.