NaNs and Infinities

NaN means "not a number." Infinity means a value exceeded representable finite range.

In training, they are alarms.

Common causes include:

• division by zero
• log of zero
• overflow in exponentials
• too-large learning rate
• unstable normalization
• invalid data values

Once NaN appears, it can spread through tensors quickly. A single non-finite value can contaminate loss, gradients, and parameters.

A practical debugging checklist:

check input values
check loss value
check logits before softmax
check gradient norms
check parameter values
check learning rate