Review

Key Ideas

  • Objective: quantity optimized by training.
  • Loss: objective usually minimized.
  • Metric: number used to evaluate behavior, not always the training loss.
  • Gradient descent: update opposite the gradient.
  • Mini-batch: small sample used to estimate an update.
  • Learning rate: scale of the update step.
  • Momentum: memory accumulated from recent gradients or updates, depending on the stated sign convention.
  • Adaptive method: optimizer that changes effective step sizes.
  • Convexity: special case where local minima are global.
  • Regularization: an explicit or implicit preference among fitted solutions.
  • Stability check: inspection for numerical training failure.

The important habit is to keep the pieces separate. A loss is not the whole task. A gradient is not the whole landscape. A learning rate is not a direction. An optimizer choice is not evidence of generalization.

Formulas to Remember

Squared loss:

L=(y^−y)2L = (\hat{y} - y)^2

Gradient descent:

wnew=w−η∇L(w)w_{\text{new}} = w - \eta \nabla L(w)

Regularized objective:

objective=data loss+λ⋅penalty\text{objective} = \text{data loss} + \lambda \cdot \text{penalty}

How To Read A Training Loop

When you see training code, identify these pieces:

  • the objective or loss being reduced
  • the data used for each update
  • the optimizer and update rule
  • the learning rate and schedule
  • any momentum or adaptive scaling
  • any regularization or penalty
  • stability checks such as loss, gradient norm, and update size

This turns a training loop from a black box into a set of choices.

Common Patterns

  • Mini-batch training: cheaper noisy estimate of the full gradient.
  • Learning-rate schedule: larger or smaller steps at different training stages.
  • Momentum: smooth recent directions.
  • Adam, RMSProp, and Adagrad: adapt effective step sizes from gradient history.
  • Weight decay or penalties: prefer some parameter patterns over others.
  • Gradient norm checks: detect unstable updates.

Checks Before You Move On

  • Optimizing a loss that does not match the desired behavior.
  • Moving with the gradient when minimizing.
  • Choosing a learning rate without checking loss behavior.
  • Treating adaptive optimizers as a substitute for understanding scale.
  • Assuming convex guarantees apply to most deep networks.
  • Treating regularization as a guarantee of generalization.
  • Ignoring validation and generalization.
  • Treating numerical failures as only modeling failures.
  • Treating NaN as a diagnosis instead of a symptom.
  • Forgetting that a useful local direction can become a bad finite step.
  • Assuming an adaptive optimizer removes the need to reason about scale.

Mental Model

Optimization is a loop: measure, compute direction, step, and check.