Review

Use this page to review the chapter before doing the exercises.

Key Ideas

  • Average slope is change in output divided by change in input. Ask: what happened over this interval?
  • A derivative is local rate of change. Ask: what happens near this input?
  • A partial derivative is one-input sensitivity. Ask: what if this input changes and the others stay fixed?
  • A gradient is a vector of partial derivatives. Ask: what is the local sensitivity vector?
  • A directional derivative is change along a chosen unit direction. Ask: what if we move this way?
  • A Jacobian is a derivative table for vector outputs. Ask: how do all outputs respond to all inputs?
  • A Hessian is a second-derivative table. Ask: how is the gradient changing?
  • The chain rule multiplies local sensitivities through composition. Ask: how does change pass through this path?
  • A computation graph is a dependency map of values and operations. Ask: which values depend on which earlier values?
  • Backpropagation is the chain rule applied backward through a graph. Ask: how did earlier values affect the loss?

Formulas to Remember

Average slope:

ΔyΔx\frac{\Delta y}{\Delta x}

Local approximation:

Δf≈f′(x)Δx\Delta f \approx f'(x)\Delta x

Gradient:

∇f=[∂f/∂x1,…,∂f/∂xn]\nabla f = [\partial f/\partial x_1,\ldots,\partial f/\partial x_n]

Directional derivative:

Duf(x)=∇f(x)⋅uD_u f(x) = \nabla f(x) \cdot u

Chain rule:

dydx=dydgdgdx\frac{dy}{dx} = \frac{dy}{dg}\frac{dg}{dx}

Gradient descent:

wnew=w−η∇L(w)w_{new} = w - \eta \nabla L(w)

Useful Distinctions

  • Average slope uses two points; a derivative is local at one point.
  • A partial derivative is a derivative for one input of a multi-input function.
  • The gradient points uphill; descent usually uses the negative gradient.
  • A gradient is for scalar output; a Jacobian is for vector output.
  • A Jacobian tracks first derivatives; a Hessian tracks second derivatives.
  • Backpropagation is organized chain rule over a graph.
  • A derivative gives a nearby estimate, not a global promise.

Checks Before You Move On

  • Treating a derivative as a global description of the whole function.
  • Forgetting that partial derivatives hold other variables fixed.
  • Comparing gradient components by sign instead of magnitude.
  • Moving with the gradient when the goal is to decrease loss.
  • Forgetting that a directional derivative usually uses a unit direction.
  • Thinking a zero gradient always means the best possible point.
  • Treating backpropagation as separate from the chain rule.
  • Forgetting that branch contributions are added during backpropagation.
  • Trusting an autodiff result without checking shapes and dependencies.
  • Forgetting that the Jacobian is the local linear map for vector-output changes.

Mental Model

Calculus tells how quantities change.

Training uses that information to adjust parameters.

Backpropagation is the bookkeeping that carries those local change signals through a computation graph.