Chapter 8

Backpropagation Through Networks

Backward passes, chain rule through layers, weight and bias gradients, ReLU backward, and gradient checking.

Subject
Deep Learning
Lessons
7 lessons
Practice
20 checks
Review
3 pages

Introduction

Backpropagation is how gradients move through a neural network.

The previous chapter explained computation graphs and automatic differentiation. This chapter applies the same ideas to layers, activations, weights, and biases.

input xlinearrelulossforward: compute valuesbackward: send gradients
Backpropagation reuses the forward graph, sending gradients backward from the loss.

Backpropagation has two passes:

  • a forward pass computes values;
  • a backward pass sends gradients from the loss back through the graph.

By the end, you should be able to follow a small backward pass and explain what each gradient measures.

Lessons

Read these in order.

The chapter opening gives the main idea. Move through these lessons next; each page reuses ideas from the pages before it.

  1. 01
    Computation Graph Review

    Reading a network forward pass as the graph used by the backward pass.

  2. 02
    Backward Pass Overview

    Starting from the loss and sending upstream gradients backward.

  3. 03
    Chain Rule Through Layers

    Combining local derivatives through a small layer computation.

  4. 04
    Gradients With Respect to Weights

    How upstream gradients and inputs produce weight gradients.

  5. 05
    Gradients With Respect to Biases

    Bias gradients as upstream gradients accumulated across examples.

  6. 06
    Backprop Through ReLU

    How active ReLUs pass gradients and inactive ReLUs block them.

  7. 07
    Gradient Checking for Parameters

    Using finite differences to verify small backpropagation gradients.

  1. •
    Conclusion

    What Chapter 8 established before training loops and optimizers.

  2. •
    Review

    A compact review of backward flow, parameter gradients, ReLU backward, and gradient checks.

  3. •
    Exercises

    Chapter-level practice for backpropagation through small networks.

You are ready when

  • Trace a small backward pass from loss to earlier values.
  • Compute scalar weight and bias gradients.
  • Explain how ReLU affects gradient flow.
  • Use finite differences to check a small parameter gradient.

Where this leads

  • Training Loops and Optimizers
  • Initialization, Scale, and Normalization
  • PyTorch as Compression

Chapter progress