Chapter 7

Computation Graphs and Automatic Differentiation

Computation graphs, forward values, local derivatives, gradient accumulation, finite-difference checks, and autodiff modes.

Subject
Deep Learning
Lessons
8 lessons
Practice
22 checks
Review
3 pages

Introduction

Training needs gradients. A gradient tells how a small change in a value would change the loss.

Automatic differentiation, or autodiff, is the machinery that computes those gradients from the computation the model actually performed.

It is not symbolic algebra. It is not numerical guessing. It is organized chain rule applied to a recorded computation.

xwbxwzL = z^2
A computation graph records how forward values were produced, so derivatives can follow the same dependencies backward.

This chapter explains what must be recorded before gradients can be computed:

  • the operations;
  • the forward values;
  • the local derivatives;
  • how gradients accumulate;
  • where gradient flow can be stopped;
  • how finite differences can check a gradient;
  • why reverse mode is useful for neural networks.

No framework is needed yet. The examples are scalar and hand-computable.

Lessons

Read these in order.

The chapter opening gives the main idea. Move through these lessons next; each page reuses ideas from the pages before it.

  1. 01
    Computation Graphs

    Recording how values depend on earlier values in a forward computation.

  2. 02
    Forward Values

    Why derivative rules need the actual values from the forward pass.

  3. 03
    Local Derivatives

    Using small derivative rules as pieces of a larger chain-rule computation.

  4. 04
    Gradient Accumulation

    Adding gradient contributions when a value affects the loss through multiple paths.

  5. 05
    Stop-Gradient and Detach

    Using a value in the forward pass while blocking selected gradient paths.

  6. 06
    Finite-Difference Checks

    Estimating derivatives by nearby function evaluations as a gradient check.

  7. 07
    Forward Mode and Reverse Mode

    Two ways to organize chain rule through a computation.

  8. 08
    What Autodiff Automates

    Derivative bookkeeping, not architecture choice or modeling judgment.

  1. •
    Conclusion

    What Chapter 7 established before backpropagation through networks.

  2. •
    Review

    A compact review of computation graphs, local derivatives, accumulation, and autodiff modes.

  3. •
    Exercises

    Chapter-level practice for computation graphs and autodiff.

You are ready when

  • Trace a small computation graph.
  • Compute local derivatives and chain-rule products.
  • Explain gradient accumulation and stop-gradient.
  • Distinguish autodiff from symbolic differentiation and finite differences.

Where this leads

  • Backpropagation Through Networks
  • Training Loops and Optimizers
  • PyTorch as Compression

Chapter progress