Stop-Gradient and Detach

Sometimes we want to use a value in the forward pass but stop gradients from flowing through it.

This idea is often called stop-gradient or detach.

Suppose:

y = x + stop_gradient(x)

In the forward pass, this behaves like:

y = x + x = 2x

But for gradients, only the first x path counts:

dy/dx = 1

The stopped path contributes no gradient.

Why this exists

Stopping gradients is useful when a value should be treated as fixed for part of a computation. It is a practical control over the graph, not a new mathematical function.

This chapter only introduces the idea. Later topics can use it when the need is concrete.

Exercise: Forward with stop-gradient

Let x = 4 and y = x + stop_gradient(x). What is the forward value of y?

Compute it first, then check your number.

HintForward still uses the value

Stop-gradient affects gradient flow, not the forward number.

SolutionWork it out

In the forward pass, stop_gradient(x) has value x, so y = 4 + 4 = 8.

Exercise: Gradient with stopped path

For y = x + stop_gradient(x), what is dy/dx?

Compute it first, then check your number.

HintIgnore stopped path for gradient

The second path contributes zero gradient.

SolutionWork it out

The first x contributes derivative 1. The stopped path contributes 0, so dy/dx = 1.