Core idea
PyTorch is a compression layer over deep-learning computation. It is introduced after the reader has seen the pieces it compresses.
Main translations
| Earlier idea | PyTorch name or pattern |
|---|---|
| numeric array | tensor |
| forward function | module or model call |
| trainable weight | parameter |
| scalar training signal | loss |
| chain-rule derivative bookkeeping | autograd |
| parameter update rule | optimizer |
| group of examples | batch from a data loader |
| learned tensor values | state dict |
Training step
optimizer.zero_grad()
pred = model(x_batch)
loss = loss_fn(pred, y_batch)
loss.backward()
optimizer.step()
Read the step as:
- clear old gradients
- predict
- measure error
- compute gradients
- update parameters
Important cautions
backward()computes gradients; it does not update parameters.optimizer.step()updates parameters; it does not compute the loss.- a module is a package around computation and trainable state.
- a data loader manages batches; it does not define the learning objective.
- saved state stores learned tensor values and related experiment state.
Before moving on
You should be able to read a small PyTorch training loop and explain each line without relying on the framework as a black box.