Learning Rates and Clipping

The learning rate is the most visible control in a training loop. It scales the update.

update = learning_rate * gradient

A smaller learning rate makes each update cautious. A larger learning rate makes each update aggressive. Neither is automatically better.

Schedules change the learning rate during training. A common pattern is to start larger, then reduce it later so training can make broad progress first and finer adjustments later.

Gradient clipping is a different tool. It limits gradients or update sizes when they become too large. This can prevent one unstable step from damaging the parameters.

Clipping does not fix every problem. If gradients are consistently huge, the cause may be scale, initialization, data, loss design, or architecture. But clipping is a useful guardrail, especially in sequence models and large networks.