Lost on a foggy hillside, you can't see the bottom but you can feel which way is downhill and step that way, over and over. Gradient descent trains a model the same way — nudging its settings downhill on the error slope until it reaches the low point.
Learning as minimizing error
A model 'learns' by adjusting its parameters to reduce a loss function — a number measuring how wrong its predictions are. Gradient descent does this by computing the gradient (the direction of steepest increase of the loss) and stepping the parameters in the opposite direction, repeatedly, until the loss stops improving.
Each step's size is set by the learning rate. Too large and the training overshoots and diverges; too small and it crawls. Choosing it well is central to training.
Picture a ball rolling down a valley: the gradient points uphill, so we move downhill; the step size is the learning rate.
- At the current parameters the gradient of the loss points 'uphill'.
- Recall gradient descent's rule.
- State the direction to update the parameters.
What you should see: Move in the direction opposite the gradient — downhill — to reduce the loss; repeat until it converges.