Module 7 · Lesson 7.4

Gradient Descent

Training a neural network means repeatedly nudging its weights a little downhill on the loss surface until the loss stops improving.

Intuition

Imagine a ball dropped on a bowl-shaped hill. It doesn't leap straight to the bottom — it rolls a little in whatever direction is steepest, then re-checks the slope from its new spot, then rolls a little more. Eventually it settles near the bottom.

Gradient descent does exactly this with a loss function. The gradient tells you the steepest uphill direction, so you step the opposite way — a small distance controlled by the learning rate — and repeat.

Core concept

Each step nudges a weight in the direction that decreases loss the fastest, by an amount scaled by the learning rate. Too large a learning rate and the ball overshoots the bottom and bounces around; too small and it takes forever to get there.

x=6 x=2 (min)
Loss L(x) = (x − 2)²start x = 6
Gradient dL/dx = 2(x − 2)2(4) = 8
Learning rate α0.25
Step = α · gradient0.25 × 8 = 2
Answer: x_new = x − step6 − 2 = 4

Interactive Lab

Set a learning rate, then press Step to watch the ball descend the bowl one gradient step at a time. Try a very large learning rate and see what happens near the bottom.

Iteration: 0 x = 6.00 Loss = 16.00

Key Idea

$$ w \leftarrow w - \alpha \nabla L(w) $$

This one update rule, applied to every weight after every batch, is the entire engine behind training a neural network. The gradient comes from backpropagation; the learning rate decides how confidently you act on it.

$w$A weight (or the whole vector of weights) being trained
$\alpha$The learning rate — how big a step to take
$\nabla L(w)$The gradient of the loss with respect to $w$ — the steepest uphill direction
$\leftarrow$"Update to" — replace the old weight with the new one

Think Further

  1. Why does the ball's step get smaller and smaller as it nears the bottom, even though the learning rate stays fixed?
  2. If the learning rate is too large, what do you predict happens to the ball as it crosses the minimum?
  3. How might a bowl with a much flatter, wider base change the number of steps needed to reach the bottom?
Show suggested answers
  1. The step size is α times the gradient, and the gradient itself shrinks as the curve flattens near the minimum — so even a constant α produces smaller and smaller moves.
  2. It overshoots the minimum and lands on the opposite side, possibly farther out than before; if the rate is large enough this can repeat and the ball bounces back and forth instead of settling.
  3. A flatter bowl means smaller gradients everywhere, so each step is smaller and it takes more iterations to travel the same horizontal distance to the bottom.