A ball three steps down a loss bowl: the tangent is the gradient, the horizontal jump is the learning rate's share of it, the dashed circle is where the update lands. Evidence: loss falls under a cosine schedule; the ladder is four optimizers by steps to converge on the same bowl.
Feel which way is down, take a small step
You're on a big hill in thick fog and want to reach the bottom. You can't see it, but you can feel which way the ground tilts under your feet. So you take a small step downhill, feel again, step again. If your steps are tiny it takes forever; if they're huge you might leap right over the bottom and up the other side.
You never need to see the bottom — only which way is down right here.
θₜ₊₁ = θₜ − η · ∇L(θₜ)
SGD
1951
~10K STEPS
Momentum
1964
~3K STEPS
RMSProp
2012
~1K STEPS
Adam
2014
~300 STEPS
Step bars are log-scale, fewer is better — each step is 10×