Gradient Descent

Imagine standing on a hillside in thick fog, trying to reach the lowest point in the valley by feel alone. You cannot see the bottom, but you can feel which way the ground slopes down right where you are standing. So you take a small step that way, feel the slope again, and repeat. Gradient descent does exactly this for a math function. It repeatedly nudges a number in the direction that makes a 'cost' smaller, a fixed number of times, until it settles near the lowest point.

1. 1Start with an initial guess for x, and pick a learning rate — how big a step to take each time.
2. 2Compute the gradient at the current x. This is a number that tells you which direction is 'uphill' and how steep it is.
3. 3Update x by moving a small step opposite the gradient: x = x - learning_rate * gradient.
4. 4Repeat the last two steps a fixed number of times. Each step should land a little closer to the minimum.

This is the engine behind training most machine learning models, including linear regression and neural networks. Use it anywhere you need to find the input that minimizes some 'error' or 'cost', and there is no simple formula to jump straight to the answer.

• ⚠Do not pick a learning rate that is too large. The steps overshoot the minimum and can bounce around or fly off to infinity instead of settling down.
• ⚠Do not pick a learning rate that is too small. The steps crawl toward the minimum so slowly that progress barely happens in a reasonable number of steps.