Lagrange Multipliers — Where Gradients Align

Intuition

In Lesson 9.2, you found the best point along a fence by watching where the fence just barely touches the smallest possible contour ring, instead of slicing through it. "Just touching" has a precise meaning: at that point, the direction the contour ring wants to shrink (the gradient of f) and the direction the fence runs are perfectly perpendicular to each other — which means the gradient of f and the gradient of the fence itself point along the exact same line.

Core concept

The gradient of a function is an arrow pointing in the direction of steepest increase, at every point. At a constrained optimum, the gradient of your objective f and the gradient of your constraint g point in the same (or exactly opposite) direction — they're parallel. If they weren't parallel, you could always slide a little further along the fence and find a smaller value of f, so the point couldn't be optimal yet. That parallel condition is the entire idea behind Lagrange multipliers: a scaling factor λ that expresses exactly how many "units of f" you'd gain for loosening the constraint by one unit.

f(x, y) = x² + y², g(x, y) = x + ysame as Lesson 9.2

∇f = (2x, 2y) at (2, 2)(4, 4)

∇g = (1, 1)constant everywhere

Check ∇f = λ∇g(4, 4) = 4 · (1, 1)

Answerλ = 4, arrows are parallel

Notice this confirms exactly the point Lesson 9.2 found by testing the contour rings directly: (2, 2). Lagrange multipliers give you a way to find that point algebraically, by solving ∇f = λ∇g together with the constraint, instead of checking every point on the fence by hand.

Interactive Lab

Slide the point along the fence and watch the two gradient arrows. Notice they only line up into one straight arrow at a single spot.

Position along fence: 0.50

Angle between gradients: 0.0° — arrows are parallel, this is the optimum.

Key Idea

$$ \nabla f(x) = \lambda \, \nabla g(x), \qquad g(x) = c $$

At the constrained optimum, the gradient of the objective is some scalar multiple λ of the gradient of the constraint. Solving this vector equation alongside the original constraint equation pins down the exact optimal point, without ever having to check contour rings by eye. The value λ itself is useful too: it measures how much f would improve if the constraint c were relaxed by a small amount, which is exactly why it's called a "multiplier."

$\nabla f$The gradient of the objective — an arrow pointing toward steepest increase in f

$\nabla g$The gradient of the constraint function, perpendicular to the fence line

$\lambda$The Lagrange multiplier — how many units of f you'd gain per unit of relaxed constraint

$c$The fixed value the constraint g must equal

Think Further

If ∇f and ∇g point in exactly opposite directions at some point on the fence (λ is negative), is that point still a valid candidate for the optimum? What does the sign of λ tell you?
Support Vector Machines find the widest possible margin between two classes of points while requiring every point to be correctly classified — a constrained optimization problem. Based on this lesson, what role do you think the "support vectors" (the points touching the margin) play, compared to points far from it?
What would it mean, geometrically, if at your candidate optimal point the two gradient arrows were perpendicular instead of parallel?

Show suggested answers

Yes — the sign of λ only tells you which direction relaxing the constraint would push f (increase or decrease); the parallel condition itself, not the sign, is what marks a candidate optimum. The sign becomes important when you also have inequality constraints, since it can then indicate whether the constraint is "active" and helpful or not.
The support vectors are exactly the points sitting on the margin boundary — they're the ones actively "touching the fence," so their gradients are what get balanced against the objective's gradient in the Lagrange condition. Points far from the margin play no role in pinning down the optimal boundary, much like fence points other than (2, 2) played no role in Lesson 9.2's answer.
Perpendicular gradients would mean the constraint direction and the direction of steepest increase in f share no component in common, which would mean you could still slide along the fence in some direction without changing f at all to first order — a sign that you have not yet found the true tangency point, so this typically wouldn't be a valid optimum unless f were locally flat.