Module 7 · Lesson 7.3

The Multivariable Chain Rule

When one input feeds an output through more than one route, its total effect is the sum of its effect along each route — this is the rule that makes backpropagation work.

Intuition

In single-variable calculus, the chain rule threads through one gear at a time: turn the first gear, it turns the second, which turns the third. But a shared weight in a neural network often drives more than one downstream path at once — like one gear meshed with two other gears side by side.

If you nudge that shared input, every path it feeds nudges a little too, and each of those nudges pushes the final output. The multivariable chain rule says: don't pick one path — add up the influence from every path the input can reach the output through.

Core concept

Suppose an output $L$ depends on two intermediate values $a$ and $b$, and both $a$ and $b$ depend on the same shared variable $w$. Changing $w$ changes $a$, which changes $L$ — and separately changes $b$, which also changes $L$. The total sensitivity of $L$ to $w$ is the sum of both chains, not just one of them.

w a b L
w2
a = w²4
b = w³8
∂a/∂w = 2w4
∂b/∂w = 3w²12
Answer: ∂L/∂w = 4 + 1216

Interactive Lab

Drag the slider to change $w$ and watch both paths update. The bars show how much each path contributes to the total gradient — notice the contributions add, they don't cancel or average.

Path through a = w² ∂L/∂a · ∂a/∂w = 1 × 4.0 = 4.0
Path through b = w³ ∂L/∂b · ∂b/∂w = 1 × 12.0 = 12.0
Total: ∂L/∂w = 16.0

Key Idea

$$ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial a}\frac{\partial a}{\partial w} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial w} $$

This is exactly what a neural network does during backpropagation: a weight often feeds into several neurons or is reused across several computations, so the gradient flowing back to it is the sum of the gradients arriving along every path it touched.

$L$The final output (for a network, typically the loss)
$w$The shared input whose total effect we want
$a, b$Intermediate values that both depend on $w$ and both feed $L$
$\partial L/\partial a$How sensitive $L$ is to $a$, holding everything else fixed
$\partial a/\partial w$How sensitive $a$ is to $w$
$+$Contributions from every path are added, never averaged

Think Further

  1. If a third path through a new variable c also depended on w and fed into L, how would the formula change?
  2. What would it mean, physically, if one path's contribution to ∂L/∂w were positive and another path's were negative?
  3. Why does it make sense that a weight shared across many neurons in a wide layer tends to accumulate a larger gradient than a weight used in only one place?
Show suggested answers
  1. You'd add one more term to the sum: ∂L/∂w = ∂L/∂a·∂a/∂w + ∂L/∂b·∂b/∂w + ∂L/∂c·∂c/∂w. Every additional path the shared variable reaches the output through adds one more term.
  2. It would mean increasing w pushes L up through one route while pulling it down through the other — the two effects partially cancel, and the net gradient reflects whichever pull is stronger.
  3. Each connection the weight participates in contributes its own term to the sum, so a weight wired into more downstream paths has more terms adding into its total gradient.