The Art of Simplifying Multivariate Gradients

Once you’ve written out the component-wise chain rule, you’re often left with nasty sums full of indices. The art of gradient derivation is recognizing patterns that collapse into clean matrix operations.

Systematic Simplification Process

When faced with a component-wise gradient, follow these steps:

Step 1: Write Out the Component Form

Start with the chain rule:

$$\frac{\partial L}{\partial {\theta }_{\text{indices}}}=\sum _{\text{all }f}\frac{\partial L}{\partial f_{\text{indices}}}\cdot \frac{\partial f_{\text{indices}}}{\partial {\theta }_{\text{indices}}}$$

Step 2: Identify Kronecker Deltas

Look for terms like $\frac{\partial W_{ij}}{\partial W_{\ell k}}$ and replace with ${\delta }_{i\ell }{\delta }_{jk}$.

Step 3: Collapse Sums Using Deltas

Use ${\sum }_k{a}_k{\delta }_{ik}=a_i$ to eliminate indices.

Step 4: Recognize Matrix Operation Patterns

Look at the structure of remaining indices:

• Single index pair → outer product
• Sum over middle index → matrix multiply
• Same indices → element-wise (Hadamard)

Step 5: Write Matrix Form

Translate the simplified component form into matrix notation.

Step 6: Verify Shapes

Check that all matrix dimensions are compatible!

Core Simplification Tools

Kronecker Delta (${\delta }_{ij}$)

Definition:

$${\delta }_{ij}=\{\begin{array}{ll}1& \text{if }i=j\\ 0& \text{if }i\ne j\end{array}$$

Power: It selects terms from sums and eliminates irrelevant indices. Key Property:

$$\sum _k{a}_k{\delta }_{ik}=a_i$$

The sum collapses to just the $i$-th term!

• notice how $a_k$ turns into $a_i$

Pattern 1: Derivatives of Parameters w.r.t. Themselves

$$\frac{\partial {\theta }_i}{\partial {\theta }_j}={\delta }_{ij}$$

For matrices:

$$\frac{\partial W_{ij}}{\partial W_{\ell k}}={\delta }_{i\ell }{\delta }_{jk}$$

Why it matters: When you see $\frac{\partial W_{ij}}{\partial W_{\ell k}}$ in a sum, it kills all terms except where $i=\ell$ and $j=k$.

Example: Linear layer $\mathbf{y}=W\mathbf{x}$ Component form:

$$y_i=\sum _j{W}_{ij}{x}_j$$

Derivative:

$$\frac{\partial y_i}{\partial W_{\ell k}}=\frac{\partial }{\partial W_{\ell k}}\sum _j{W}_{ij}{x}_j=\sum _j{x}_j{\delta }_{i\ell }{\delta }_{jk}={\delta }_{i\ell }{x}_k$$

The double Kronecker delta collapses the sum instantly!

Pattern 2: Identity Functions

For a function $\mathbf{y}=\mathbf{x}$ (identity):

$$\frac{\partial y_i}{\partial x_j}={\delta }_{ij}$$

In chain rule:

$$\frac{\partial L}{\partial x_j}=\sum _i\frac{\partial L}{\partial y_i}{\delta }_{ij}=\frac{\partial L}{\partial y_j}$$

This is useful for linear layers like ReLU.

Recognizing Matrix Products

I got a good number of them in Matrix and Component Forms

Component Pattern	Matrix Form	Fundamental Pattern
$C_{ij}=a_i{b}_j$	$C=\mathbf{a}{\mathbf{b}}^T$	Outer product
$y_i={\sum }_j{A}_{ij}{x}_j$	$\mathbf{y}=A\mathbf{x}$	Matrix-vector product
$Y_{ik}={\sum }_j{A}_{ij}{B}_{jk}$	$Y=AB$	Matrix-matrix product
$z_i=f(x_i)$	$\mathbf{z}=f(\mathbf{x})$	Element-wise unary operation
$z_i=x_i\circ y_i$	$\mathbf{z}=\mathbf{x}\odot \mathbf{y}$	Element-wise binary operation
$y={\sum }_i{x}_i$	$y=1^T\mathbf{x}$	Reduction operation
$Y_{ij}=X_{ji}$	$Y=X^T$	Transpose
${\sum }_k{a}_k{\delta }_{ik}$	$a_i$	Kronecker delta collapse

All of the gradient component simplifications come from this core set!

For example when I was deriving the weight gradient of a linear layer, I got to

$$\frac{\partial \mathcal{L}}{\partial W_{lk}}=\frac{\partial \mathcal{L}}{\partial y_l}\cdot x_k$$

This is an outer product!!!!!

$$\frac{\partial \mathcal{L}}{\partial W}=\frac{\partial \mathcal{L}}{\partial y}\ast x^T$$