Before differentiating anything, I want to see the actual shapes. Imagine a tiny grayscale image row with five pixels. We move a three-number filter across it, one pixel at a time. The filter produces three output values. Then we ask: if I change one input pixel or one filter weight slightly, which outputs move, and by how much? That question is the Jacobian. Backprop uses those answers to carry a loss gradient back through the layer.

In the earlier \(wx+b\) training loop, one parameter use produced one gradient contribution. A convolution reuses that parameter at many positions. This is the exact point where the scalar story becomes a kernel story. I will use a single-channel, single-filter, one-layer Conv1D. There is no hidden activation, normalization, or optimizer state in this first example.

1. Draw the input row, patch matrix, shared filter, bias and output shapes.
2. Calculate the first patch by hand, then repeat the identical operation at the other two positions.
3. Build the small Jacobians: which output depends on which input or parameter?
4. Send the loss gradient backward through those Jacobians; add every shared and overlapping contribution.
5. Update the filter and test whether the loss fell. Then train a second toy filter to recognize a visible pattern.

Step 1: draw the shapes before the calculus

We will use exactly the numbers from the first post: x=[1,2,3,4,5], k=[2,0,-1], b=1, stride one, no padding. Treat \(x\) as one row of five grayscale pixel values. This is a 1D teaching slice of an image, not a full 2D photograph. The filter has three taps, and the same taps are used at every position.

There is one input channel and one output channel. The conceptual tensor shapes are input \([N,C_{\rm in},L]=[1,1,5]\), weight \([C_{\rm out},C_{\rm in},K]=[1,1,3]\), bias \([1]\), and output \([1,1,3]\). If we remove the batch and channel dimensions for hand calculation, these become a five-element row, a three-element filter and a three-element result.

The three patches overlap. Writing each patch as a row makes a \(3\times3\) window matrix:

\[X_{\mathrm{win}}=\begin{bmatrix}x_0&x_1&x_2\\x_1&x_2&x_3\\x_2&x_3&x_4\end{bmatrix}=\begin{bmatrix}1&2&3\\2&3&4\\3&4&5\end{bmatrix},\qquad X_{\mathrm{win}}\in\mathbb R^{3\times3}.\]

Rows answer where the filter is; columns answer which tap of that filter is reading a value. The repeated \(x_2=3\) is not duplicated input data in the mathematical problem: three windows read the same input position. An implementation may generate these window values on demand rather than allocating this matrix.

Step 2: do one patch, then reuse the operation

Let \(i\) identify an output position and \(r\) a filter tap. With input length five and filter length three, valid stride-one output length is \(5-3+1=3\):

\[y_i=b+\sum_{r=0}^{2}k_r x_{i+r},\qquad i\in\{0,1,2\}.\]

y0 = 1*2 + 2*0 + 3*(-1) + 1 = 0
y1 = 2*2 + 3*0 + 4*(-1) + 1 = 1
y2 = 3*2 + 4*0 + 5*(-1) + 1 = 2
y  = [0, 1, 2]

Step 3: what is a Jacobian here?

For a tenth-grade intuition, a derivative asks: “If I nudge this one number, how much does that output change?” A Jacobian places all of those small-change answers in a table. Its rows are outputs; its columns are the input numbers we might nudge. For this linear layer, the answers do not depend on how large the nudge is, but we still use the derivative language because a real network chains many nonlinear operations.

Holding \(k\) and \(b\) fixed, the input-to-output Jacobian has shape \(3\times5\). Each row places the same filter under one patch:

\[J_x\equiv\frac{\partial y}{\partial x}=\begin{bmatrix}2&0&-1&0&0\\0&2&0&-1&0\\0&0&2&0&-1\end{bmatrix}.\]

Read its middle column, for \(x_2\): if that one pixel rises by a tiny \(\epsilon\), the outputs move by \([-\epsilon,0,2\epsilon]\). The first and third windows read \(x_2\), but the second window multiplies it by the zero middle tap. A zero Jacobian entry means no first-order influence through that particular path.

Holding \(x\) fixed instead, the filter-to-output Jacobian has shape \(3\times3\). It is exactly the patch matrix, because changing a filter tap changes every window that uses it. The bias-to-output Jacobian has shape \(3\times1\):

\[J_k\equiv\frac{\partial y}{\partial k}=X_{\mathrm{win}}=\begin{bmatrix}1&2&3\\2&3&4\\3&4&5\end{bmatrix},\qquadJ_b\equiv\frac{\partial y}{\partial b}=\begin{bmatrix}1\\1\\1\end{bmatrix}.\]

These are local Jacobians of the layer. Training does not normally build a giant dense Jacobian for an entire model. The next step is to use the arriving loss gradient to multiply these tables efficiently, usually as vector-Jacobian products.

3 × 5Three output values, five input pixels. A Jacobian row says which pixels affect one output; a column says which outputs one pixel can affect.

Step 4: start backprop at the loss

Give the three outputs a target t=[1,1,1]. To keep differentiation simple, use L = 0.5 * sum((y-t)^2). The factor 0.5 cancels the derivative's factor of two. The loss here is 1, and the gradient arriving at the convolution output is g = dL/dy = y-t = [-1,0,1].

Writing the loss and its derivative explicitly makes the first backward edge visible:

\[L=\tfrac12\sum_{i=0}^{2}(y_i-t_i)^2=\tfrac12\bigl[(-1)^2+0^2+1^2\bigr]=1,\qquadg_i\equiv\frac{\partial L}{\partial y_i}=y_i-t_i.\]

The three arriving gradients are \(g=[-1,0,1]\). A negative value at \(y_0\) says increasing that output would reduce this loss; a positive value at \(y_2\) says decreasing that output would reduce it. The middle output already matches its target, so it contributes zero for this example. Those signs are about this loss, not generic labels for good and bad features.

Before writing any matrix product, take the highlighted pixel \(x_2\). It can affect several outputs, and each output affects the loss. Multiply the two changes along each path and add the paths:

\[\frac{\partial L}{\partial x_2}=\sum_{i=0}^{2}\frac{\partial L}{\partial y_i}\frac{\partial y_i}{\partial x_2}=(-1)(-1)+(0)(0)+(1)(2)=3.\]

That one line is the chain rule for a fan-out: a pixel was read in overlapping windows, so the loss has multiple paths back to it. The matrix expression below performs that same path accounting for all inputs and parameters at once.

Now the chain rule has a compact matrix form. A Jacobian maps small changes forward; its transpose carries the scalar loss gradient backward:

\[\boxed{\nabla_x L=J_x^{\mathsf T}g},\qquad\boxed{\nabla_k L=J_k^{\mathsf T}g},\qquad\boxed{\frac{\partial L}{\partial b}=J_b^{\mathsf T}g}.\]

The bias gradient is a sum

The same bias was added to all three outputs. Each output therefore contributes its upstream gradient to that one parameter:

\[\frac{\partial y_i}{\partial b}=1\quad\Longrightarrow\quad\frac{\partial L}{\partial b}=\sum_{i=0}^{2}\frac{\partial L}{\partial y_i}\frac{\partial y_i}{\partial b}=\sum_{i=0}^{2}g_i=-1+0+1=0.\]

dL/db = g0 + g1 + g2 = -1 + 0 + 1 = 0

Zero here does not mean the model is correct. It means the bias contributions cancel for this particular input and target. Other parameters still have nonzero gradients.

Shared weights collect work from every window

For each filter tap, look at the input value that tap multiplied at each output position. Multiply by that output's gradient and sum over positions. For the first tap, the visited values were [1,2,3]; for the second, [2,3,4]; for the third, [3,4,5].

Tap \(k_r\) changes every output, but the amount depends on the input beneath that tap at each window:

\[\frac{\partial y_i}{\partial k_r}=x_{i+r}\quad\Longrightarrow\quad\boxed{\frac{\partial L}{\partial k_r}=\sum_{i=0}^{2}g_i x_{i+r}}.\]

dL/dk0 = -1*1 + 0*2 + 1*3 = 2
dL/dk1 = -1*2 + 0*3 + 1*4 = 2
dL/dk2 = -1*3 + 0*4 + 1*5 = 2
dL/dk  = [2, 2, 2]

This is the weight-sharing idea turned around. The filter appears once in the parameter table but participates in three forward computations. Its gradient is the sum of all three uses. If we updated the filter after each window independently, we would not be differentiating the complete loss as written.

Input gradients overlap

Now follow each output gradient back to the input positions it read. Output zero read x0,x1,x2, so g0=-1 contributes [-2,0,1] there. Output one has g1=0, contributing nothing. Output two read x2,x3,x4, so g2=1 contributes [2,0,-1] there. Position x2 is in both the first and last windows: 1+2=3.

Input \(x_j\) is read wherever a window satisfies \(j=i+r\). Its gradient sums exactly those paths:

\[\boxed{\frac{\partial L}{\partial x_j}=\sum_{\substack{i\in\{0,1,2\}\\0\le j-i<3}}g_i k_{j-i}}.\]

At the overlap \(j=2\), two nonzero paths survive: \(g_0k_2+g_1k_1+g_2k_0=(-1)(-1)+(0)(0)+(1)(2)=3\). These are contributions to the same input activation, so the backward kernel must add them. A write that overwrites the earlier contribution would produce an incorrect gradient.

from y0: [-2,  0, 1, 0,  0]
from y1: [ 0,  0, 0, 0,  0]
from y2: [ 0,  0, 2, 0, -1]
  dL/dx: [-2,  0, 3, 0, -1]

You may hear this input-gradient calculation called a transposed convolution. More precisely, it is the adjoint of this forward linear map under the chosen layout, stride, padding and boundary conventions. The word transposed refers to the operator, not to reversing the training process. The filter gradient above is a separate computation.

This also explains the apparent filter flip in some derivations. The forward operation here reads \(x_{i+r}\) with \(k_r\). The input-gradient expression asks which output positions read a given \(x_j\), so its indexing traverses the filter in the opposite spatial direction. Do not change the forward convention halfway through the derivation. A transposed convolution is not generally an inverse that recovers the original input.

Step 5: implementation views of the same gradient

The Jacobian view gives \(\nabla_kL=J_k^{\mathsf T}g=X_{\rm win}^{\mathsf T}g\). The same formula is a matrix multiplication over patches and may be mapped to GEMM. A separate backward kernel may visit the patches directly without allocating the unfolded matrix. Both must agree on the accumulation order and numerical tolerance.

The parameter gradient is then one matrix multiplication. The input-window gradient is another; its first and third rows both touch the original \(x_2\):

\[\nabla_k L=X_{\mathrm{win}}^{\mathsf T}g=\begin{bmatrix}2\\2\\2\end{bmatrix},\qquad\frac{\partial L}{\partial X_{\mathrm{win}}}=gk^{\mathsf T}=\begin{bmatrix}-2&0&1\\0&0&0\\2&0&-1\end{bmatrix}.\]

Reverse-unfolding the right-hand matrix scatters each row into its original window location and adds overlaps. The result is \(\nabla_xL=[-2,0,3,0,-1]\). This is the same vector as \(J_x^{\mathsf T}g\). The two routes are not two different derivatives: one thinks in terms of the full input Jacobian, while the other thinks in terms of patch gradients and a reverse mapping, often called col2im.

Step 6: perform one training step

We have the trainable parameter gradients: \(\nabla_kL=[2,2,2]\) and \(\partial L/\partial b=0\). Suppose stochastic gradient descent uses learning rate \(\eta=0.01\). The input gradient is available for an earlier layer, but \(x\) itself is fixed example data here, so we do not update it.

\[k^{\mathrm{new}}=k-\eta\nabla_kL=[1.98,-0.02,-1.02],\qquad b^{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b}=1.\]

Run the same forward operator again with the new filter:

\[y^{\mathrm{new}}=[-0.12,0.82,1.76],\qquadL^{\mathrm{new}}=\tfrac12\bigl[(-1.12)^2+(-0.18)^2+(0.76)^2\bigr]=0.9322.\]

The loss fell from \(1\) to \(0.9322\) in this one step. This is a local check, not a guarantee that every learning rate works: with \(\eta=0.1\), the same update overshoots and the loss rises to \(5.02\). The gradient gives a local direction; the learning rate decides how far we travel.

1 → 0.9322One Conv1D layer, one squared-error loss, one SGD step at learning rate 0.01. This is the training loop, not only a derivative table.

Step 7: what can a tiny filter learn?

The first exercise used one row and a deliberately simple target so that every derivative fit on the page. Now reset the parameters and ask a different question: can one shared three-tap filter learn to mark a brightness change along a row of pixels? A row is still a 1D slice of an image; a real image has a second spatial axis and usually colour channels.

Use all \(2^5=32\) possible five-pixel black-and-white rows as the training set. There are three valid windows per row, so the loss sees \(M=32\times3=96\) window responses. We define the teaching target at each window as right pixel minus left pixel:

\[t_i(x)=x_{i+2}-x_i,\qquad\hat t_i(x)=k_0x_i+k_1x_{i+1}+k_2x_{i+2}+b,\qquadL=\frac{1}{2M}\sum_{m=1}^{M}(\hat t_m-t_m)^2.\]

That target makes the desired meaning precise: zero on a flat patch, positive when brightness rises from left to right, negative when it falls. The middle pixel is available to the learner, but this particular target does not depend on it. Start with \(k=[0,0,0]\) and \(b=0\). For each training step, calculate all predictions, send the mean residual backward, sum the weight contributions across all 96 windows, and update the four trainable numbers:

\[\nabla_k L=\frac{1}{M}X_{\rm train}^{\mathsf T}(\hat t-t),\qquad\frac{\partial L}{\partial b}=\frac{1}{M}\sum_m(\hat t_m-t_m),\qquad(k,b)\leftarrow(k,b)-\eta\left(\nabla_k L,\frac{\partial L}{\partial b}\right).\]

With \(\eta=0.2\) for 300 full-batch gradient steps, the retained run goes from loss \(0.25\) to about \(1.1\times10^{-14}\). Its filter is approximately \([-1,0,1]\), with bias approximately zero. On a longer row \([0,0,0,1,1,1,1,0,0]\), its seven outputs are approximately \([0,1,1,0,0,-1,-1]\): two windows notice the rising boundary, two notice the falling boundary, and the flat windows return zero.

from itertools import product
import numpy as np

rows = np.array(list(product((0., 1.), repeat=5)))
patches = np.lib.stride_tricks.sliding_window_view(rows, 3, axis=1)
patches = patches.reshape(-1, 3)  # 32 rows * 3 windows = 96 patches
target = patches[:, 2] - patches[:, 0]
k, b = np.zeros(3), 0.

for step in range(300):
    error = patches @ k + b - target
    dk = patches.T @ error / len(patches)
    db = error.mean()
    k -= 0.2 * dk
    b -= 0.2 * db

print(k, b)  # approximately [-1, 0, 1], 0
probe = np.array([0., 0., 0., 1., 1., 1., 1., 0., 0.])
probe_patches = np.lib.stride_tricks.sliding_window_view(probe, 3)
print(probe_patches @ k + b)  # approximately [0, 1, 1, 0, 0, -1, -1]

This is a synthetic supervised task: I chose the target formula, so the experiment proves that the gradient machinery can recover that known local rule from examples. It does not prove that the layer independently discovered an edge detector from image labels or learned a useful feature for a real vision model. In real training, the target and upstream gradient come from a broader task and many layers; the learned filter is judged by held-out task performance, not by resemblance to a hand-picked kernel.

For a batch, sum parameter gradients over examples as well as spatial positions. For multiple input and output channels, the same rule sums over the channel indices that contributed to each result. A depthwise filter has fewer channel connections, so its gradient follows the same chain rule over a smaller connectivity pattern. The tensor shapes and summation axes change; the principle does not.

From one row to a full Conv2D tensor

An image or feature map adds a second spatial coordinate and usually channels. For batch example \(n\), output channel \(o\), input channel \(c\), output location \((u,v)\), and filter offsets \((r,s)\), valid stride-one cross-correlation is:

\[Y_{n,o,u,v}=b_o+\sum_c\sum_r\sum_sK_{o,c,r,s}\,X_{n,c,u+r,v+s}.\]

Suppose later layers send \(G_{n,o,u,v}=\partial L/\partial Y_{n,o,u,v}\) back to this convolution. The bias adds all uses of an output channel; a kernel element adds all batch and spatial uses of its particular input-channel connection:

\[\frac{\partial L}{\partial b_o}=\sum_{n,u,v}G_{n,o,u,v},\qquad\frac{\partial L}{\partial K_{o,c,r,s}}=\sum_{n,u,v}G_{n,o,u,v}X_{n,c,u+r,v+s}.\]

The input gradient reverses the read relationship. With out-of-range \(G\) treated as zero:

\[\frac{\partial L}{\partial X_{n,c,p,q}}=\sum_o\sum_r\sum_sG_{n,o,p-r,q-s}\,K_{o,c,r,s}.\]

Compare these three equations with the Conv1D example: the derivatives did not acquire new rules. We added indices for batch, channels and the second spatial axis. Stride, dilation, padding, groups and nontrivial layouts change which indices are legal and where values land; they are part of the operator contract, not details to hand-wave away. For a grouped or depthwise convolution, only the declared channel connections enter these sums.

For a kernel engineer, these formulas also describe different work patterns. \(\partial L/\partial K\) is a reduction over examples and output positions; parallel workers must combine their partial sums. \(\partial L/\partial X\) accumulates contributions where output windows overlap; a kernel can scatter-add or arrange an equivalent gather, but it cannot silently discard a contribution. The order of floating-point additions may differ between those implementations, so a numerical tolerance and a reference layout are part of a credible backward-kernel test. The mathematics identifies the answer; the schedule determines how we reach it on a CPU or accelerator.

If this layer feeds an activation \(a=\phi(Y)\), the gradient entering these formulas is first transformed by the chain rule: \(G=(\partial L/\partial a)\odot\phi'(Y)\). For ReLU, positive pre-activations pass their gradient and negative ones block it; the exact choice at zero is an implementation convention. In a deeper network, the computed \(\partial L/\partial X\) is passed to the preceding layer. That is how one trained convolution joins a full backpropagation graph.

Our fully worked numbers remain the simple Conv1D fixture. The Conv2D expressions are the generalization under the stated valid, stride-one convention, not a claim that the example script has tested all image layouts and boundary modes.

Check the math rather than trusting the diagram

A finite-difference check perturbs one parameter by a small epsilon, reruns the forward loss and estimates its derivative from the two neighbouring losses. It is not a replacement for an independent framework oracle or a full model test, but it is an excellent first check for a small kernel.

import numpy as np

x = np.array([1., 2., 3., 4., 5.])
k = np.array([2., 0., -1.])
b = 1.
target = np.ones(3)

def forward(signal, weight, bias):
    windows = np.lib.stride_tricks.sliding_window_view(signal, 3)
    return windows @ weight + bias

def loss(signal, weight, bias):
    error = forward(signal, weight, bias) - target
    return 0.5 * np.sum(error * error)

y = forward(x, k, b)
g = y - target
windows = np.lib.stride_tricks.sliding_window_view(x, 3)
dk = windows.T @ g
db = g.sum()
dx = np.zeros_like(x)
for i, upstream in enumerate(g):
    dx[i:i + 3] += upstream * k

np.testing.assert_allclose(y, [0., 1., 2.])
np.testing.assert_allclose(dk, [2., 2., 2.])
np.testing.assert_allclose(dx, [-2., 0., 3., 0., -1.])
np.testing.assert_allclose(db, 0.)

eps = 1e-5
plus, minus = k.copy(), k.copy()
plus[0] += eps
minus[0] -= eps
finite_difference = (loss(x, plus, b) - loss(x, minus, b)) / (2 * eps)
np.testing.assert_allclose(finite_difference, dk[0], atol=1e-8)

eta = 0.01
new_k, new_b = k - eta * dk, b - eta * db
np.testing.assert_allclose(loss(x, new_k, new_b), 0.9322)
print("before:", loss(x, k, b), "after:", loss(x, new_k, new_b))

The code above is enough to reproduce the article's complete training step with NumPy. My retained local fixture additionally checks every input, weight and bias gradient by centred differences before drawing the figure. In a production kernel, I would also test boundary shapes, multiple channels, padding, stride, dilation, non-contiguous layouts and numerical precision separately. This small fixture proves only the formula shown here.

It also asserts the SGD update above. On this fixture the largest absolute difference between an analytic gradient and its finite-difference estimate is about \(5.8\times10^{-11}\). That tight agreement is evidence for this scalar fixture, not a promise of bit-identical results for larger floating-point kernels.

What the training run actually tells us

The hand calculation tells me that the chain rule reaches the right shared parameters and overlapping inputs. The first SGD step tells me a small update can reduce its stated loss. The second experiment shows a filter recovering a known local difference rule from an informative synthetic dataset. None of those is a full-image classifier, proof of generalization to photographs, or a complete training claim for CKE.

A real network continues the chain: an activation transforms the arriving gradient, an earlier layer receives \(\partial L/\partial X\), and the optimizer applies the accumulated parameter gradients under its own update rule. This is the same chain rule as the original scalar \(wx+b\) training loop, now with shared parameters and spatial overlap.

That is the useful mental model for me as a kernel engineer: forward convolution gathers local values and reuses a filter; backward convolution distributes gradients and accumulates every shared use. The Jacobian tells us which paths exist. The data and loss tell us which direction to train. Shape, layout, accumulation order and numerical contract make the implementation hard, but the calculus remains one multiply and one sum at a time.

Related Notes and Sources

• Convolution Is Wx+B Moving Across a Signal - part one, with the forward windows, image filters and GEMM view.
• Backpropagation From wx+b: The Smallest Training Loop
• PyTorch Conv1d reference - its stated forward convention is valid cross-correlation.
• PyTorch Conv2d reference - channel and spatial shapes for the 2D operator.
• PyTorch ConvTranspose1d reference - the input-gradient relationship and why this is not an inverse convolution.
• PyTorch vector-Jacobian products - the reverse-mode operation behind the compact Jacobian notation.
• A Guide to Convolution Arithmetic for Deep Learning - stride, padding and transposed-convolution geometry.
• CKE Audio Kernels Deep Dive