In the first two parts of this series I took convolution from wx+b to a moving window, then followed its shared weights backward through a gradient. Now I want to see what happens when the window does not touch every adjacent sample. This is dilated convolution. It is an old idea that turns out to be useful in a modern speech generator, and CKE's recent Kokoro work gives me actual kernels and circuit tests to examine rather than just a diagram in a paper.

The specific example here is Kokoro's iSTFTNet-style generator. This is the text-to-speech output side: it helps turn already-computed acoustic features into a waveform. It is not the FastConformer encoder used for transcription by Parakeet, and it would be misleading to say all audio models use this particular residual block. I will cover that encoder separately.

Before the model: what is audio in a computer?

Sound in air is a changing pressure. A microphone and an analog-to-digital converter give us a sequence of measurements, x[0], x[1], .... If a recording has N samples at f_s samples per second, its duration is N/f_s seconds. At 16,000 samples per second, one sample is 62.5 microseconds apart from the next. A sample tells me the measured amplitude at one instant; it does not tell me whether I am hearing a vowel, a click, or background noise. Those are patterns across time.

Here is a synthetic 0.2-second signal I made from 220 Hz and 440 Hz tones, an amplitude envelope, and a brief transient. It is not speech, and its frequencies are just convenient teaching choices. The top plot shows the whole sampled sequence. The second magnifies five milliseconds and overlays a nine-sample local average. The bottom plot reduces each 100-sample chunk to a toy energy value, then shows how three taps can visit neighbouring frames or more widely separated frames. Those energy values are not Kokoro's learned features; they are a bridge from a waveform to the idea of a time-indexed feature map.

The nine-sample average asks a deliberately simple local question: m[t] = (x[t-4] + ... + x[t+4])/9, treating out-of-range samples as zero. It smooths rapid changes. A difference filter could instead emphasize a change; a learned filter may mix several channels and learn very different responses. The common operation is a weighted sum over a window that moves. The signal-processing version of convolution often starts with a chosen filter; a neural-network Conv1D learns its weights. In the neural-network convention used below, the stored taps are applied in order as cross-correlation, not mathematically reversed convolution.

For a three-tap filter on a feature sequence h[t], I can write y[t] = w[0]h[t-d] + w[1]h[t] + w[2]h[t+d] + b. At d=1 it reads adjacent positions; at d=4 it reads positions four feature steps apart. Same three weights, a wider span. That is why dilation is useful in this particular speech generator: its learned feature maps carry structure at several temporal scales, and the residual blocks need local and wider context. It does not mean convolution is required to represent digital audio or that four feature steps equal a fixed number of waveform samples throughout Kokoro's upsampling graph.

I kept the plot reproducible. The earlier convolution article develops the moving-window arithmetic; this short version builds the same progression from sound to a toy feature timeline. The retained plotting script also writes a PCM16 WAV so the numerical signal is inspectable, but the plots alone make no speech-quality claim.

import numpy as np

fs = 16_000
t = np.arange(int(0.20 * fs)) / fs
envelope = np.minimum(t / 0.02, 1.0) * np.exp(-3 * t)
x = envelope * (0.58 * np.sin(2 * np.pi * 220 * t)
              + 0.22 * np.sin(2 * np.pi * 440 * t))
x += 0.16 * np.exp(-((t - 0.12) / 0.0005) ** 2)

local_average = np.convolve(x, np.ones(9) / 9, mode="same")
toy_frames = np.sqrt(np.mean(x.reshape(32, 100) ** 2, axis=1))
weights = np.array([1.0, 0.0, -1.0])
for dilation in (1, 4):
    positions = 16 + dilation * np.array([-1, 0, 1])
    print(dilation, positions, weights @ toy_frames[positions])
# These are illustrative energy frames, not Kokoro's feature tensors.

First, see the whole Kokoro graph

I find it easier to understand a kernel once I know which problem the entire model is solving. Kokoro takes prepared phoneme IDs and a selected voice-style vector, then predicts how long each phoneme lasts. Let T be the number of input IDs and A the sum of their checked durations in acoustic frames. It must produce two separate feature streams at that new length. One stream carries the duration/prosody information; another carries the text features. They use the same timing map but do not become the same tensor. This is the important architectural boundary that a simple “text → audio” diagram hides. See the upstream model call graph and CKE's pinned graph inventory for the source-level version.

In order, the model does the following:

1. Prepare the input. A text frontend turns supported written text into phoneme IDs. In the pinned model graph those IDs are already prepared; a dictionary-only native frontend is a narrower capability than full context-sensitive pronunciation. A voice-bank row supplies style. Neither a WAV file nor a prompt string is fed directly into the Conv1D kernel.
2. Give the phonemes context and time. ALBERT encodes the IDs, and a style-conditioned duration path predicts per-token lengths. The checked sum A=sum_t d[t] is the first important data-dependent extent. If that sum exceeds planned capacity, the graph must reject the request before an expansion writes output.
3. Expand two different streams. The duration features and the parallel convolution-plus-bidirectional-LSTM text features are each copied from T token positions to A frames according to the same duration mapping. The text stream is not just an ALBERT tensor with a new view.
4. Predict acoustic controls. The prosody path turns expanded duration features into pitch F0 and noise features. The acoustic decoder brings those controls together with the aligned text features and decoder-style slice.
5. Generate the waveform. The generator uses acoustic features and a direct pitch-controlled harmonic/noise source, upsamples in stages, and runs residual convolution branches at several time scales. A final spectral head supplies magnitude and phase to inverse STFT, which produces floating-point audio samples. Writing PCM16 WAV and playing it are separate host/application jobs.

Zoom in: generator, branch, pair, tap

After the acoustic decoder, Kokoro's generator increases time resolution with two transposed-convolution upsampling stages. At each stage, multiple residual branches use kernel widths 3, 7, and 11; their outputs are averaged. Within each branch, three residual pairs use dilations 1, 3, and 5. A pair contains the style normalization and Snake activations as well as the two convolutions shown later. The final generator head does not output WAV bytes directly: it produces spectral magnitude and phase parameters for inverse STFT. Those details come from Kokoro's generator implementation and decoder configuration.

This is the nesting I want to keep in mind: model → acoustic decoder → waveform generator → upsampling stage → residual branch → residual pair → one dilated Conv1D. The next seven-number example is that last item, not an entire speech synthesizer.

Start with seven numbers

Let a single channel contain x=[0,1,2,3,4,5,6]. Use three learned taps w=[1,0,-1], stride one, and zero padding so we keep seven output positions. Ordinary neural-network Conv1D uses the stored tap order as a cross-correlation: at output position t, it reads x[t-1], x[t], x[t+1]. At t=3, the answer is 1*2 + 0*3 - 1*4 = -2.

Now set the dilation to two. Keep the same three taps. Instead of positions t-1,t,t+1, read t-2,t,t+2. The middle output is 1*1 + 0*3 - 1*5 = -4. With dilation three, positions 0,3,6 give -6. The weights did not multiply. The distance between sampled positions changed.

For input channel i, output channel o, tap k, stride s, padding p, and dilation d, the operation CKE implements is:

y[o,t] = b[o] + sum_i sum_k W[o,i,k] * x[i, t*s - p + k*d], treating out-of-range input positions as zero. The input has shape [I,T], weights [O,I,K], bias [O], and output [O,U]. This is dense channel mixing: every output channel can use every input channel. Do not confuse it with a depthwise convolution, where a filter is tied to one channel.

For the usual discrete Conv1D shape rule, U = floor((T + 2p - d*(K-1) - 1)/s) + 1. With odd K=3, stride s=1, and symmetric padding p=d, the seven-position example stays seven positions long. With two input channels, the kernel would perform two three-tap sums for every output channel and add them to one bias. Dilation only changes which time indices those sums read. It does not remove the cross-channel sum.

Here is that fact in the language of a C loop. This is a shortened teaching version of CKE's checked provider; the production function also validates spans, finite values, overflow and caller-owned staging before committing results.

for (size_t oc = 0; oc < O; ++oc)
  for (size_t t = 0; t < U; ++t) {
    float sum = bias[oc];
    for (size_t ic = 0; ic < I; ++ic)
      for (size_t k = 0; k < K; ++k) {
        int64_t q = (int64_t)t * stride
                  + (int64_t)k * dilation - padding;
        if (q >= 0 && q < (int64_t)T)
          sum = fmaf(input[ic * input_stride + q],
                     weight[(oc * I + ic) * K + k], sum);
      }
    staged[oc * U + t] = sum;
  }

Why speech needs several time scales

A waveform changes rapidly, while the structure that makes a vowel or consonant recognizable persists over a longer interval. One filter width is a narrow way to ask both questions. A dilated filter reaches farther without increasing the number of taps, while a second, ordinary convolution can recombine nearby results. Stack the pair and add the input back as a residual connection. The next pair can see what earlier pairs found. This is a way to collect evidence over different spans of time, not a claim that a dilation of five directly “recognizes a phoneme.”

Kokoro's published decoder configuration selects residual kernel widths 3, 7, 11, with a dilation sequence 1, 3, 5 for each branch. In the reference generator, each residual pair is not merely “conv then conv”: style-conditioned instance normalization (AdaIN), Snake activation, a dilated Conv1D, another AdaIN and Snake, an ordinary Conv1D, then a residual addition. Three such pairs occur in a block. Several kernel-width branches are combined after each upsampling stage. This lineage draws on multi-receptive-field speech generators, but Kokoro's actual source and configuration, not an assumed generic HiFi-GAN diagram, determine the details here. The HiFi-GAN paper is useful historical context.

Written as a compact calculation for one pair, the upstream sequence is u = Conv_d(Snake(AdaIN(x, style))), v = Conv_1(Snake(AdaIN(u, style))), and out = x + v. AdaIN first normalizes each channel over time and then applies style-derived scale and shift, approximately (1+gamma(style))*InstanceNorm(x) + beta(style). Snake contributes the periodic term z + sin(alpha*z)^2/alpha with learned per-channel alpha. This explains why a matching convolution output is necessary but not sufficient: a small mismatch in normalization or activation enters the next convolution and then the residual sum. The reference AdaINResBlock1 fixes the order.

For a simplified chain of three kernel-three pairs, with dilations 1, 3, and 5 on the first convolution in each pair and dilation one on the second, the maximum receptive-field span is 1 + [(2*1+2) + (2*3+2) + (2*5+2)] = 25 positions, assuming stride one. That arithmetic explains the growing span; it is not the total receptive field of Kokoro's generator. Its upsampling, other blocks, branches, and final transforms change that overall number.

What CKE has actually added

CKE now has an explicit dilated-Conv1D kernel map. It declares the input, weight, output and scratch layouts; runtime dimensions include stride, padding and dilation. The current map is FP32, inference-only, with a checked C function in the audio kernel source. The independent PyTorch fixture test checks numerical output, padded row strides, malformed capacities and recovery after rejected calls. It allows up to 3e-5 absolute error for its pinned cases; it is not a bit-identical claim.

The work then moved beyond an isolated C function: PR #675 added the checked provider; #677, #679, and #680 connected successively larger Kokoro source-residual boundaries in generated circuits. Later #742 and #748 added checked x86 FMA/AVX2 and channel-parallel providers, and #764 selected a certified AVX2 provider in native bundles. Those are separate kinds of evidence: numerical kernel correctness, generated graph connection, and a selected optimized provider. None alone proves the synthesized voice matches the reference for arbitrary text.

How one Kokoro operation becomes generated C

There are several distinct implementation questions here. A C function can get the dot products right and still be unusable in a model if the weight layout is wrong, the circuit connects the wrong producer, a valid length is mistaken for allocated capacity, or a generated call selects a different numerical provider. I would build the boundary in this order, and CKE's merged work shows much of that sequence in real code:

1. Pin the exact model question

Choose the upstream Kokoro code revision, decoder config, model weights, voice row, fixed test utterance, and the tensors to compare. Capture the input and output at the specific residual pair in PyTorch before writing a C implementation. For a stochastic waveform path, either pin a declared random-source contract or feed the same captured source tensors to both implementations for the arithmetic test. “Same seed” does not by itself make two different random-number generators equivalent. The dilated-convolution reference builder and pinned fixture metadata show the oracle side of this work.

2. Specify the primitive before optimizing it

The checked C provider implements the index expression above for channel-major FP32 buffers. Its call takes explicit element capacities and physical row strides for input and output, plus I, O, T, K, stride, padding, dilation, and U. The caller supplies scratch storage. The provider must reject zero dilation, bad geometry, short buffers and non-finite values without half-publishing an output. The oracle test compiles both -O0 and -O3 and compares against committed PyTorch results. That is stronger than seeing one plausible waveform.

3. Declare the kernel-map contract

The kernel map declares input:[I,T], weight:[O,I,K], bias:[O], output:[O,U], FP32 layouts, scratch lifetime, the exact C symbol, and the runtime scalars the generated call must bind. operation_interface identifies the semantic operation; the selected numerical capability describes the scalar FMA order. That is the reusable contract. A Kokoro-specific weight filename should not be hard-coded into the compiler to make this operator work.

4. Connect the circuit, not another Python runtime

The source-residual-block authoring script assembles the three pairs with explicit graph_slots for producer/consumer edges and weight_refs for the pinned parameters. It declares AdaIN, Snake, dilated and ordinary convolution, and each residual sum in sequence. In the actual circuit JSON, a later residual pair uses a header for two style projections, a body for ordered normalization, Snake and convolution on both sides, and a footer for the residual addition. Those are compiler scheduling declarations, not three separate Python-driven inference programs. The existing compiler lowers the circuit; Python authors and verifies it but must not execute a hidden model stage when native C runs. The compiler plans activation and scratch lifetimes from these declarations.

5. Compare each boundary, then widen it

Compile the circuit to generated C and compare the first dilated output, second convolution, residual sum, and final block output with the pinned oracle. A failed later checkpoint points back to the first bad edge. The source-block test includes direct-input checkpoints and a connected test that keeps the known upstream source-phase failure explicit. Once the scalar route is understood, select the checked AVX2/FMA provider and rerun the numerical and native-bundle gates. A provider swap is not a license to silently change a numerical bound.

6. Carry the proof to the full request

The generator is downstream of duration, two expanded streams, F0/noise, and the acoustic decoder. Each new consumer needs the correct valid-frame count and declared maximum capacity. The full waveform circuit's spectral head supplies magnitude and phase to inverse STFT; the C host converts samples into PCM16 WAV. The native request package also checks weight and voice-bank provenance. To make this generally usable, CKE must connect runtime-variable lengths through the remaining prosody, decoder and waveform stages, not just compile a separate graph for each approved length.

The next connected boundary

There are already exact-length generated circuits for the downstream stages. The remaining model-level problem is not to invent another dilated convolution. It is to carry the checked, request-dependent lengths and the same tensor contracts through the complete graph while resolving the existing waveform mismatch. In practical order:

1. Join the variable-length prefix to F0 and noise. Use the checked A from duration expansion and shared prosody to size both branches, then verify their valid outputs at 2A. Keep their independent oracle checkpoints; a branch that executes safely is not yet numerically correct.
2. Join the acoustic decoder. Bind expanded text, F0, noise and the correct decoder-style slice to its declared inputs. Compare each residual block and its last [512,2A] feature output against pinned hooks. A wrong concat order can produce a valid shape and invalid speech.
3. Join the generator and source. Carry pitch to the direct harmonic/noise source, validate the upsampling extents, run the residual convolution branches and their average, then the magnitude/phase head and inverse STFT. Declare random-source provenance so an arithmetic comparison is not confused by unrelated noise draws.
4. Close the evidence loop. Compare intermediate tensors first, then the FP32 waveform, then PCM16 conversion and WAV structure. Keep execution success, numerical parity, audible usefulness, speed and accessibility controls as separate results. Only after that should a general text frontend and cancellable playback session be described as a working voice workflow.

That is also why the architecture matters to me. A new model stage should become a bounded operation, kernel map or circuit edge with a retained regression, not a model-name branch that hides missing work. If a new connection really requires a compiler change, the smallest model-neutral graph that reproduces it should become a compiler test before the full Kokoro graph relies on it.

Try the arithmetic yourself

This tiny Python version deliberately has one input and one output channel. It uses the same index rule as the kernel map, with symmetric zero padding for odd K, and makes the centre calculation visible. It is a teaching fixture, not a Kokoro inference command.

def conv1d_same(x, weights, dilation):
    pad = dilation * (len(weights) - 1) // 2
    out = []
    for t in range(len(x)):
        total = 0
        for k, weight in enumerate(weights):
            index = t - pad + k * dilation
            if 0 <= index < len(x):
                total += weight * x[index]
        out.append(total)
    return out

x = [0, 1, 2, 3, 4, 5, 6]
w = [1, 0, -1]
for dilation in (1, 2, 3):
    print(dilation, conv1d_same(x, w, dilation)[3])
# 1 -2
# 2 -4
# 3 -6

A useful next experiment is to replace the ramp with a short oscillation or a sharp transient and plot all seven output positions. Then change only d. You will see not just a wider view but different samples selected; dilation can also skip a pattern. The surrounding branches and ordinary convolutions matter partly because one sparse view is not sufficient.

Where to go next

If you are following the mathematics, read the forward convolution lesson and its backward pass first. For the broader model, read How Kokoro TTS Works; for its contextual phoneme stage, see my ALBERT deep dive. The first diagnostic WAV investigation is a dated milestone, while the CKE Kokoro bring-up documentation records the newer gates. To contrast generation with recognition, see CKE's audio-kernel deep dive; Parakeet's FastConformer is an encoder problem and deserves its own worked example.

For me the interesting part is how a small old signal-processing rule becomes one reusable piece in a much larger generated program. The three-tap calculation is easy. The hard engineering is preserving its numerical contract while normalization, periodic activation, upsampling, residual addition and waveform reconstruction all have to agree with the reference.