An earlier article explained how an audio transformer hears: waveform to frames, frames to log-Mel features, features to encoder states, states to decoded text. This article is the implementation sequel. It follows the exact Whisper Tiny circuit now running through C-Kernel-Engine: every tensor shape, kernel boundary, memory lifetime, parity gate, and generated-C stage. If you want the conceptual introduction — timestamps, hallucination, why audio is not a vision-language-model patch — start there. If you want to watch the computation execute, stay here.

C-Kernel-Engine (CKE) is my CPU-first model runtime: a model is described as an explicit circuit, the circuit resolves into kernel-map provider selections, lowering passes schedule it into call IR (intermediate representation), and the output is a generated C artifact with a fixed memory plan. The documentation site carries the full architecture. Until now, that pipeline ran language families (decoder-only and hybrid text models) and one vision family (Qwen3-VL, through an encoder-plus-prefix bridge). Whisper adds three things the pipeline had never carried at once: raw signal processing before the transformer, a bidirectional encoder, and a causal decoder bound to that encoder by cross-attention.

One clarification before the details: “v8” is the generation of CKE's execution pipeline — the circuit, kernel-map, lowering, and codegen stack — not a version of Whisper. The model is OpenAI's Whisper Tiny, unchanged. And “end to end” has a precise meaning here: PCM16 (pulse-code modulation, 16-bit) WAV bytes enter generated C, and transcribed tokens leave it. Python manages artifacts and execution; it does not implement the model arithmetic.

WAV bytes
  → normalized samples
  → 80 × 3000 log-Mel features
  → 1500 × 384 encoder memory
  → decoder cross-attention
  → output tokens
  → text
Whisper Tiny, End to End on v8 diagram: PCM16 mono WAV bytes flow through a generated audio frontend of seven circuit operations — decode, resample, pad, STFT tables, STFT, mel filters, log-mel — producing an 80 by 3000 log-mel tensor, then a four-block pre-LN encoder with six heads and dense bidirectional attention, then a decoder with causal self-attention and cross-attention to the encoder. A persistent cross-attention K/V cache projects encoder keys and values once at prefill and reuses them for every decoded token, yielding a 79 times faster decoder: 16.57 seconds down to 0.21 seconds, total 28.64 down to 12.22 seconds, with identical 23 JFK tokens across three runs. Parity gates compare the log-mel frontend and primitives against PyTorch nightly and the full WAV-to-transcript path against Hugging Face. Current contract chips read: Whisper Tiny FP32, English no timestamps, PCM16 WAV 16 kilohertz mono, 30 seconds maximum truncated not chunked, planned for v10 landed in v8.
The whole path: WAV bytes to tokens, all arithmetic in generated C, with the measured 79× decode win and the parity gates that make the claim falsifiable.

Whisper Tiny, one kernel boundary at a time

This is the complete circuit in execution order. Each stage names its kernel, its shapes, and the lifetime of the memory it touches.

1. WAV decode

The artifact receives raw WAV bytes. The decode kernel walks the PCM16 payload and converts each 16-bit integer to a normalized FP32 sample, scaled by 1/32768. The result is a one-dimensional f32 signal of n samples. It is not yet an embedding, not yet a token tensor — it is still just sound, represented as numbers the rest of the circuit can contract over.

2. Resampling

Whisper expects 16 kHz (kilohertz) mono audio. The circuit makes the two paths explicit: audio already at 16 kHz passes through untouched; anything else runs a windowed-sinc resampling kernel. Resampling is a separate numerical contract rather than something hidden inside file loading, because interpolation is arithmetic — it has error behavior, and CKE's rule is that anything with error behavior gets a kernel boundary and a parity test.

3. Pad or truncate

Whisper Tiny was trained on 30-second windows, so the circuit pads or truncates every input to exactly 480,000 samples. Longer recordings are currently truncated rather than chunked — a stated limitation, not a silent one.

4. STFT tables and STFT power

The short-time Fourier transform (STFT) stage is split into two circuit operations: table construction (precomputing the Hann window and the discrete Fourier transform basis for a 400-point fast Fourier transform) and execution. With a 400-sample window and a hop of 160 samples, 480,000 samples become 3,000 frames; the 400-point FFT yields 201 frequency bins. The output is a 201 × 3000 power spectrogram. Table construction and execution are separate ops because they have different memory lifetimes — tables are built once in planned memory and reused by every frame.

5. Mel filter construction

The 80-band Slaney mel filterbank is not a learned tensor and does not ship with the model weights. CKE builds it in planned memory at runtime, from the same Slaney equations the reference implementations use. This is a deliberate architectural choice: a deterministic transform belongs in the circuit, not in the artifact.

6. Log-Mel

The power spectrogram passes through the mel filters, then a log10 compression with Whisper's exact normalization: find the global maximum, clamp every value to at least max − 8, and rescale. The result is the 80 × 3000 log-Mel tensor — the only representation of the audio the transformer ever sees.

7. Conv1D stem

Two one-dimensional convolutions with GELU (Gaussian error linear unit) activations turn spectral channels into model width and halve the time axis:

80 × 3000
  → Conv1D k=3, stride=1 → GELU
  → Conv1D k=3, stride=2 → GELU
  → 384 × 1500

Stride two in the second convolution is exactly what halves 3,000 frames into 1,500 positions.

8. Token transpose and position embeddings

The conv stem produces channel-major 384 × 1500; attention wants token-major 1500 × 384. The transpose is an explicit layout seam in the circuit, not an implicit view — memory layout is part of the contract. Learned absolute position embeddings are then added to every token. Whisper predates the rotary fashion: there is no RoPE (rotary position embedding) anywhere in this model.

From Waveform to Tokens diagram: the shape journey inside the generated frontend across two domains. Signal domain — WAV decode (PCM16 scaled by 1/32768 to n samples of f32), resample and pad (windowed sinc to 16 kilohertz, exactly 480,000 samples for 30 seconds), STFT power (FFT 400, hop 160, Hann window, 201 by 3000 power), log-mel (80-band Slaney, log10, clamp, 80 by 3000). Tensor domain — conv stem (k3s1 plus k3s2 plus GELU, 384 by 1500), transpose to token-major (1500 by 384 tokens), encoder times four blocks (dense bidirectional, learned positions, 1500 by 384 memory), decoder (causal plus cached cross-attention, tokens to text). Panels note that 480,000 samples become 1,500 tokens, a 320 times compression before attention runs, and that one entry point ck_model_run_audio_wav owns all seven circuit ops in generated C while Python only orchestrates.
Stages 1 through 8 in one picture: 480,000 samples become 1,500 tokens before attention ever runs.

9. Encoder block, ×4

The encoder is four identical pre-LayerNorm blocks. LayerNorm here is the per-token normalization that keeps activations numerically well-behaved; “pre” means it runs before each sub-layer rather than after. One block expands to:

LayerNorm
  → Q/K/V projections
  → six attention heads
  → dense bidirectional attention
  → output projection → residual
LayerNorm
  → MLP up → exact erf GELU → MLP down
  → residual

The dimensions: 1,500 tokens, model width 384, six heads of 64 dimensions each. The attention score tensor is 6 × 1500 × 1500, and each head computes the familiar scaled dot-product attention:

\[\mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{64}}\right)V\]

The GELU is the exact erf (error function) variant, matching the PyTorch reference bit-for-bit behavior rather than the faster tanh approximation — a numerical-contract decision, not a performance one.

The property that matters most is what is missing: there is no causal mask. Every audio position attends to every other audio position, past and future alike. That is what “dense bidirectional” means, and it is the structural difference between this encoder and every decoder-only family v8 has hardened — those constrain each token to its own past.

Whisper Tiny encoder anatomy diagram from the CKE documentation: one pre-LayerNorm block expanded — LayerNorm, Q/K/V projections, six heads of 64 dimensions, dense bidirectional attention with a 6 by 1500 by 1500 score tensor, output projection and residual, then LayerNorm, MLP up, exact erf GELU, MLP down, and residual. The diagram shows the tensor dimensions at each stage: 1500 tokens of width 384 throughout.
One encoder block, from the audio kernels deep dive — width 384 flows through the whole block; only the score tensor explodes to 6 × 1500 × 1500.

10. Encoder memory

After four blocks, the encoder has converted 30 seconds of sound into a fixed 1500 × 384 representation. Two properties of this memory matter for everything downstream: it is not text — it is the audio, digested — and it does not grow during decoding. It is computed once and consulted on every generated token.

11. Decoder self-attention

The decoder is a causal language model over Whisper's token vocabulary. It generates one token at a time, and its self-attention uses a persistent KV (key/value) cache that grows by one row per generated token — the same contract v8 already uses for text models.

12. Decoder cross-attention

Cross-attention is the piece audio adds. For each output token, the decoder produces a single query row (Q = 1) and attends across all K = 1500 encoder positions. This creates two memory lifetimes in one model:

  • Decoder self-attention KV grows with generated text — one new row per token.
  • Encoder cross-attention K/V is fixed for the entire transcription — the encoder output cannot change while decoding proceeds.
Cross-attention memory lifetimes diagram from the CKE documentation: the encoder runs once, producing 1500 by 384 memory projected into a persistent FP32 K/V cache at prefill and never recomputed during decode, while the decoder runs per token — one query row per decode step after causal self-attention, whose own KV cache grows by one row per generated token. The cross-attention core computes softmax of Q times K-transposed over the cached encoder keys.
Two memory lifetimes in one model: the encoder's cross-attention K/V is immutable after prefill; the decoder's self-attention KV grows one row per token.

The original implementation ignored that second fact and re-projected the encoder's keys and values on every decode step — the same projections, recomputed per token, per layer. The corrected circuit projects them once during prefill, stores them in a persistent layer-major cache, and lets every decode step read the cache. That single observation is the 79× result, and the performance section below picks up exactly there.

The whole circuit in one table

StageInput shapeOutput shapeKernel or circuitMemory lifetimeParity
WAV decodePCM16 bytes[n] f32audio_wav_decodeper runPyTorch
Resample[n] f32[m] f32 @ 16 kHzaudio_resample (sinc)per runPyTorch
Pad / truncate[m] f32[480000] f32audio_pad_or_­truncateper runshape
STFT tables—window + DFT basisaudio_stft_tablesplanned, reusedNumPy
STFT power[480000] f32[201 × 3000]audio_stft (FFT400)per runPyTorch
Mel filters—[80 × 201]audio_mel_filters (Slaney)planned, reusedref
Log-Mel[201 × 3000][80 × 3000]audio_log_melper runnightly
Conv stem[80 × 3000][384 × 1500]conv1d ×2 + GELUper runkernels
Transpose + positions[384 × 1500][1500 × 384]layout + embedding addper runlayout
Encoder ×4[1500 × 384][1500 × 384]LN, proj, attn, MLPper runX-ray
Decoder self-attntokens so far[t × 384]causal attn + KV cachegrows per tokenkernels
Decoder cross-attnQ [1 × 384], enc [1500 × 384][1 × 384]cross-attn + persistent cachefixed after prefillkernels
LM head[1 × 384]token idprojection + argmaxper tokenJFK

Every row has its own walkthrough — layouts, contracts, and why the kernel exists — in the audio kernels deep dive.

Shared kernels versus Whisper-specific structure

CKE did not grow twenty new “Whisper kernels” for this milestone. The work divides into four buckets, and the sizes of the buckets are the point:

  • Genuinely new signal-processing kernels: WAV decode, windowed-sinc resampling, pad/truncate, STFT tables and execution, Slaney mel construction, log-Mel. Audio needed these; nothing else in the tree did.
  • Shared transformer kernels, reused untouched: LayerNorm, GEMM (general matrix multiply) projections, residual addition, softmax, GELU, the MLP shape, KV-cache machinery. Whisper's transformer arithmetic is the same arithmetic every other family runs.
  • Whisper-specific structure, no new math: the tensor dimensions, the channel-to-token ordering, the two memory lifetimes, and the numerical contracts that pin them down.
  • The circuit itself: the template that stitches reusable kernels into Whisper Tiny — seven frontend ops lowered into one generated entry point, ck_model_run_audio_wav.

This is the CKE thesis working as designed: a new model family should primarily introduce a new circuit and only the genuinely missing mathematical kernels — not another standalone runtime.

Performance and the theory of constraints

The 79× number, restated with its bookkeeping: decoder cross-attention fell from 16.57 seconds to 0.21 seconds when the repeated encoder K/V projections became a prefill-once persistent cache. Total transcription time for the JFK sample fell from 28.64 to 12.22 seconds, with byte-identical 23 tokens across three repeated runs.

But eliminating one repeated computation only ever exposes the next one. Of the remaining 12.22 seconds, the encoder now consumes roughly 11.15 — dense bidirectional attention over 6 × 1500 × 1500 score tensors, four blocks deep, is the dominant cost of transcribing 30 seconds of audio on a CPU. That is the next optimization target: the encoder's GEMMs and attention, not the decoder. Whisper Tiny on v8 is correct and dramatically less wasteful than it was; it is not yet fully optimized, and the profile now says exactly where the work is.

Evidence

Every claim above is tied to a test that can falsify it:

  • The log-Mel frontend runs nightly against a PyTorch reference.
  • Kernel-level tests cover Conv1D, exact-erf GELU, layout transforms, self-attention, persistent-KV behavior, and cross-attention.
  • The 79-edge encoder X-ray compares the generated encoder against the reference computation edge by edge: for the documented FP32 run, a maximum absolute difference of approximately 5.05e-4 and an RMSE (root-mean-square error) of 6.7e-6.
  • The JFK sample transcribes to exactly the same 23 tokens across three repeated runs.
  • The full Hugging Face transcription comparison remains an opt-in artifact test, because CI runners do not carry the generated Whisper artifacts.

And the honest contract, unchanged: Whisper Tiny only. FP32 only. English transcription without timestamps. PCM16 mono WAV at 16 kHz — ffmpeg -i your-audio.mp3 -ac 1 -ar 16000 -c:a pcm_s16le out.wav converts anything else. Maximum 30 seconds, truncated rather than chunked. The JFK sample is corpus-exact; arbitrary recordings are functional, not yet corpus-certified. CKE has not “solved audio” — it has carried one audio model through the same gates every other modality passes.

Trying it is two commands — the v8 runbook carries the canonical audio section:

version/v8/scripts/cks-v8-run audio \
  --encoder-run-dir /path/to/whisper-tiny-encoder \
  --decoder-run-dir /path/to/whisper-tiny-decoder \
  --wav /tmp/test-audio.wav

Why this milestone matters

Audio was a v10 roadmap item because I expected the encoder–decoder shape and the signal-processing frontend to fight the v8 pipeline. The opposite happened: the circuit abstraction already knew how to describe the graph, the kernel maps already knew how to select providers, and lowering already knew how to schedule it. The work was writing the audio kernels and their contracts, not reinventing plumbing.

One Engine, Three Senses diagram: three modality lanes enter the same compile path. Text (GGUF artifact, decoder-only and hybrid families — Qwen, Gemma, GLM, LLaMA, Nemotron), vision (GGUF plus mmproj, Qwen3-VL, SigLIP encoder to decoder prefix bridge), and audio, highlighted as new in v8 (safetensors to BUMP, Whisper Tiny, generated frontend to encoder to decoder, planned for v10 and landed two versions early). All three flow through circuit, kernel maps, lowering to call IR, and generated C, running on the CPU you already own with runtime ISA dispatch from SSE4.2 through AVX2, AVX-VNNI, AVX-512, AMX BF16 and INT8, and NEON on ARM, parity-checked against llama.cpp/ggml and PyTorch. A panel contrasts what changed per modality — decoder templates, an encoder plus prefix bridge, a frontend plus encoder-decoder pair — with what never changed: circuit, maps, lowering, C, memory plan, and parity gates. A timeline shows v8 hardening text, then learning to see, then learning to listen.
Three senses, one compile path — and the short list of what actually had to change per modality.

Whisper Tiny is evidence that CKE's circuit, kernel-map, lowering, memory-planning, generated-C, and parity architecture can carry an encoder–decoder model with a signal-processing frontend. The achievement is not merely that Whisper runs. It is that audio joined text and vision without requiring a separate execution engine.

Text was the first sense, vision the second. Audio is the third — and it arrived not with a new engine, but with the same engine proving it was never modality-specific in the first place.

The code is at github.com/C-Kernel-Engine, the documentation at c-kernel-engine.github.io/C-Kernel-Engine, and the contributor path describes how to reproduce, bound, validate, and submit evidence. The engineering Discord is open at discord.gg/J7wNx8xS — bring questions, skepticism, or a model family you want to see run on CPUs.

Related notes and further reading