If an assistant writes “Hello, this is CKE,” what has to happen before we hear it?
We have been bringing Kokoro through CKE's generated C path, while a new Qwen3-TTS reference capture has just landed. Both turn text into speech, but they choose different intermediate languages. Kokoro moves from phonemes to predicted timing and an acoustic waveform. Qwen3-TTS predicts discrete speech codes and uses a speech decoder to turn those codes into audio. Understanding that split makes the engineering work, and the limits of today's results, much clearer.
Start with the output, not the model name
Our accessibility loop needs to take an agent's written answer, prepare it without changing its meaning, synthesize audio, play it through the headset, and stop promptly when the user interrupts. A WAV file proves only that samples were written. It does not prove that numbers were spoken correctly, that a long answer was complete, that the voice sounds comfortable, or that the user can interrupt it.
That distinction has shaped the CKE work. In the speech bring-up ledger, text preparation, individual arithmetic checkpoints, generated execution, whole-utterance output, connected numerical parity, and real listening are separate gates. Passing one does not automatically pass the next.
Path one: Kokoro decides how long each sound lasts
Kokoro is an 82-million-parameter open-weight TTS model. Its public pipeline converts written text into phonemes using language-specific grapheme-to-phoneme processing, then the model synthesizes audio conditioned on a selected voice. In the pinned path CKE is testing, a context encoder processes those phonemes; a predictor estimates durations; length regulation expands token-level information into frame-level sequences; style and prosody shape the acoustic representation; and the decoder plus inverse-STFT path produces waveform samples. These are concrete operations, not just a generic “text in, speech out” call. See the upstream pipeline, model implementation, and CKE component ledger.
Suppose the prepared input has phonemes p1, p2, p3 and the duration predictor assigns 2, 1, 3 frames. The aligned stream has six positions: [p1, p1, p2, p3, p3, p3]. This toy example is not Kokoro's complete feature tensor, but it shows why a duration sum is a runtime shape, not a decorative number. A wrong duration or unchecked buffer length can corrupt the rest of the graph before we ever hear a sound.
Frames = sum(durations) = 2 + 1 + 3 = 6
Token-level features → checked frame-level extent → waveform synthesis
Path two: Qwen3-TTS speaks in discrete audio codes
The Qwen3-TTS family has 0.6B and 1.7B variants, with different CustomVoice, Base, and VoiceDesign capabilities. Here we mean one precise target: Qwen3-TTS 12Hz 0.6B CustomVoice, the checkpoint in CKE's first reference capture. Claims about voice cloning or voice design on other variants should not be transferred to this one without checking its interface.
In the pinned reference path, text is tokenized and paired with a preset speaker and language. A talker predicts the speech sequence autoregressively. A code predictor fills out the groups of discrete speech codes. The separate speech-tokenizer decoder converts those codes into 24 kHz waveform samples. “12 Hz” describes the coarse cadence of codec frames, not a 12-sample-per-second WAV: the final audio is still sampled at 24,000 samples per second. The CKE oracle PR records the exact model and reference-code revisions.
For the pinned sentence “Hello, this is CKE.”, that official CPU reference produced 15 text token IDs, 46 codec frames with 16 code groups per frame, and 88,320 samples of 24 kHz audio. The decoder's configured expansion is 2 × 2 × 8 × 5 × 4 × 3 = 1,920 samples per frame. Thus 46 × 1,920 = 88,320 samples, or 3.68 seconds at 24,000 samples per second. This exact equality describes this pinned full-decoder fixture; it is not a general promise about chunk boundaries or every model called “12Hz.” The reference took 9.47 seconds to generate in one recorded CPU FP32/eager setup with eight threads. It is a reproducible fixture for CKE bring-up, not a CKE throughput result and not a Kokoro-versus-Qwen benchmark.
Written sentence15 text token IDs in this fixture
Predicted speech representation46 frames × 16 code groups
Decoded audio88,320 samples / 24,000 samples per second = 3.68 seconds
Follow Qwen3-TTS kernel by kernel
“Encoder” and “decoder” are easy to misuse here. For this CustomVoice text-to-speech request, there is no separate speech-input encoder: we are not feeding reference audio. Text IDs are embedded and projected into the talker, which acts as the autoregressive speech-code generator. A five-layer code predictor fills the remaining codebooks for each frame. Only then does the speech-tokenizer decoder turn codes into samples. The speech tokenizer's audio encoder is relevant to other paths, not required to synthesize this preset speaker. The pinned talker source and pinned decoder source are the code-level references below.
1. Text and speaker preparation
The host tokenizes the sentence with the model snapshot's tokenizer and constructs a prompt with language, speaker, and codec control IDs. Embedding lookups turn IDs into vectors. The text embedding starts at 2,048 channels, then a biased linear → SiLU → biased linear projection brings it to the talker's 1,024-channel space. Here the operations are token lookup, range checking, two matrix-vector/matrix-matrix projections, bias add, SiLU, and embedding combination. The chosen speaker affects the prompt; it is not a waveform encoder running on a recorded voice. For CKE, every ID table and valid length is a separate contract.
2. Talker prefill and autoregressive frame generation
The 0.6B snapshot declares 28 talker layers. One layer performs RMSNorm, Q/K/V linear projections, per-head Q/K RMSNorm, interleaved multimodal rotary position embedding, causal grouped-query attention, output projection, residual add, another RMSNorm, gated MLP (SiLU/SwiGLU-style gate and up/down projections), and a second residual add. Prefill processes the prompt; each later step uses cached K/V history rather than recomputing the entire prefix. A final norm and codec-vocabulary projection produce logits for the first codebook. Masking, sampling policy, and the EOS code decide whether generation continues. A correct GEMM without the same positions, cache update, or stop rule does not reproduce this model.
3. Fifteen more codebooks for each frame
The first code selects one speech code. A separate five-layer code-predictor transformer then predicts the other 15 groups sequentially, using group-specific embeddings and output heads. Its kernels look familiar — embedding, RMSNorm, Q/K/V GEMM, ordinary RoPE, causal attention, gated MLP, and logits — but the codebook index, short-lived predictor cache, and reset at a new frame are part of the computation. The reference packs 16 IDs per frame; those IDs are not PCM samples.
4. Turn 16 IDs into one acoustic latent
The speech decoder first uses a split residual vector quantizer: one semantic book and 15 further books. Each codebook row is formed from an embedding sum divided by usage clamped to at least 1e-5, looked up by ID, optionally projected, then accumulated in codebook order. This requires bounded lookup, clamp/divide, projection, ordered vector addition. It is tempting to call it just “an embedding layer”; that would miss the effective-weight formula and reduction order. A causal convolution prepares the resulting latent stream.
5. Codec transformer and waveform generator
The decoder next runs an eight-layer pre-transformer with RMSNorm, rotary attention, MLP, residual adds and a 72-token sliding-attention window in the pinned configuration. After that, two causal transposed-convolution upsampling stages, each followed by a ConvNeXt block, expand the time axis. ConvNeXt uses depthwise causal Conv1D, LayerNorm, pointwise linears, exact GELU, per-channel gamma and a residual add. A final stack uses causal convolution, four more transposed-convolution upsampling stages and residual units with dilated causal Conv1D. Its periodic SnakeBeta activation is not ordinary Snake: it has separately exponentiated alpha and beta parameters. The output is clamped to [-1, 1] before the host converts samples to PCM/WAV. This Qwen decoder does not run Kokoro's duration expansion, harmonic source or inverse STFT.
46 code frames × (2 × 2 × 8 × 5 × 4 × 3) = 88,320 samples
88,320 / 24,000 = 3.68 seconds of output in the pinned full-decoder example
Which CKE kernels can we start from?
The following are CKE v8 maps and remaining contracts, not an end-to-end Qwen compatibility claim. PR #875 exercised some of the new checked maps on an oracle-fed first block; later speech stages remain candidates. A kernel can have the right name yet the wrong layout, padding, reduction order, or live-state contract. The kernel map inventory is the source for the CKE side; the pinned upstream files above define Qwen's side.
| Qwen operation | CKE starting point | Contract to prove |
|---|---|---|
| Text, speaker and codebook ID lookup | embedding_forward_fp32 | Each table's bounds, token-major layout, valid length and lookup value. |
| Text projection; Q/K/V, MLP and logits GEMM | linear_rows_checked_f32 for biased rows; new linear_rows_checked_no_bias_f32 for the first block's biasless rows | Distinct bias contract; both checked linear maps use FP64 ordered accumulation then FP32 output. Logits and repeated decoding remain untested. |
| RMSNorm and per-head Q/K norm | New rmsnorm_rows_checked_f32 in the first-block circuit | FP32 square/ascending sum and correct axis: 1,024-channel rows versus 128-channel head rows. The older FP64-sum map is a different provider. |
| Talker M-RoPE; predictor and codec RoPE | New rope_qk_split_bounded_f32 for the tested text-only block; existing multimodal/ordinary RoPE maps | PR #875 preflights angle and signed positions before in-place Q/K writes. General prompt positions and later cache offsets still need proof. |
| Causal grouped-query attention | New attention_causal_head_major_bounded_f32 wraps existing GQA flash arithmetic | 16 Q heads share eight K/V heads; test mask and QK/softmax/PV stage errors. One prefill block is not a live KV-cache decode path. |
| Gated MLP, SiLU and residuals | New silu_mul_rows_checked_f32 and add_rows_checked_f32 in the first block | Check finite gate/up products, no unintended aliases, two residual joins and padding-safe dimensions. |
| Logits, first-code choice, remaining groups | audio_argmax_first_f32 for a deterministic first lane | Separate top-k/top-p/temperature/RNG, EOS, codebook scheduling and predictor-cache reset. |
| 16 residual-vector-quantizer codebooks | Embedding, projection and add primitives | Effective row = embedding_sum / max(usage, 1e-5); semantic/acoustic split and ordered 16-book accumulation. |
| Codec sliding-attention transformer | Attention and normalization maps | 72-token window, LayerScale where present, mask and context behavior at codec frame rate. |
| Causal and depthwise Conv1D | Checked audio Conv1D maps | Explicit left zero-pad, possible short right tail, dilation/groups, stride and valid output extent. |
| Transposed Conv1D and ConvNeXt | Checked deconvolution, LayerNorm and GELU maps | Right crop by kernel size minus stride; per-channel gamma; Qwen's erf GELU, not tanh-GELU. |
| SnakeBeta, clamp and final PCM | Existing Snake map and audio host are only starting points | New SnakeBeta formula, finite sample check, exact 1,920-sample expansion per frame, buffer capacity and PCM conversion. |
Numerical parity should be built at the first divergent boundary, not by replacing all arithmetic until the WAV sounds plausible. PR #875 supplies a bounded projection and oracle-fed first-block comparison. Next connect the real prompt, prove prefill and one decode step with KV state, then repeat all 28 layers and test EOS. Only then connect the five-layer predictor and 16-book codec. Inside the codec, compare quantizer output, transformer output, each upsampling stage and final samples separately.
Six kernels, six mathematical contracts
Here is what the new CKE code actually computes, followed by the kernel work that still has to be added. Each formula is a small contract. Shapes and rounding belong beside it, because two programs can implement the same real-number equation and still disagree as FP32 programs. The checked C functions and the first-block circuit make those contracts inspectable.
A. A projection is a dot product with an optional bias
yj = roundFP32(bj + Σi=0..K-1 xi Wj,i)
Q/K/V and MLP rows use bj = 0; the text projection has a real bias.
The new linear_rows_checked_no_bias_f32 map gives zero-bias projections their own operation identity, rather than letting them accidentally resolve to Kokoro's biased contract. The current checked C implementation forms products and accumulates the sum in FP64 in ascending channel order, then casts the result to FP32. This is a numerical choice, not the same thing as PyTorch's unspecified CPU GEMM dispatch or an FP32 fused multiply-add kernel. For an intuition-only example, adding 100,000,000 + 1 - 100,000,000 left-to-right in FP32 can lose the 1; a wide accumulator can retain it. That toy sum is not a Qwen weight row. It explains why the reduction tree must be measured at each projection checkpoint. Read the C provider.
B. RMSNorm is a reduction over the right axis
s = 1 / sqrt((Σi xi2) / D + ε)
yi = xi × s × γi
The new rmsnorm_rows_checked_f32 accumulates squared channels in FP32. The first talker block normalizes 1,024 channels per token, then normalizes Q and K per 128-channel head. Reducing across tokens or across all 16 heads would be a different operation. The existing FP64-sum RMSNorm map is not silently substituted. The checked provider verifies finite inputs and the computed scale before writing; the oracle comparison determines whether its FP32 order is close enough for this bounded graph. The visual below follows one token through projection, per-head norm, positional rotation, attention and the two residual joins.
C. RoPE rotates pairs; checked state prevents partial damage
For a split-half pair (a, b) at position p: (a', b') = (a cos θ - b sin θ, a sin θ + b cos θ).
Frequency depends on pair index: θi = p × base-2i/D, with the pinned first block's base 106.
The bounded first-block circuit uses a split-pair RoPE provider for the tested text-only positions. That is not a blanket proof for the talker's full multimodal position scheme or for continuing at arbitrary cache offsets. The new wrapper checks signed position range, representable angles, finite Q/K and frequency base before rotating in place. This matters because a rejected request must not leave half-rotated Q/K values for the next operation. Read the checked wrapper.
D. Attention is two matrix products with a softmax between
scoret,u = dot(Qt, Ku) / sqrt(128), with u ≤ t only.
Pt,u = exp(scoret,u - m) / Σv≤t exp(scoret,v - m); Ot = Σu≤t Pt,u Vu.
Sixteen query heads share eight key/value heads, so two Q heads read each K/V head. A token-major ↔ head-major layout conversion is part of the executable graph, not free notation. The new attention_causal_head_major_bounded_f32 checks shapes and finite inputs, then calls CKE's existing online/flash GQA kernel. Its softmax reduction order differs from the pinned PyTorch eager path; the first-block result is therefore judged by a stated tolerance and intermediate checkpoints, not a byte-for-byte WAV claim. This bounded test has no production KV-cache lifecycle yet.
E. The gated MLP and residual are more than another GEMM
g = Wgatex, u = Wupx; h = SiLU(g) ⊙ u = (g / (1 + exp(-g))) ⊙ u.
y = xafter-attention + Wdownh.
CKE's new silu_mul_rows_checked_f32 and add_rows_checked_f32 validate finite operands and results. The block has two residual joins, one after attention and one after the MLP; dropping either changes the model even if every matrix product is correct. In the pinned test, 8- and 18-token oracle-fed block cases stayed under a 5e-5 maximum-absolute-error gate; the PR reports a worst stage difference of 9.54e-6. The separate 3/7-token text projection test reports 1.20e-7. These are bounded PR results, not a claim that 28-layer streaming speech works. Inspect the first-block test.
F. The still-missing codec kernels change the mathematics again
After code generation, the decoder is not “just another transformer.” The 16-code residual quantizer, causal padding, transposed-convolution crop and SnakeBeta activation need explicit new contracts. The existing CKE maps supply pieces but not this complete decoder. These formulas come from the pinned Qwen codec source:
| Missing / adapting kernel | What the arithmetic does | What CKE must prove |
|---|---|---|
| Residual codebook decode | For each selected row, e = embedding_sum / max(cluster_usage, 1e-5); project and sum one semantic plus 15 acoustic contributions in order. | Exact code IDs, effective-weight rounding, optional projection and accumulation order. |
| Causal Conv1D | Pad zeros on the left by effective_kernel - stride; add only the short right tail needed for stride alignment. | No future-sample leakage; exact valid length, groups, dilation and boundary taps. |
| Transposed Conv1D | Write the raw upsampled convolution, then crop kernel_size - stride samples on the right. | The crop is not symmetric padding; preserve frame/sample count and channel layout. |
| ConvNeXt | Depthwise causal Conv1D → LayerNorm → pointwise linear → erf-GELU → pointwise linear → per-channel gamma → residual. | LayerNorm axis and epsilon, exact GELU variant, gamma broadcasting and no aliasing. |
| SnakeBeta | y = x + sin²(exp(alpha) x) / (exp(beta) + 1e-9), with per-channel alpha and beta. | A distinct checked kernel: ordinary Snake's single parameter cannot stand in for SnakeBeta. |
A safe bring-up order is: codebook values and ordered sum → eight-layer codec transformer with its 72-frame attention window → causal ConvNeXt stages → each cropped transposed convolution and SnakeBeta residual unit → final clamp and exact sample count. For the pinned 46-frame fixture, the full decoder should produce 46 × 1,920 = 88,320 samples. A sample-count pass is necessary, but waveform parity, listening and playback are separate tests.
What actually differs?
| Question | Kokoro v1.0 | Qwen3-TTS 0.6B CustomVoice |
|---|---|---|
| What enters the model? | Phoneme IDs and a chosen voice style after G2P. | Text tokens with language and preset-speaker conditioning. |
| What is the central timing decision? | Predicted durations expand token features into acoustic frames. | Autoregressive generation advances through discrete speech-code frames. |
| What becomes a waveform? | Acoustic features through a synthesis and inverse-STFT path. | Groups of codec tokens through a separate speech decoder. |
| Where can meaning break? | Normalization, pronunciation, duration, style and spectral math. | Token formatting, sampling or stop behavior, code prediction and codec decoding. |
| What has CKE demonstrated? | Bounded generated requests, retained worker and diagnostic whole-text WAV; connected waveform parity still fails. | Pinned official PyTorch CPU oracle, plus a generated text projection and oracle-fed first talker block under bounded numerical tests; the rest is unconnected. |
Neither row proves a winner for our headset. Kokoro's smaller parameter count suggests a different resource profile, but model size alone is not a first-audio latency, throughput, or listening-quality measurement. Qwen's upstream streaming capability does not mean our CKE port streams today. Those are experiments to run under matched conditions.
Read the recent CKE commits as an evidence ladder
There is a useful progression in the October 21–22 CKE work. PR #850 added bounded planning for real Kokoro text. PR #855 retained source-intervention evidence while debugging the first connected waveform divergence. PR #865 kept a checked waveform worker loaded across requests, and PR #867 aggregated checked chunks into one WAV instead of separate files. These make a longer spoken reply more practical to test; they do not erase the parity failure or certify cancellation and headset playback.
Then PR #873 pinned Qwen3-TTS's official reference, including token IDs, codec codes, waveform and artifact hashes. PR #875 added the first generated text projection and an oracle-fed talker layer-zero circuit, with explicit biased/biasless map separation and RoPE failure checks. This is now a real bounded CKE numerical comparison, but it starts at a captured block input. Prompt assembly, talker recurrence and K/V state, the five-layer predictor, codec decoding, and whole-speech parity remain separate future gates.
What should we test next?
First, keep improving the Kokoro path we can actually execute: identify the first connected numerical mismatch, prove long-answer text preparation does not omit symbols, and measure first audible speech, interruption, and repeated requests through the Sony headset. Separately, connect Qwen3-TTS's new text projection and first block to real prompt assembly, then prove the remaining talker layers, K/V decode state, code predictor and codec-to-waveform path with independent checkpoints at each boundary. Only after both have an end-to-end path should we compare voices or latency using the same sentences and hardware.
The goal is not “which model has the nicer model card?” It is a voice loop that reliably reads what the agent actually wrote, starts in time to feel interactive, and stops when asked. Kokoro currently gives us the closer CKE integration path; Qwen3-TTS gives us a different architecture and a pinned reference to build toward. Both still have work ahead.
Continue the build log
- CKE's First Kokoro WAV Is Not Ready Yet explains why diagnostic audio and numerical parity are separate claims.
- ALBERT: How Kokoro's Phoneme Encoder Works explores the text-side graph in more detail.
- I Want To Use My Linux Workstation Without Looking At It describes the accessibility goal.
- CKE TTS bring-up ledger tracks the latest tested and untested boundaries.
