For anyone new to the project, C-Kernel-Engine (CKE) is an open-source CPU runtime and kernel compiler for transformer-based language, vision, and audio models. We describe a model as smaller mathematical circuits, connect those circuits to C kernels, and generate the runtime. PyTorch and llama.cpp are important references for checking whether that generated runtime is still doing the same mathematics.

During the recent Qwen3.6 work, comparing only the final text was not helping much. A model can produce a reasonable sentence even when small numerical differences are accumulating inside it. It can also produce one wrong token after hundreds of operations worked correctly. We needed a way to open the run, compare useful points inside each layer, and connect a mismatch back to the generated call and kernel that produced it. That is the job of CKE X-Ray.

We Had The Wrong Token. Where Did It Start?

A token mismatch can start in many places. The tokenizer may have produced different input IDs. RoPE may have used the wrong dimension. A projection may have loaded the wrong dtype. Attention or recurrent state may have drifted. The output projection may be fine and simply receive a bad residual from an earlier operation. Final logits combine all of those possibilities into one large result, so they are useful for detecting a failure but poor at locating it.

We therefore added checkpoints at points that mean the same thing in both runtimes: after Q/K normalization, after RoPE, after a recurrent-state update, after the block residual, and before sampling. I use the phrase semantic checkpoint for this. The buffers may have different names or layouts, but both checkpoints represent the same step in the model's mathematics.

wrong token
    |
    v
first logit-ranking disagreement
    |
    v
last passing semantic checkpoint
    |
    v
first failing semantic checkpoint
    |
    v
same-input replay of the suspected provider
    |
    +-- replay passes  -> upstream or propagated drift
    |
    +-- replay fails   -> bounded kernel, ABI, layout, or topology fault

There is one more complication. The first checkpoint that fails may not be the operation that created the difference. It may simply be the point where an earlier, smaller difference became easy to measure. To separate those cases, we can capture the input to the suspected kernel and give that exact input to both implementations. If they now agree, the difference arrived from upstream. If they still disagree, we have narrowed the problem to that kernel, its ABI, its layout, or the model topology it was told to use.

How X-Ray Looks Inside A Run

X-Ray does not automatically understand every runtime. We still need a small adapter for each reference, or oracle, that captures the matching checkpoints. Once those results exist, X-Ray connects them to information CKE already produces while compiling the model.

The visualizer shows the complete operation path even when we have not captured every intermediate value. Operations without evidence stay gray. This makes it easier to see the difference between “every checkpoint we captured passed” and “we compared every operation.” Those are not the same statement.

The implementation and operator workflow are documented in the CKE X-Ray visualizer runbook. PR #250 added the integrated visual surface and then hardened it against misleading joins.

When X-Ray Entered CKE

Qwen3-VL was the forcing function. CKE already had generic first-divergence checks, parity tests, and an IR visualizer, but they answered separate questions. The parity harness could tell us that two values differed. The visualizer could show the generated circuit. During Qwen3-VL vision prefill and decoder debugging, final-prefix cosine similarity could not tell us which graph edge had changed the token ranking. That limitation started X-Ray inside CKE v8, the current inference lane that lowers model circuits and kernel contracts into generated C. The implementation lives in the version/v8 source tree; “v8” is a CKE software-generation label, not a CPU or hardware generation.

The earlier article What Numerical Parity Actually Requires documents the Qwen3-VL and Qwen3.5 parity work that made these diagnostics necessary. The present article continues that history through the integrated visualizer and the later Qwen3.6 investigation; Qwen3.6 did not originate X-Ray.

That sequence is still continuing. Every model family exposes another ambiguity: batched prefill versus sequential recurrence, persistent decode versus full replay, BF16 storage versus FP32 accumulation, or one logical tensor represented by different physical layouts. X-Ray is useful precisely because it is being hardened by real failures rather than designed only around a clean synthetic example.

Why Short Agreement Is Not Enough

Autoregressive generation feeds each selected token back into the next model step. If two deterministic greedy runs match for the prompt and the first few generated tokens, that is useful evidence, but it does not certify the rest of the trajectory. A small difference in a reduction, storage cast, cache update, recurrent state, mask, or positional calculation can remain below the top-1 decision boundary for many steps. If it eventually changes one selected token, the two runs now have different histories and later divergence is expected.

same prompt
  -> small hidden-state difference
  -> same top-1 token for many steps
  -> one ranking boundary is crossed
  -> different token becomes the next input
  -> the two trajectories are no longer the same experiment

Even before a token flips, growing hidden-state or recurrent-state drift can reveal a real implementation defect. This matters more as context windows grow. Qwen3-Next documents a native context of 262,144 tokens, while Google's Gemma 3 model card specifies 128K for its larger models. A runtime that agrees for three decode steps but applies a slightly wrong recurrent update, KV-cache cast, RoPE width, or reduction order has not yet shown that it preserves the model's intended long-context behavior.

CKE therefore pursues the strongest parity available at each boundary:

• Bit-exact primitive parity when CKE deliberately implements a pinned llama.cpp production arithmetic contract.
• Scoped numerical parity when BF16, FP16, compiler choice, or a different valid reduction schedule makes byte equality the wrong requirement.
• State and ranking parity across prefill, persistent decode, long trajectories, thread counts, and execution phases.
• Deterministic token parity only when the tokenizer, prompt, model bytes, cache state, sampling settings, and oracle are all fixed.

The target is not to pretend that every valid PyTorch kernel and every valid llama.cpp kernel must produce identical floating-point bytes. The target is to avoid dismissing unexplained drift as “normal.” If a selected CKE route claims compatibility with a particular production contract, we try to match it exactly. Where exactness is not the contract, we record the dtype, tolerance, schedule, and ranking behavior instead of quietly widening a global threshold.

Making Sure We Are Comparing The Same Run

The first version of X-Ray was useful, but it could join evidence too eagerly. For example, it could label mean absolute error as RMSE, infer an execution identity that the producer never recorded, or attach a vision report to decoder IR. The screen would still look convincing, but the comparison could be wrong.

This is where provenance matters. Here, provenance simply means the fingerprints that tell us which model, generated runtime, subject runtime, oracle, and experiment produced an artifact. Commit f2516797 changed the evidence contract:

• Unknown metrics and execution identities remain null instead of being guessed.
• Ranking correlation requires complete run, model, generated-runtime, subject-runtime and oracle fingerprints.
• Circuit IR is selected by modality, preventing a vision report from silently joining decoder operations.
• An explicit X-Ray directory is exclusive unless fallback loading is deliberately enabled.
• Empty panels provide the producer command and prerequisites instead of merely saying evidence is absent.

We later found another edge case. An early checkpoint could be slightly different but still inside its accepted numerical gate. When a later checkpoint finally failed, X-Ray temporarily blamed that later kernel. Commit b3bba165 changed the classification back to DOWNSTREAM_OR_PROPAGATED_DIVERGENCE. In ordinary language: we can see the error here, but we cannot yet say this operation created it. We need the same-input replay before making that conclusion.

What Went Wrong In Qwen3.6

The recent Qwen3.6-27B bring-up gave us a practical test of this workflow. The quantized checkpoint stored recurrent Q/K projections as Q6_K weights. CKE first needed explicit Q6_K-weight by Q8_K-input contracts for prefill GEMM and decode GEMV. That work reached bit-exact primitive parity against the pinned llama.cpp oracle across 1, 16, 20, and 24 threads.

But the recurrent block still diverged. The projection kernels were now correct, so we followed their outputs into DeltaNet. That exposed a different problem: the values were correct, but CKE was interpreting their shape incorrectly. Qwen3.6 stores 16 compact recurrent Q/K groups but maintains 48 value/state heads. The DeltaNet provider assumed that Q/K and value head extents were identical. Passing a 16-group Q/K buffer to a 48-head contract caused logical out-of-bounds reads during both prefill and decode.

We could have hidden the problem by expanding the buffer or adding a Qwen3.6 conditional inside one C kernel. Instead, we made group_count part of the DeltaNet ABI and sourced it from the model metadata. The kernel now selects the compact Q/K row with head % group_count, and prefill keeps separate strides for Q/K and value data.

for (int head = 0; head < num_heads; ++head) {
    const int group = head % group_count;
    const float *q_head = q + group * q_stride;
    const float *k_head = k + group * k_stride;
    float *state_head = state + head * state_stride;

    recurrent_update(q_head, k_head, value_head, state_head);
}

This is a simplified loop, not the optimized provider copied from the repository. It shows the relationship we needed to represent. Model metadata provides the group count, the circuit asks for grouped recurrent attention, the kernel map selects an ABI that supports it, and generated C receives both num_heads and group_count.

How We Reached The Fix

This took several pull requests because each fix exposed the next problem. We first made BF16 recurrent behavior explicit. Then we corrected full-attention dtype and RoPE metadata. After that came batched prefill policy, the missing Q6 projection route, and finally the grouped Q/K topology. Saying “DeltaNet was broken” would hide all of those separate causes.

What Works Now, And What Still Does Not

After the grouped-topology fix, the pinned llama.cpp DeltaNet oracle became bit-exact for attention output and recurrent state at the real Qwen3.6 geometry: 48 heads, 16 Q/K groups and head dimension 128. The matrix covered decode and 18-row prefill at 1, 16, 20, and 24 threads. Layer-0 token-0 recurrent maximum error fell from 0.151899 to 4.47e-8, and the first three greedy tokens matched.

The short test is now much better, but the long-context work is not finished. At 353 tokens, logit cosine was 0.94280; at 1000 tokens it was 0.87854, even though top-1 remained stable in the measured cases. llama.cpp evaluates recurrent work in 64-token chunks, and CKE has not yet reproduced the complete chunked evaluation order. We fixed the grouped shape problem and moved the first important mismatch further into the run. We have not completed Qwen3.6 parity.

Why This Work Matters For CKE

I do not think CKE is interesting simply because it runs AI on CPUs. llama.cpp already does CPU inference extremely well. Generated C by itself is also not enough. What I want CKE to provide is a connected view from the model architecture to the numerical contract, selected kernel, generated call, memory layout, and comparison evidence.

This investigation is a small example of that idea working. Reading the tensor headers showed that the recurrent projections were Q6_K. The kernel maps showed which GEMM and GEMV providers were selected for prefill and decode. X-Ray then showed that their correct outputs entered the wrong recurrent topology. We updated the circuit and ABI, tested the real 48-head/16-group shape, and could still see the remaining long-context drift afterward.

A lot of CKE is now developed with AI assistance, so this kind of investigative surface is increasingly important to me. The model can read the code, run tests, and propose a repair. I still need a way to inspect what it changed and connect a failure to the right part of the system. X-Ray is one step toward making that practical.

Code, Documentation, And Related Posts

• CKE X-Ray Visualizer runbook
• CKE v8 Numerical Contracts
• CKE IR Pipeline
• What Numerical Parity Actually Requires
• How DeltaNet State And Full Attention Share One Decoder
• Memory Planning In C-Kernel-Engine