I am revisiting this gap today because it is the measurement I keep being asked about, and because the honest answer has changed shape since I first wrote it down. The numbers below — 281 seconds down to 76.7 seconds on the Qwen3-VL OCR path, against a llama.cpp baseline around 55.8 — are the historical record from early July, and I am leaving them untouched. What machine produced them? A Kubernetes pod on an Intel Xeon Gold 6542Y host — Emerald Rapids, so AVX-512 with AMX on die. The pod could see all 24 physical cores (48 logical with Hyper-Threading) and about 250 GiB of host memory behind a 56 GiB container limit, but a cgroup quota capped execution at roughly 14 CPU equivalents through CFS time-throttling rather than a pinned cpuset. The July ladder runs used 20 threads on that host; the later Qwen3.6-VL runs used 14. That machine description is itself part of the lesson: I did not record it in the original notes at all. That omission bugged me enough that it later became its own post — the nightly report now names its machine precisely because this one didn't. What has changed since July is everything around the numbers: the harness got stricter, a second vision family arrived, and several mechanisms I listed as future work have shipped.

The lesson this build log taught me, twice now: a gap with a name, a tensor shape, and a measurement harness is engineering. A gap without those is folklore. When I wrote the first version, the gap had just gotten its name — Q4_K × Q8_K Gate-Up SwiGLU in mixed prefill. In the weeks since, the harness became the more interesting half of the story, because a number you cannot re-certify is a number you will eventually stop believing.

This post is for CPU inference and runtime engineers, for people following C-Kernel-Engine as it chases llama.cpp, and for anyone who has ever asked whether a young runtime's gap to a mature one is a wall or a map. It is a map.

C-Kernel-Engine v8 · Qwen3-VL · CPU performance hardening

This post is a performance hardening note about C-Kernel-Engine chasing the Qwen3-VL OCR gap against llama.cpp on CPU. It is intentionally detailed. Performance work without measurements becomes folklore. The useful question is not whether CKE is already faster than llama.cpp. It is not. The useful question is whether the gap has become measurable, localized, and attackable.

The short version: CKE moved a Qwen3-VL OCR path from roughly 281 seconds to roughly 76.7 seconds on a Kubernetes-hosted Intel Xeon Gold 6542Y pod, while a llama.cpp report baseline is still around 55.8 seconds per sample. That is not a victory lap. It is a map. The bottleneck moved from "the whole bridge is slow" to specific quantized projection kernels, especially Q4_K x Q8_K Gate-Up SwiGLU in mixed prefill. Once a bottleneck has a name, a tensor shape, and a measurement harness, it becomes engineering.

Section 1: Qwen3-VL Is Not Ordinary Text Decode

A small text-only GGUF model is already enough work for a young runtime. Qwen3-VL adds another layer of difficulty (I wrote the vision-side mechanics up separately in How Qwen3-VL Vision Works). A real image has to become patches. Those patches become a vision encoder sequence. The encoder output crosses a projector. The projected rows become a visual prefix. That prefix must be stitched into the decoder prefill path with text tokens, MRoPE-style grid metadata, and a persistent KV cache that can continue generation one token at a time.

That means the bridge is not decorative glue. The bridge is the runtime contract that decides whether image-derived rows enter the decoder as a valid prefix or as broken tensor noise. If the bridge is wrong, speed measurements are not meaningful. If the KV cache handoff is wrong, the first generated token may look plausible while the continuation is already poisoned. If the visual prefix is scheduled like one-token decode, the runtime can lose throughput before the hot kernel optimization even starts.

The important engineering shift is that CKE is no longer only proving text inference. The v8 lane now has to prove that a model artifact can be converted, lowered, memory-planned, code-generated, compiled, bridged, executed, inspected, and measured across both vision and language components.

Section 2: The Speed Ladder

The raw numbers are useful because they prevent vague optimism. The Qwen3-VL OCR path started painfully slow. On the Kubernetes Xeon host, the historical wall-clock ladder looks like this:

The first big improvement was not a magic SIMD trick. It was a runtime-shape fix. Qwen3-VL OCR is dominated by prefix work: many image-derived rows plus text prompt rows. Treating that path too much like decode is structurally wrong. The staged bridge made the runtime behave more like prefill for the prefix, then continue as decode after the prefix state exists.

Section 3: What The Bridge Fix Actually Changed

Before the staged bridge work, the mixed visual/text prefix was too close to a decode-shaped activation plan. That is the wrong mental model. Decode is the single-token continuation phase. Prefill is the phase where the model processes many tokens together and fills the KV cache. A multimodal visual prefix belongs much closer to prefill than decode.

The current staged path prepares a hybrid decoder runtime: decode IR plus prefill-sized activations for the prefix. The mixed visual prefix runs batched. Then generation continues through incremental decode. The fix that copied staged prefill KV into the f16 decode cache is part of that seam. It is not glamorous, but it matters: after the prefix is processed, the decode runtime must continue from the correct cached state.

This is why I do not like treating "the bridge" as a small wrapper around the real model. The bridge is a model execution boundary. It controls row semantics, grid metadata, prefix length, KV continuation, prompt segmentation, and runtime selection. If that boundary is weak, everything downstream looks slow and confusing.

Section 4: The Bottleneck Moved

After the bridge became less wrong, the profile became more useful. The current CK gap is specific and measurable: decoder mixed prefill remains largely Q4_K/Q6_K x Q8_K projection work. That is good news. It means the next speed work is no longer "make Qwen3-VL faster" as a vague wish. It is "attack the quantized projection kernels that dominate mixed prefill."

This is one of the reasons CKE keeps investing in visualizer and report surfaces. The runtime should not only emit tokens. It should tell me where the time went. It should expose the IR, dtype decisions, bridge reports, per-op wall-clock CSVs, and run artifacts. Without that, performance work becomes vibes.

Section 5: Why Q4_K Gate-Up SwiGLU Is Part Of The Qwen3-VL Story

The Q4_K Gate-Up SwiGLU work is directly related to Qwen3-VL. It is not a random microbenchmark. The research harness targets the real Qwen3-VL OCR MLP hot path.

M = 1028 mixed visual/text tokens
D = 12288 intermediate half-dim
K = 4096 input dim

weights ~= 54 MiB Q4_K gate+up matrix
output  ~= 48 MiB FP32
scratch ~= 96 MiB if gate/up are materialized unfused

That shape explains why this is hard. The kernel is not operating on a cute toy vector. It is moving a large quantized weight matrix, quantizing or consuming Q8 activations, accumulating enough output to feed SwiGLU, and doing it while cache capacity, memory bandwidth, register pressure, thread scheduling, and output-loop vectorization all matter.

The current x16 path is intentionally opt-in behind CK_ENABLE_Q4K_GATEUP_SWIGLU_X16=1. That is the right discipline. A kernel can look good in a standalone benchmark and still fail as a default runtime path if packing cost, cache pressure, CPU family differences, or model-level wall-clock do not hold up.

Section 6: First Principles, Then Measurement

The part I care about most is not only that CKE got faster. It is how the runtime studied the problem. The useful workflow is first principles, then measurement.

The thread sweep is a good example. The naive idea is that more threads should help. But this kernel is constrained by cache/core/memory-bandwidth balance. Every worker needs useful cache capacity and enough bandwidth to consume packed weights and activations. At some point, more workers create pressure faster than they create useful throughput.

This is the kind of rule CKE should eventually turn into runtime policy. Not hard-coded "always use all cores." Not blind OpenMP. Not one magic number copied from one server. The runtime should derive a default from the shape and the CPU topology, then allow override and measurement.

Section 7: Rejected Ideas Are Part Of The Runtime

One of the more useful outcomes was a rejected idea. The dual gate/up accumulator helper was numerically clean. It shared one Q8 activation traversal while updating both gate and up accumulators. That sounds attractive. On this Xeon, it was slower.

The likely reason is register and instruction pressure. Saving a traversal is not automatically a win if the resulting loop is harder for the CPU to schedule. That does not mean the idea is globally bad. It means this CPU rejected it. CKE should retest it on Ryzen/X3D, EPYC, and larger-cache Xeon systems before deleting the idea from the mental model.

This is why CPU diversity matters. A runtime targeting CPUs cannot only tune for one Kubernetes Xeon pod. Different cache sizes, memory channels, SIMD width, frequency behavior, and SMT tradeoffs can change which kernel idea wins.

Section 8: Why llama.cpp Is Still Ahead

llama.cpp is still faster on this Qwen3-VL OCR report. The honest explanation is not mysterious: llama.cpp has mature packing, mature quantized projection kernels, mature executor scheduling, and years of model-family edge cases behind it. Its hot loops are closer to the practical cache/memory roofline.

That is the bar. CKE is not competing against a toy. It is chasing a mature CPU inference runtime. The useful comparison is therefore not "CKE has a clever idea." The useful comparison is "where does CKE still lose wall-clock time, and can the runtime now name the exact region?"

On this path, the answer is yes. The gap is now concentrated in places CKE can attack:

• packed/reused Q4_K/Q6_K x Q8_K batched prefill kernels
• conversion/load-time prepacking instead of lazy runtime packing
• cache-derived active-thread defaults
• model-level sweeps across CPU families
• continuing bridge timing splits for Qwen3-VL, Gemma4-VL, and larger image-token workloads

Section 9: Since This Was First Written (August 2026)

The ladder above is the historical record, and I am leaving it exactly as measured. But the repo did not stand still, and an honest build log has to say where things stand now. Three threads moved.

The evidence system got stricter. Readers who followed theWhisper evidence arcwill recognize the pattern. On the vision side, the July 11 X-Ray handoff notes (QWEN3VL_BF16_*_2026-07-11) put Qwen3-VL BF16 under the same parity microscope thatfound Qwen3.6's first bad circuit. The 40-image private OCR corpus became a real certification lane — certify_qwen3vl_llamacpp_corpus_v8.py runs three-way native CLI / Python / llama.cpp checks and fails closed — and PRs#358 and#359added an explicit Qwen3.6-VL profile and registered both corpus lanes in the nightly. PR#347hardened X-Ray runtime alignment, and docs/notes/MODEL_AGNOSTIC_VISION_CORPUS_CERTIFICATION.md turned the whole thing into a reusable design instead of a Qwen-only script.

New performance mechanisms landed. PR#329stopped vision attention from repacking invariant BF16 K/V tiles per query block and dropped a head-major→token-major transpose per layer. Commit 6bbc4ebcc eliminated attention output layout copies — prefill attention providers now write the consumer layout directly (on a controlled Qwen3-0.6B text-prefill shape that measured a narrow 965.8 ms vs 990.46 ms same-host edge over llama.cpp; a microbenchmark, not the OCR path). The bridge itself became a first-class citizen: eb56ef355 moved multimodal prefix-insert and M-RoPE position construction into DSL-resolved kernel-map providers, and 3ccd3c523 composed Qwen3.6-VL explicitly as a circuit — vision encoder, decoder, and visual-prefix stitch declared, zero implicit DeepStack injection. PR#357fixed segmented prefill parity. One newer data point belongs to this family context, not to the ladder above: the Xeon handoff for the DSL-resolved bridge ran a real 1,027-token Qwen3.6-VL OCR prefill in 117.36 seconds, and the known teacher-forced step-3 token moved from [ to llama.cpp's { (cosine 0.998983 → 0.999516). That is a different model and a different measurement boundary than the 76.7s Qwen3-VL ladder, so I am reporting it as its own fact, not as an update to the old numbers.

The family context widened. Qwen3.6-VL now runs alongside Qwen3-VL, and the publicnightly test reporttracks vision rows and names the runner hardware that produced them — that story got its own post,A Green Checkmark Should Name Its Machine.

What has not changed: there is no newer apples-to-apples Qwen3-VL OCR timing against llama.cpp in the repo. The last certified numbers — 76.7s versus 55.8s — stand as the most recent honest measurement of that specific gap. What exists now that did not exist in July is the harness that will produce the next ones: the certified corpus lane, in the nightly, on named hardware. When the gap number moves, it will move on the record.

Section 10: Why This Matters For CKE As A Product

The larger point is that CKE is becoming both an execution engine and a measurement engine. That matters because CPU AI runtime work is not only about emitting tokens. It is about converting a model artifact into an inspectable execution artifact.

For Qwen3-VL, that means CKE needs to expose:

• the model conversion path from GGUF into CKE's runtime format
• the vision encoder graph and decoder graph
• the bridge report that proves image rows crossed into the decoder correctly
• the IR report that shows lowering, memory layout, and kernel flow
• the dtype audit that shows quantized tensor formats
• the per-op wall-clock profile that names the bottleneck
• the benchmark harness that tests kernel ideas outside the full model
• the parity checks that prevent speed from corrupting the output

This is also why the IR Visualizer and IR Hub are part of the product story, not just internal debugging UI. When I say CKE makes a model inspectable, I mean the runtime should make it possible to ask: what graph did we lower, what dtype did each tensor use, where did the bridge put the visual prefix, what kernel ran, how much time did it take, and what should be optimized next?

Section 11: Summary

The important result is not that CKE is already faster than llama.cpp. It is not. The important result is that CKE can now describe the gap with engineering precision — and, since July, re-certify that description every night. The path moved from roughly 281 seconds to roughly 76.7 seconds. The llama.cpp target is still around 55.8 seconds per sample. The bridge is no longer the only story. The current hot region is quantized projection work, especially Q4_K x Q8_K Gate-Up SwiGLU in mixed prefill.

That is a much better place to be. The runtime can now say: this many milliseconds are in the encoder, this many are in mixed prefill, this many are in Q4_K Gate-Up SwiGLU, this thread count regresses because cache/core pressure wins over parallelism, this kernel idea was numerically clean but slower on this CPU, and this is the next target.

CKE Qwen3-VL performance hardening =
  bridge correctness
  + staged mixed prefill
  + measurable encoder/projection splits
  + Q4_K/Q8_K kernel research
  + cache/core ratio modeling
  + empirical sweeps
  + parity gates
  + CPU-family retesting

Once the bottleneck has a name, a tensor shape, and a measurement harness, it becomes engineering. That is the real progress.