C-Kernel-Engine (CKE) is advancing pretty well. I now have more compute to parallelize the work across the P3 and Ryzen nodes, and that has made it possible to add more coverage across very different model families. There is still a lot of work left, but the architecture is holding up decently so far.
Much of this work is being done with help from AI. We inspect a model, identify the kernels it needs, implement or adapt those kernels, add the model template, and create the kernel maps that represent which kernels the circuit is allowed to use. The DSL then stitches those pieces together, plans the memory, and generates the C runtime.
It is slowly becoming that simple. It is not quite there yet, but it is getting closer to the point where most of the work can focus on the mathematics, the kernel, how the kernels stitch together, and how the DSL needs those kernels represented in the kernel map before it can generate code.
As a result, over the last week CKE added or hardened support for AMD Instella-MoE 16B-A3B Think, Cohere Command R7B, Poolside Laguna-XS 2.1, NVIDIA Nemotron Nano 9B v2, the text decoder from Moonshot AI's Kimi-VL A3B Instruct, and Qwen3.8-27B. I was so impressed by the Qwen3.8 result that I wrote a complete post about it yesterday. The point of this work is not only to collect model names. It is to let me, contributors, and anyone using CKE study the kernels these models use and see how those kernels form a complete model circuit.
Frontier labs usually do not publish the complete architecture, weights, and current production kernels behind their strongest models. There are important exceptions. OpenAI shared GPT-2 and later GPT-OSS, and Google publishes the Gemma family. But the open model work from Qwen, Kimi, DeepSeek, GLM, Poolside, NVIDIA, Cohere, and others gives CKE a much broader set of modern circuits to study. One objective of CKE is to make those kernels explicit and understand how they are stitched together to form the models.
Surprisingly, the CKE architecture is holding up decently well. I think. So in this post I want to go through how each of these recently added models works, which kernels and kernel maps it needed, what level of evidence CKE currently has, and which pull requests you can study to see exactly what changed.
The implementation path is documented in the CKE architecture, IR pipeline, and kernel-map documentation. The rest of this post connects those general pieces to the actual model families and merged code.
What CKE Means By Model Support
I do not think a single green checkmark is enough anymore. A runtime can recognize a checkpoint without compiling it. It can compile without producing coherent output. It can produce coherent output while drifting from the reference after a few tokens. It can match a short numerical checkpoint without surviving a long trajectory. It can also be perfectly correct and still be much slower than the established runtime.
The updated CKE model-kernel matrix therefore separates five evidence levels:
- Contract-only: the circuit, model map, shapes and provider interfaces are validated, but a real checkpoint run is not being claimed.
- Coherent end to end: a real model converts, compiles and produces sensible text. This proves the full path is alive, not that every logit matches an oracle.
- Numerically sampled: selected logits, hidden states, tokens or replay positions agree with PyTorch or llama.cpp.
- Trajectory exact: a complete tested token trajectory and its logits match the chosen oracle at the stated boundary.
- Planner-only: the memory planner certifies a context or shape that has not yet completed a corresponding runtime execution.
These labels are not bureaucracy. They stop an eight-token replay from being described as equivalent to Qwen3.8's 4,096-token bit-exact run. They also let useful intermediate work remain visible without inflating the claim.
Cohere2 Changes The Shape Of The Transformer Block
Merged PR #401 added the Cohere2 Command R lane. The important difference is not a new brand name in a switch statement. Cohere2 uses one shared pre-block LayerNorm, sends the normalized stream into attention and the MLP in parallel, and then combines both results with the original residual:
n = LayerNorm(x)
y = x + Attention(n) + MLP(n)That differs from the common serial pattern where the attention residual is completed before a second normalization and MLP. CKE's Cohere2 circuit has to preserve this parallel ownership explicitly. It also applies RoPE only on sliding-attention layers and runs the model-declared logit_scale as a final footer operation after the language-model head.
The current evidence is an eight-position tokenizer-free replay agreement with llama.cpp on Command R7B Q4_K_M. That is useful sampled numerical evidence. It is not yet a long-trajectory certification or a matched performance result, and the public matrix says so.
Laguna Combines Several Ideas In One Circuit
Laguna-XS 2.1 is a more complicated runtime test because it combines several features that are individually easy to describe and difficult to stitch correctly.
Its layer policy alternates global and sliding attention. The RoPE rule changes with that policy: global layers use YaRN while sliding layers use a full rotary path. Q and K are normalized. Attention produces the normal weighted value output, but a separate learned projection produces one scalar per head. Softplus turns that scalar positive and scales the corresponding attention head before the output projection:
A = softmax(QK^T / sqrt(d)) V
g = softplus(H W_g)
Y = concat(g * A) W_oThis is why Laguna needed an explicit attn_gate_softplus_mul operation rather than pretending softplus replaces softmax. Softmax still performs competition across token positions. Softplus creates an independent positive strength for each head output. I derived that distinction and its backward pass in Softmax vs Softplus.
The feed-forward side also changes after the first dense layer. Laguna uses sigmoid routing with a correction bias, selects eight routed SwiGLU experts, and adds a shared SwiGLU expert. Its quantized artifact mixes Q4 and Q6 expert storage. Merged PR #404 expresses those choices in the Laguna circuit, not as Laguna conditionals in the compiler.
The tested 20.27 GB Q4_K_M artifact converted all 684 mapped entries and generated coherent text on the Ryzen node. Embedding and the first layer's RMSNorm are bit-exact, and an identical-input router replay preserves the selected experts with routing weights within one ULP. Long-trajectory parity is still open because small attention and projection differences can flip a near-tied expert route. That is a real remaining numerical problem, not a reason to hide the successful end-to-end bring-up.
Instella Makes BF16 Storage Part Of The Circuit
Instella-MoE 16B-A3B exercises a different branch of the portfolio. Its attention path is gated MLA rather than ordinary GQA. It compresses the KV representation through a LoRA-style latent, reconstructs key and value heads, concatenates a partial rotary section, and stores specific intermediate values at BF16 boundaries.
Those storage boundaries matter. Computing a dot product in FP32 and then storing BF16 is not numerically equivalent to keeping every intermediate in FP32. Repeatedly ignoring where PyTorch rounds to BF16 can produce a small hidden-state difference that grows layer by layer. CKE therefore records the projection, partial-RoPE and KV-decompression storage contracts in the Instella circuit.
Instella also uses FarSkip-style stream handling around its routed and shared experts. The compiler must know which stream is the main residual, which stream bypasses the routed path, and where they are combined. This is exactly the kind of graph that becomes fragile when stream ownership is hidden inside mutable buffers.
Merged PR #389 certified the BF16 safetensors execution lane. At the tested 32-token checkpoint, the full-logit cosine is approximately 0.99998 and top-1 matches PyTorch. Quantized GGUF and long trajectories are not yet certified.
Nemotron Is Recurrent State, Attention And Feed-Forward Policy
Nemotron-H remains one of the clearest reasons CKE cannot assume every layer is attention plus SwiGLU. The model's layer policy can select Mamba2, attention, dense ReLU2 MLP or routed ReLU2 MoE work.
A Mamba2 layer has persistent recurrent state instead of a growing attention KV cache. It performs an input projection, a convolution/state update, a softplus time-step transform, selective state scan, gated RMSNorm and output projection. Attention layers still exist, but they are sparse in the stack. Feed-forward layers can use ReLU(x)^2 rather than SwiGLU. Routed layers add group-limited expert selection and shared-expert work.
I covered those kernels in more detail in the Nemotron architecture report and the Mamba, attention and MoE follow-up. The current Nemotron-H circuit carries those layer kinds as contract data.
The current Q4_K_M lane produces coherent end-to-end text and has PyTorch reference coverage for the Mamba2 state path. Merged PR #402 also fixed the embedded chat protocol so the runtime formats the same conversation the checkpoint expects. Broader long-trajectory and performance sweeps remain open.
Kimi And Qwen3.8 Show Two Different Maturity Points
Kimi-VL A3B's text decoder now has repeatable BF16 execution through merged PR #403. The first-token comparison preserves the top-20 ranking with cosine approximately 0.997696, and repeat runs are bit-identical. The remaining internal BF16 drift begins near layer seven, and the MoonViT vision bridge is not part of the certified lane. CKE therefore says the text decoder is numerically sampled; it does not say the complete vision-language model is finished.
Qwen3.8 is currently at the other end of the evidence spectrum. Its dense hybrid circuit reuses recurrent DeltaNet and full-attention providers CKE had already hardened. Merged PR #387 added the explicit model identity and circuit. Merged PR #388 then compared 4,000 generated positions and found every complete vocabulary row bit-exact with llama.cpp.
That result is why I described Qwen3.8's fast bring-up as an architecture test. The model did not need a new runtime fork because its required arithmetic already existed. It needed a truthful identity, an explicit circuit and a hard numerical gate.
The Same Compiler Path Does Not Mean The Same Model
The reusable path now looks like this:
- Inspect the GGUF or safetensors checkpoint and identify every tensor and configuration field.
- Select a model-owned circuit that states the actual layer policy, residual flow, state and tokenizer/chat contract.
- Map every source tensor into BUMP or fail with an accounting report.
- Resolve each operation against the kernel registry using dtype, layout, phase, ISA and numerical requirements.
- Plan temporary, persistent, KV, recurrent and weight storage.
- Generate C, compile it for the machine, and run the artifact.
- Compare the right checkpoints with PyTorch or llama.cpp and publish the strongest evidence actually earned.
Merged PR #406 updated the public documentation so these families and evidence levels are no longer buried in individual pull requests. The v8 runbook now gives the current execution commands and caveats. The model-kernel matrix shows the circuit, required kernels, strongest evidence, tested context, oracle and remaining gap for each family.
Merged PR #407 adds a resumable long-context certification harness for Qwen3.8, Qwen3.5, Gemma4, GLM4, Nemotron, Instella, Kimi and Laguna. Registering a model in that catalog is not itself a long-context pass. It gives every configured checkpoint the same staged test surface and writes partial results incrementally so multi-day CPU runs can resume rather than start over.
Where The Matrix Stands Today
| Family | Distinct circuit work | Current strongest evidence | Important open boundary |
|---|---|---|---|
| Qwen3.8 27B | Dense hybrid DeltaNet/full attention | 4,096-position trajectory exact with llama.cpp | Long runtime contexts and remaining performance gap |
| Cohere2 Command R7B | Parallel attention/MLP residual, sliding-only RoPE, logit scaling | Eight-position llama.cpp replay agreement | Long trajectory and matched performance |
| Laguna-XS 2.1 | Global/sliding attention, softplus head gate, mixed-quant routed MoE | Coherent E2E plus strong leaf and router evidence | Near-tied route flips and long trajectory |
| Instella-MoE 16B-A3B | BF16 gated MLA, partial RoPE, FarSkip routed/shared streams | 0.99998 full-logit cosine and top-1 match at 32 tokens | Quantized GGUF and long trajectory |
| Nemotron Nano 9B v2 | Mamba2 state, attention, ReLU2 and hybrid layer policy | Coherent Q4_K_M E2E plus PyTorch reference lanes | Broader trajectory and optimization |
| Kimi-VL A3B text | Gated MLA decoder and BF16 KV-LoRA decompression | First-token/top-20 sampled agreement, repeatable execution | Layer-seven drift and MoonViT vision bridge |
This matrix will keep changing. The useful part is that it now tells me what kind of work is missing. A family may need a new circuit, a missing primitive, a storage-boundary correction, a chat-template fix, a longer oracle run, or a faster provider. Those are different engineering problems and should not be collapsed into one unsupported/supported label.
Why I Think This Direction Is Holding Up
CKE still has a long way to go. Several families need longer numerical trajectories. Vision is not complete across every multimodal model. Many kernels remain slower than llama.cpp. Distributed execution and training are active research rather than finished product claims.
But I am increasingly comfortable with the core architecture. New model work is expanding a shared portfolio instead of turning the compiler into a collection of model-name branches. Cohere contributed a different residual graph. Laguna contributed softplus-gated attention and a mixed global/sliding MoE path. Instella forced exact BF16 MLA storage and FarSkip stream ownership. Nemotron forced recurrent state and heterogeneous layer policies. Kimi added another MLA contract. Qwen3.8 showed how quickly an existing portfolio can compose a new model when the required kernels are already present.
This also gets closer to the longer-term reason I am building CKE. Model compatibility is useful evidence, but it is not the final objective. I want kernels, dtypes, quantization formats, gradients and circuits to become reusable trainable building blocks. Today I can study architectures funded by much larger labs, make their mathematical contracts explicit, and run them on CPUs I own. Over time, I want to be able to compose those kernels into smaller experimental circuits, inspect the forward and backward paths, train them on commodity hardware, and learn which combinations actually work.
I do not have the budget to invent and train every architecture from scratch. Open model families give CKE access to the output of very expensive research. The work here is to understand that output deeply enough to reproduce its execution, measure it honestly, and make the pieces reusable without hiding them behind a large framework.
That is why this week's small wins matter to me. CKE does not merely have more model names on a page. It has a broader and better-tested language for describing how modern models are built.