I have been testing Kimi as a long-horizon engineering model and calling the property I care about stamina: can the model read a large repository, keep a numerical contract in mind, run experiments for hours, and still finish coherently? I documented that practical workflow in Why I Am Testing Kimi And ChatGPT Together For Real Work. Kimi was already doing useful engineering work there. With the Kimi K3 weights now released, we can move from judging that behavior to inspecting the machine behind it.

The release includes the weights, model configuration, reference Python implementation, tokenizer, quantization metadata, and a technical report. That does not make the model small or easy to run. It does make its claims inspectable. We can ask a much better question than “is K3 frontier?”: what mathematical circuits produce the computation, and what kernels would a runtime need to execute them?

The honest answer: architecture, scale, training, and harness

Kimi K3's reported capability cannot be assigned to one cause. Four systems changed together:

Moonshot reports an approximately 2.5× improvement in scaling efficiency over Kimi K2. Their own report attributes that result collectively to architecture, data, and training changes. It does not isolate a clean percentage for each component. Any claim that “KDA caused the 2.5× gain” or “2.8 trillion parameters caused it” would go beyond the published evidence.

Start with the released configuration

The released config.json describes a 93-layer, native multimodal MoE model with a hidden width of 7,168 and a context limit of 1,048,576 tokens. One layer is dense. The remaining layers use sparse experts. The language backbone alternates three recurrent KDA layers with one global MLA layer, then ends with one additional global layer:

Layer pattern:
KDA → KDA → KDA → Gated MLA
KDA → KDA → KDA → Gated MLA
...
final layer → Gated MLA

Total: 69 KDA layers + 24 Gated MLA layers = 93 layers

1. Kimi Delta Attention is a real new kernel family

Full attention compares tokens with previous tokens. Its score matrix grows with sequence length. KDA instead maintains a recurrent state for each head. Every token decides how much of that state to retain, what new key/value information to write, and what to read with its query. The report gives the single-head recurrence as:

\[S_t =\left(I-\beta_t k_t k_t^{\top}\right)\operatorname{Diag}(\alpha_t)S_{t-1}+\beta_t k_t v_t^{\top},\qquad\widetilde{o}_t=S_t^{\top}q_t\]

The operational idea is simpler than the notation: S is a fixed-size recurrent memory, alpha forgets part of it, beta controls the write, and the query reads the updated state. During prefill, tokens are processed in parallel within chunks. During one-token decode, the implementation switches to a fused recurrent kernel.

This creates a concrete runtime split:

• KDA prefill: short convolutions, Q/K normalization, decay products, triangular chunk interactions, state update, and dense matrix tiles.
• KDA decode: one recurrent-state update per layer and token, with convolution state and recurrent state kept persistent.

K3 also changes Kimi Linear's unbounded negative-softplus decay into a lower-bounded sigmoid mapping:

\[g_t=g_{\min}\,\sigma(e^A z_t), \qquad\alpha_t=e^{g_t}, \qquad g_{\min}=-5.\]

This is not cosmetic. Bounding the decay keeps reciprocal rescaling within BF16 range over a 16-token tile. Moonshot says that allows both diagonal and off-diagonal causal tiles to use dense matrix multiplication instead of a special position-pair diagonal path. In kernel-engineering language: they changed the mathematics partly so the numerically safe implementation maps better onto the hardware.

2. Gated MLA keeps periodic global attention

KDA is efficient, but recurrence compresses history into state. Every fourth layer therefore uses Multi-head Latent Attention, adapted from the DeepSeek family, to restore unrestricted token-to-token interaction.

MLA compresses each token's key/value content into a 512-dimensional latent vector instead of caching independent full K/V vectors for all 96 heads. K3 uses a 1,536-dimensional low-rank query path, 128 content query/key dimensions, an additional 64-dimensional query/key partition retained in the configuration surface, and 128 value dimensions. Despite the legacy qk_rope_head_dim field name, the released forward path applies no rotary transform. The report describes the K3 MLA path as NoPE: KDA supplies sequence sensitivity, while MLA supplies global content lookup without depending on RoPE scaling for the million-token extension.

K3 adds a full-rank output gate to MLA:

\[y_t=W_o\left[\sigma(W_gx_t)\odot\widetilde{o}_t\right].\]

That gate is another projection plus sigmoid plus elementwise multiplication. It lets each token decide which channels from global attention should pass onward. For a runtime, “MLA support” is therefore not enough; the exact decompression, cache representation, position policy, output gate, and numerical accumulation contract all matter.

3. Attention Residuals make depth an attention problem

A conventional residual stream repeatedly adds each layer's result into one running vector. K3 argues that this compresses all earlier depth into one state, much like an RNN compresses earlier time. Attention Residuals let a layer select among earlier representations instead.

Full depth attention would keep every layer output. K3 uses a block form. The 93 layers are grouped in blocks of 12. Outputs within a block are accumulated, while block summaries remain available as a small set of depth-wise values. A learned scalar projection scores the embedding, previous block summaries, and the current partial block; softmax produces the mixture.

\[\alpha_{i\rightarrow l}=\frac{\exp\left(w_l^\top\operatorname{RMSNorm}(v_i)\right)}{\sum_j\exp\left(w_l^\top\operatorname{RMSNorm}(v_j)\right)},\qquad{}h_l=\sum_i\alpha_{i\rightarrow l}v_i.\]

This is a small attention kernel across depth, not sequence. It adds persistent block representations, normalization, scalar scoring, online softmax, and a weighted reduction. The model code exposes attn_res_block_size: 12. The report says this reduces retained depth state from one value per layer to one per block while recovering most of the full method's benefit.

4. Stable LatentMoE is more than “top-k experts”

K3 contains 896 routed experts per MoE layer and selects 16 for every token. That is 56× sparsity, but “only 16 experts run” should not be confused with “the model is cheap.” Each token still activates 104.2 billion parameters across the full network, and the runtime must find, load, execute, weight, and reduce 16 expert paths at nearly every layer.

The routed experts do not operate at the full 7,168 model width. A down-projection maps the token into a 3,584-dimensional latent space. The selected experts use 3,072-wide feed-forward interiors. Their weighted result is normalized and projected back to model width. Two shared full-width experts run alongside the routed path.

x [7168]
 ├─ shared expert 1 [full width]
 ├─ shared expert 2 [full width]
 └─ latent down projection [7168 → 3584]
      ├─ sigmoid router over 896 experts
      ├─ select 16 experts
      ├─ grouped expert GEMMs with SiTU-GLU
      ├─ weighted reduction
      ├─ RMSNorm
      └─ latent up projection [3584 → 7168]
sum shared + routed paths

This creates several separate kernel contracts: router GEMM, sigmoid, bias-adjusted top-k, token permutation, grouped expert GEMM, SiTU activation, weighted expert reduction, RMSNorm, and latent up/down projections. In distributed execution it also creates expert placement and all-to-all communication problems.

5. SiTU-GLU bounds the expert activation

K3 replaces SwiGLU with Sigmoid Tanh Unit GLU. SwiGLU multiplies two unbounded branches, so coincident activation outliers can become dangerous during low-precision training. SiTU smoothly caps both:

\[\operatorname{SiTU\text{-}GLU}(x)=\left[4\tanh\left(\frac{W_gx}{4}\right)\odot\sigma(W_gx)\right]\odot\left[25\tanh\left(\frac{W_ux}{25}\right)\right].\]

Near zero, the function behaves similarly to SwiGLU. At large magnitudes, its elementwise output is bounded by \(4\times25=100\). This is both a training-stability idea and an inference kernel: a runtime must reproduce the two tanh operations, sigmoid, products, FP32 promotion policy, and final dtype conversion accurately.

6. Native MXFP4 is part of the released model, not an after-market quant

MXFP4 means Microscaling four-bit floating point. It is easiest to understand as a two-level number format. Each individual weight receives a tiny four-bit floating-point payload, while a nearby group of weights shares one larger scale. The payload describes the values relative to one another; the shared scale places the entire group at the right magnitude.

What do the four bits contain?

The Open Compute Project MX specification defines MXFP4 elements using the E2M1 layout:

• 1 sign bit: positive or negative.
• 2 exponent bits: a very small dynamic range.
• 1 mantissa bit: one bit of precision inside that range.

Ignoring the sign, E2M1 can represent only a small set of magnitudes: {0, 0.5, 1, 1.5, 2, 3, 4, 6}. A four-bit payload therefore cannot represent an arbitrary BF16 weight by itself. The missing range comes from a shared E8M0 scale: an eight-bit exponent-only value that represents a power-of-two multiplier.

One scale serves 32 weights

K3's released quantization configuration records group_size: 32, num_bits: 4, floating-point payloads, symmetric encoding, a min/max observer, and a uint8 scale. For a group g, reconstruction is conceptually:

\[\widehat{w}_{g,i}=S_g\,P_{g,i},\qquad{}i\in\{0,\ldots,31\},\]

Here, P[g,i] is one E2M1 payload and S[g] is the shared E8M0 scale. Suppose S[g] = 0.5. Payloads 1.5, -3, and 6 reconstruct as 0.75, -1.5, and 3.0. A source weight between those available values must round to the nearest representable payload; a value outside the block's range may clip. Choosing the block scale well is therefore part of preserving accuracy.

What happens during the expert GEMM?

The runtime should not expand an entire expert matrix to BF16 and then call an ordinary GEMM. That would spend memory bandwidth writing and rereading the expanded weights. A useful kernel performs the conversion inside its matrix tile:

1. Load packed bytes containing two four-bit weight payloads each.
2. Unpack or table-decode the E2M1 values.
3. Load the E8M0 scale shared by the 32-weight block.
4. Decode the corresponding activation block. Moonshot describes the execution recipe as MXFP4 weights with MXFP8 activations.
5. Combine the weight and activation scales with the payload products.
6. Accumulate the dot product using the kernel's higher-precision numerical contract, then convert at the output boundary.

This is why MXFP4 support is a real kernel contract, not merely a file loader. Packing order, scale interpretation, rounding, subnormal behavior, accumulation order, and conversion dtype can all change the output. The official K3 summary describes MXFP4 weights and MXFP8 activations with quantization-aware training. The Hugging Face checkpoint configuration separately records the persistent MXFP4 weight packing; its input_activations: null field should not be misread as a denial of Moonshot's runtime/training activation recipe.

How is this different from Q4_K or ordinary integer Q4?

All three approaches reduce weight bandwidth, but their bit meanings and kernels differ. A Q4_K kernel cannot read MXFP4 bytes correctly just because both names contain “4.” MXFP4 gives every payload a tiny local exponent and then gives the block another shared exponent. Integer Q4 instead distributes a finite set of integer levels across a scaled interval.

Why K3 can use four-bit expert weights

Moonshot did not simply train a BF16 model and compress everything afterward. It reports quantization-aware training from supervised fine-tuning onward. During that process, the model experiences the rounding and range limits it will face at deployment, so its parameters can adapt. K3 also applies MXFP4 selectively. The routed experts that dominate storage use it, while the released configuration excludes attention projections, shared experts, latent-MoE projections, the dense MLP path, language-model head, vision tower, and multimodal projector.

That selective boundary is the practical compromise: compress the enormous routed-expert pool aggressively while retaining more precision in smaller or more numerically sensitive paths. It is not equivalent to converting the entire checkpoint to a generic four-bit format after training.

What is training-only or mostly training-side?

Quantile Balancing

The router uses sigmoid scores, adds a learned expert-specific correction bias for top-k selection, and then normalizes the original selected scores. During training, Quantile Balancing estimates score quantiles so each expert receives a target load without a manually tuned fixed-step bias rule. At inference, that final correction bias is frozen. The runtime needs the biased top-k operation; it does not run the histogram-based quantile update.

Per-Head Muon

Per-Head Muon is an optimizer policy. It applies Muon independently to attention heads so their updates can adapt separately. It affects the weights we receive, but an inference engine does not execute the optimizer.

Native multimodal and long-context training

MoonViT-V2 was trained from scratch with the same next-token objective as the language backbone rather than attached later from a contrastive vision model. Pre-training begins at 8K context, extends to 64K, and is later trained for much longer context. Data includes web text, code, mathematics, knowledge, images, OCR, video, SVG, webpages, games, 3D assets, and CAD. These choices can plausibly contribute as much to behavior as any isolated kernel, but they cannot be reconstructed from config.json.

Why “104B active” still understates the systems problem

Sparse activation reduces arithmetic relative to executing all 2.78T parameters, but the complete expert pool still has to live somewhere. At four bits, 2.8 trillion raw weight values alone imply roughly 1.4TB before scales, non-expert BF16 tensors, metadata, recurrent state, caches, and runtime workspace. The 96 released weight shards are a useful reminder that open weights do not automatically mean laptop weights.

K3 also has two different long-lived sequence memories:

• KDA state: fixed-size recurrent and short-convolution state per request and layer.
• MLA cache: compressed token-wise cache that still grows with sequence length in the 24 global layers.

Moonshot's serving design packs both into one paged cache manager, but their lifetimes remain different. Prefix reuse is valid only when the MLA prefix and the corresponding KDA state checkpoint agree at the same token boundary. This is exactly the kind of problem discussed in Memory Planning in C-Kernel-Engine: the arithmetic can be correct while cache identity, lifetime, or movement is wrong.

What would CKE need to support?

C-Kernel-Engine does not currently support Kimi K3. The useful exercise is to treat the released model as a circuit-specification test. A serious CPU path would need:

This is why I view models as circuits rather than monoliths. CKE's template and kernel-map approach should let a new family declare these operations explicitly, lower them into inspectable IR, bind them to numerical contracts, and generate C. K3 is a severe test of that thesis because it combines recurrent attention, global attention, depth attention, extreme MoE sparsity, multimodality, and native low precision in one decoder.

Could a distributed CPU system run it?

Comparing Kimi K3 with one workstation would be meaningless. One GPU cannot run it either. Moonshot recommends a supernode containing at least 64 accelerators. K3 is a distributed-systems workload on every hardware architecture, so the fair comparison is:

distributed CPU cluster versus distributed accelerator supernode — measured by usable model capacity, memory bandwidth, network traffic, power, capital cost, time to first token, and tokens per second.

The released checkpoint gives us enough information to state the CPU hypothesis concretely. The 2.78T parameters require at least about 1.4TB even if every value occupied exactly four bits. The actual requirement is higher because only routed-expert weights use MXFP4; attention, shared experts, latent projections, the language head, the vision tower, scales, caches, and workspace use other representations. No single component needs to hold all of that if the model is partitioned across nodes.

Decode viability can be framed as a bandwidth lower bound instead of a slogan:

\[t_{\mathrm{token}}\;\gtrsim\;\frac{B_{\mathrm{active\ weights}}} {\mathrm{BW}_{\mathrm{effective,\ cluster}}}+t_{\mathrm{collectives}}+t_{\mathrm{state}}+t_{\mathrm{imbalance}}.\]

Here, \(B_{\mathrm{active\ weights}}\) is the actual number of weight bytes read for the 104.2B activated parameters under K3's mixed-precision layout. Effective cluster bandwidth is not the sum printed on DIMM boxes: it is the bandwidth the complete routed circuit sustains after NUMA effects, cache misses, dequantization, thread scheduling, and imperfect expert reuse. The remaining terms charge for network collectives, KDA/MLA state work, and whichever node receives the slowest expert assignment.

This is not a peripheral experiment for CKE. It is one of the project's central research programs. CKE is being built to determine how far modern AI can be carried by distributed CPU systems assembled from open, expandable server hardware. CPUs bring large aggregate DRAM capacity, substantial memory bandwidth, large LLC, many PCIe lanes, local NVMe tiers, and standard high-speed Ethernet or RDMA. Those properties create a different systems design space from an accelerator supernode: more emphasis on placement, movement, sparsity, memory hierarchy, and cooperation between nodes rather than peak arithmetic throughput alone.

The motivation is accessibility across hardware. Open weights are only partly accessible if useful execution still requires a narrow class of systems that few individuals, laboratories, schools, or small organizations can own. CPU servers are widespread, incrementally upgradeable, and available across generations and vendors. CKE is aggressively investigating whether quantization, sparse expert placement, pipeline and tensor parallelism, tiered memory, generated kernels, and explicit communication plans can turn that installed hardware base into a serious AI platform.

The practical program begins with two nodes, not because two nodes are the final target, but because that is the smallest system that exposes the real distributed problems: transport, partitioning, numerical parity across boundaries, failure recovery, scheduling, and communication hidden behind computation. From there, CKE can add memory channels, faster networking, storage tiers, and nodes methodically while measuring local weight streaming, expert dispatch, collectives, recurrent state, global attention, and pipeline bubbles.

CPU execution is routinely underestimated because comparisons often place one ordinary workstation against an accelerator cluster. CKE rejects that comparison. K3 must be evaluated as a distributed workload on both sides. We are not neutral about the research direction: we believe CPUs have much more practical AI capacity than current software exposes. But that belief must become generated code, reproducible parity tests, profiler traces, scaling curves, and working multi-node demonstrations. Producing that evidence is the core work.

How to interpret the benchmark claims

Moonshot reports frontier-level results across coding, reasoning, vision, and knowledge-work evaluations, while also stating that the overall user experience still trails its strongest proprietary comparisons. The detailed tables matter, but so do their footnotes. Different models sometimes use different agent harnesses; K3 commonly uses Kimi Code, OpenAI models use Codex, and Claude models use Claude Code or other harnesses. Reasoning effort, tool availability, refusals, fallbacks, context management, and repeated-run policy can all change a score.

The correct conclusion is not that benchmarks are useless. It is that a model plus harness is the evaluated system. K3's preserved-thinking-history requirement makes this unusually explicit: omit earlier reasoning content or switch models halfway through a session, and Moonshot warns that quality can become unstable.

My conclusion

Kimi K3 is not “just scale.” Its released implementation contains meaningful new circuits whose numerical and memory behavior would force real work in any inference engine. It is also not “one architectural breakthrough.” The frontier gap narrowed through a coordinated system: 2.8T parameters, 104B active computation, hybrid attention, depth retrieval, extreme sparse experts, bounded activations, native multimodality, quantization-aware post-training, data, optimization, and an agent harness designed for long work.

The most important part for independent engineering is the release itself. We are no longer limited to a product demo and a benchmark chart. We can read the configuration, inspect the model code, trace the equations, see which weights are quantized, reproduce small kernels, and ask what it would take to make this class of intelligence practical on hardware people can own.

Sources and related reading

• Moonshot AI: Kimi K3 model card and open weights
• Kimi K3 technical blog
• Kimi K3 technical report
• Released KDA, MLA, MoE, SiTU, and AttnRes reference code
• Why I Am Testing Kimi And ChatGPT Together For Real Work
• What Is the C-Kernel-Engine?
• What Numerical Parity Actually Requires in CKE
• How CKE Kernel Maps Bind Exact C Functions to Numerical Contracts
• DeltaNet vs State Space Models
• Memory Planning in C-Kernel-Engine
• Pipeline vs Tensor Parallelism
• C-Kernel-Engine documentation