DeepSeek-V4 is useful to study because it shows where modern transformer architecture is going once the simple dense-attention block stops being enough. The core transformer operations are still familiar: projection GEMMs, RMSNorm, RoPE, softmax, attention, MLP/MoE, residual streams, and optimizer updates. But the way those pieces are arranged is changing.

The headline is this: DeepSeek-V4 is not just a bigger transformer. It is a long-context kernel and memory architecture. mHC changes the residual highway. CSA and HCA change attention memory. DeepSeekMoE changes FFN compute. Muon changes the training update. FP8/FP4 paths change storage and deployment. core idea At million-token context, architecture becomes systems design. The model is deciding what information stays exact, what gets compressed, and what gets sparsely retrieved.

0. Glossary For The Sprint Review

These names are easy to mix up because DeepSeek combines architecture, training, inference, and serving-system ideas. This table is the short version I would keep open during the sprint review.

Multi-head Latent Attention in one paragraph

Multi-head Latent Attention is DeepSeek’s KV-cache compression idea from V2/V3 lineage. In normal attention, the model stores keys and values for every token and every relevant head. MLA stores a smaller latent representation and uses projection/reconstruction paths so attention can still compute useful query-key/value interactions. The practical reason is memory: at long context, KV cache becomes a serving bottleneck. MLA says, “do not store the full expensive KV form if a smaller latent cache can recover what the attention layer needs.”

Cross-entropy formula

If the team asks what CE means in the MTP section, this is the formula. The model outputs logits, softmax converts logits into probabilities, and cross-entropy penalizes the probability assigned to the correct token.

1. The DeepSeek Line: What Changed From V2 To V4?

DeepSeek-V2 introduced two ideas that matter for runtime design: Multi-head Latent Attention, or MLA, and DeepSeekMoE. MLA attacks KV cache growth by storing a compressed latent representation instead of a full key/value cache. DeepSeekMoE attacks dense FFN cost by activating only a subset of routed experts plus a shared expert path. If the KV-cache part feels abstract, read the earlier KV Cache Memory post first; if the MoE/runtime part feels abstract, compare this with the Nemotron architecture note.

DeepSeek-V3 keeps that direction and scales it aggressively. It adds stronger MoE training infrastructure, auxiliary-loss-free balancing, multi-token prediction, and serious FP8 training work. DeepSeek-R1 is mainly a reasoning/post-training story, but it matters for systems because reasoning workloads create long decode traces and repeated rollout pressure. The optimizer side connects directly to my Muon Optimizer post, while the low-precision deployment side connects to K-quants and the broader quantization deep dive.

DeepSeek-V4 moves the bottleneck to million-token context. At that length, ordinary attention and ordinary KV cache layout are the problem. The V4 answer is hybrid attention: CSA plus HCA plus a sliding-window branch. The V4 stability answer is mHC. The V4 training answer includes Muon and additional MoE/router stability machinery. This is also why the post belongs near the DeltaNet versus SSM discussion: modern architectures are all asking how much full photographic attention can be replaced, compressed, streamed, or interleaved without destroying model quality.

2. MTP: Multi-Token Prediction

MTP stands for multi-token prediction. The simple version is: instead of training the model only on the next token, the model also learns to predict one or more future tokens from the same hidden stream. If normal language-model training asks, “given tokens up to position t, predict token t+1,” MTP adds auxiliary prediction targets such as t+2 or t+3.

Baseline: the final LM head

Before MTP makes sense, the team needs the baseline shape clear. At the end of a decoder-only transformer, the model has one hidden vector per input token. If the context length is T, hidden width is D, and vocabulary size is V, the final matrix multiply is:

This means the final layer takes [tokens × embed_dim], multiplies by [embed_dim × vocab_size], and produces [tokens × vocab_size]. Softmax is then applied row-wise across the vocabulary dimension. Each row answers: “at this position, what should the next token be?”

H[0] -> logits[0] -> softmax over vocab -> target token[1]
H[1] -> logits[1] -> softmax over vocab -> target token[2]
H[2] -> logits[2] -> softmax over vocab -> target token[3]
...

training uses all rows
inference usually uses only logits[T-1]

The important correction is that normal cross-entropy already trains on every position in the sequence. If the context has T tokens, the model predicts token[1] from H[0], token[2] from H[1], and so on. Those losses already accumulate gradients into the shared transformer weights. MTP changes the horizon attached to each individual hidden state.

This is easy to misunderstand. MTP does not automatically mean inference always accepts multiple tokens in one step. It gives the model extra future-token prediction paths. A serving system can then use those paths for speculative-style acceleration, but correctness still requires an acceptance or verification rule. important distinction MTP is first a training signal. Multi-token decode speedup is a serving algorithm built on top of that signal.

A useful way to think about MTP is as “more gradient per token.” The hidden state is not only punished or rewarded for predicting the immediate next token. It also receives pressure to encode information that helps predict the next few positions. That can improve representation learning and can also create a native draft path for future-token proposals.

MTP training algorithm

1. Run the normal transformer forward pass and obtain hidden states H[t].
2. Compute the main LM head logits for token t+1.
3. For each future step k, run an MTP projection/head that produces logits for token t+k.
4. Compute cross-entropy for each future target.
5. Weight and sum the auxiliary losses with the main next-token loss.
6. Backpropagate all losses into the shared hidden stream and into the MTP heads.

H = transformer(tokens)

# normal CE already does this over every sequence position
logits_1[t] = lm_head(H[t])            # predict token[t+1]
loss = cross_entropy(logits_1[:, :], tokens[1:])

# MTP adds extra shifted targets from the same H[t]
for k in 2..K:
    H_k[t] = mtp_block_k(H[t])
    logits_k[t] = mtp_head_k(H_k[t])  # predict token[t+k]
    loss += alpha_k * cross_entropy(logits_k[:-k], tokens[k:])

backward(loss)

How MTP can help inference

During ordinary autoregressive decoding, the model samples one token, appends it, and runs again. With MTP, the model may have additional heads that propose future tokens. A serving engine can treat those proposed tokens like a draft:

1. Main head proposes token t+1.
2. MTP head proposes token t+2 or beyond.
3. The engine checks whether the future proposal remains valid after the accepted previous token.
4. If valid, it accepts more than one token for that round.
5. If invalid, it falls back to normal one-token decode from the accepted prefix.

This is why MTP belongs near the CKE inference roadmap but should not be confused with the attention kernel itself. MTP touches the LM head, loss path, decode scheduler, KV-cache update policy, and acceptance logic. It does not replace CSA/HCA. It sits above them in the token-generation loop.

For the C-Kernel-Engine build path, the clean first version is not speculative decoding. The clean first version is training parity: attach one auxiliary MTP head, compute two cross-entropy losses, and verify that the combined backward pass correctly accumulates gradients into the shared hidden state. After that, a decode experiment can use the MTP head as a draft proposal path.

3. mHC: Manifold-Constrained Hyper-Connections

mHC is not attention. It is not MoE. It is not a tokenizer or embedding trick. It is a residual-connection algorithm. Normal transformers carry one residual stream per token. mHC carries multiple residual lanes per token, commonly four.

A normal Pre-LN residual block has this shape:

mHC expands the residual state from [T, D] to [T, n, D]. With n = 4, every token carries four residual lanes:

The layer still runs a normal hidden-width block. Attention or MoE does not suddenly become four times wider. The width expansion lives in the residual highway. The algorithm learns how to read from the lanes, how to write back into the lanes, and how to carry/mix old lane state forward.

The 8-step mHC process

1. Start with the multi-lane residual state \(X_l \in \mathbb{R}^{T \times n \times D}\).
2. Flatten per token from [n, D] into [nD] for a lightweight controller path.
3. Apply RMSNorm to the flattened residual controller input.
4. Project the controller input to raw H_pre, H_post, and H_res coefficients.
5. Gate H_pre and H_post with sigmoid-style transforms so the read/write weights stay controlled.
6. Exponentiate raw H_res so entries become positive.
7. Run Sinkhorn-Knopp row/column normalization, often 20 iterations, so H_res is approximately doubly stochastic.
8. Compute the next residual state from stable carry plus layer write-back.

The reason DeepSeek constrains H_res is stability. If the residual carry matrix is unconstrained, repeated multiplication across many layers can amplify or attenuate the residual signal. A doubly stochastic matrix has non-negative entries, rows that sum to one, and columns that sum to one. This makes the residual lane mixer behave like stable convex mixing instead of an arbitrary amplifier. hc_mult = 4 DeepSeek-V4 config uses four residual lanes per token, with Sinkhorn iterations to stabilize the carry matrix.

CKE reference target for mHC

The current DeepSeek scalar-reference patch adds a direct contract for lane mixing:

void deepseek_mhc_mix_f32(const float *streams,
                          const float *mix,
                          float *out,
                          int tokens,
                          int n_streams,
                          int dim);

// streams: [tokens, n_streams, dim]
// mix:     [tokens, n_streams, n_streams]
// out:     [tokens, n_streams, dim]

That is the correct first CKE step: pin parity for the residual mixer before optimizing. The full mHC path then adds controller RMSNorm, projection, sigmoid transforms, Sinkhorn, read, block function, post write, backward, and recompute strategy.

4. Sinkhorn-Knopp: Why The Carry Matrix Does Not Blow Up

The Sinkhorn step is easy to describe but important. Start with a raw 4 × 4 matrix. Exponentiate it so every entry is positive. Then repeatedly normalize columns and rows until the matrix is approximately doubly stochastic.

def sinkhorn(raw, iters=20, eps=1e-6):
    M = exp(raw)
    for _ in range(iters):
        M = M / (M.sum(axis=0, keepdims=True) + eps)  # columns
        M = M / (M.sum(axis=1, keepdims=True) + eps)  # rows
    return M

From a kernel standpoint this is not a large GEMM. It is a tiny matrix normalization kernel repeated across tokens and layers. The engineering problem is memory traffic, deterministic reduction order, and whether to save or recompute the tiny controller intermediates during backward.

5. DSA: DeepSeek Sparse Attention

DeepSeek Sparse Attention, or DSA, is a separate line of work from the V4 hybrid CSA/HCA system, but it is conceptually important. DSA asks: instead of attending densely to all keys, can we cheaply select the useful keys or blocks first? The key primitive is top-k selection followed by softmax over selected scores.

In kernel language, DSA is not magic. It is a sparse attention pattern:

CKE already has the generic building blocks: topk_kernels.c implements top-k and top-k softmax for MoE-style routing, and the DeepSeek patch adds deepseek_dsa_topk_softmax_f32 plus a backward wrapper. This is exactly the kind of scalar reference kernel that should exist before SIMD or threaded versions.

void deepseek_dsa_topk_softmax_f32(const float *scores,
                                   int *indices,
                                   float *weights,
                                   int tokens,
                                   int heads,
                                   int key_count,
                                   int top_k);

6. CSA: Compressed Sparse Attention

CSA is the more detailed long-context branch in DeepSeek-V4. The idea is to compress nearby token groups into compressed KV entries, then use an indexer to select which compressed entries matter for a query. V4’s local report uses compression rate m = 4. That means the long context is shortened by about 4× before sparse selection.

CSA has four conceptual stages:

1. Project hidden states into candidate compressed KV streams and compression logits.
2. Compress each local token group into fewer KV entries using learned softmax weights.
3. Run an indexer: query-to-compressed-key scores, then top-k compressed block selection.
4. Run attention only over selected compressed entries, then add the sliding-window branch.

The compression step is important. It is not just average pooling. The compressed entry is a learned weighted sum over local candidate KV entries. From a kernel perspective, this is a blockwise softmax plus weighted reduction.

The current CKE DeepSeek patch adds the attention-side reference contract:

void deepseek_csa_attention_f32(const float *q,
                                const float *k,
                                const float *v,
                                const int *indices,
                                float *out,
                                float *attn,
                                int query_tokens,
                                int key_tokens,
                                int heads,
                                int dim,
                                int top_k,
                                float scale);

This does not yet mean the whole CSA system is complete. It means the sparse attention math can be pinned against PyTorch before adding the compression kernel, indexer-key construction, top-k indexer scoring, sliding-window merge, grouped output projection, and cache manager.

7. HCA: Heavily Compressed Attention

HCA is the more aggressively compressed long-memory branch. The report uses compression rate m' = 128. Instead of top-k sparse retrieval, HCA can attend densely over the heavily compressed stream. One million original tokens become roughly 7813 compressed entries.

CSA and HCA are both compression-based, but they are not redundant. CSA is higher-resolution compressed sparse memory. HCA is lower-resolution compressed dense memory. The sliding window is exact local memory. The point of the hybrid is to cover three regimes:

8. Interleaving: Why Some Layers Use CSA And Some Use HCA

The interleaving is the architecture-level answer to a tradeoff. If every layer used only HCA, the model would have cheap global memory but could lose important medium-resolution information. If every layer used only CSA, the model would preserve more detail but pay more indexer/top-k/sparse-attention cost. Interleaving lets some layers retrieve detailed compressed blocks while other layers perform cheaper global integration.

A simplified mental model:

early layers:
    local or HCA warmup

middle/deep layers:
    layer l:   CSA + sliding window
    layer l+1: HCA + sliding window
    layer l+2: CSA + sliding window
    layer l+3: HCA + sliding window

This is similar in spirit to what other modern models do when they mix full attention and sliding-window attention. The dense matrix algebra does not break. The selected key dimension changes. The attention output still returns [tokens, heads, head_dim]. The projection after attention still sees the same hidden width.

9. Heterogeneous KV Cache: The Real Serving Problem

With normal attention, the KV cache is conceptually uniform: keys and values for each layer grow with sequence length. With V4-style hybrid attention, the cache is heterogeneous. A request may carry recent sliding-window KV, CSA compressed entries, HCA compressed entries, and incomplete tail tokens that are not yet ready to form a compression block.

request_state:
    swa_recent_kv[layer]       # exact recent window
    csa_compressed_kv[layer]   # one entry every 4 tokens
    hca_compressed_kv[layer]   # one entry every 128 tokens
    csa_tail[layer]            # 0..3 uncompressed tokens
    hca_tail[layer]            # 0..127 uncompressed tokens
    indexer_cache[layer]       # compressed index keys for CSA

This is why a generic PagedAttention layout is not the whole answer. PagedAttention is excellent for uniform KV pages. V4-style attention needs block lifecycle rules: every 4 tokens create CSA entries, every 128 tokens create HCA entries, keep last 128 exact, and manage tails during prefix cache hits. The simpler baseline is covered in KV Cache Memory; this DeepSeek-style cache is the next step because the cache is no longer one contiguous conceptual tensor.

This is also where C-Kernel-Engine can develop a useful CPU-first artifact: a deterministic heterogeneous KV cache simulator. The point is to prove the memory accounting and block update rules before optimizing sparse attention throughput.

10. Muon: Training Update, Not Inference Kernel

Muon matters because the architecture is trained around it, but it is not an inference kernel. The update uses momentum and an orthogonalization-style step, often described through Newton-Schulz iterations. DeepSeek-V4 uses Muon for most parameters while keeping AdamW for sensitive parameters such as embeddings, prediction heads, mHC static/gating factors, and RMSNorm weights.

For CKE, the practical order is simple: do not implement full Muon first. First implement a small Newton-Schulz demo, then a matrix-update parity test, then training-loop integration. The immediate inference work is CSA/HCA/mHC forward, not Muon.

11. How This Maps To C-Kernel-Engine Today

CKE already has many of the primitives needed to study these papers. The important thing is to state this carefully. CKE does not yet mean “drop in DeepSeek-V4 production inference.” It means the kernel surface is moving toward the right contracts. The compiler/runtime side of that claim is expanded in the v8 IR pipeline post and the runtime-ownership side is expanded in Threadpools and Memory Pools.

12. Build Order: Do Not Build V4 All At Once

The useful build path is staged. Each step should have a scalar reference, a PyTorch parity test, then an optimized kernel.

1. RoPE refresher: pairwise, split-half, partial RoPE, forward/backward.
2. Normal attention: dense QK, causal mask, softmax, weighted V, decode KV cache.
3. Sliding-window attention: exact recent-token ring buffer.
4. Compression kernel: softmax-weighted pooling for m = 4 and m = 128.
5. HCA toy: dense attention over heavily compressed entries plus sliding window.
6. CSA toy: compressed entries, index scores, top-k, sparse selected attention, sliding window.
7. Heterogeneous KV cache: CSA blocks, HCA blocks, SWA state, tail state.
8. mHC forward: lane state, controller maps, Sinkhorn, read, write, carry.
9. MoE routing: top-k experts, histogram, dispatch, grouped GEMM, combine.
10. Low precision: FP8/FP4 simulation, scale tracking, deterministic comparisons.
11. Determinism: batch-invariant reductions and replay tests.
12. Reasoning workload simulation: long decode, shared-prefix reuse, rollout scheduling.

13. One-Page Mental Model

DeepSeek-V4 is a transformer MoE model designed around very long context. It keeps the DeepSeekMoE and MTP ideas from earlier versions, but ordinary attention and ordinary KV cache cannot survive at million-token scale. So the attention stack changes. CSA compresses every few tokens and sparsely retrieves selected compressed blocks. HCA compresses much more aggressively and attends densely over that shorter memory. Both keep a sliding-window branch so recent local tokens remain exact.

mHC gives every token multiple residual lanes, then constrains the residual carry matrix so those lanes remain stable across depth. Muon changes how most large matrices are updated during training. Low precision changes the deployment memory equation. The architecture is therefore not just neural-network math. It is math plus kernels plus memory layout plus cache lifecycle plus precision plus training recipe.

That is exactly why it belongs in the C-Kernel-Engine study path. The point is not to chase model names. The point is to learn the invariant: every new frontier architecture eventually becomes a set of memory layouts, reductions, selections, projections, normalization kernels, and parity tests.