Introduction — From Attention to Vision

In the earlier attention note, we derived attention as a routing rule over tokens. In the residual-connections note, we saw why deep stacks stay trainable. In the full-backprop note, gradients were flowing end to end. In the optimizer note, AdamW was nudging those parameters into something useful. And in the DeltaNet note, we looked at recurrent alternatives when full attention is not always the cheapest memory contract.

The natural next question is: what does all of that look like when the input is not text, but an image? How does a real vision-language model turn pixels into tokens that a language decoder can reason over? Qwen3-VL is one of the strongest open multimodal systems in that class, and C-Kernel-Engine v8 gives us an unusually clear answer because the architecture is exposed as templates plus generated C.

The crucial idea is simple. The image encoder is not a mysterious monolith. It is a pipeline with a header, a repeated body, and a footer. Header means “convert image grid into transformer-ready patch tokens.” Body means “run the repeated ViT layers.” Footer means “compress and project those vision features into the embedding space the language model expects.”

The Template Architecture — Header, Body, Footer

C-Kernel-Engine v8 organizes model definition in three conceptual stages described in version/v8/templates/PIPELINE.md: Template → GraphIR → LoweredIR. The template stage is architecture-agnostic. It names the ops and their order. Later stages bind those ops to actual kernels and generated C code.

For Qwen3-VL vision, the template file is version/v8/templates/qwen3_vl_vision.json. It explicitly declares one block type called vision_encoder. That block type contains a sequence field set to ["header", "body", "footer"]. There is no metaphor here. This is the source of truth that tells the runtime what must happen first, what repeats, and what gets emitted at the end.

{
  "version": 3,
  "name": "qwen3_vl_vision",
  "family": "vision_transformer_with_branches",
  "flags": {
    "patch_frontend": "dual_patch_proj_sum",
    "activation": "gelu",
    "normalization": "layernorm"
  },
  "block_types": {
    "vision_encoder": {
      "sequence": ["header", "body", "footer"],
      "header": [
        {"id": "patchify", "op": "patchify"},
        {"id": "patch_proj", "op": "patch_proj"},
        {"id": "patch_proj_aux", "op": "patch_proj_aux"},
        {"id": "patch_sum", "op": "add_stream", "params": {"merge_size_from_config": "spatial_merge_size"}},
        {"id": "patch_bias", "op": "patch_bias_add"},
        {"id": "position_embeddings", "op": "position_embeddings", "params": {"merge_size_from_config": "spatial_merge_size"}},
        {"id": "vision_position_ids", "op": "position_ids_2d", "params": {"position_rank": 4, "merge_size_from_config": "spatial_merge_size"}}
      ],
      "body": {
        "type": "dense",
        "ops": [
          {"id": "attn_norm", "op": "layernorm"},
          {"id": "qkv_packed_proj", "op": "qkv_packed_proj"},
          {"id": "split_qkv", "op": "split_qkv_packed"},
          {"id": "vision_mrope", "op": "rope_qk", "params": {"rope_mode": "vision", "position_rank": 4, "section_policy": "equal_4way"}},
          {"id": "attn", "op": "attn"},
          {"id": "attn_out_proj", "op": "out_proj"},
          {"id": "attn_residual", "op": "residual_add"},
          {"id": "ffn_norm", "op": "layernorm"},
          {"id": "mlp_up", "op": "mlp_up"},
          {"id": "mlp_gelu", "op": "gelu"},
          {"id": "mlp_down", "op": "mlp_down"},
          {"id": "mlp_residual", "op": "residual_add"}
        ]
      },
      "footer": [
        {"id": "final_norm", "op": "layernorm"},
        {"id": "merge_main", "op": "spatial_merge"},
        {"id": "projector_fc1", "op": "projector_fc1"},
        {"id": "projector_gelu", "op": "projector_gelu"},
        {"id": "projector_fc2", "op": "projector_fc2"},
        {"id": "deepstack_concat", "op": "branch_concat", "output": "vision_embeddings"}
      ]
    }
  }
}

Header Step 1 — Patchify (im2patch)

Vision transformers do not ingest a [3, 448, 448] image the way a CNN does. They first cut it into fixed-size patches. In the Qwen3-VL setup described in the prompt, patch size is P = 14. So a 448 × 448 image becomes a 32 × 32 patch grid because 448 / 14 = 32. That yields 32 × 32 = 1024 patches.

Each patch contains 3 × 14 × 14 = 588 scalar values. So after patchification the image is no longer stored conceptually as height-width pixels. It is stored as a token matrix of shape [1024, 588]. This is the exact moment vision becomes “tokenizable” and therefore transformer-compatible.

void im2patch(const float *image, float *patches, int C, int H, int W, int P)
{
    const int grid_h = H / P;
    const int grid_w = W / P;
    const int patch_dim = C * P * P;
    for (int ph = 0; ph < grid_h; ++ph) {
        for (int pw = 0; pw < grid_w; ++pw) {
            const int patch_idx = ph * grid_w + pw;
            float *out = patches + (size_t)patch_idx * patch_dim;
            int k = 0;
            for (int c = 0; c < C; ++c) {
                for (int py = 0; py < P; ++py) {
                    for (int px = 0; px < P; ++px) {
                        const int iy = ph * P + py;
                        const int ix = pw * P + px;
                        out[k++] = image[(size_t)c * H * W + (size_t)iy * W + ix];
                    }
                }
            }
        }
    }
}

Header Step 2 — Dual Patch Projection

Once the image has become [1024, 588] patches, Qwen3-VL does something richer than a standard single projection. The template declares two parallel ops: patch_proj and patch_proj_aux. Both read the same patch matrix. Both project it into the encoder embedding width. In the generated configuration, that width is EMBED_DIM = 1152.

So the header performs two GEMMs of the form [1024, 588] × [588, 1152] → [1024, 1152]. The two outputs are not kept separate forever. They are merged by patch_sum, whose runtime helper is described as add_stream_reorder_2d in src/kernels/vision_kernels.c lines 384-412. That merge is not a naïve row-wise add. It includes merged-tile spatial reordering.

// Patches from im2patch
float patches[1024][588];

// Main learned projection
float patch_proj[1024][1152] = GEMM(patches, W_main);

// Auxiliary learned projection
float patch_proj_aux[1024][1152] = GEMM(patches, W_aux);

// Spatially reordered merged-tile addition
float patch_sum[1024][1152] = add_stream_reorder_2d(
    patch_proj,
    patch_proj_aux,
    merge_size);

// Bias then positional information
float patch_bias[1024][1152] = patch_sum + b_patch;

That ordering matters. The model does not add position first and project later. It first learns a token embedding from pixel content, then merges the two streams, then applies a bias, then injects positional structure. In other words, content channels are formed before spatial context is layered on top.

Header Step 3 — Position Embeddings (2D M-RoPE)

Text is naturally one-dimensional. Token 7 comes after token 6. Images are different. A patch has both a vertical coordinate and a horizontal coordinate. So the vision encoder needs a 2D position system, not just a single running index.

Qwen3-VL handles that in two related ways. First, position_embeddings_add_tiled_2d in src/kernels/vision_kernels.c lines 158-239 adds learned position embeddings with bilinear interpolation, so the runtime can adapt when the actual patch grid differs from the training grid. Second, vision_position_ids_2d_merge in lines 250-287 emits four position streams for M-RoPE.

void vision_position_ids_2d_merge(int32_t *positions, int grid_h, int grid_w, int merge_size)
{
    // Output: [4, T] where T = grid_h * grid_w
    // Four streams: [y, x, y, x] for M-RoPE sections
    const int T = grid_h * grid_w;
    int idx = 0;
    // Merged-tile traversal order
    for (int bh = 0; bh < grid_h; bh += merge_size) {
        for (int bw = 0; bw < grid_w; bw += merge_size) {
            for (int dh = 0; dh < merge_size && (bh + dh) < grid_h; ++dh) {
                for (int dw = 0; dw < merge_size && (bw + dw) < grid_w; ++dw) {
                    int y = bh + dh;
                    int x = bw + dw;
                    positions[0 * T + idx] = y;  // stream 0: y
                    positions[1 * T + idx] = x;  // stream 1: x
                    positions[2 * T + idx] = y;  // stream 2: y (duplicate)
                    positions[3 * T + idx] = x;  // stream 3: x (duplicate)
                    idx++;
                }
            }
        }
    }
}

The line that matters most is the output convention: [y, x, y, x]. That is the multi-section rotary layout for vision. Instead of one scalar position per token, each patch gets four coordinated streams that will later rotate different quarters of the query and key vectors.

Body — The Vision Transformer (27 Layers)

Once the header is done, the encoder is holding a patch-token matrix of width 1152 with content and spatial information already attached. The body of the model then repeats a dense transformer block 27 times. In the generated file build/regression-runs/qwen3vl/multimodal_bridge/encoder/encoder_v8.c, those 27 layers are unrolled. The configuration given in the prompt is EMBED_DIM 1152, NUM_HEADS 16, HEAD_DIM 72, and ROTARY_DIM 72.

This is a normal transformer rhythm, but applied to image tokens rather than text tokens: normalize, build QKV, rotate Q and K, attend across tokens, project back, add residual, normalize again, run the MLP, and add the second residual. If you remember the attention and MLP posts from the series, you already know the conceptual pieces. The novelty here is how they are staged in a real vision encoder.

{
  "type": "dense",
  "ops": [
    {"id": "attn_norm", "op": "layernorm"},
    {"id": "qkv_packed_proj", "op": "qkv_packed_proj"},
    {"id": "split_qkv", "op": "split_qkv_packed"},
    {"id": "vision_mrope", "op": "rope_qk", "params": {"rope_mode": "vision", "position_rank": 4, "section_policy": "equal_4way"}},
    {"id": "attn", "op": "attn"},
    {"id": "attn_out_proj", "op": "out_proj"},
    {"id": "attn_residual", "op": "residual_add"},
    {"id": "ffn_norm", "op": "layernorm"},
    {"id": "mlp_up", "op": "mlp_up"},
    {"id": "mlp_gelu", "op": "gelu"},
    {"id": "mlp_down", "op": "mlp_down"},
    {"id": "mlp_residual", "op": "residual_add"}
  ]
}

Body Detail — Vision M-RoPE

Standard text RoPE rotates query and key coordinates according to a one-dimensional token index. Vision M-RoPE is more structured. According to the prompt, the relevant implementation is mrope_qk_vision() in src/kernels/rope_kernels.c. It consumes the four position streams [y, x, y, x] and divides the head dimension into four equal sections.

Here the head dimension is 72. With equal_4way section policy, the split is 72 / 4 = 18 features per section. Section 0 is rotated with the first y stream. Section 1 uses x. Section 2 uses y again. Section 3 uses x again.

Body Detail — Bidirectional Attention

Text generation uses causal masking because future tokens are supposed to stay hidden. Vision encoding has no such constraint. The whole image is already present. A patch in the top-left corner must be able to compare itself with a patch in the bottom-right corner immediately.

That is why the vision attention in this stack is dense and bidirectional. The prompt explicitly frames it as attn_variant: "dense_bidirectional" with causal: false. The referenced execution path uses attention_forward_full_head_major_gqa_ggml_strided.

This is also why a vision encoder does not need a long-lived autoregressive KV-cache in the way a decoder does. The image is processed in one pass. The attention workspace is temporary. Once the encoder has produced the final vision tokens, the decoder no longer needs per-layer image KV state from the encoder.

DeepStack Branches — Multi-Layer Feature Extraction

Qwen3-VL adds something more specialized than a plain ViT footer. The template contains a branch named deepstack. Its tap point is declared as body.mlp_residual.out at a set of deepstack_layer_indices from configuration. So intermediate layer outputs are not discarded. Selected ones are harvested and fed into their own mini-projector path.

This is a strong design choice. Instead of trusting only the final layer to summarize the entire image hierarchy, the encoder preserves some intermediate representations. That lets the eventual multimodal projector see a blend of lower-level and higher-level features.

{
  "name": "deepstack",
  "kind": "fixed_branch",
  "tap": {
    "from": "body.mlp_residual.out",
    "layers_from_config": "deepstack_layer_indices"
  },
  "producer": {
    "ops": [
      {"id": "merge", "op": "branch_spatial_merge"},
      {"id": "norm", "op": "branch_layernorm"},
      {"id": "fc1", "op": "branch_fc1"},
      {"id": "gelu", "op": "branch_gelu"},
      {"id": "fc2", "op": "branch_fc2"}
    ]
  },
  "collect": {
    "mode": "concat",
    "axis": "feature",
    "target": "branch.deepstack"
  }
}

Footer — From Vision Tokens to Language Space

After 27 layers of bidirectional visual reasoning, the encoder still holds patch-grid features in its own vision width. The footer is the conversion zone. It takes those features and turns them into the final vision_embeddings buffer consumed by the multimodal bridge.

The footer sequence is: final_norm → merge_main → projector_fc1 → projector_gelu → projector_fc2 → deepstack_concat. In words: normalize, spatially compress, project toward the decoder embedding space, and then concatenate the DeepStack branch features.

void spatial_merge_2x2(const float *input, float *output, int grid_h, int grid_w, int embed_dim)
{
    const int out_h = grid_h / 2;
    const int out_w = grid_w / 2;
    for (int oh = 0; oh < out_h; ++oh) {
        for (int ow = 0; ow < out_w; ++ow) {
            float *dst = output + ((size_t)oh * out_w + ow) * embed_dim * 4;
            // Pack: [top-left, top-right, bottom-left, bottom-right]
            const float *tl = input + ((size_t)(2*oh) * grid_w + 2*ow) * embed_dim;
            const float *tr = input + ((size_t)(2*oh) * grid_w + 2*ow + 1) * embed_dim;
            const float *bl = input + ((size_t)(2*oh+1) * grid_w + 2*ow) * embed_dim;
            const float *br = input + ((size_t)(2*oh+1) * grid_w + 2*ow + 1) * embed_dim;
            memcpy(dst, tl, embed_dim * sizeof(float));
            memcpy(dst + embed_dim, tr, embed_dim * sizeof(float));
            memcpy(dst + 2*embed_dim, bl, embed_dim * sizeof(float));
            memcpy(dst + 3*embed_dim, br, embed_dim * sizeof(float));
        }
    }
}

That packing order matters. Spatial merge does not average four neighboring patches into one smaller vector. It concatenates their embeddings in a fixed order: top-left, top-right, bottom-left, bottom-right. So the number of tokens drops by 4, while the local feature width grows by 4 before the projector MLP reshapes it again.

The Multimodal Bridge

At this point the vision encoder has emitted vision_embeddings. Those are not end-user words. They are already vector embeddings. The bridge logic in version/v8/scripts/vision_bridge_runtime_v8.py decides how they are handed to the text decoder.

The key parameters named in the prompt are prefix_tokens, embed_dim, and projector_total_out_dim. The bridge resolves how many visual tokens exist after merging and projection, what their width is, and where they are written inside the decoder's input embedding buffer.

// Decoder embedded input buffer
float decoder_input[T_total][4096];

// Prefix region comes from vision encoder output
copy_vision_prefix(
    decoder_input,
    vision_embeddings,
    prefix_tokens,
    projector_total_out_dim);

// Text token embeddings are appended after the prefix
copy_text_embeddings(
    decoder_input + prefix_tokens,
    text_token_embeddings,
    text_token_count);

// Decoder sees one unified sequence:
// [vision_tokens... | text_tokens...]

This is the multimodal trick in its cleanest form. The decoder does not need one attention mechanism for images and another for text. It just receives a single sequence of embeddings in which the first chunk came from the vision encoder. That is why the bridge matters so much. It translates between the vision template's output contract and the decoder template's input contract.

The Text Decoder — What Changes for Multimodal

The decoder side is still a transformer language model. The generated file build/regression-runs/qwen3vl/multimodal_bridge/decoder/decoder_v8.c contains 36 unrolled layers with EMBED_DIM 4096 and NUM_HEADS 32. Its attention is causal for text generation. Its MLP uses the familiar Qwen-style gate_up → silu_mul → down SwiGLU pattern from the text template.

What changes in multimodal mode is not the decoder's identity. What changes is its input sequence. Instead of beginning with only learned token embeddings for text, it begins with a block of vision-derived embeddings that have already been projected into its embedding space.

The subtle but important decoder behavior is this: text tokens cannot look into their own future, but they can attend backward to the vision prefix because that prefix is already part of the past from the decoder's perspective. So multimodal generation is still causal, just with a richer past.

SigLIP vs Qwen3-VL Vision — Two Approaches

The prompt gives us a useful comparison point in version/v8/templates/siglip_vit.json. SigLIP ViT uses the same general header/body/footer template pattern, but the design is simpler. It uses an im2patch frontend, a single projection path, RMSNorm, and GeGLU. Qwen3-VL uses dual patch projection, LayerNorm, GELU, 2D M-RoPE, and DeepStack branches.

That comparison is valuable because it shows the template language is not specific to one architecture. CKE v8 can describe both a cleaner SigLIP-style ViT and the richer Qwen3-VL vision frontend with branches. Same structural grammar. Different model family decisions.

Memory and Compute Profile

Let us make the buffer story concrete. The raw image is [3, 448, 448] or 602,112 floats. Patchify preserves that total float count but reorganizes it into [1024, 588]. After either projection path, the header is working with [1024, 1152] activations, which is 1,179,648 floats.

At FP32, one such activation matrix is roughly 1,179,648 × 4 ≈ 4.5 MB. That is only one stage. Internally, each layer also creates Q, K, and V views across 16 heads of width 72. Because the encoder is bidirectional and non-autoregressive, those attention intermediates are ephemeral rather than stored as a growing decode cache.

Conclusion — Everything Connects

The biggest takeaway from Qwen3-VL in C-Kernel-Engine v8 is not merely that a strong open model uses patchify, RoPE, and attention. It is that the entire path is declared in a structure we can read: header, body, footer.

Header turns pixels into positioned patch tokens. Body applies the same transformer ideas we derived earlier in the series: LayerNorm, packed QKV projection, attention, output projection, GELU MLP, and residual addition. Footer compresses and projects those features into a decoder-facing embedding sequence. Then the multimodal bridge prepends that sequence to text embeddings so one causal decoder can reason over both image and language.