One support ticket, three ways to answer

Consider the state: “Checkout is failing for customers. Orders have been blocked for an hour.” We want two decisions: which team owns it, and is it urgent?

The examples are conceptually comparable but not identical products. A fixed three-label classifier does not automatically accept a new fourth team at inference time. A schema-conditioned head can score options supplied in the request, but only if its training taught it to understand such questions and option descriptions. A language model can produce far richer free text, at the cost of decoding and validation.

The common starting point: hidden states

Let the input token sequence have length \(T\) and the transformer's hidden width be \(d\). The backbone emits \(H \in \mathbb{R}^{T \times d}\): one contextual vector per token. An image-capable backbone may also incorporate visual input before those final hidden states. None of these vectors is yet a human-readable answer. The output head decides what gets scored.

Path 1: an LM head scores vocabulary tokens

The output at a scored position has |V| scores, where V is the tokenizer vocabulary. The mapping may be a linear projection or include an intermediate transform; some models tie its output weights to input embeddings. BERT's masked-LM head predicts deliberately hidden tokens using bidirectional context. A causal decoder's LM head predicts the next token from earlier context; at inference it selects a token, appends it, and repeats. These are different training and inference contracts, even though both score the vocabulary. The diagram and support-ticket example use the causal case. To generate technical, that model may need more than one token; braces, field names, and punctuation cost additional decode steps.

Prompting a causal LM to return JSON changes the requested format, not its next-token calculation. Structured decoding can restrict the allowed token sequence and improve validity, but it still emits tokens. A BERT encoder with a new d → K label head is instead a classifier, not an LM head merely because it is attached to a language-model backbone. The BERT sequence-classification class makes that task distinction explicit.

Path 2: a classifier scores a fixed taxonomy

The pooling rule is part of the architecture: last token, a designated token, or a masked average can yield different representations. For our team router, K=3. With illustrative logits [2, 1, 0] for technical, billing and sales, softmax gives approximately [0.665, 0.245, 0.090]. Those are normalized scores, not proof of a calibrated 66.5% real-world success rate.

Fine-tuning on labelled tickets updates the classifier and often at least some backbone weights or weight adapters. The loss is cross-entropy on the known team. If the business adds “fraud review,” the fixed classifier's output dimension and training data must change. A second binary head could predict urgency, but independently trained heads may miss relationships between fields. Conversely, for a stable taxonomy with plenty of labelled data, this small fixed head may be simpler and cheaper than a schema model.

Path 3: Clef scores the options in a request

Cloudflare describes Clef as a Qwen-derived decision model. Its request provides a state plus typed questions, with allowed choices or an ordered score scale. The backbone processes the state and schema in a prefill-only pass. The released joint schema head then uses spans for questions and options, routes option representations through attention over the backbone's hidden states, lets fields interact, and computes one logit per allowed option. It applies a softmax separately within each question's option set. There is no autoregressive answer sentence to parse.

{
  "state": "Checkout has failed for an hour and orders are blocked.",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Who should own this incident?",
      "criteria": {
        "billing": "Invoices and refunds",
        "technical": "Outages and errors",
        "sales": "Plans and upgrades"
      }
    },
    "urgent": {
      "type": "noul",
      "instructions": "Does this need immediate attention?"
    }
  }
}

In this format, noul is the API's yes/no type. A possible invented head output is team logits [0, 2, -1], becoming roughly [0.114, 0.844, 0.042] after softmax, alongside urgency logits [-0.1, 1.1], becoming roughly [0.231, 0.769] for no/yes. These numbers teach the arithmetic; they are not a Clef run or accuracy claim. The application still decides whether 0.769 crosses its escalation threshold.

Clef is not a plain d → K linear classifier. In the published implementation, option vectors are built from contextual spans and lexical information, evidence-routing layers attend to the sequence, field vectors interact, and a scorer combines a lexical prior with learned joint features. The code reads the model's output-embedding weights for that lexical prior, even though it does not run the usual LM-head vocabulary projection to generate a reply. This is why “Clef has no LM head” is too blunt: the generative output path is unused, but vocabulary-linked weights still contribute to scoring.

Inside the head: where attention actually happens

The Clef head does not attend to another LLM. It attaches to one Qwen backbone and reads that backbone's final hidden states. The record encoder remembers token spans for each question and option. In the released head code, those spans become queries while a projection of the full hidden-state sequence becomes attention memory.

The released Cloudflare head configuration is small enough to read in full. These are Cloudflare's dimensions, not the smaller model we trained:

{
  "hidden_size": 5120,
  "width": 1024,
  "routing_layers": 2,
  "layers": 4,
  "heads": 16,
  "feedforward": 4096
}

Read it as a module-level computational graph: Qwen yields a 5,120-wide hidden state at each input position; the joint head projects to width 1,024. It has two option-to-evidence attention layers and four question-field transformer decoder layers, each with 16 attention heads and 4,096-wide inner FFNs. This is indeed a transformer extension, not a mysterious new species of neural network. Routing means attention over the input evidence here, not MoE expert selection. The per-option scoring contract makes it a decision head.

Here E is Qwen's pretrained output-embedding matrix, reused as a lexical clue about the option wording. The W matrices and routing-attention weights in our newly attached head began untrained. The equation omits layer normalization, multiple attention heads, residual paths and FFNs for readability; those are present in the implementation. The crucial operation is option-specific evidence lookup: “different person” and “insufficient evidence” can attend differently to the same record. A fixed pool(H) → Linear(d, 3) instead hands every label the same pooled vector and fixed learned class weights.

Next the head summarizes each question from its routed options, adds a global record vector and question-type embedding, and sends the question-level vectors through decoder-style transformer layers. Those layers include self-attention among question fields and cross-attention back to the backbone memory. Finally, each option receives a score combining a lexical prior, similarity to its question vector, and a small residual FFN:

The residual FFN is one component of the head, not the whole architecture. The head also contains learned projections, multi-head cross-attention, field-interaction layers, norms, embeddings and scoring scales. This extra structure may help when a request changes its questions or option descriptions. But an option-conditioned FFN could also be designed; attention is not magic. Its value must be measured against such a baseline, on data that actually varies the schema.

What was the new head initialized to?

Unsloth loaded pretrained Qwen3-0.6B, then created a fresh JointSchemaHead using random seed 3407; our training script did not provide a head_init checkpoint. The Unsloth loader shows both branches: supply head_init to load existing head weights, or omit it to construct a seeded new head. It warns that even a released Clef head on a different backbone is only a warm start, not a matched decision model. For the 1,024-wide Qwen hidden states, its default design chose width 512, two evidence-routing layers, four field layers, eight attention heads and a 2,048-wide feed-forward sublayer. PyTorch initialized the new linear and attention weights in the ordinary module constructors. The released head code explicitly starts three scoring scalars at zero, so the two exponentiated scales initially equal 1 and the sigmoid residual gate initially equals 0.5. These are starting values, not trained knowledge. The pretrained Qwen representations and its output embeddings are not random. [Head implementation]

The rank-16 LoRA matrices were also newly attached. During training, gradients reached the head and LoRA updates, while Qwen's base weights stayed frozen. The fresh system therefore was not a model from scratch: it was a pretrained language backbone with an untrained decision readout. On our measured balanced 90-pair test, that starting system scored 33.3% with cross-entropy 1.0988, close to the three-way chance-level loss ln(3) ≈ 1.099. That aggregate result does not mean every individual prediction was uniform.

Head versus adapter: two different modifications

A low-rank update can be written \(W' = W + BA\), where \(B\) and \(A\) are narrow learned matrices. It changes what the backbone represents while the head changes how those representations are read out. Cloudflare says it trained Clef's routing head together with rank-256 low-rank adapters while freezing the Qwen base weights, using label-smoothed cross-entropy plus a Brier objective for probability calibration. The released model packages a post-trained backbone and a separate joint-head weight file. Therefore “just bolt Clef's head onto stock Qwen” is not an equivalent model. [training account, model card]

Cloudflare's model and our Unsloth run are not the same checkpoint

Cloudflare designed and released Clef, including a Qwen-derived 27B model, a smaller Clef-Flash variant, a joint schema head, and a typed decision API. Its announcement describes evidence routing, joint attention across fields, rank-256 adapter training, a calibration-aware supervised loss, and an additional reinforcement-learning objective. Those are claims about Cloudflare's training program. Unsloth provides a way to train other backbones with a compatible Clef-style head. Its guide covers dataset conversion, LoRA, head training, calibration, saving, and serving. It can also fine-tune an existing decision checkpoint. Our experiment instead selected the plain-LM-plus-new-head route.

“Clef-style” describes the architecture and API shape here, not inherited general decision ability. The saved training artifacts consist of a roughly 39 MB LoRA adapter and 53 MB joint-head file; the merged checkpoint also includes backbone weights. The artifact sizes describe files, not model quality.

Follow one fictional identity pair through training

The dataset generator produced 15,000 fictional record pairs: 9,129 train, 2,927 validation, 2,944 test. Each exported row contains two source records in a state, one question asking whether they refer to the same natural person, descriptions of the three allowed options, and a gold label. Hidden synthetic person IDs are not included in the model input. The three outcomes are same_person, different_person, and insufficient_evidence. This is a supervised decision task, not next-token language-model training.

1. Encode the record. Unsloth serializes the state, question, and option descriptions into one token sequence and records where the question and each option occur. The training script rejects skipped or truncated decisions at a 1,024-token limit.
2. Read once with Qwen. Qwen3-0.6B computes contextual hidden states. There is no answer-token decode loop. The pretrained backbone was loaded in 4-bit, with trainable rank-16 LoRA matrices on selected attention and MLP projections.
3. Route evidence in the new head. The 512-wide head makes option representations from their contextual spans and lexical information, routes them against the hidden-state sequence, lets field information interact, and returns one logit for each option. Its output-embedding lexical prior does not mean it generated an answer through the LM head.
4. Measure the error. Softmax turns the three logits into a distribution for this question. With gold option different_person, the basic loss is \(L=-\log P(\texttt{different\_person})\). Backpropagation updates the fresh head and LoRA parameters, not the frozen Qwen base weights.
5. Fit and save. The run used one epoch, batch size two with four gradient-accumulation steps, LoRA learning rate 2e-4, and the trainer's separate default head learning rate 1e-4. It fitted a temperature on validation decisions, evaluated the synthetic test split, saved adapter plus head, and later saved a merged serving checkpoint.

What changed before and after training?

The saved trainer log contains 114 loss entries over 1,142 optimizer steps. At step 10, logged training loss was 1.0667; at step 50, 0.2772; at step 100, 0.00265; and at step 1,140, about 0.00000214. The plot uses a logarithmic y-axis, because the final values would otherwise disappear against the initial loss. A low training loss establishes that the training examples became easy to fit; it does not measure robustness.

To measure a true starting point rather than guess it, we reloaded pretrained Qwen3-0.6B with the same deterministic fresh-head seed and untrained rank-16 LoRA configuration. We evaluated it and the saved adapter checkpoint on the same 90 held-out synthetic test rows, selected deterministically with 30 per label. The fresh head scored 33.3%, and the trained checkpoint scored 100%. This was an evaluation of the existing trained run, not a new training run or a Cloudflare benchmark.

The complete synthetic test report also recorded 100% on 2,944 generated examples. Yet a manual name-only probe was incorrectly labelled same_person when the appropriate answer was insufficient_evidence. The generator's sparse-evidence cases share a conspicuous pattern, so the model may have learned that pattern instead of the intended epistemic rule. A perfect score against a narrow generator can coexist with a dangerous failure on a simple counterexample. The failure is more informative for our next experiment than another decimal place of synthetic accuracy.

Would a freshly trained FFN beat the joint head?

We do not know yet. For exactly three fixed identity labels, a 1,024 → 3 linear head has 3,075 learned parameters; a 1,024 → 512 → 3 FFN has 526,339. Both are much simpler than the joint head. A randomly initialized vocabulary-sized LM head would score tokens, not these three decisions, and would throw away the pretrained output mapping. If it emits only three outputs, it is a classifier head, regardless of what we call it.

A fair comparison would keep the same Qwen checkpoint, record serialization, pair splits, LoRA budget, and training schedule, and vary only the head: frozen-backbone linear, frozen-backbone FFN, LoRA plus FFN, and LoRA plus joint schema head. Report false-merge rate, abstention quality, calibration, latency, and performance on an independently designed counterfactual holdout, not just generated accuracy. The joint head's advantage may appear when option wording or the number of fields changes per request; the simple head may win on a stable taxonomy. Until that experiment exists, either outcome is plausible.

Can we interpret the decision?

We can test behavioral dependencies, but an attention map is not a certified explanation. Create controlled record pairs: same name but conflicting birth dates; name only; swap record order; remove a field; change only source metadata; rewrite option descriptions; add a fourth distractor option. For each, record the logit change for all three labels, not just the winning label. Then compare the direction of those changes against an explicit rule and human-reviewed examples. If changing irrelevant form metadata flips the answer, we have found a shortcut. Gradient attribution or head-attention visualizations can guide debugging, but counterfactual intervention and independent labels provide stronger evidence of what the system actually uses.

In an identity workflow, a confident same_person is not authorization to merge records. Evaluation must put false merges, missing evidence, shared households, common names, OCR corruption, provenance, and reviewer override ahead of a single headline accuracy. The experiment uses fictional data and is not an operational identity-decision service.

Why change the head at all?

For a typed decision, an LM's flexibility may be unnecessary. It pays for each generated token and can return a valid-looking but unsupported answer unless constrained and checked. A fixed classifier avoids that loop but bakes in its taxonomy. A schema head occupies the middle: the application specifies the allowed answers now, and the model scores them directly. Clef's runtime still pays for reading the whole input and running a nontrivial joint head; “one pass” is not “free.” Cloudflare reports latency comparisons for its tested workloads, but those numbers should not be generalized to every hardware, context length, or output length. [Cloudflare measurements]

The correct output contract is a product choice. Choose generation when the answer is open-ended prose or code. Choose a fixed classifier when the label set is stable and the data supports it. Choose a schema-conditioned decision model when a workflow asks several typed questions with changing allowed options and wants scores rather than text. For high-impact actions, none of these probabilities should bypass policy checks, human review where warranted, or calibration testing on the site's own data.

What a runtime would have to implement

A generic autoregressive server usually stops at tokenization, prefill, KV-cache decode, and LM-head logits. A Clef-compatible path needs a different graph boundary: encode the record and option spans, run the Qwen-derived backbone without a decode loop, retain relevant hidden states, execute the joint head's routing and cross-field operations, produce variable-length option logits, then normalize per question. The serving API must preserve the question IDs and option ordering. Treating the head as a single linear projection, or applying one softmax across all questions, would implement the wrong model.

For CKE, this is an architectural checklist, not a statement that Clef runs on CKE today. The practical work would be a pinned model/tokenizer reference, a record-encoding oracle, per-stage head checkpoints, generated operators with exact shape contracts, and tests for multiple questions, changed option order, empty or invalid schemas, numerical parity, and calibration. The lesson is broader than Clef: a model's head is part of its executable contract, not merely a decorative layer at the end.

The takeaway

Backbone: what the input means as contextual vectors. Head: what is scored. Adapter: which internal representations are retuned. Loss: what good outputs mean during training. Runtime: whether we generate tokens or return task-specific scores. Clef is interesting because it changes several of these together. A model becomes a useful decision model through that complete contract, not by renaming an LM response “classification.”

Read the sources: Cloudflare's Clef announcement, model card and API example, released head implementation, and Transformers output shapes.