First, what does a tokenizer actually promise?

Before a transformer can multiply anything, the frontend has to produce a sequence of integer IDs. A useful mental model is text → normalization → pre-tokenization → subword algorithm → IDs → embeddings. Special-token insertion, unknown-byte fallback, and decoding are additional parts of the contract. For the same string, two implementations can disagree because one lowercases, one treats the leading space as part of a piece, or one applies learned merge ranks where the other simply finds the longest vocabulary entry. A correct runtime must match the tokenizer shipped with the model; “both are BPE” is not enough. The Hugging Face Tokenizers pipeline makes these stages explicit. Quick read / contractText is not the model inputThe model receives IDs. Normalization, spaces, merge order, and special tokens must all match its saved tokenizer.

Why use subwords at all? Whole-word vocabularies become enormous and fail on unseen names, while single characters or bytes make sequences long. Subwords trade a bounded vocabulary against sequence length. That trade affects embedding memory, context use, throughput, multilingual coverage, and the model's learned patterns. None of the segmentation methods guarantees that a piece is a meaningful linguistic morpheme.

Imagine labeling pieces in a box of building blocks. One block might be a whole common word, another a reusable word ending, and another a single byte. The tokenizer chooses blocks and hands their ID numbers to the model. Start with that picture; we will measure the compute cost only after the three algorithms make sense. For a broader introduction, see Tokenization: The First Decision That Shapes Everything.

WordPiece, one cursor move at a time

Suppose the word is playing and our toy vocabulary contains play and ##ing, but not playing. First, look for the longest vocabulary piece that fits at the beginning: play. Next, move the cursor past those four letters. The leftover text is ing. Because this is no longer the beginning of the word, look for a continuation piece marked ##. Choose ##ing, then emit the two IDs. The ## marker is a vocabulary convention, not something the user typed.

If no continuation piece fits, we cannot quietly skip letters. The checkpoint may use an unknown token or another declared fallback. WordPiece training builds the vocabulary, but the encoding rule here is greedy longest-match. The Hugging Face tokenizer guide gives a longer WordPiece example; below we will see why this rule is not the same as BPE's ranked merge program. WordPiece / one ruleChoose, move, repeatAt each cursor position, choose the longest valid piece. After the first piece, only continuation pieces such as ##ing qualify.

BPE, a recipe learned from repeated neighbors

Now put three words on the table: low, lower, lowest. Begin with small tiles: l o w, l o w e r, l o w e s t. Count neighbors. Both l+o and o+w occur three times. A fixed tie rule chooses one, say l+o, so we make the new tile lo. Count again. Now lo+w occurs three times, making low. We save not just the new tiles but the order in which joins were learned.

Think of the recipe as numbered instructions. When new text arrives, start from the base tiles and apply the highest-priority eligible join, then the next, until no learned join fits. Merely finding the longest tile in a dictionary can give a different answer. The CKE tokenizer walkthrough draws the same distinction for its C implementations; the original subword BPE paper gives the research context. BPE / key distinctionA vocabulary is not a recipeTraining saves merge priorities. Encoding replays those priorities; it does not recount pairs in the new sentence or simply choose its longest known piece.

input: a | b | c | d
rank 1: a + b -> ab
rank 2: c + d -> cd
rank 3: ab + c -> abc

SentencePiece, the box that can hold different recipes

SentencePiece is not the name of one joining rule. It is a toolkit that can train and run BPE or Unigram piece models on raw text. For a toy sentence I like cats, the model first applies its declared normalization, then may mark spaces with a visible symbol such as ▁. This helps it learn that like after a space differs from letters inside another word. The actual model file decides the pieces; the drawing is illustrative.

With a BPE model, follow the learned merge ranks. With a Unigram model, imagine several possible paths of pieces across the sentence and choose the complete path with the best score. Finally, map the chosen pieces and any special symbols to IDs. Google's SentencePiece documentation documents both model types. CKE's Unigram-focused deep dive shows the path search in more detail. Name checkSentencePiece is the toolkitRead the model file before naming the algorithm: a SentencePiece tokenizer can use BPE or Unigram.

playing
play | ##ing
emit ID(play), ID(##ing)

Where do agent and tool-call tokens fit?

Picture a conversation as letters in envelopes. BPE or Unigram cuts the letter's words into pieces. A registered special token can mark the envelope boundary: a message begins, a message ends, or the model has reached an end-of-turn marker. These control IDs are part of the model's protocol, not frequent word fragments that BPE happened to learn. But not every agent-looking string is special. The words assistant, a tool name such as calculator, JSON arguments, and even a visible <tool_call> tag may be ordinary text, depending on the model's chat template. The Qwen3-Coder template is a concrete example of dedicated message-boundary spellings alongside textual tool-call markup.

How does the boundary become one ID? The model's tokenizer artifact registers an exact spelling with an ID (or a post-processor inserts the ID directly). At encoding time the tokenizer recognizes that registered boundary as an atomic unit and segments the ordinary text around it with BPE or Unigram. In CKE's true-BPE encoder, the special-token scan emits a registered ID before sending the surrounding spans through the normal BPE path. Hugging Face's special-token API describes the same principle: registered special tokens are not split by the subword model. Agent boundaryOne marker, one IDOnly a registered control marker gets this treatment. A tool name or JSON field can still be ordinary text, and a separate harness executes the call.

Adding a new special token is more than adding a string to C code. First register its spelling, unique ID, and matching behavior in the tokenizer artifact or runtime; this is model-specific configuration, not necessarily a hard-coded branch. Next, if the ID enlarges the vocabulary, resize the model's input embeddings and usually its output projection, then train or fine-tune so that ID has learned meaning. If the model already reserves that ID, use its existing contract rather than inventing another one. Finally update the chat template, parser, stop conditions, and exact-ID tests together. Hugging Face's tokenizer API explicitly calls out embedding resize for newly added IDs. Merely defining <new_tool> in a tokenizer cannot teach an unchanged model to call a tool.

There is also a trust boundary. A user typing the visible characters of a control token should not silently acquire the authority of a template-created message boundary. Applications must handle untrusted text so it cannot be mistaken for control structure, and test the actual model/template/runtime combination. This is why CKE's complete Qwen tool-prompt parity PR, Gemma BOS ownership PR, and AddedToken capacity PR matter: the first wrong ID may be a control boundary rather than a word.

The math contract: from characters to the loss

Let x be a UTF-8 string. A tokenizer is a composition, not a single lookup: normalize N, split at permitted boundaries P, segment with learned or fixed rules S, then map pieces to IDs I. A model's embedding table consumes those IDs. Changing any stage changes the model input, even if the visible text remains identical.

Here θ means learned merges or piece scores when those exist; for a fixed byte alphabet it can be empty. The position term p_i is schematic: some models add position vectors, others encode position in attention. The training objective is over the resulting IDs, not over an abstract word:

For chat, the template and its registered control markers wrap this ordinary-text segmentation; they are additional terms in the input contract. A tokenizer swap without retraining is therefore not a cosmetic change. The embedding rows, output IDs, probabilities, and sometimes chat-control tokens no longer mean the same things. This is why parity at the ID boundary comes before kernel parity.

Now train BPE on a table small enough to count

Imagine a corpus containing one occurrence each of low, lower, and lowest. Ignore end-of-word markers for this teaching example. Start with character sequences:

(l,o) and (o,w) each appear three times. A real trainer needs a deterministic tie rule; for this example choose (l,o). Replace that adjacent pair everywhere with the new piece lo. Now (lo,w) appears three times, so the next merge can make low. More iterations can make lower or lowest if their frequencies and stopping policy warrant it. This is bottom-up compression: frequent neighboring units become reusable larger units. The original subword BPE paper adapted this idea for rare-word translation. Sennrich et al. (2016)

Corpus      low | lower | lowest
Initial     l o w | l o w e r | l o w e s t
Count       (l,o): 3     (o,w): 3     (w,e): 2
Rank 1      l + o → lo       [tie rule chooses this pair]
New corpus  lo w | lo w e r | lo w e s t
Rank 2      lo + w → low
New corpus  low | low e r | low e s t

Training gives us a vocabulary and an ordered merge program. During encoding, the model's BPE tokenizer applies those ranks to input pieces. In byte-level BPE, the base alphabet covers bytes, so unfamiliar UTF-8 text can still be represented, though some strings may use many tokens. OpenAI's tiktoken implementation shows pre-tokenization followed by ranked byte-pair merging. That byte-level guarantee belongs to that design; it should not be ascribed to every BPE configuration.

Write the BPE trainer as an algorithm

Suppose each training word w has corpus frequency f(w) and current piece sequence s(w). At one training step, count each adjacent pair across that multiset, then merge the pair with the largest count. The toy low example makes the tie visible; a real trainer must also fix its tie-breaking, boundaries, initial alphabet, special tokens, and stopping rule.

Then replace eligible occurrences of a*,b* by the new piece a*b* and repeat until the merge or vocabulary budget is exhausted. Inference does not recount the input corpus: it begins at the model's base symbols and repeatedly applies the eligible adjacent merge with the best learned rank. The ranking is an executable artifact, not merely a set of vocabulary strings.

pieces = initial_symbols(pretokenize(normalize(text)))
while an adjacent pair has a learned rank:
    pair = eligible pair with the smallest rank
    merge that occurrence (respecting the tokenizer's tie rule)
return [vocabulary_id(piece) for piece in pieces]

This pseudocode deliberately leaves byte mapping, pre-token boundaries, and ties to the model artifact. It is the distinction between a family of algorithms and the exact tokenizer you must port.

The trap: a trie can be fast and still be wrong

Let the toy vocabulary contain a, b, c, d, ab, cd, and abc. Suppose merge rank 1 is c+d→cd, rank 2 is a+b→ab, and rank 3 is ab+c→abc. This is a possible training history if abc appeared elsewhere without a following d. For input abcd, a longest-match lookup sees abc then d. Ranked BPE starts with [a,b,c,d], must choose c+d first, then a+b, and cannot apply ab+c. It emits [ab,cd]. Same input, same vocabulary, different IDs. The example is synthetic, but the failure mode is real: vocabulary membership does not encode merge history.

CKE's history illustrates the distinction. The older greedy C tokenizer can choose trie or hash lookup. CKE also has a separate ranked BPE implementation that applies merge priorities. The public tokenizer documentation walks through both. I would not describe the greedy path as equivalent to GPT/Qwen-style ranked BPE merely because both consume subword vocabularies. The criterion is exact token-ID parity on the intended model's real tokenizer fixtures.

Where WordPiece, BPE, and Unigram really differ

Here is the subtle correction that matters in model bring-up: SentencePiece is not a fourth algorithm that always means Unigram. It is a toolkit and serialization ecosystem that supports both BPE and Unigram, with its own normalization, whitespace marker, and optional byte fallback. Its project documentation says so explicitly. Google's Gemma PyTorch tokenizer loads a SentencePiece model; the Gemma tokenizer implementation notes describe BPE with byte fallback and space normalization. So “Gemma uses SentencePiece” and “Gemma uses BPE” can both be true. “Gemma therefore uses Unigram” does not follow.

For Unigram, if a word admits segmentations [low, er] and [lo, wer], a model score over pieces chooses the path with the best combined score. In log space, a simplified decision is argmax over paths Σ log P(piece); actual implementations also encode unknown/fallback and normalization rules. This is different from replaying an ordered BPE merge list. The Unigram subword regularization paper explains why one may sample alternative segmentations during training rather than always take the best path.

Unigram is a shortest-path problem in log space

For Unigram, make a graph whose vertices are character offsets and whose edges are candidate pieces spanning those offsets. If edge e has piece probability p(e), its cost is -log p(e). The best segmentation is the minimum-cost path from offset zero to the end. Dynamic programming implements this without enumerating all segmentations:

That is why [low, er] and [lo, wer] can be compared as whole paths. Greedy WordPiece instead commits to a longest piece at each cursor; ranked BPE replays merge priorities. These are distinct inference contracts even if all three return integer IDs.

Why decoder models often choose BPE, but did not converge on one tokenizer

Byte-level BPE is a strong engineering default for many decoder models: fixed vocabulary, deterministic merge rules, reusable pieces for common code and prose, and a byte base that avoids an unknown word for arbitrary UTF-8. That helps explain its prevalence, not a universal victory. The exact pre-tokenizer, normalizer, special-token inventory, and merge ranks still vary between model families. WordPiece persists in BERT-style encoder checkpoints. SentencePiece packages both BPE and Unigram, and Unigram remains useful where raw-text training or segmentation sampling matters. “The industry uses BPE” is too broad; the more defensible observation is that subword vocabularies with explicit fallback and model-specific contracts became the practical middle ground between words and bytes. tiktoken, SentencePiece, and the Hugging Face tokenizer summary illustrate the diversity.

When fewer tokens do not mean a better model

Loss per token is not comparable across tokenizers because their tokens cover different amounts of source text. For the same held-out UTF-8 bytes, use a byte-normalized measure. If a passage has B bytes, the bits per byte (BPB) under an autoregressive model is:

Worked example for a 20-byte held-out passage: tokenizer A yields 12 tokens at 1.2 nats/token, total 14.4 nats or about 1.039 BPB. Tokenizer B yields only 8 tokens but 1.7 nats/token, total 13.6 nats or about 0.981 BPB. B has the worse per-token loss yet the better byte-normalized likelihood. This toy arithmetic is not a CKE benchmark. Compare on exactly the same raw held-out bytes, with a valid probability model, before claiming a tokenizer improves modeling quality. Also measure throughput and context cost separately: compression, likelihood, and speed are different axes. Compare fairlySame bytes, then compareToken counts and loss per token change with the tokenizer. Bits per byte uses the same held-out text as the denominator.

Only now: what does faster tokenization buy us?

My first serious C tokenizer was a hand-built WordPiece frontend for a CPU text pipeline. It lowercased ASCII, separated punctuation, scanned spaces with AVX-512 where available, looked up candidate pieces in a hash table, and built overlapping token windows. The central algorithm was still the simple cursor rule above: choose the longest initial piece, then the longest allowed ## continuation. The Fast WordPiece Tokenization paper explains a linear-time approach to that lookup.

Vector instructions can accelerate byte cleanup or whitespace scanning, but they do not settle which IDs are right. Hash and trie lookups change the cost of finding a piece; they do not turn longest-match WordPiece into ranked BPE. My early experiment even skipped unmatched bytes, a reminder that a fast incomplete frontend is unsafe as a general model tokenizer. Later CKE work had to make byte handling, ranks, and reference parity explicit. The greedy C path and ranked BPE path are separate for that reason.

Sequence length matters downstream too. If one tokenizer produces 400 IDs and another 250 for the same passage, a conceptual full-attention score grid has 400² / 250² = 2.56 times as many positions in the longer case. The KV-cache length factor is 400 / 250 = 1.6. These are arithmetic illustrations, not latency benchmarks: fused attention may not materialize that grid, and a larger vocabulary increases embedding and output-head storage by roughly V × d parameters before weight tying. For the actual matrix computation, see Attention: The Core of the Transformer. Correctness and the compression/compute trade must be measured separately.

CKE's tokenizer log: from a fast lookup to an exact contract

The interesting history is not a single speedup. It is a sequence of fixes to what counts as the same tokenization. The early C hash/trie path made vocabulary lookup practical; the separate ranked BPE path addressed a different semantic problem. Later work moved the acceptance criterion from plausible English output to exact model-specific token IDs, prompt controls, Unicode boundaries, and capacity behavior. The links below are selected engineering milestones, not an exhaustive PR list.

Start at the first source files, not the first PR number

The first CKE tokenizer work predates the numbered PR trail. On January 7, a direct commit introduced the C tokenizer with vocabulary hashing, UTF-8 support, memory helpers, and a test program. Its encoder used greedy longest-match; it was not yet proof of exact ranked-BPE parity. On January 9, another direct commit introduced true_bpe.c with an explicit merge table and priorities. The January 28 update extended that path. These are the honest starting points; calling them early PRs would be inaccurate.

From there the files separate by responsibility. Read the greedy vocabulary frontend beside its basic C tests; compare it with the ranked BPE encoder. The SentencePiece/Unigram path and its parity tests arrived in February; the pure-C BPE trainer followed in its own direct commit. These are tokenization frontends and training tools, not GEMM-style numerical kernels. The performance question comes after choosing the correct segmentation contract.

The PRs below are the traceable pull-request contributions I found across tokenizer implementation, training, model import, and prompt serving. Some are broader PRs with tokenizer changes inside them. The direct-commit trail also includes ASCII identity and true-BPE loading, SVG DSL atoms, and byte/UTF-8 parity. None of these is mislabeled as a PR.

What this history taught me: the dangerous bug is often silent success. The model still returns text, but its first divergent ID may be a leading-space marker, a byte continuation, an added tool token, or a truncated prompt. That is why model-specific fixtures and fail-closed capacity checks deserve the same status as kernel numerical tests.

What if the training target is SVG?

“SVG training needs no tokenizer” is almost right only if we mean no learned subword tokenizer. A discrete language model always needs a representation-to-ID contract. There are at least three honest routes, and they solve different problems:

For a tiny formal grammar, I would begin with the scene DSL rather than assume BPE is required. Reserve only reusable structural atoms, serialize variable parameters separately, validate the compiled SVG, and compare against raw-byte and hybrid BPE baselines on held-out compositions. CKE's spec-training method describes that compositional design; its SVG tokenizer ablation is a proposed controlled comparison, not a measured verdict. A later raw-SVG fixture also shows why complete-document validity and truncation checks matter. A tokenizer can improve compression yet make a formal output task harder if it obscures the grammar's useful boundaries.

What I test before trusting a tokenizer port

A tokenizer is a numerical input dependency. If the IDs differ, downstream model parity is impossible even if every GEMM and attention kernel is perfect. My practical gate is therefore an encode/decode fixture matrix, not a single English sentence:

For each probe, compare the full ID sequence against the reference tokenizer shipped with the checkpoint, then decode where the format promises reversibility. Run the same test on the actual generated runtime boundary, not just a helper function. CKE's tokenizer guide is a useful visual starting point; source and parity fixtures determine what is actually certified.

The CKE SentencePiece deep dive focuses on its Unigram/Viterbi path; the SentencePiece toolkit itself also supports BPE. That naming distinction is exactly the sort of contract detail worth carrying into model integration.