Oct 24, 2026
From WordPiece to Ranked BPE: What a Tokenizer Really Does
A visual math log of WordPiece, BPE, SentencePiece, special tokens, and exact token-ID contracts in CKE.
Read post →Tagged With
tokenization
3 posts connected to this tag.
Subscribe
Get my rants delivered to your inbox
I will send new posts as and when I write. No fixed cadence, just engineering notes, rants, and things I am thinking through.
Subscribe to emails from Anthony
First Name
Last Name
Email
Submit
Need an intelligent system to work on real hardware?
Embedded systems · Robotics · Constrained AI · CPU and HPC · Accelerators · Distributed systems
Work With Us / Antshiv Robotics
ShivasNotes
Independent notes on software, systems, AI workflows, robotics, and building in public.
