Watch a tokenizerlearn language
A hands-on playground for a custom subword tokenizer: paste any corpus, watch the merge log fill up in real time, then encode and decode text with animated token chips, live statistics and a searchable vocabulary.
0 base slots + 0 learned merges
whitespace-split training tokens
50 characters → 51 pieces
higher is better compression
Train the tokenizer
Paste a corpus, pick a vocabulary budget and watch byte-pair merges happen live.
Most frequent pairsmerged next
No mergeable pairs left — corpus fully compressed.
Merge log0 merges
Press Train to watch byte-pair merges happen live.
Encode & decode
Encoding is live — every keystroke re-runs the merge stack and animates the token stream.
Token stream
round trip differsEncoded ids
[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0]
Decoded text
—
Vocabulary explorer
Everything the tokenizer learned, ranked by id — filter it, search it, export it.
- No vocabulary entries match “”.
How the merges work
Three steps, repeated until the vocabulary budget is spent.
Split & count pairs
Every word becomes its characters plus a </w> end marker. We count how often each adjacent pair occurs across the whole corpus.
Merge the winner
The most frequent pair is glued into a brand-new vocabulary entry, and every occurrence in the corpus is rewritten in one pass.
Repeat, then encode
Loop until the vocabulary hits its target size. Encoding replays merges by rank; decoding is a lossless join of the pieces.
Word-level reference demo
The original frequency-based tokenizer from src/tokenizer.ts, side by side with the BPE run.