Byte-pair encoding · learned in the browser · zero dependencies

Watch a tokenizerlearn language

A hands-on playground for a custom subword tokenizer: paste any corpus, watch the merge log fill up in real time, then encode and decode text with animated token chips, live statistics and a searchable vocabulary.

Vocab
0
Merges
0
Special tokens
4
tokenizer.session
scroll
Vocabulary size
0

0 base slots + 0 learned merges

Corpus words
0

whitespace-split training tokens

Tokens for input
0

50 characters → 51 pieces

Chars per token
0.00

higher is better compression

1

Train the tokenizer

Paste a corpus, pick a vocabulary budget and watch byte-pair merges happen live.

Corpus presets
ready
0vocab / 150
Awaiting corpus

Most frequent pairsmerged next

No mergeable pairs left — corpus fully compressed.

Merge log0 merges

Press Train to watch byte-pair merges happen live.

2

Encode & decode

Encoding is live — every keystroke re-runs the merge stack and animates the token stream.

<BOS>SpecialtheMergedxCharacter␣Word end?Unknown

Token stream

round trip differs
<BOS>0T1h1e1␣1q1u1i1c1k1␣1b1r1o1w1n1␣1f1o1x1␣1j1u1m1p1s1␣1o1v1e1r1␣1t1h1e1␣1l1a1z1y1␣1d1o1g1!1␣1@121012151␣1<EOS>0

Encoded ids

[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0]

Decoded text

—
Raw text50characters
Subword pieces51merges applied
Token ids53integers
Decoded—round trip
3

Vocabulary explorer

Everything the tokenizer learned, ranked by id — filter it, search it, export it.

0 entries
0 / 0 tokens
  • No vocabulary entries match “”.

How the merges work

Three steps, repeated until the vocabulary budget is spent.

01

Split & count pairs

Every word becomes its characters plus a </w> end marker. We count how often each adjacent pair occurs across the whole corpus.

02

Merge the winner

The most frequent pair is glued into a brand-new vocabulary entry, and every occurrence in the corpus is rewritten in one pass.

03

Repeat, then encode

Loop until the vocabulary hits its target size. Encoding replays merges by rank; decoding is a lossless join of the pieces.

4

Word-level reference demo

The original frequency-based tokenizer from src/tokenizer.ts, side by side with the BPE run.

InputHello world from tokenizer
Tokens[2, 4, 5, 6, 8, 3]
DecodedHello world from tokenizer
Vocab size11
Special tokens<PAD>, <UNK>, <BOS>, <EOS>
CorpusHello world from the tokenizer demo corpus