Where does the text split?
Compare subword tokenizers and learned patching methods on the same text.
Choose an example
Choose a saved passage or add your ownWrite your own example
Add a Hugging Face tokenizer Optional · BPE, WordPiece, Unigram
Bring your own tokenizer
BPE · WordPiece · UnigramPaste a public Hugging Face model link or ID. Downloads tokenizer files only; your text stays in this browser. Add up to four to compare.
An internet connection is needed to add a tokenizer. No model weights or login required.
Compare the boundaries
Each colored transition marks a group boundaryThree experiments to try
Add a Hugging Face tokenizer No tokenizers loaded
Read across to compare examples. Each column has its own input; the selected method and display controls apply to all columns.
Swipe or scroll horizontally for more columns →Custom text runs with the space baseline and your loaded tokenizers. Saved BPE, BLT, H-Net, and Bolmo outputs apply to their original passages only. Token counts do not measure model quality or runtime.
How a hierarchical Transformer processes bytes
This example illustrates a common local encoder → global Transformer → local decoder design. Watch 16 input bytes get processed in three chunks, then produce byte-level outputs. The drawing is a conceptual animation, not a live model run or a measurement of compute.
1
2See the pre-tokens
BPE merges never cross a pre-token boundary3
Flags: gu. Alternatives are tried left to right at each position; the first one that matches wins. Each line is a separate split applied to the pieces left by the line above; DeepSeek V3 uses three. In hand-edit mode, named groups (?<name>…) label the pre-tokens.