Tokenization Visualizer

From characters to tokens

A large language model never sees your text the way you do. Before a single word reaches the network, a tokenizer chops the raw string into a sequence of tokens — small, reusable chunks drawn from a fixed vocabulary of tens of thousands of entries. Each token maps to an integer id, and it is that stream of integers, not the letters, that the model reads, predicts, and bills you for. Tokenization is the invisible first step that shapes everything downstream: how long your prompt counts as, how much it costs, and even how well the model reasons about the pieces of a word.

The dominant scheme for modern models is Byte Pair Encoding, or BPE. It starts from individual bytes and repeatedly merges the most frequent adjacent pair into a new symbol, learned once over a giant training corpus. Common words like the, and, or token survive as a single token because they appear constantly; rarer or longer words get assembled from several sub-word fragments. This is a deliberate trade-off — a vocabulary small enough to be efficient, yet expressive enough to spell out any word, name, or typo it has never seen by falling back to smaller pieces, all the way down to raw bytes if needed. Nothing is ever truly out-of-vocabulary; worst case, a string is encoded one byte at a time.

Spacing matters more than people expect. In these encodings a leading space is usually glued to the following word, so "token" at the start of a string and " token" mid-sentence are different tokens with different ids. That is why this visualizer marks spaces with a middle dot and newlines with an arrow: the whitespace is part of the token, not a gap between tokens, and seeing it explains a lot of otherwise baffling counts.

Why "strawberry" splits oddly — and why it matters

The famous example is strawberry. Ask a model how many letter "r"s it contains and it often stumbles — not because it can't count, but because it never sees the individual letters. At the start of a text, cl100k_base splits strawberry as str + aw + berry and o200k_base as st + raw + berry. In the middle of a sentence, " strawberry" with its leading space is a single token in both, so the model gets one id and no letters at all. The example in the box above shows the second case; delete "The " to see the first. The same effect makes models weak at spelling, reversing strings, and character-level arithmetic: those tasks live below the resolution of the token.

This has very practical consequences. Token boundaries decide your real cost, since APIs charge per token, and your real context limit, since the window is measured in tokens, not words. Non-English text, especially scripts far from the training data, can use several tokens per character and quietly blow past budgets. Code tokenizes unevenly because of symbols and indentation. Even a stray emoji can expand into many tokens. Seeing the split — rather than trusting a rough "four characters per token" rule — lets you trim prompts intelligently, design better chunking, and understand why a model behaves the way it does on the letters inside a word. This tool shows you the genuine article, computed locally from the same encoding tables the OpenAI API uses.

Disclosure: RECATOOLS is written with the assistance of Claude, made by Anthropic. Claude's tokenizer is not shown here, and the page names it only to say so.