If you write prompts in Thai, or Khmer, or Lao, you may have noticed that a document which fits comfortably in English does not fit in your language, and that the bill at the end of the month is larger than the equivalent work in English would have been.
This is not an impression; it is a measurable effect, and the multiples are larger than most people expect.
How you measure this fairly
A fair comparison cannot use just any text. A paragraph of English news against a paragraph of Thai cooking instructions measures the topics, not the languages. You need one document, professionally translated, that means the same thing in every language.
The Universal Declaration of Human Rights is the standard answer. It is professionally translated into hundreds of languages, and the Unicode Consortium publishes the whole set in a machine-readable form. We took it in eleven languages — every ASEAN member state, plus Singapore's four official languages — and counted the tokens in each.
These are exact counts, not estimates. They come from the same tiktoken tables that power our own Token Counter, so you can verify any number below by pasting the source text into the tool yourself.
The same document, eleven prices
The English text of the declaration is 1,888 tokens. Here is what the identical document costs in each of the others, under the o200k encoding used by current GPT models:
| Language | Script | Tokens | Times English |
|---|---|---|---|
| Lao | Lao | 19,026 | 10.08× |
| Burmese | Myanmar | 8,061 | 4.27× |
| Vietnamese | Latin with diacritics | 6,693 | 3.55× |
| Khmer | Khmer | 6,271 | 3.32× |
| Tamil | Tamil | 4,552 | 2.41× |
| Thai | Thai | 3,779 | 2× |
| Tagalog | Latin | 3,549 | 1.88× |
| Malay | Latin | 2,924 | 1.55× |
| Indonesian | Latin | 2,779 | 1.47× |
| Chinese (Simplified) | Han | 2,192 | 1.16× |
| English | Latin | 1,888 | 1× |
Lao is the outlier and it is not close. The same declaration costs ten times more to send, and fills ten times more of the context window, than it does in English.
Why it happens
A tokenizer is trained on text. Whatever it saw a lot of, it learned to compress into long pieces; whatever it saw little of, it has to fall back on splitting into bytes.
The fallback to byte-level splitting shows up directly in the data. Divide the size of the text by the token count and you get the average number of bytes each token carries. English gets 5.45 bytes per token. Thai does better still at 7.03, because Thai characters are three bytes each and the tokenizer has learned useful multi-character pieces. Lao gets 1.61 — barely more than a byte and a half, which means the tokenizer is chopping three-byte Lao characters into two tokens apiece and has learned almost no Lao words at all.
Chinese is the instructive opposite, sitting at just 1.16× the English cost. The writing system is completely different, but a single Han character often carries the weight of an English word, and the tokenizer has seen a vast amount of Chinese. Script is not the problem. Training data is.
Newer models fixed some of this, and skipped Lao
The current o200k encoding, which replaced the older cl100k used by GPT-4 and GPT-3.5, is much more efficient for most Asian scripts. The improvement is not evenly distributed.
| Language | Tokens saved by the newer encoding |
|---|---|
| Tamil | 75.4% |
| Burmese | 73.2% |
| Khmer | 62.7% |
| Thai | 56.7% |
| Chinese (Simplified) | 32.8% |
| Indonesian | 23.3% |
| Vietnamese | 20% |
| Lao | 13.3% |
Burmese cost dropped by nearly three quarters. Tamil by more. Lao got 13.3%, which is why it now stands so far outside the rest: the upgrade that rescued its neighbours largely passed it by.
What it means for your bill and your context window
This has two consequences, and the second is the one people miss.
Cost. API pricing is per token. A Lao-language support bot processing the same conversations as an English one pays about ten times as much for the input. A Vietnamese one pays about three and a half times.
Context. A model's context window is measured in tokens too. A 128,000-token window holds roughly 67 English copies of the declaration and fewer than 7 Lao ones. In retrieval systems, chunking to a fixed token size means each chunk in these languages carries far less actual meaning, which quietly degrades answer quality without ever throwing an error.
What you can do
- Measure before you budget. Estimating tokens as characters divided by four is roughly right for English and badly wrong for every language in the top half of that table. Paste a real sample into our Token Counter, or the tokenizer your provider publishes, and use the actual number.
- Do not size chunks in characters. A retrieval pipeline that splits at 1,000 characters produces wildly different token counts per language. Split on tokens.
- Check the encoding, not just the model name. Moving from a cl100k-era model to an o200k-era one is worth up to three quarters of your input cost in some of these languages, and almost nothing in others.
- Weigh English as a pivot honestly. Translating to English, prompting, and translating back can cost less than prompting directly in Lao. It also adds two points of failure and loses nuance, so it is a trade-off, not a universal fix.
- Compare providers on your own text. These figures are OpenAI encodings. Other vendors tokenize differently, and the ranking above is not guaranteed to hold for them.
If you are budgeting a project around these numbers, our LLM cost calculator turns token counts into monthly figures.
The same scripts are penalised a second time, in a place nobody bills you for: form fields that count storage units rather than letters, so a forty-character box holds twenty-one Khmer characters. That is why your name does not fit in the box. And before either question arises, there is the matter of whether the page can display the script at all, which is why some languages come out as empty boxes.
What is measured here
The token counts are measured, on 20 August 2026, from the Universal Declaration of Human Rights as published in the Unicode Consortium's own repository, tokenized with the vendored tiktoken tables this site already serves. The script refuses to emit results unless every one of the eleven languages fetched and decoded to a full document, so a partial run cannot quietly become a published table.
One document is one document, and this one is formal, legal prose. A chat message or a product description would produce different ratios. What the test establishes is the direction and the rough scale of the difference, both large enough that no plausible corpus would reverse them.
For a quick check you can run in a minute, Article 1 alone is 33 tokens in English, 35 in Chinese, 72 in Thai and 273 in Lao. These are OpenAI encodings only. Anthropic and Google do not publish browser tokenizers, so we make no exact claim about Claude or Gemini here, and neither should anyone else without naming their method.
