Using AI in Your Own Language — What the Machine Can Actually See

Type The company reviewed the quarterly accounts into our token counter and it comes to seven tokens. Type the same sentence in Malay — Syarikat itu telah menyemak akaun suku tahunan — and it comes to fifteen. Same meaning, roughly the same number of letters, more than twice the tokens.

That gap is usually explained as a money problem, and we have already written that guide. What an LLM actually costs you measured the tokenisation tax across eight languages and put a number on it. This is the other half of the question, and nobody in this region has written it. What can the machine actually see of your language?

The model does not read words. In English, it nearly does.

A token is a chunk of text, and which chunks exist was decided once, by counting what turned up most often in the training text. Tokens and context windows, from zero covers that mechanism. The consequence worth measuring is what those chunks line up with.

We have an unusual thing to measure it on. Our own About page exists in nine languages. Each was written as a transcreation rather than run through a translator, which gives us a parallel corpus of 102 strings carrying the same meaning in every language. We ran all nine through the same tokeniser our token counter ships to you, and asked a question the cost guide did not: how often does one word come out as one token?

In English, 96.3% of words are a single token. In Malay, 60.2%. In Indonesian, 63.6%. Four words in ten arrive at the model as rubble, in a language with something like three hundred million speakers.

The count is not the interesting part. Look at what the pieces are. Malay carries a great deal of its grammar in affixes: membiayainya is mem- plus biaya plus -i plus -nya, four morphemes, marking voice and object and possession. The tokeniser also gives four pieces — " memb" "i" "ay" "ainya" — and not one of them is one of those morphemes. It cuts across the grammar rather than along it.

The worst in our corpus is sekurang-kurangnya, which means "at least" and is about as ordinary as Malay gets. Six tokens: " sek" "ur" "ang" "-k" "ur" "angnya". Nothing in that list is a Malay word, a Malay prefix or a Malay root.

Then there is Cantonese, which is worse than anything else we measured

Our About page has three Chinese versions: Simplified for the mainland, Traditional for Taiwan, and a Hong Kong version written in actual Cantonese rather than Standard Written Chinese. That third one turned out to be the finding.

When a character is too rare to have earned a token of its own, the tokeniser falls back to cutting it into pieces of its own byte encoding — fragments that are not characters and cannot be printed. Here is how much of each version arrives that way:

Twenty-eight per cent. More than a quarter of the Hong Kong document reaches the model as pieces of UTF-8 rather than as text.

A number that large usually points to a mistake in the measurement, so we went looking for one and found none. What it points to instead is that the characters carrying Cantonese grammar are entirely absent from the vocabulary.

嘅 is the possessive particle — Cantonese for "of". 唔 is "not". 哋 makes a pronoun plural. 咗 marks a completed action. 佢 is "he" or "she". Each of them costs two tokens, and both of those tokens are byte fragments. Their Mandarin equivalents — 的, 不, 們, 了, 他 — are each one whole token.

These characters appear 211 times in our Hong Kong copy, accounting for 415 tokens — 19.3% of the whole document. A fifth of the file is spent on the connective tissue of the language, broken into pieces the model has never seen as units.

And the pattern is not confined to vernacular Cantonese. Take ordinary Traditional vocabulary. 稅 is the character for tax. It has no token of its own, so it arrives as two fragments. 税 — the Simplified form of the same character, the same word — is a single whole token. So is 里 against 裡, 译 against 譯, 隐 against 隱, 审 against 審. If you write about tax in Taipei, the machine reads you in pieces. If you write about tax in Shenzhen, it does not.

You can check any of this yourself. Paste a single character into our token counter and read the count.

So does any of that make the answers worse?

An article like this normally stops measuring at about this point and starts asserting. We are going to be more careful than that.

The most common complaint about using these systems in a South-East Asian language is that they will not stay in it: you ask for Bahasa Malaysia and get something Indonesian, you ask for Traditional Chinese and Simplified characters creep in. It is a specific claim, and it is the one claim in this area we could score objectively, so we tested it.

Two tasks, two non-Anthropic models, three repeats each, and each instruction given once in English and once in the target language — which also tests the other question people ask, whether you should prompt in your own language or in English. We scored the results with our own tools as the ruler, so you can repeat the scoring: every Han character in the Chinese outputs was run through the OpenCC mapping inside our Chinese converter, and every Malay output was searched for the Indonesian side of all 97 word pairs in our Bahasa Indonesia ↔ Bahasa Malaysia converter.

It did not reproduce.

Not one Simplified character across 1,741 Han characters of output. Not one Indonesian-standard word: the models used mesyuarat, kakitangan, akaun, pengurus, syarikat, invois — the Malaysian form every time, never rapat, karyawan, rekening, manajer, perusahaan. And giving the instruction in English rather than in the target language made no measurable difference in either test.

Two of the twelve Malay samples are missing from that count. We are saying why rather than quietly running the numbers on ten. Both were the same model on the third repeat, and both came back with an empty body while still billing for output tokens — a reasoning model that spent its whole budget thinking and returned nothing. They are excluded, not scored as clean.

Our own scorer was wrong first, and the way it was wrong is the useful part

The first scoring pass reported an Indonesian leak in every single Malay sample. The word was pejabat.

It was a false positive, and it landed on precisely the word our own dictionary warns about. Pejabat is the correct Bahasa Malaysia word for office. It is also the correct Bahasa Indonesia word for official. Our converter carries both pairs and flags the collision in both directions; a scorer looking only for the Indonesian surface form therefore condemns perfectly good Malay.

Corrected by ignoring any form valid on both sides — pejabat is the only one of the 97 pairs that is. We are leaving the mistake in because it is the more useful half: the trap that caught an automated check is the same trap that catches a reader, and it is the reason a word-level converter shows you the note instead of silently swapping.

What that result does not license

It establishes that on these two tasks, these two vendors and these instruction languages, on 2 August 2026, the drift did not happen. It does not establish that the answers are as good in Malay or Cantonese as in English. Judging that means judging fluency, register and factual accuracy in each language, which needs a native reader and not a script. We did not have one, so we are not claiming it — and a fragmentation measurement is a mechanism, not a substitute.

The useful summary is narrower than the usual one. We did not find the failure people expect. What we did find is a measurable problem a layer below the output — the machine works in units that do not correspond to your language, most severely for Cantonese and Traditional Chinese.

What follows from this, practically

We will keep this to the things the measurements actually support.

Script choice is a lever where you have one. Simplified costs 1.19× English on the same content in our corpus; Traditional costs 1.43×, Cantonese 1.63×. If you are writing an instruction block that you will reuse a thousand times and the audience is indifferent to script, that is a fifth of your context back.

Keep local names spelled consistently. Regional institution names do not survive as single tokens the way Western ones do — across 28 ASEAN names we averaged 2.21 tokens against 1.50 for 24 US and UK equivalents, with only 3 of the 28 arriving whole. BPJS Kesehatan comes to five pieces. That is not a reason to avoid them, but it is a reason not to also spell them three different ways in the same prompt.

Your context window is smaller than the number on the box. The same document is 1.34× longer in Malay, 1.63× in Cantonese, 1.77× in Thai. A limit quoted in tokens is a different limit in every language.

And ask in whichever language you think in. On the only version of that question we could score, it made no difference — so use the one where you can tell whether the answer is any good.

We are not claiming that a persona, a worked example or a glossary in the target language improves the output. Those are plausible and widely repeated, and we have not tested them. They belong to a different guide, and it is coming.

What this audit found in our own tools

We read the source of the tools a guide will link before writing it. This time both Chinese text tools had the same defect, and it is the same shape as the subject.

Our Chinese converter counted UTF-16 code units in three places: the diff highlighter, the "characters changed" figure, and the length cap. That is wrong for any character outside the Basic Multilingual Plane — and for this tool that is not an edge case, because CJK Extension B holds characters in daily Hong Kong use (𨋢 for lift, 𠮶 for that, 𠝹 for cut), and OpenCC maps 633 single Traditional characters in the main CJK block onto Extension B forms. 嗰 → 𠮶 is one of them.

Converting 嗰個𡃁妹好靚 — a perfectly ordinary Cantonese sentence — the tool split 𠮶 across two highlight elements, leaving half a character in each, so it rendered as two replacement boxes. Highlighting is on by default, so that was the default experience. 𡃁, which had not changed at all, was split and marked as changed. The length shift then knocked every later position out of alignment, and the tool reported 8 characters changed in a 6-character sentence while counting the sentence as 7 characters long.

Our Chinese prompt normalizer had the same root cause in its before, after and changed tiles. No garbled output, because it writes to a plain text box, but it told you a 15-character prompt was 19 characters long. We fixed both together rather than one at a time.

The symmetry here is uncomfortable, and worth stating plainly. The tokeniser shatters rare characters because it was built by counting text in which they were rare. Our converter shattered the same characters because it was written by someone assuming a character is one unit. Neither is malice and neither is exotic; both are the ordinary consequence of building against the common case and never testing the other one.

General information about how text-processing software handles these languages, not advice about any particular product. Figures describe one tokeniser, one corpus and two models on one day, and every limit on what we measured is stated above rather than left for you to infer. Where this guide says it did not test something — output quality per language, and whether personas or glossaries help — it means it did not test it, and you should treat any claim to the contrary, including ours in future, as needing its own evidence.