A text splitter cuts a long document into chunks small enough to embed, and where it cuts decides whether a word arrives in one piece. We ran three splitters over 13 texts in seven languages at 512 tokens per chunk with 64 of overlap, counting chunk edges, each chunk's start and end. LangChain's recursive splitter, measuring in tokens, cut inside a word at 2 of 1,134 edges. LlamaIndex's SentenceSplitter did so at 5 of 930. A plain token window cut inside a word at 422 of 892, and inside a visible character at 78, all of them in Thai or Khmer, where neither library cut one.
Neither library defaults to 512 and 64. At its own 4,000 characters with 200 of overlap, LangChain cut inside a word at 0 of 124 edges, and LlamaIndex at its 1,024 tokens with 200 at 1 of 500. The texts were eight Universal Declaration of Human Rights translations from the UDHR in Unicode repository (formerly unicode-org/udhr; commit 588b3f4b; counts only) and five Wikipedia articles on Thai, Khmer, Malay, Chinese and Japanese, as retrieved on 16 November 2026. LangChain ran as the langchain-text-splitters 1.1.3 package on Python 3.12.16; LlamaIndex as its unmodified SentenceSplitter source at v0.14.26, on the same Python with nltk 3.10.4 and stubbed plumbing. Tokens are cl100k_base (tiktoken 0.14.0), and edges were scored with Intl.Segmenter in Node 26.13 (ICU 79.1, Unicode 18), so "a word" is what ICU calls one. Shresth Shukla's chunking analysis for LanceDB (27 January 2025) compared methods in English, Hindi, French and Spanish without scoring edges.
LangChain's separator list ends with the empty string
LangChain's recursive splitter measures length with Python's len, which counts code points, and tries four separators in turn: a blank line, a newline, a space and the empty string. The last splits any piece still too long between every character. LangChain's documentation, as read on 16 November 2026, calls it "the recommended one for generic text".
LlamaIndex's SentenceSplitter at v0.14.26 runs a longer chain. Its paragraph break is three newlines, so a blank line never triggers it, and nltk's untrained Punkt tokenizer ends sentences only at a full stop, question mark or exclamation mark. A regular expression then knows the ideographic full stop 。 but lacks the full-width comma , and the Khmer khan ។. Last come spaces and single code points.
Unspaced text longer than the chunk is the trigger
All 949 mid-word edges in our LangChain runs fell inside a stretch with no space or newline at least as long as the chunk, or within one token of that length. The Chinese Wikipedia article's longest one is 641 characters. Chunked at 250 characters, it took 8 mid-word edges, and at 100 it took 59.
Over all 13 texts, LangChain at 1,000 characters cut no word at its 670 edges; at 100 it cut 410 of 7,120. The libraries' clean record on visible characters lasts down to 256 tokens: at 128, LlamaIndex cut inside one 13 times, 7 of them in the Khmer UDHR.
Phrase spaces carry Thai and Khmer through the recursive splitter
Thai and Khmer write most words unspaced, but these texts put spaces between phrases: 251 in the Thai UDHR and 436 in the Khmer. The Khmer declaration's longest unspaced stretch is 95 characters, so it took no mid-word edge even at 100. The Thai one's is 154, and at 100 it took 7 in 286. The rule is the same; what differs is the text.
Adding Chinese punctuation removes most Japanese mid-word edges
LangChain's documentation warns that for Chinese, Japanese and Thai the default list "can cause words to be split between chunks", and suggests adding punctuation including two full-width and two ideographic marks and the zero-width space. On the Japanese article at 100 characters that list took mid-word edges from 330 to 22, and at 250 from 44 to 6. On the Chinese article the drops were 59 to 2 and 8 to 2. Thai, Khmer and Malay were already at zero. The list has no Khmer khan and no Thai punctuation, and its zero-width space helps only text that carries one: the Khmer article holds 487, the UDHR files and the Thai article none.
Plain token windows cut inside words at almost half their edges
The plain token window in these counts was our own text chunker before its revision dated 16 November 2026. It refused only a cut inside a UTF-8 byte sequence, so no chunk held a broken "�" at any size we tried, but it cut inside a word at 47.3% of edges, Malay and Indonesian included. On the Khmer UDHR, 31 of 76 edges fell inside a visible character: 14 chunks opened on a vowel sign, coeng or other mark, and 17 ended on a consonant whose mark went to the next chunk. The page promised no edge split a character, which held only for code points.
The revised version cuts only where Intl.Segmenter starts a new grapheme cluster, and a "Keep words whole where possible" option, on by default, moves each edge back to a word gap within half a step. Over the 13 texts at 512 with 64, grapheme boundaries alone took mid-character edges from 78 to 0. With word edges on, mid-word edges fell from 422 of 892 to 0 of 896 (the plan changes), while repeated tokens rose 2.4%, from 28,647 to 29,321. In our runs no chunk ran over its size. Both kinds of edge come from the reader's browser, whose ICU version can differ from ours. To see it, paste a few thousand characters of Khmer at the default 512 and 64, then switch the option off and on; the note under the results counts the edges moved back.
Overlap repeats tokens and can still lose a sentence
Overlap keeps a cut sentence whole in the next chunk only if the overlap is longer than the part before the cut. In the plain token window at 512 with 64, 28 of the 45 Khmer UDHR sentences that crossed an edge were whole in no chunk. Thai lost 12 of 26 that way, English none of 8. At 800 with 400 of overlap, Khmer fell to 2 of 39 and Thai to 0. These are ICU's sentences, and with no full stop in Thai, one is often a whole paragraph.
Token-measured chunks can run a few tokens over
A token-measured splitter counts pieces, then joins them, and the join can count to more than the parts. Across the 13 texts that pushed a chunk past its size twice, both at 128 tokens: a LangChain chunk on the Malay article came to 129, a LlamaIndex chunk on the Japanese article to 132.
Measuring in characters, LangChain's default, hides a wider spread. Our median for a 1,000-character LangChain chunk of the UDHR was 165 cl100k tokens in English and 271 in Malay. Thai came to 902, and Khmer to 1,505. Those tokens meet the limits set out in One embedding model, four different maximum lengths and Tokens and context windows, explained from zero; what a cut does to a Khmer cluster is in What a character limit does to Khmer.