Text Chunker for RAG & Embeddings
Split long text into token-sized chunks with overlap for RAG and embeddings, using exact GPT tokenization. Copy as JSON. Free, runs in your browser.
Text Chunker for RAG and Embeddings
How to Use the Text Chunker
Paste your document
Drop in the text you want to split, and pick the tokenizer: cl100k_base is what OpenAI's embedding models count with, o200k_base is GPT-4o, GPT-4.1 and GPT-5. Counts are exact for that encoding, and a chunk edge never falls in the middle of a character.
Set chunk size and overlap
Choose how many tokens each chunk holds and how many tokens neighbouring chunks share, or start from a preset: 800 / 400 is OpenAI's default for its vector stores, 1024 / 200 is LlamaIndex's sentence splitter default (which also respects sentences; this tool does not).
Review the chunks
See the token count, the number of chunks, the largest chunk, and how many tokens you will actually embed once overlap is counted twice. Each chunk shows its token range. Tune until the splits make sense for your content.
Copy as JSON
Export every chunk as a JSON array of strings, ready for an embedding pipeline or vector store — the copy includes all chunks, not only the ones shown. Everything stays in your browser.
Chunking: The Quiet Decision That Makes or Breaks RAG
Why documents get chunked at all
Retrieval-augmented generation (RAG) works by embedding your documents into vectors, storing them, and at query time retrieving the most relevant pieces to feed the model. But you can't embed a whole book as one vector — embedding models have input limits, and a single giant vector blurs everything together so retrieval becomes useless. So documents are split into chunks: passages small enough to embed cleanly and specific enough that a match means something. Chunk by tokens rather than characters or words, because tokens are the unit an embedding model's input limit is set in — but only the model's own tokenizer counts them exactly, which is why this page asks which one you mean.
The two dials that matter are size and overlap. Smaller chunks give precise retrieval but can lose surrounding context; larger chunks keep context but dilute relevance and cost more to embed. Overlap — repeating a slice of tokens between neighbouring chunks — stops an idea that straddles a boundary from being cut in half, so a sentence split across two chunks still appears whole in at least one. There's no universal best setting; it depends on your content and your model. Two published defaults give a sense of the range: OpenAI's vector stores split files into 800-token chunks with 400 tokens of overlap unless told otherwise (and require the overlap to be no more than half the chunk), and LlamaIndex's sentence splitter defaults to 1,024 tokens with 200 of overlap.
"Retrieval can only return what chunking preserved. Split badly and the right answer is sitting in a chunk your search will never surface."
Why exact token boundaries matter
This tool chunks on real tiktoken boundaries, so each chunk's token count is exact for the encoding you chose and never over your size limit — important when an embedding model has a hard input cap. One subtlety matters for Asian-language text: tiktoken works on bytes, and a single Chinese, Japanese, Korean or Thai character, or an emoji, is often two or three tokens. Cutting blindly every N tokens can end a chunk half-way through a character, which decodes as "�" and loses it. This tool only cuts where a character starts, moving an edge back by a token or two when needed; the note under the results says when that happened. If one character is larger than the whole chunk size, that chunk runs over rather than break it. It uses a fixed-size sliding window (advance by size minus overlap) and exports the result as JSON. Estimate the resulting cost with our Embedding Cost Calculator, and check how the retrieved chunks stack up against a model's limit with the Context Window Visualizer.
Other models count differently
Anthropic and Google do not publish a tokenizer that runs in a browser, so for Claude and Gemini this page can only estimate, using the same per-script estimates as our Token Counter: Anthropic says a Claude token is about 3.5 English characters on models before Claude 4.7 and that 4.7 and later produce about 30% more tokens for the same text, and the Gemini estimate is fitted to Google's own token counter. The note under the results gives those estimates for your whole text, marked as such. If you are chunking for one of those models, leave a margin below its limit.
Disclosure: RECATOOLS is written with the assistance of Claude, made by Anthropic, whose models are mentioned on this page. Readers should weigh that interest.
Frequently Asked Questions
- It's splitting a document into smaller passages so each can be embedded into a vector and retrieved independently. Retrieval-augmented generation depends on it: well-sized chunks make search precise, while a whole document embedded as one vector retrieves poorly.
- There is no universal best, so test retrieval on your own questions. For reference, OpenAI's vector stores default to 800-token chunks with 400 tokens of overlap, and LlamaIndex's sentence splitter to 1,024 with 200; both are one click away in the presets. Smaller chunks retrieve more precisely; larger ones keep more context.
- Input limits are set in tokens, and characters per token vary several-fold between languages. Chunking by the model's own tokenizer means each chunk fits its budget exactly; character or word counts only approximate it. This is exact only when the encoding you pick is the one your model uses.
- Overlap repeats a slice of tokens between adjacent chunks so context that straddles a boundary isn't lost. A sentence split across two chunks will still appear complete in at least one of them, which improves retrieval at the cost of embedding a little text twice.
- OpenAI's tiktoken, with the encoding you pick. By tiktoken's own model table, cl100k_base is used by OpenAI's embedding models (text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002) and GPT-4, and o200k_base by GPT-4o, GPT-4.1 and GPT-5. Other providers' models use their own tokenizers and can count the same text quite differently, especially outside English; the page shows Claude and Gemini estimates for your text so you can see the gap.
- Yes. Click "Copy chunks as JSON" to get a JSON array of strings you can paste straight into an embedding script or vector-store loader. For long documents only the first 60 chunks are shown on the page, but the copy always contains every chunk.
- Yes. Overlapping tokens are embedded in more than one chunk, so you pay for more tokens than the document has — the "tokens embedded, with overlap" figure is the number to use. At OpenAI's 800 / 400 default, almost every token is embedded twice. Put that figure into our Embedding Cost Calculator to price it.
- It can help. This tool uses fixed-size token windows, which is the standard and most predictable approach. Sentence- or paragraph-aware splitting avoids mid-sentence cuts but produces uneven chunk sizes; many pipelines combine both. Fixed-size with overlap is a strong, reliable default.
- No. The tokenizer and chunking run entirely in your browser from a locally-served library. Your text is never uploaded; the only network request is your browser fetching the tokenizer file from our own domain.
Method & sources
How it computes
Text is tokenized with OpenAI tiktoken (cl100k_base, used by OpenAI embedding models, or o200k_base). Chunks are a sliding window of the chosen size advancing by size minus overlap, with every cut moved back to the nearest token that starts a UTF-8 character, so no character is split and no chunk exceeds the size. Claude and Gemini figures are labelled estimates (TextChunkerKernel, TokenEstimateKernel).
What this tool implements
- Counts are exact for the encoding chosen: cl100k_base, which tiktoken's model.py maps to text-embedding-3-small, text-embedding-3-large and text-embedding-ada-002 (and GPT-4, GPT-3.5); or o200k_base, mapped to GPT-4o, GPT-4.1 and GPT-5. Other providers' models are estimated with the Token Counter's per-script rates and marked as estimates.
- Character safety: a cut may fall only before a token whose first byte is not a UTF-8 continuation byte (10xxxxxx, RFC 3629). A chunk end moves back and a chunk start moves back, so a chunk never exceeds the size and the overlap is never less than asked; a single character larger than the size is kept whole in an over-size chunk that the page reports. Tested on the vendored tokenizer: every chunk of mixed Chinese, Japanese, Korean, Thai, emoji and astral-plane text decodes without U+FFFD and the chunks rebuild the input exactly.
- Presets, with their sources: 800 / 400 is OpenAI's vector-store default (the auto strategy "currently uses a max_chunk_size_tokens of 800 and chunk_overlap_tokens of 400", openai-openapi spec); 1024 / 200 is LlamaIndex's SentenceSplitter default (DEFAULT_CHUNK_SIZE = 1024, SENTENCE_CHUNK_OVERLAP = 200). 512 / 64 is this tool's own starting point, not a cited norm.
- "Tokens embedded, with overlap" is the sum of chunk lengths — what an embedding call is billed on; at 800 / 400 that is about twice the document.
- Corrected 2026-11-03: chunks cut mid-character (U+FFFD in output) → cuts only at character starts; o200k_base only → cl100k_base default for embeddings, o200k_base selectable; "<|endoftext|>" in the text crashed the tool → encoded as text; "Tokens / chunk" showed the setting → largest chunk and tokens embedded; copy disabled past 60 chunks → copies all; unsourced "512 with 10-15% overlap is common" and "o200k is a solid basis across the board" → sourced presets and labelled estimates.
Sources
- OpenAI. tiktoken — model-to-encoding map (tiktoken/model.py). https://github.com/openai/tiktoken/blob/main/tiktoken/model.py
- OpenAI. OpenAPI specification — AutoChunkingStrategyRequestParam and StaticChunkingStrategy (max_chunk_size_tokens 800, chunk_overlap_tokens 400). https://raw.githubusercontent.com/openai/openai-openapi/master/op…
- LlamaIndex. llama_index core constants.py (DEFAULT_CHUNK_SIZE = 1024) and node_parser/text/sentence.py (SENTENCE_CHUNK_OVERLAP = 200). https://github.com/run-llama/llama_index/blob/main/llama-index-co…
- Yergeau F. RFC 3629: UTF-8, a transformation format of ISO 10646. 2003. https://www.rfc-editor.org/rfc/rfc3629.txt
- Anthropic. Glossary — tokens (about 3.5 English characters per token; Claude 4.7 and later about 30% more), the basis of the Claude estimate. https://platform.claude.com/docs/en/about-claude/glossary
What can make this go out of date
- None at runtime beyond the vendored tokenizer files served from this site; nothing the reader pastes leaves the browser.
Pick up where you left off
Stored only in this browser — never sent to our servers.