DeutschAI-Designers
A language model never reads your whole document. It reads bite-sized pieces. These pieces are called chunks, and how you cut a document into them decides whether your AI search finds the right paragraph or half a sentence from the wrong chapter.
Many teams spend weeks choosing the embedding model and the vector database. Splitting the documents they handle on the side, with three lines of code. Yet this is often exactly where a RAG application starts giving weak answers. This article shows which methods exist, where their limits lie, and how to find the right one for your data.
Why the chunk is the unit that matters
When you build a RAG pipeline, you break every document into smaller sections. Each section is embedded on its own and stored in the vector database. When a user asks a question, the system retrieves the best-matching chunks and hands them to the language model as context. So the model answers based solely on what sits inside those pieces.
That leads to an uncomfortable truth. Cut a thought in the middle of a sentence, and the model only sees half of it. Pack five topics into a single chunk, and the meaning blurs so the search grabs at it vaguely. The chunk is the smallest unit your application ever sees – which makes it the lever with the most riding on it.
The good news: you can gain a lot here without swapping out a single model. Better chunking costs no extra GPU budget. It only takes an understanding of how your documents are built.
The naive route and what it costs
Most libraries suggest the same default: fixed length. They cut the text every 500 or 1,000 characters, no matter what sits there. It is quick to wire up and works surprisingly often on smooth prose. The catch shows the moment your documents have structure.
A cut by fixed character count separates a table from its heading. It tears an enumerated list apart. It slices a contract clause down the middle, so the condition lands in one chunk and the legal consequence in the next. The search then finds a fragment stripped of its decisive context, and the model pieces together the rest by guessing.
It gets especially treacherous with technical documents, invoices, or forms. There, the position of a number carries its entire meaning. Separate it from its column header, and it is worthless. Fixed length is therefore a fine starting point for a prototype, but rarely the answer for production.
Follow the structure of the document
The next step is obvious: split where the document itself draws boundaries. A recursive splitter first tries to break at paragraphs. If that is not enough, it drops down to sentences, then to individual words. What belongs together stays together, and a chunk ends at a paragraph break instead of mid-word.
You get even more out of it when you read the document’s markup. Markdown knows headings, HTML knows sections, PDFs often carry a recognizable outline. Cut along this hierarchy, and you get chunks that each map to one topic. A chapter, a subsection, a clause – each as its own unit.
The effort pays off where your documents are cleanly structured: manuals, policies, knowledge bases. The clearer the source is laid out, the better the splitter follows it. On chaotically scanned legacy files the method hits a wall, simply because the structure is not there.
Split by meaning, not by form
Sometimes the form tells you nothing about the content. A transcript from a meeting runs as a wall of text with no headings. Semantic chunking helps here. The method embeds the text sentence by sentence and measures how closely two neighboring sentences resemble each other in meaning. When that similarity drops, there is probably a topic shift – and that is exactly where the cut goes.
The result is chunks that hold together in content, even when the document offers no outline on the surface. When the conversation moves from budget to schedule, the method spots the transition and splits there. Each section then covers exactly one topic.
The price is compute time. Semantic chunking embeds the text already while splitting, not only when indexing. On large collections that can weigh in. For valuable, unstructured sources the extra effort still often pays off, because retrieval quality rises noticeably.
Overlap, metadata, and the parent context
Whichever method you pick, three tricks improve almost any result. They cost little and prevent the typical errors at the edges of chunks.
- Overlap: let neighboring chunks share a few sentences. That way a thought sitting right on the cut line is lost to neither side.
- Metadata: attach source, chapter, date, and document type to every chunk. With that you pre-filter the search and can name the source to the user at the end.
- Parent context: search with small, precise chunks, but give the model the larger section around them. Small units hit more accurately, the wider context makes the answer complete.
That last point, often called parent-document retrieval, resolves a real conflict. Small chunks find the right spot but give the model too little surrounding. Large chunks provide context but blur the search. Separate the two: search fine, answer broad.
Late chunking: embed first, then split
A newer approach flips the usual order. Instead of cutting the text first and embedding each snippet in isolation, late chunking sends the whole document through the embedding model. Only then does it divide the resulting vectors into chunks. Each section carries the information of the entire document within it.
The benefit shows with references that span sentence boundaries. If paragraph three only says the pronoun “the company” and the name appeared in paragraph one, a classically isolated chunk knows nothing of that name. With late chunking the link is already anchored in the vector, because the model saw the whole text at once.
The prerequisite is an embedding model with a long enough context window. Modern, freely available models bring that along. The method is young, but it fits well with a self-hosted setup, where you hold full control over the pipeline anyway.
How to find the right strategy
There is no single correct chunking method. What fits depends on your documents and on the kind of questions. A manual with a clear outline calls for a different approach than a stack of informal minutes. So do not start with the technique, but with a look at your data.
After that, only measuring helps. Put together a small set of real questions along with their expected source passages. Test two or three chunking variants against each other and check which brings the right paragraphs to the top. This one day of effort saves you weeks later of puzzling over why the answers are off.
Treat chunking as what it is: a decision of its own with large impact, not a footnote in the setup. Those who work cleanly here pull markedly better answers out of the same model and the same database – without a cent more for hardware.
Looking to build a RAG solution tailored to your documents and your infrastructure? We help companies design and run self-hosted AI systems – from document preparation to production search. Talk to us at ai-designers.eu.
Images: AI-Designed