DeutschAI-Designers
One million tokens in a single prompt. Entire contract bundles, whole codebases, a year of support tickets – all in context at once. The new long-context models promise to solve an old problem: the AI no longer has to look things up piece by piece, it simply has everything in view. Sounds like the end of retrieval pipelines. It isn’t. Look at the technology soberly and the picture becomes clear: large context windows solve some problems elegantly and create new ones in return.
What a million-token context actually means
A token is roughly half a word to a whole word. A million tokens, then, add up to around 700,000 words – about seven fat novels, or a few hundred business documents. Models like the current Qwen, Llama and GLM generations now reach windows of that size as open weights you can host yourself. You no longer depend on the big commercial APIs when you want to process very long inputs.
The appeal is obvious. Instead of chopping a document into fragments, indexing them and fishing out the right pieces for every query, you simply hand over the whole thing. The model sees connections that chunk-based search tears apart: the reference in clause 14 to a definition in clause 2, the table on page 80 that leans on an assumption from page 3. These cross-references are exactly where classic retrieval keeps falling short.
Where long contexts shine in daily work
Take the legal team. A lawyer uploads a 120-page framework agreement with four appendices and asks: „Which clauses contradict our standard liability terms?“ A long-context model can review the document as a whole, including the interplay between the main contract and the annexes. A retrieval pipeline would find individual clauses but often miss the overall picture.
Software development is similar. A developer submits a complete module with twenty files and asks for a refactoring. The model needs the full context so it doesn’t break dependencies. Or in support: you feed the model a customer’s entire conversation history over two years and let it trace the real cause of a recurring problem. These are tasks where the answer depends on the complete picture, not on three well-chosen snippets.
Financial analysis benefits too. Present an annual report, the prior-year figures and the transcript of the last investor call together, and you get comparisons and context that fragments can barely produce.
The catch: cost and latency grow with it
This is where it gets uncomfortable. The compute for a transformer’s attention mechanism grows quadratically with input length. Double the context and, in the worst case, the compute for that step quadruples. In practice that means a 500,000-token prompt doesn’t cost you half as much, it costs a multiple compared with a lean query – in GPU time, in memory, in waiting.
The so-called KV cache, where the model stores its intermediate results for every token, devours serious amounts of VRAM on long inputs. On a single GPU you hit limits fast. And the first answer keeps you waiting, because the model has to „read“ the entire million tokens before it produces a single word. For an interactive application where users expect quick responses, that’s a real problem.
Lost in the middle: long doesn’t mean thorough
A large context window is no guarantee the model actually uses everything inside it. Research and practice point to the same effect: information at the start and end of a long input gets processed reliably, while the material in the middle is readily skipped. „Lost in the middle“ is the name for it. The longer the input, the more pronounced it gets.
For you this means: just because a detail sits somewhere in those 300 pages doesn’t mean the answer will account for it. A model that can process a million tokens often works more precisely with 50,000 well-curated ones than with a million raw. More context is not automatically better context. That insight saves you expensive disappointments in production.
Long context or RAG? Usually both
The debate often gets framed as either-or. In practice the two approaches complement each other. Retrieval-augmented generation shines when your knowledge base is huge, changes constantly and you need to trace where an answer came from. You search millions of documents, pull the relevant passages and place only those in the context window. That stays cheap and brings the evidence along.
Long contexts play to their strength where a bounded but sizeable set of documents has to be understood as a whole – the one contract, the one project, the one case. Many teams therefore combine the two: a retrieval step narrows the overall corpus down to the relevant fifty documents, and only that selection goes, in full, into a long-context model that evaluates it in context. That way you get the precision of the large window without paying for an indiscriminately stuffed prompt.
What this means for self-hosting
If you host long-context models yourself, plan around memory. Techniques like prefix caching help you compute recurring inputs – a fixed system prompt, say, or a reference document shared by many queries – only once and reuse the result. That cuts cost and latency noticeably as soon as inputs overlap.
Set realistic limits. Not every query needs the full window. Cap the input length per use case, measure the length at which answer quality tips over, and route short queries to a smaller, faster model. Also check whether your model truly commands the stated context length – many advertise a number they reach through after-the-fact scaling without using it cleanly across the whole span. One test with your real documents tells you more than any datasheet.
The bottom line: a tool, not a cure-all
Long context windows are a genuine win for tasks that demand the full picture – contracts, codebases, sprawling cases. They don’t replace retrieval, they complement it. Keep an eye on cost, budget for the lost-in-the-middle effect, and curate inputs deliberately instead of burying the model, and you’ll get the most out of it. The more interesting question is rarely „How much fits in?“ but „What belongs in at all?“.
Want to find out whether long-context models, a retrieval pipeline or a combination of both fits your documents and your budget? At AI-Designers we design self-hosted AI solutions tailored to your data and your infrastructure – get in touch.
Images: AI-Designed