DeutschAI-Designers
One GPU, two completely different jobs
Anyone running a language model in-house knows the puzzle: the expensive graphics cards sit at 100 percent utilisation, and users still wait for their answers. The reason lives deep in how the models work. A request moves through two phases that could hardly be more opposite. First the model reads the entire prompt and builds what engineers call the KV cache — the prefill phase. Then it produces the reply one token at a time — the decode phase.
Prefill is compute-heavy. The GPU grinds through large matrices and pushes its arithmetic units to the limit. Decode is the opposite: each step yields a single token, yet the model must pull the whole cache out of memory every time. The arithmetic units sit idle while memory bandwidth becomes the bottleneck. Force both phases onto the same card, and one long prompt blocks everyone else’s answers — and the reverse happens too.
That coexistence costs companies real money. They buy hardware for peak load and never use it efficiently. The fix is disaggregation: split prefill and decode, then spread them across specialised machines.
What llm-d 1.0 changes
On 20 October 2026, llm-d 1.0 arrived — an open-source stack for distributed inference. IBM, Red Hat and more than 500 contributors from over 25 organisations stand behind the project. This is no longer an academic experiment; it is cleared for production. For context-heavy, agentic workloads IBM cites cost reductions of up to five to tenfold and triple the output throughput.
llm-d builds on vLLM and uses its native ability to pull prefill and decode apart. The actual handoff of the KV cache between machines runs over NIXL, a library for fast memory transfer. On top sits a router with a scheduler that decides, for each request, which machine does the prefill work and which generates the answer.
The clever part hides in the orchestration. Pods — individual compute units on Kubernetes — carry the role prefill or decode as a label. A proxy sidecar sits in front of every engine. When a request comes in, the sidecar first sends it to a prefill pod, catches the transfer parameters of the finished cache, and passes them to a decode pod. To your application the whole dance stays invisible — it still talks to a single endpoint.
KV-cache-aware routing: the real lever
The biggest gain comes not from the split alone but from smart routing. Many requests share text: the same system prompt, the same document template, the same conversation history. Compute that part once and it already exists as a cache. A naive router hands out requests in turn and throws that advantage away. llm-d checks where matching cache fragments already live and sends the request there.
Work it through for a support chatbot. Thousands of conversations open with identical instructions and product details. Route each follow-up to the machine that already holds the conversation context, and the expensive recompute disappears. Prefill load drops noticeably, and users see the first word of their answer sooner.
The effect scales with the length of the shared prefixes. The larger your system prompts and the longer your document contexts, the more you save. For RAG applications that stuff heavy context into every request, this is the difference between affordable and ruinous running costs.
When the split pays off — and when it doesn’t
Disaggregation is no cure-all. It adds a network hop: the cache has to travel from the prefill machine to the decode machine. With short prompts and light load, that overhead outweighs the benefit. A small model that runs smoothly on a single card does not need the mechanism.
The approach earns its keep under high throughput and uneven load profiles. Picture a platform that analyses long documents in the morning and fields many short questions in the afternoon. With separate pools you scale prefill capacity and decode capacity independently. When parallel conversations surge, you add decode pods without buying expensive prefill hardware alongside.
Hardware choice gets sharper too. Prefill wants raw compute; decode wants memory bandwidth. You can put older or cheaper cards on one role and your top-tier hardware where it truly counts. That reshapes procurement — away from the blanket “another big GPU” towards a targeted investment.
Getting it into production
llm-d runs on Kubernetes and slots into the usual gateway infrastructure. If you already serve models with vLLM, the ground is familiar: same engine, same model weights, just wired differently. Start small — one model and a modest number of pods — before you fine-tune the roles.
Measure before and after. The two numbers that matter are time-to-first-token and tokens per second. The first tells your users how quickly an answer starts; the second, how smoothly it keeps flowing. Disaggregation improves both — but only when your load profile fits. Skip the measurement and you are working blind.
Watch the network between the pods. The KV transfer needs bandwidth; over slow links, the transfer eats the gain right back. In practice that means planning fast connections between nodes and not scattering the pods across the whole data centre.
Sovereignty as a side effect
That the stack is open source and permissively licensed feeds into a concern many European companies share: control over their own AI. You run models in your own data centre, decide for yourself over data and hardware, and tie yourself to no single cloud provider. Distributed inference makes exactly that in-house operation economically viable.
Because the most common objection to self-hosted AI is that it costs too much next to an API. That objection loses weight once you use your GPUs twice as well and serve far more requests on the same hardware. “Too expensive” turns into “cheaper once you do the maths” — provided you run enough volume to justify the complexity.
Distributed inference is therefore less a toy for hyperscalers than a practical tool for mid-sized operators with serious throughput. A technique that a year ago seemed reserved for large labs now sits ready as a finished building block.
Wondering whether running a language model in-house adds up for your business, and want a realistic read on architecture, hardware and cost? Talk to us at ai-designers.eu — we will plan a self-hosted AI with you that fits your load profile and your budget.
Images: AI-Designed