Speculative Decoding: Make Self-Hosted AI Noticeably Faster

Why your AI server spends most of its time waiting

A language model writes its answer one word at a time. More precisely, one token at a time. For every single token the graphics card loads the full model weights from memory, runs one step, appends the result, and starts over. On a model with 70 billion parameters that means dozens of gigabytes travel across the memory bus per token. The GPU’s compute units, meanwhile, sit idle. They wait for data.

This is the real bottleneck of self-hosted AI. Compute power isn’t the brake; memory bandwidth is. Your expensive H100 or your RTX 6000 cluster often runs at a fraction of its capacity during text generation. Anyone serving many users at once or producing long answers notices it immediately: replies trickle out, latency climbs, and sooner or later you buy the next card.

Speculative decoding turns this problem around. It fills the GPU’s idle time with useful work and returns the exact same text your model would have written anyway. No loss of quality, no compromise on output. Just faster.

A small model guesses, the large one checks

The idea is surprisingly simple. Instead of letting the large model produce each token on its own, a small, fast draft model steps in. This draft model proposes several tokens in one go, say four or five. Because it is small, that costs almost no time.

Then the large model takes over. It checks all the proposed tokens in a single pass, in parallel rather than one after another. If a proposal fits, it accepts it and skips the individual steps. If the large model disagrees at some point, it discards the rest from there and continues on its own. So the GPU can process several tokens for the price of one whenever the draft and target models agree.

Here’s the crucial part: the method is lossless. The large model has the final say on every token. The draft only speeds up the cases where the answer was obvious anyway, and in ordinary prose, code, or structured output there are plenty of those. Practical reports cite speedups roughly in the range of one-and-a-half to three times, depending on the task.

Three variants, three trade-offs

A draft model doesn’t have to be a separate neural network. In practice three approaches have taken hold, and each fits a different situation.

The classic route uses a small, standalone draft model from the same family, say a 1-billion model feeding a 70-billion target. It works broadly but costs a bit of GPU memory for the second model. The second variant drops the model entirely: n-gram matching simply searches already-generated text for repetition. That costs next to nothing and shines on repetitive tasks, for instance when a model quotes from a long document or rewrites code.

The most advanced option is the EAGLE approach, which adds a lightweight prediction head to the target model itself. It makes especially good guesses because it learns directly from the large model’s internal states. In return, you have to train it once. The “speculators” library from the vLLM project now bundles several of these methods under one roof, so you don’t have to wire up each variant by hand.

What it means for your budget

Let’s do the math without inventing numbers. If the same graphics card answers twice as fast, you serve twice as many requests on the same hardware. Or you cut your users’ waiting time in half. Both feed straight into operating costs, especially when you run a model around the clock for internal chatbots, document analysis, or developer tools.

For many mid-sized companies this shifts the line at which owning hardware pays off. If you were just short of a card’s capacity, speculative decoding may let you avoid a new purchase altogether. And if you are scaling anyway, you simply need fewer GPUs for the same load. The technique demands no new hardware, no cloud contract, and no vendor switch. It gets more out of what already sits in the rack.

Where it excels, and where it stumbles

Speculative decoding works best where individual answers need to finish quickly and the GPU isn’t already fully loaded. Interactive assistants are the prime case: a developer waiting on a code completion feels every half second. Structured output like JSON, or answers that stick closely to a template, also tend to match the draft model’s guesses well, so the speedup is correspondingly large.

There is a flip side, though. If you process very many requests at once and your GPU already runs at the limit, the technique brings less. In that case the idle time it usually exploits simply isn’t there. A poorly matched draft model can even slow things down, because the target model keeps discarding its bad guesses. The key metric is the acceptance rate: it measures how often the large model takes the draft’s proposals. When it’s high, you gain a lot. When it’s low, the effort barely pays.

How to get started in practice

The barrier to entry is low if you already run an inference server like vLLM. There you enable speculative decoding through a handful of configuration parameters and point it at a draft model or the n-gram method. Test first with your own typical requests, not synthetic examples, because the acceptance rate depends heavily on how predictable your text is.

Measure before and after. Record tokens per second, latency, and acceptance rate for a realistic load profile. Only these numbers reveal which variant suits your use case. Start with the n-gram approach, since it needs no extra model and takes minutes to set up. If it delivers too little, move on to a draft model or EAGLE.

What matters stays the same: you change only the speed, not your model’s behavior. Answers, safety rules, and fine-tuning remain untouched. That makes the technique suitable even for regulated environments where every output has to be reproducible.

The takeaway: more speed from the hardware you own

Speculative decoding is one of those rare optimizations without a real catch. It costs no answer quality, demands no new graphics card, and lets you test it step by step. For any company running its own AI models and depending on fast answers or low costs, it belongs on the checklist.

Are you planning a self-hosted AI solution or looking to make your existing inference more efficient? Talk to us. We guide you from architecture to production. Learn more at www.ai-designers.eu.

Images: AI-Designed