Mixture of Experts: More AI Power at Lower Compute Cost

The most capable open model you can run on your own hardware today carries more than a trillion parameters. Yet for every word it writes, it computes with only a small fraction of them. That apparent contradiction has a name: Mixture of Experts, or MoE. It explains why inference costs for large models have dropped so sharply over the past two years — and why you still need to be careful when you self-host.

Anyone running a flagship model on their own servers in 2026 will run into this architecture. DeepSeek, Kimi, Qwen, Mistral, the open models from OpenAI — they all rely on MoE. Understanding what sits underneath pays off, because the architecture shifts the cost to a place many teams overlook at first.

What Mixture of Experts actually means

A classic language model pushes every token through all of its weights. If the model has 70 billion parameters, it computes with all 70 billion for every single word. That is thorough, but expensive. Each percentage point of extra quality costs you compute time in a straight line.

MoE breaks that rule. Instead of one large network, the model holds many smaller sub-networks — the experts. A router decides, for each token, which two or four experts handle it. The rest stay silent. DeepSeek-V3, for instance, carries 671 billion parameters but activates only about 37 billion per token — just over five percent. Kimi K2.6 reaches a trillion parameters and computes with roughly 32 billion per pass.

The effect: you get the knowledge of a huge model but pay only for a mid-sized one in terms of compute. The model specializes internally, and you avoid the full price on every word.

Why this cuts your inference costs

GPU compute time is the line item that drives your operating costs. Fewer active parameters mean fewer floating-point operations per token, shorter response times, and more requests served in parallel on a single card. This is exactly where MoE pays off.

Take an everyday example. A support assistant answers customer queries around the clock. With a dense 70-billion model, every request pins your GPUs. An MoE model of comparable answer quality computes with far fewer weights per token — so you serve more users at once on the same hardware. Keep the server fleet fixed and throughput rises, or you get by with fewer cards.

Training shows the advantage too, though that matters less for most companies. What counts for you in operation is this: the running cost per answered request falls without quality suffering. That is why prices for top open models have come down.

The catch: MoE eats memory, not compute

Here lies the trap many teams walk into the first time. MoE saves compute — but not memory. The router picks only a few experts per token, yet it must be able to reach all of them at any moment. So every expert has to sit in GPU memory simultaneously.

In concrete terms: a model with a trillion parameters needs the memory for a trillion parameters, even when it uses only three percent of them per token. DeepSeek-R1 takes up around 800 gigabytes of GPU memory in FP8 quantization. Running that locally calls for a server with at least eight H200 cards. The compute load is moderate; the memory requirement is enormous.

For the large providers’ API operations this is no problem, because the memory cost spreads across thousands of concurrent requests. When you self-host at modest throughput, the math flips: you pay for expensive high-end memory you never fully use. Check this point before you commit to a trillion-parameter model.

Which MoE model fits which hardware

The good news: you do not have to reach straight for Kimi K3 with its 2.78 trillion parameters. The field runs from giants down to models that fit on two cards. DeepSeek V4-Flash and Mistral Small 4 bring near-frontier quality to two-GPU systems. For many domain applications, that is plenty.

Ask the question the other way around. Not: which is the strongest model? But: what answer quality does my use case actually need, and how much memory do I have? An MoE model that just fits on your cards will serve you better than a larger one that constantly offloads and turns sluggish.

Work out the memory requirement in advance. Multiply the total parameter count by the byte size of your quantization — an FP8 weight takes one byte, a four-bit weight half a byte. Add headroom for the context cache. Only when that sum fits comfortably into your memory does the model make sense for continuous operation.

Quantization as a lever

Because memory is the bottleneck with MoE, quantization becomes the most important lever. Storing each weight at lower precision — eight or four bits instead of sixteen — halves or quarters the memory footprint. Those 800 gigabytes in FP8 shrink to roughly 400 in four-bit quantization.

With modern methods the quality loss usually stays small, as long as you do not compress too aggressively. Four bits is a sensible compromise for many enterprise tasks. But test with your own data rather than trusting generic benchmarks. A model that shines in English can noticeably weaken on German technical text after heavy quantization.

Combine quantization with the MoE property and the picture turns attractive: a trillion-parameter model, compressed to four bits, suddenly runs on a single well-equipped server instead of a full rack. For mid-market budgets, that is the difference between feasible and unaffordable.

What this means for your AI strategy

MoE is not a passing fad but the standard architecture for open top-tier models. Anyone planning an AI infrastructure today should grasp the logic: quality comes from the breadth of many experts, speed from sparse activation, and the cost sits in memory.

Translate that into your procurement. With MoE, do not invest in the fastest compute units but in cards with plenty of memory and high memory bandwidth. A system built for raw compute quickly stalls on MoE, because it cannot even load the models in the first place.

And plan throughput realistically. MoE plays its strength where many requests run at once. Run a huge model for a handful of daily queries and you throw money away. In that case a smaller dense model, or a mid-sized MoE, is the smarter choice.

Choosing the right architecture, the matching hardware, and a clean quantization is rarely obvious — and costly when you get it wrong. If you want to bring your AI infrastructure onto your own servers and align model, memory, and cost cleanly, the team at AI-Designers supports you from selection through to production.

Images: AI-Designed