Mixture of Experts (MoE)

A mixture-of-experts (MoE) transformer replaces the dense feed-forward (FFN) layer in each transformer block with a sparse mixture: FFN “experts” plus a small router that, for each token, picks of them to actually compute.

The total parameter count is × larger; the active parameters per token are only × the size of one expert. With , , you get a model with the parameter capacity of an 8×-bigger dense model, at the compute of a 2×-bigger dense model. This is the entire economic value of the architecture.

How it works mechanically

Each transformer block normally has:

1. Attention.
2. Dense FFN (a 2-layer MLP, ~4× embed dim hidden).

MoE replaces step 2 with:

1. Router. A small linear layer . Computes scores over the experts.
2. Top- gating. Pick the highest-scoring experts. Send the token through only those.
3. Weighted sum. Combine the expert outputs weighted by their gate scores.

Each expert is just a normal FFN — same shape as the dense one would be. The router is tiny (~ parameters).

Why scaling helps

Empirical scaling laws say more parameters means better quality, holding compute fixed. MoE decouples parameters from compute: at the same FLOPs/token, a sparse model has many more parameters to encode knowledge in. Mixtral 8×7B (Mistral, 2023) has 47B total parameters but ~13B active per token, and substantially outperforms dense 13B models. DeepSeek-V3 takes this further: 671B total, 37B active.

The serving problem

MoE is great in training, hard in production. Three issues:

• VRAM. All experts must be loaded even if only are computed per token. A 47B-total model needs 94GB at fp16 — same as a dense 47B. The compute savings don’t extend to memory.
• Routing variance. Different tokens go to different experts; batched inference over many queries hits different expert patterns, breaking the contiguous-matrix-multiply pattern that makes dense transformers fast on GPUs.
• Load imbalance. If the router sends 90% of tokens to expert 0, expert 0 is the bottleneck and the others are idle. Auxiliary load-balancing losses during training mitigate this.

The serving stacks (vLLM, SGLang) have specialized MoE kernels by 2025 — fused gating + grouped GEMM — but throughput per active parameter is still worse than dense, even when total throughput is competitive. See continuous batching for how production schedulers cope.

Why MoE matters strategically

The frontier-LLM scaling shape since ~2024: MoE for parameter count, dense for active compute. GPT-4 is reportedly MoE; Claude is reportedly dense; the open-weight frontier (DeepSeek, Mixtral, Qwen MoE variants) is solidly MoE. The bet is that knowledge density per FLOP is the binding constraint, and sparse routing is the architecture that solves it.

For specialized small models — rerankers , embedders , classifiers — MoE generally isn’t the right call. The whole point of specialization is that the model doesn’t need broad knowledge; a dense FFN is fine. MoE is a frontier-LLM technique.