Model architectures
Why do large models use a mixture-of-experts design?
Mixture-of-experts is a sparse architecture: a router sends each token to a few of many expert sub-networks instead of running them all. This decouples a model's total capacity from its per-token compute, so a very large model can answer at the cost of a much smaller one. The trade is more memory to hold and harder training.
A mixture-of-experts (MoE) model takes the feed-forward part of a transformer block and splits it into many smaller copies called experts. A small router then looks at each token and picks just a few experts to run, leaving the rest idle. The sizes page covers how to read the result in a name like 235B-A22B, where the first number is the total stored and the second is the roughly active count per token. This page is about why you would build a model this way in the first place.
The design exists to break a link that dense models can’t escape. In a dense transformer, every parameter runs for every token, so capacity and compute rise together: a bigger model is always a more expensive model. MoE separates the two. The model can hold the full capacity of a very large network, since different experts specialize in different things, while each token only pays for the handful that fire. The Switch Transformer made this practical by routing each token to a single expert, which cut routing cost and let the authors train models up to a trillion parameters with large gains in pre-training speed over a dense baseline of the same compute.[1] Later models such as Qwen3-235B-A22B route to several experts, 8 of 128 in that case, for the same reason.[2]
The trade shows up in two places. You still have to load every expert into memory, so an MoE model’s memory footprint matches its full total, not its active slice. Routing adds its own problems: the router can send too much traffic to a few popular experts, so training needs load-balancing pressure to keep experts busy, and that balance can make MoE models more delicate to train and fine-tune than a plain dense model. You are buying compute efficiency, and paying for it in memory and in complexity.
The router is doing more work than the simple description suggests. It has to make a fresh routing decision for every single token, which means two tokens next to each other in the same sentence can fire completely different experts. Nothing about MoE guarantees related tokens share a path through the network — the routing is learned from what balances training, not designed by hand around any notion of what “should” go together.
References
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (arXiv:2101.03961) — Fedus, Zoph & Shazeer, Google
- Qwen3-235B-A22B Model Card — Qwen (Hugging Face)
Common questions
- How is this different from the sizes-and-parameters page on MoE?
- That page explains how to read an MoE name like 235B-A22B and what total versus active parameters mean. This page is about MoE as a design choice: why you would build a model this way, how routing works, and what it costs you in return.
- Does an MoE model use less memory than a dense model?
- No. It uses less compute per token, not less memory. You still have to load every expert into memory even though only a few run per token, so an MoE model's memory footprint matches its full total parameter count.