Model architectures
What does a model's architecture decide?
Architecture is the structural blueprint a model is built on. It decides what the model can do, how fast it runs, and what it costs to run. The main families today are dense transformers, mixture-of-experts, state-space models, diffusion models, and the encoder, decoder, and hybrid shapes built from them.
A model’s architecture is its structural blueprint: how the parts are wired, what flows through them, and in what order. It is a separate question from how big the model is or what data it saw. Two models with the same parameter count can behave very differently because they are built on different blueprints, and a single blueprint choice tends to settle three things at once. One is the ceiling on what the model can learn: some shapes handle long inputs or parallel work better than others. Another is speed, since the math the design demands per token is what the hardware has to grind through. The last is cost, in both the memory needed to hold the model and the compute needed to run it.
Almost every large language model in wide use today descends from the transformer, introduced in 2017 in a design “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”[1] That paper is the common ancestor, but the family has branched. The standard dense transformer runs every parameter for every token. Mixture-of-experts keeps most of those parameters idle on any given token and routes the work to a few. State-space models such as Mamba drop attention’s quadratic cost for something closer to linear, which matters most on very long sequences.[2] Diffusion models build text or images by refining noise instead of writing left to right. And the older split between encoder, decoder, and encoder-decoder shapes still decides which jobs a model is suited to.
The pages in this topic walk through each of these families. None of them is simply the best. Each trades something away to gain something else, and knowing which trade a model made tells you a lot about how it will behave before you ever run it.
Reading the timeline as a strict handoff would be a mistake, though. Dense transformers did not retire when mixture-of-experts arrived; they are still the right choice below a certain scale, and the dense block is the unit both MoE and hybrid designs are built from. What actually happens release over release is accumulation: each new idea solves a cost the previous generation couldn’t, and gets folded in alongside the others rather than replacing them outright. Jamba, covered on the encoder-decoder and hybrids page, is the clearest example — attention, state-space, and mixture-of-experts, in one model.
References
- Attention Is All You Need (arXiv:1706.03762) — Vaswani et al., Google
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces (arXiv:2312.00752) — Gu & Dao
Common questions
- Is architecture the same as model size?
- No. Size is the parameter count; architecture is how those parameters are wired together. Two models of the same size can run at different speeds and handle different jobs because they use different blueprints.
- Are all modern language models transformers?
- Most are, but not all. The transformer is the common ancestor, and dense transformers and mixture-of-experts are both branches of it. State-space models like Mamba and diffusion-based text models are newer designs that step outside the standard transformer recipe.