What is a dense transformer?

A dense transformer is the plain version of the design, and the one most people mean when they say “a transformer.” It is a stack of identical blocks. Each block has a self-attention step, where every token looks at the tokens before it, and a feed-forward step, which is a small network applied to each position. The word dense is the important part: every parameter in every block takes part in processing every single token. Nothing is routed away or left idle.^[1]

Modern chat models are decoder-only dense transformers. They use masked attention so each position sees only the words that came before it, which is exactly what next-token prediction needs.^[2] The appeal of this shape is its plainness. There is one path through the model, so training is stable, serving is predictable, and the memory and speed of the model are easy to reason about. If you know the parameter count and the input length, you have a good idea of what an answer will cost.

That plainness is also the limit. Because every parameter fires for every token, the compute per token rises in step with the model’s size. A dense model with twice the parameters costs roughly twice as much to run, with no way to dodge that bill. As frontier models pushed past the hundreds of billions of parameters, this got expensive enough that the largest of them moved to mixture-of-experts, which keeps the capacity of a big model while paying the compute of a much smaller one. The dense block is still the building unit inside those models, so it is worth understanding first.

Where a model sits on this trade also shapes how forgiving it is to run on your own hardware. A dense model’s cost is easy to predict from its parameter count alone, which is exactly the arithmetic the memory page walks through. An MoE model breaks that shortcut: you still have to hold every expert in memory even though most sit idle, so its hosting cost tracks its full size, not the smaller number the “active” figure in its name suggests.