Model architectures
When do you use encoder-only, decoder-only, or hybrid models?
Three transformer shapes suit different jobs: encoder-only models such as BERT read text bidirectionally for understanding, decoder-only models generate text one token at a time and power today's chatbots, and encoder-decoder models such as T5 turn one sequence into another. Hybrid designs combine attention with state-space layers to balance recall against speed.
The transformer comes in three shapes, and the difference is what each half is for. An encoder reads a whole input at once, letting every token see every other token in both directions, which is good for building a rich understanding of text that already exists. A decoder generates, with each position seeing only the tokens before it so it can predict the next one. Put both together and you have an encoder-decoder, which reads one sequence and writes a different one.
Each shape found its job. BERT is encoder-only: it reads text bidirectionally and was built to be fine-tuned for understanding tasks like classification and question answering, not to write fluent paragraphs.[1] T5 is encoder-decoder, and it reframed every task as turning one piece of text into another, “text-to-text,” which fits translation, summarization, and similar mapping problems where there is a clear input and a separate output.[2] Decoder-only models, the GPT line and the chat models that followed, won the generative side: when the task is to continue a conversation token by token, you only need the decoder, and that simplicity scaled well. This is why almost every chatbot today is decoder-only.
The newest category is hybrid. As state-space models like Mamba showed they could carry long sequences cheaply, designers stopped treating attention and state-space as a choice. Jamba interleaves Mamba layers with a smaller number of attention layers, and folds in mixture-of-experts on top, to get long-context efficiency without losing attention’s exact recall.[3] The pattern is a useful summary of this whole topic: architectures are not a ranking but a set of trades, and the strongest recent models increasingly combine them rather than commit to one.
Decoder-only won the popularity contest because chat is the dominant product shape, not because it is the better design in the abstract. Encoder-only and encoder-decoder models never went away; they are still the sharper tool whenever the job has a clear, separate input and output rather than an open-ended conversation, which is most of what BERT-style classifiers and translation systems do in production today.
References
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805) — Devlin et al., Google
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) (arXiv:1910.10683) — Raffel et al., Google
- Jamba: A Hybrid Transformer-Mamba Language Model (arXiv:2403.19887) — Lieber et al., AI21 Labs
Common questions
- Why are chatbots decoder-only rather than encoder-decoder?
- Chat is next-token generation, which is exactly what a decoder does: each position attends only to earlier tokens and predicts the next one. The encoder half mainly helps when you have a distinct input to read and a separate output to produce, as in translation.
- What makes a model a hybrid?
- A hybrid mixes more than one mechanism in the same model. The common case today is interleaving attention layers with state-space layers, so the model keeps attention's exact recall while using cheaper state-space layers for most of the long-range work.