What are the three families of transformer models?

The original transformer was an encoder-decoder built for translation. Later models specialized the design into three shapes:^[1]

• Encoder-only (such as BERT): bidirectional, tuned for understanding text.
• Decoder-only (such as GPT): each token sees only earlier tokens, tuned for generating text one token at a time.
• Encoder-decoder: the full original, for translation-style tasks.

Today’s generative chatbots are overwhelmingly decoder-only transformers, using masked self-attention so each position attends only to the words before it, which is what makes next-token prediction work.

The split comes down to which directions a token is allowed to look. An encoder-only model lets every token attend to every other token in both directions, good for building a representation of text that already exists in full. A decoder-only model applies a causal mask: attention scores for any later position are forced to negative infinity before the softmax, so after softmax they land at exactly zero. A token can only pull information from itself and what came before it, the property that makes next-token prediction well-defined at every position during training, not only at generation time (see one token at a time for how that plays out step by step).

Decoder-only did not win by default. BERT, an encoder-only model, needed a different fine-tuned head bolted on for every task: one for classification, another for question answering, another for named-entity recognition. GPT-2 and GPT-3 showed that a single decoder-only model, trained on nothing but next-token prediction and then simply prompted, could do translation, summarization, and question answering with no task-specific retraining at all. That flexibility, more than raw language-modeling accuracy, is most of why decoder-only architectures (GPT-4, Llama, Claude) now dominate general-purpose use, while encoder-only models (BERT, RoBERTa) stay common for classification and search, and encoder-decoder models (T5, BART) persist for translation-style tasks with a clear input-to-output shape.