What is a transformer block, and what is inside it?

A transformer is not one big thing; it is one block stacked many times. Each block has two main parts: a multi-head self-attention layer and a small position-wise feed-forward network. Each part is wrapped in a residual connection (the input is added back to the output) followed by layer normalization, which keeps training stable through a deep stack. The original model stacked six such blocks.^[1]

Two more pieces make it complete. Multi-head attention runs several attention computations in parallel, so different heads can track different relationships at once, such as syntax in one head and which noun a pronoun refers to in another.^[2] And because attention treats the input as a set with no inherent order, a positional encoding is added to each word so the model knows that “dog bites man” differs from “man bites dog.”^[3]

The feed-forward network is not a footnote. It typically expands each token’s representation to a wider intermediate size (2,048 dimensions in the original model, four times the block’s 512-dimensional width) before projecting it back down, and across the whole model it accounts for roughly two-thirds of a transformer’s total parameters.^[4] Researchers who probed what these layers actually store found that they behave like key-value memories: each learned “key” correlates with a pattern seen during training, and its paired “value” pushes the output toward particular vocabulary, with lower layers picking up shallow surface patterns and upper layers learning more semantic ones.^[4] Attention moves information between positions; the feed-forward network is closer to where that information gets looked up and acted on.

The residual connections are what make stacking so many blocks survivable. Without a path for the input to skip forward unchanged, a gradient has to pass back through every nonlinear transform in every block during training, and in a deep enough stack that signal can shrink to almost nothing before it reaches the earliest layers. The residual path gives it a direct route back, which is a large part of why models can stack dozens of blocks instead of a handful. The original transformer used six; GPT-3’s largest version uses 96.^[5]