What do layers do, and what does depth mean?

A language model does not process text in one step. It passes the text through a stack of layers, one after another. Each layer takes the running internal representation of the text, reworks it, and hands the result to the next layer. The parameters live inside these layers, so the layers are where the actual work happens.

The number of layers in the stack is the model’s depth. Bigger models tend to be deeper. For example, the LLaMA 2 family at 7B, 13B, and 70B parameters “consist of 32, 40 and 80 transformer layers.”^[1] So the 70B model is not just wider, it also runs the text through more than twice as many processing steps as the 7B.

Why stack so many? The rough intuition is that early layers tend to handle surface details and later layers build toward more abstract meaning, with each layer working from the output of the one before it. More layers give the model more chances to refine its understanding before it produces an answer. Depth is one of the main knobs a model’s designers turn, alongside how wide each layer is.

Directly probing a model’s layers shows a more specific picture than a smooth gradient from surface to abstract. A study that tested LLaMA layer by layer with targeted probing tasks found that the lower layers hold comparatively little arithmetic or factual knowledge but already show real logical-reasoning and multilingual ability, while the upper layers are where most factual knowledge and computational power actually concentrate.^[1] Depth is not just “more of the same refinement” repeated at every step; different bands of the stack appear to specialize in different kinds of work, holding different pieces of what the model knows and how it reasons.