What is a parameter in a language model?

A parameter is a single number stored inside a model. Most parameters are weights, and a weight is the strength of a connection between two of the model’s internal units. As one glossary puts it, each weight “signifies the strength and direction (positive or negative) of the influence one neuron has on another.”^[1] A second, smaller kind of parameter is the bias, a number added on top that shifts the result.

These numbers are not written by a programmer. They start out random and are adjusted during training. The network “learns by iteratively adjusting these weights to predict the correct output,” and “the set of weights in the network encapsulates what the network has learned from the training data.”^[1] So when people say a model “knows” something, that knowledge is not stored as text or rules. It is spread across the values of its parameters.

When a model finishes training, those values are frozen and saved to a file. That file is the model. Running it means reading the parameters back and doing arithmetic with them. The count of parameters is how a model’s size is usually reported, which is why model names so often end in a number followed by a “B” for billion.^[2]

The scale these counts reach is hard to picture in the abstract, but the trajectory is telling. GPT-2’s smallest published configuration held 117 million parameters; GPT-3, three years later from the same lab, held 175 billion, roughly a 1,500-fold jump within a single model lineage.^[3] Every one of those parameters is an individual number nudged a little during training, and the count keeps climbing not because more parameters guarantees a smarter model, but because, so far, more capacity paired with proportionally more training data has kept translating into measurably better performance.