What do people mean by a model, its parameters, and training?

A model is the trained program at the center of any AI system. A large language model, the kind behind chat assistants, is “a very large deep learning model that is pre-trained on vast amounts of data.”^[1] You can think of it as a single large mathematical function: text goes in, and a prediction about the next piece of text comes out.

A parameter is one of the numbers inside that function. Modern models have billions of them. In AWS’s words, “weights and biases along with embeddings are known as model parameters.”^[1] These numbers are not written by a person. They start out close to random and get shaped by the data the model sees. The count you hear quoted, like “70 billion parameters,” is roughly a measure of the model’s size and capacity.

Concrete examples make the scale easier to picture: OpenAI’s GPT-3 has 175 billion parameters, and AI21 Labs’ Jurassic-1 has 178 billion, two separate teams landing in almost the same range.^[1] Reaching that scale takes an enormous amount of raw text. Training data is often pulled from the open internet, including Common Crawl, a dataset built from more than 50 billion web pages, plus more curated sources like Wikipedia’s roughly 57 million pages.^[1]

Training is the process that sets those numbers. “During training, the model iteratively adjusts parameter values until the model correctly predicts the next token from the previous sequence of input tokens.”^[1] So training is not the model memorizing answers. It is a long loop of guessing the next bit of text, checking how wrong the guess was, and nudging the parameters to be a little less wrong next time, repeated across a huge amount of text.