Does a bigger model always mean a better model?

As a rule of thumb, more parameters give a model more room to capture patterns, so larger models tend to do better on hard tasks. But parameter count is not the whole story, and treating it as a single quality score leads people astray.

The clearest evidence comes from a 2022 DeepMind study often called Chinchilla. It compared a 70B-parameter model against an earlier 280B-parameter model trained with the same compute budget. The smaller model, trained on four times as much data, won, including a notable jump on a broad knowledge benchmark.^[1] The lesson was that many large models had been undertrained: they had plenty of parameters but had not seen enough text to fill them usefully.

The takeaway is balance. The study’s guiding rule is that model size and the amount of training data should grow together, roughly doubling the data every time you double the size.^[1] So when you compare two models, the parameter count is one input among several. How much data the model saw, how that data was curated, and how the model was tuned afterward all shape what it can actually do.

A year later, Meta’s LLaMA gave the lesson a second, independent data point at a different scale: trained on trillions of tokens rather than the smaller sets typical at the time, its 13B model outperformed GPT-3 despite GPT-3 having roughly 13 times the parameters, and its 65B model held its own against Chinchilla-70B and PaLM-540B.^[2] Two results, two different labs, the same direction: past a certain point, more training data on a smaller model beats more parameters on a starved one.

There is a wrinkle worth knowing, because production teams don’t always follow Chinchilla’s rule as written. Chinchilla’s ratio is optimal for training compute alone; it says nothing about the cost of running the model afterward. A 2024 analysis found that once you account for a realistic volume of inference requests, the better move is often to train a smaller model for far longer than Chinchilla-optimal, because training is a one-time bill while inference cost accrues with every request the model ever answers.^[3] Meta’s own Llama 3 put this into practice: Chinchilla-optimal training for an 8B model calls for roughly 200 billion tokens, but Meta trained its 8B and 70B Llama 3 models on over 15 trillion — two orders of magnitude past that point — and reported that both continued to improve log-linearly the whole way, meaning each further multiplication of the training data bought a steady increment of capability rather than tapering off.^[4] A smaller model that’s cheaper to serve at scale can be worth the extra training compute it takes to get there.