Choosing a model
Why are public benchmarks not enough?
Public benchmarks are a useful first filter but a weak final answer. Scores can be inflated by training on test data, and they measure broad skills rather than your specific inputs. Use them to narrow, then test on your own cases.
Public benchmarks are worth a glance, but a high score is a weak promise. They are good for a first cut and bad as a final answer.
One problem is that benchmark scores can be inflated rather than earned. A 2025 review of the field argues that “benchmark performance should not be used as a reliable indicator of general LLM cognitive capabilities,” partly because models can pick up the test questions during training and partly because strong scores often fail to carry over to real tasks.[1] When a model has effectively seen the answer key, the number tells you about memory, not skill.
That risk is not hypothetical. In April 2025, Meta submitted an unreleased, specially tuned chat variant it called “Llama-4-Maverick-03-26-Experimental” to LMArena, a popular crowdsourced leaderboard, and it scored well. The model Meta actually shipped, Llama-4-Maverick-17B-128E-Instruct, ranked 32nd on that same leaderboard once people tested the real thing.[3] LMArena changed its evaluation policy in response. The lesson generalizes past this one leaderboard: the entry that tops a chart and the model you can actually download are not guaranteed to be the same model.
The other problem is fit. Standardized benchmarks “test broad capabilities, not the specific inputs your system might handle.”[2] A model can top a reasoning chart and still stumble on the particular phrasing, format, and edge cases of your work.
A benchmark run in good faith can still mislead through the comparison it implies. When OpenAI launched GPT-4 in 2023, it advertised a “90th percentile” score on the Uniform Bar Exam.[4] A later reanalysis found that figure held up only against a skewed reference group — February bar exam sittings, dominated by repeat test-takers who had already failed once and score lower than the general pool. Measured against first-time test-takers using official exam-board data, the same GPT-4 result lands closer to the 62nd percentile overall, and the 42nd on the written-essay half.[4] The model’s score never changed between the two claims. The comparison group did.
So treat benchmarks as a way to build a shortlist, then settle the choice the way the previous page describes: run the candidates on your own cases and see which one actually does your job.
References
- Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models — James Fodor, arXiv 2025
- LLM evaluation: a beginner's guide — Evidently AI
- Meta's 'vanilla' Maverick AI model ranks below rivals on a popular chat benchmark — TechCrunch
- Why ChatGPT-4's Score on the Bar Exam May Not Be So Impressive — New York State Bar Association