How do you test a model on your own data?

Once you have a shortlist, the way to decide between models is to run them against your own data, not a public score. Anthropic’s guidance is to create tests specific to your use case and notes that “having a good evaluation set is the most important step in the process,” then test with your actual prompts and data and compare accuracy, quality, and edge-case handling.^[1]

This evaluation set, often called an “eval,” is just a collection of real cases paired with what a good answer looks like. The point is that it reflects your work. Evidently AI draws the line clearly: “LLM product evals are more like job performance reviews,” checking whether the system “excels in the specific task it was ‘hired’ for.”^[3]

How you score each case matters as much as collecting it. The Pragmatic Engineer’s guide to evals pushes toward a plain pass/fail per case rather than a 1-5 rating scale, on the grounds that binary decisions force clarity: a “fail” is an unambiguous signal to fix something, where a middling number is not.^[2] For mechanical checks — does the output parse as valid JSON, does it include a required field — a small script can score faster and more consistently than a person. For checks that need judgment, like whether a summary stays faithful to its source, the common middle ground is “LLM-as-judge”: a second model call grades the first one’s output, which Evidently AI calls one of the most popular approaches in practice today.^[3]

You do not need a huge set to start. Evidently AI’s own advice is to begin informally, running a handful of sample inputs and reading the outputs, before building a proper held-out set once you’ve settled on a candidate model and prompt.^[3] From there, the highest-value move is to look at your own outputs and let the failures guide you. Hamel Husain suggests reading at least 100 real traces end to end during this pass, noting anything that looks wrong before trying to categorize or fix it — in his words, this bottom-up error analysis “is the single highest-ROI activity in AI development.”^[2] A few dozen real cases, scored honestly, will tell you more about which model to pick than any leaderboard can.