How do you weigh capability against cost and privacy?

Selecting a model means “balancing three key considerations: capabilities, speed, and cost.”^[1] These pull against each other. The most capable model is usually the slowest and most expensive, and the cheapest is rarely the smartest. The skill is matching the model to the task rather than always reaching for the top of the leaderboard.

A concrete case makes the trade-off obvious. A support-ticket classifier sorting messages into five known categories needs a fraction of the reasoning a multi-file coding assistant does — pointing the classifier at a frontier model mostly buys latency and cost, not better sorting. The coding assistant is the opposite case: skimping there shows up later as bugs a stronger model would have caught.

A useful habit is to start cheap and only move up if you have to. Anthropic suggests one path of beginning with a fast, low-cost model, testing it, and upgrading “only if necessary for specific capability gaps.”^[1] Many real tasks are simple enough that a smaller model handles them at a fraction of the price. What “necessary” looks like in practice is usually specific, not a vague sense that the answers could be better: a smaller model that summarizes cleanly but garbles a multi-step instruction, or one that writes working code but misses the edge case the task actually depends on. Those concrete failures, caught on your own eval set, are the signal to move up a tier — not a hunch that the top-tier model would probably do a bit better everywhere.

Privacy is the third pull, and it can override the other two. If your data cannot leave your own systems, a model you run yourself becomes the answer regardless of cost. A self-hosting guide makes the case with real hardware: “a single 4090 plus Llama 3 22B or Mixtral 8x7B gets you fair quality for <$1/h, no data leaks.”^[2] One consumer graphics card, running an open-weight model, at a fraction of what a comparable hosted API call would cost per hour of steady use. The trade is real work to operate it. Sending data to a hosted API is simpler and often more capable, but it means your data passes through someone else’s servers.