Local vs cloud
Which is cheaper and more capable, local or cloud AI?
Cloud APIs cost nothing upfront but charge for every request, so heavy use adds up. Local costs money for hardware first and then very little per query, but the models you can run at home are usually weaker than the best cloud models.
Cost works in opposite directions for the two options. A cloud API has no setup cost: you sign up and start sending prompts. But you pay for usage, billed by the token (a token is roughly a few characters of text). Input and output are priced separately, and output tokens cost more than input. OpenAI’s price list, for instance, shows its standard models charging several dollars per million input tokens and more per million output tokens.[1] For light use that is cheap. For an application that runs millions of queries, the bill grows with every one.
Local cost is front-loaded. You need a machine with enough memory to hold the model (see the model size calculator), which can be a real expense. After that, each query is close to free, because you are only paying for electricity. So local tends to win once your usage is high and steady, while cloud tends to win when usage is low or unpredictable.
“Close to free” is worth one caveat. A power-hungry GPU running near its limit for hours at a stretch draws a meaningful amount of power, and in a market with expensive electricity that adds up over a year in a way a single per-token cloud invoice never makes obvious. It is still almost always far cheaper than equivalent cloud usage at real volume, but the honest comparison is hardware amortized over its useful life plus real, if modest, electricity, set against a cloud bill that never has an endpoint of its own.
Capability is where cloud usually pulls ahead. The largest, strongest models are run by providers on hardware far bigger than a home computer, and those are typically the most capable models available. A model small enough to run on a laptop is real and useful, but it is usually weaker than the top cloud models on hard tasks.
Latency, the time to get a reply, cuts both ways. A cloud call has to make a round trip over the internet, which adds delay; a local model skips that, but it can still be slow if your hardware is underpowered for the model you chose. Neither side is simply faster.
References
- API pricing — OpenAI