What does a local AI runtime actually do?

A model on disk is just a big file of numbers. It does nothing on its own. The program that loads those numbers into memory, feeds in your prompt, and produces the reply one token at a time is the runtime (also called an inference engine or a server). Running AI yourself means picking one of these and pointing it at a model file.

Runtimes differ in what they are tuned for. Some are friendly desktop apps. Some are bare command-line engines. Some are built to serve many people at once. But most of them share one important habit: they speak the same network language.

That shared language is the OpenAI-compatible endpoint. The OpenAI API has become a common shape that most tools already understand, so local runtimes copy it. Ollama, for example, ships built-in compatibility with the OpenAI Chat Completions API at a local address like http://localhost:11434/v1.^[1] llama.cpp’s server is described as “a lightweight, OpenAI API compatible, HTTP server for serving LLMs.”^[2]

Nothing about this particular shape is technically special, and it did not win because it was the best possible design on paper. OpenAI’s API reached broad adoption first, while the ecosystem of client libraries, agent frameworks, and example code was still being built, so that shape became the one everything else got written against. Copying it is the cheapest way for a new runtime to be immediately useful to everyone who already has code written for a chat completions endpoint, whether or not a cleaner design existed somewhere else.

The pages that follow look at the four runtimes people reach for most: Ollama and LM Studio for easy everyday use, llama.cpp as the engine underneath much of it, and vLLM for serving a model to many users at once.