AI stacks
What components make up a local-inference stack?
The local-inference stack runs on your own hardware: an open-weight model file you can download, a runtime (Ollama, llama.cpp, or vLLM) that loads it and does the math, and an app on top that gives you a way to use it. The runtime is the part that talks to your GPU or Mac chip.
Running AI on your own hardware is the same three layers as a hosted setup, just with the serving layer pulled onto your machine. From the bottom: an open-weight model file you download, a runtime that loads that file and does the math, and an app layer you actually interact with. The local-inference-basics topic calls the lower two the two-layer stack, and the same split holds here.
The model has to be open-weight, meaning the weights are published for download, because a runtime needs the file to load. Families like Llama and Qwen ship their weights for exactly this; a frontier model offered only through an API does not, so it cannot live in a local stack. Once you have a model file, the runtime is the piece that opens it, loads the weights into memory, and turns prompts into tokens while talking to your GPU or your Mac’s chip. The common choices split by purpose: Ollama bills itself as the easiest way to build with open models[2], llama.cpp is the lightweight engine many other tools are built on[3], and vLLM is a fast library for inference and serving aimed at high throughput when many users hit one model at once.[1]
The app layer is whatever you point at the runtime: a desktop chat window, a script, or an entire product. A useful detail ties the stack together: most local runtimes copy the OpenAI-compatible endpoint shape, so an app written against a cloud model often works against a local one after you change a single base-URL setting. That one setting is the whole cost of the swap. The serving and runtimes pages go deeper on each engine, and the hardware-and-VRAM topic covers the hard limit underneath all of it, which is whether the model fits in your memory at all. That ceiling applies at the runtime layer specifically: the same model file that runs fine through vLLM on a data-center GPU can fail to load through Ollama on a laptop even though the file and the math are identical, because only the memory available to the runtime differs.
References
- vLLM documentation — vLLM
- Ollama — Ollama
- llama.cpp README — ggml-org
Common questions
- Why open-weight models for a local stack?
- Because you need the actual weights file to run a model on your own machine. Frontier models served only through an API never ship their weights, so you cannot host them locally. Open-weight families like Llama and Qwen are what this stack runs on.
- Which runtime should I pick?
- For everyday single-user chat, Ollama or LM Studio are the easy path. llama.cpp is the lean engine underneath much of the ecosystem. vLLM is built for serving one model to many users at once, so reach for it only when you have that need.