What is llama.cpp and why does it show up everywhere?

llama.cpp is the engine many of the friendlier tools are built on. It is a C/C++ implementation for running models locally, and its stated goal is “to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.”^[1]

The “wide range of hardware” part is the reason it spread so far. It runs on Apple Silicon, on ordinary x86 processors, and on GPUs from NVIDIA, AMD, and Intel, among others.^[1] The reason it reaches that far is a design choice stated plainly in its own description: llama.cpp is a “plain C/C++ implementation without any dependencies.”^[1] A framework built on top of PyTorch or CUDA inherits whatever hardware those support and no more. A project with no framework underneath is free to write a dedicated backend for whatever chip shows up next, which is how llama.cpp ended up with Metal, CUDA, HIP, Vulkan, SYCL, and close to a dozen other backends maintained as first-class targets rather than one generic path with everything else bolted on.^[1] If you have a laptop or an older desktop with no fancy graphics card, llama.cpp is often the thing that lets it run a model at all.

It also has an answer for the in-between case: a model too big for your VRAM but small enough for system RAM. CPU+GPU hybrid inference splits the model’s layers between the two, running what fits on the GPU and the rest on the CPU, so an oversized model still runs instead of simply refusing to load.^[1] It will be slower than running entirely on a GPU, but slower and working beats a clean failure.

It reads models in the GGUF format, a single portable file that packs the weights, the tokenizer, and the metadata together. The project provides scripts to convert models from other formats into GGUF.^[1] GGUF is also where quantization lives, the trick that shrinks a model to fit in less memory, which has its own topic on this site.

llama.cpp can also serve over the network. It ships llama-server, described as “a lightweight, OpenAI API compatible, HTTP server for serving LLMs,” with a basic web UI included.^[1] So you can use llama.cpp directly, or use it without realizing it through a tool like LM Studio that wraps it in a window.