How do I trade size against quality?

Every step down in precision is a deal: you give up a bit of quality to save a lot of space. The savings are real. The llama.cpp project measured Llama 3.1 8B at full 16-bit precision taking 14.96 GiB, while the 4-bit Q4_K_M version of the same model is 4.58 GiB.^[1] That is roughly a third of the size, which is the difference between a model that fits on a typical GPU and one that does not.

The cost shows up as a small drop in how accurately the model reproduces what its full-precision version would have said. Reducing the weights to a lower precision “naturally gives rise to a drop in the performance of the model.”^[2] llama.cpp’s own documentation names how that drop actually gets measured: usually perplexity or Kullback-Leibler divergence against the full-precision original, a number that shrinks further if you quantize using a calibration file (an imatrix) instead of skipping that step.^[1] The size of that drop depends on how far you go. Going from 16-bit down to 8-bit or 4-bit usually costs very little that you would notice in normal use. Pushing to 2-bit or lower starts to hurt more, and very small models tend to feel the loss sooner than large ones.

There is a second axis to this trade that is easy to miss: a quantized model is not just smaller, it is often faster. Token-by-token generation is bounded by how fast weights can be streamed out of memory, not by raw compute (the running on a Mac topic covers why), so fewer bytes per weight means less to stream for every single token. In llama.cpp’s own benchmark on this same Llama 3.1 8B model, generation measured about 72 tokens per second at Q4_K_M against about 29 tokens per second at full 16-bit precision on identical hardware, roughly two and a half times faster.^[1] Size, quality, and speed move together here, not just size against quality.

The practical guidance most people follow: Q4_K_M is the common sweet spot for running models at home, where the file is small enough to fit and the quality stays close to the original. If you have extra memory to spare, Q5_K_M or Q6_K give you a little more headroom on quality. Drop to Q3 or Q2 only when a model would not otherwise fit, and expect the answers to get rougher as you do. The right pick is the highest precision that still leaves room in your memory once the model and its context are loaded.