What do labels like Q4_K_M mean?

When you download a model to run at home you will see files named things like Q4_K_M, Q5_K_M, Q8_0, or Q2_K. These come from GGUF, the file format used by llama.cpp and the tools built on it.^[1] The label tells you how aggressively the model was compressed. Reading it is simple once you know the parts.

The number after the Q is roughly how many bits each weight uses. Q4 is about 4 bits, Q8 is 8 bits, Q2 is 2 bits. Fewer bits means a smaller file. The exact figure is a little higher than the headline number because some bookkeeping values are kept at higher precision: Q4_K works out to 4.5 bits per weight, Q5_K to 5.5, Q6_K to about 6.56, and Q2_K to about 2.63.^[1]

The K means the model uses the modern “k-quant” method, which groups weights into blocks and stores some parts of each block more precisely than others.^[1] The trailing letter is the size variant: S for small, M for medium, L for large. So Q4_K_M is 4-bit, k-quant method, medium variant. The _0 style labels (like Q8_0) are older “legacy” methods that GGUF notes are not used widely today.^[1] You will also see IQ labels, an even more compact family that leans on an importance matrix to decide which weights to protect.^[1]

One real example makes the block idea concrete instead of abstract. Q6_K groups weights into super-blocks of 256, split into 16 blocks of 16 weights each; every block gets its own scale factor stored at 8-bit precision, landing at 6.5625 bits per weight once those scales are counted in.^[1] Q5_K covers the same 256-weight super-block with a coarser split, 8 blocks of 32 weights, each carrying a 6-bit scale.^[1] Smaller blocks track a local range of values more closely and waste less precision on outliers, at the cost of storing more scale factors; that trade-off is why the K-variants exist as a family rather than one fixed scheme. IQ formats go a step further: instead of relying only on a block’s own scale, they also draw on a separately computed importance matrix to help decide each weight’s value.^[1] In practice you supply that as an imatrix file when quantizing, and the llama.cpp project notes it measurably reduces accuracy loss compared to skipping it.^[2]