What is quantization?

A model is a giant pile of numbers called weights. By default each weight is stored at high precision, usually a 32-bit or 16-bit floating-point number. That precision is accurate, but it takes up space: every weight costs several bytes, and a model can have billions of them.

Quantization is the trick of storing those same weights at lower precision. Instead of 16 or 32 bits each, you keep them at 8 bits, 4 bits, or even fewer. Hugging Face describes it as “a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types like 8-bit integer (int8) instead of the usual 32-bit floating point (float32).”^[1] Fewer bits per weight means a smaller file and less memory needed to load it.

How does it keep the numbers usable? The original weights cover a wide range of values. Quantization finds that range and maps it onto the small set of values a low-precision format can hold, recording a scale factor so the values can be read back approximately.^[1] A 4-bit number can only represent sixteen distinct levels, so each weight gets rounded to the nearest available one. The rounding is where quality loss comes from, but for large models the loss is usually small.

A concrete pass through the math helps. Say a layer’s weights range from -2 to 2. A 4-bit format has sixteen slots to divide that range into, so each slot sits about 4 ÷ 15 ≈ 0.27 apart, and every weight snaps to whichever slot is closest, off by at most half that gap. That per-weight error is small and mostly independent from one weight to the next rather than compounding, which is part of why the measured quality drop stays small even though every weight in the model moved.

The same math exposes a real weakness in using one scale for an entire tensor: a single outlier weight ten times the size of its neighbors forces the scale wide enough to cover it, wasting precision on every ordinary weight sharing that range. Real formats avoid a single tensor-wide scale for exactly this reason. The quantization labels page covers the block-by-block scheme that limits how far one outlier’s damage can spread.