Sizes & parameters
How much memory does it take to run a model?
Memory needed is roughly parameter count times bytes per parameter: 2 bytes each in half precision, so a 70B model needs about 140GB. Quantization shrinks each parameter to cut that down.
The parameter count also tells you, fairly directly, how much memory a model needs to run. The estimate is simple: multiply the number of parameters by the number of bytes used to store each one. In full precision each parameter takes 4 bytes; in the half-precision formats used for most models today, 2 bytes each.[1]
So a 70B model in half precision needs roughly 70 billion times 2 bytes, about 140GB, just to hold the parameters. That is far more than a single consumer graphics card, which is why large models are often split across several. The same arithmetic explains a 176-billion-parameter model needing about 352GB in half precision.[1]
The common way to fit a model into less memory is quantization: storing each parameter with fewer bits. Moving from 2 bytes to 1 byte (8-bit) halves the size, and 4-bit storage uses half a byte each, which brings a 70B model down toward 35GB.[1] The trade is a small loss of precision in each number, which usually costs a little accuracy. This is also why a Mixture-of-Experts model is no lighter to load: memory follows the total parameter count, even when only a few experts compute per token.
The precision you pick is a real accuracy decision, not just a storage one, so it is worth being deliberate about it rather than always reaching for the smallest number. 8-bit quantization is close enough to lossless for most everyday use that it is a reasonable default. 4-bit is where the trade becomes noticeable enough to test for your specific task before committing to it, especially on anything requiring precise arithmetic or long chains of reasoning, where small per-step errors have more room to compound.
Put a real card against real numbers. A consumer RTX 4090 has 24GB of memory. At 4-bit, that comfortably holds a 22B-parameter model — about 11GB for the weights alone — with room left over. It cannot hold a 70B model at 4-bit, which needs roughly 35GB, without a second card or a still-smaller quantization format. That gap between “room left over” and “no room at all” also explains why quoted VRAM requirements always leave a margin above the raw weight size: the context window and intermediate activations need memory of their own, on top of the parameters, and that overhead grows with how much context is actually in use at the time.
Tools for running and shrinking models
- Hugging Face bitsandbytes ↗
Library and guide for loading models in 8-bit and 4-bit to cut memory use.
- llama.cpp ↗
Open-source runtime that runs quantized models on ordinary CPUs and GPUs.
- Ollama ↗
Local runner that downloads and serves quantized open models with one command.