Why are Macs good for running AI models locally?

On a typical PC, memory is split into two pools. The CPU uses system RAM, and a separate graphics card has its own VRAM. When a model needs to run on the graphics card, its data has to be copied from one pool to the other across a connection called the PCIe bus, and that copying is slow.^[1]

Apple silicon does not split memory this way. The CPU, the GPU, and the memory all sit on the same chip and share one pool, often called unified memory. As one explainer puts it, “There is no VRAM. There is no system RAM. There is one pool of unified memory.”^[1] A model loaded into that pool is reachable by the GPU with no copy step in between.

This fits how language models actually run. Once a model has read your prompt, producing each new word is mostly a matter of reading the model’s weights out of memory, not heavy calculation. Apple’s own research puts it plainly: “Generating subsequent tokens is bounded by memory bandwidth, rather than by compute ability.”^[2] So a machine where memory is fast and close to the processor is well suited to the job, which is why a Mac can run sizable models without a separate graphics card.

The reason it works this way is arithmetic intensity. Generating one token for one person touches every weight in the model exactly once, in a single multiply-add, so almost nothing gets reused once it is loaded. The processor spends most of its time waiting for the next weight to arrive rather than computing. Serving many people at once looks different, because the same loaded weight can be reused across a batch of concurrent requests, which is why data-center serving leans on raw compute and batching instead of unified memory the way a single Mac session does.

Apple’s own numbers show the bandwidth relationship holding almost one-to-one. Comparing the M5 chip to the M4, memory bandwidth rose from 120GB/s to 153GB/s, a 28% increase, and measured generation speed rose 19 to 27% across the model architectures Apple tested.^[2] A bandwidth increase in that range producing a speed increase in almost the same range, rather than one far smaller or far larger, is itself evidence for the bottleneck Apple describes in words: the number moves with memory speed because memory speed is what it is waiting on.