Why is memory the limit on a phone, not processing power?

Memory paths and their trade-offs

Many fast runtimes keep weights in RAM while generating. Some systems can stream or offload parameters from storage, trading lower resident memory for performance. Weight-size estimates depend on the numeric format and its overhead; runtime memory also includes cache and working state.

Only part of a phone’s RAM is available to an app; the operating system and other apps share the same pool. The amount an app can use depends on the device and current system activity.

Storage trades capacity for speed; it is not a drop-in substitute for RAM. Phone flash is typically slower than RAM in bandwidth and latency, and every read costs power. A model streamed from disk can ease a capacity limit, but storage latency and bandwidth may reduce generation speed.

Every word generated reads the model again

Text comes out one token at a time, a token being a word or a fragment of one, and each token is a fresh pass through the whole network. On-chip cache is measured in megabytes and the weights in gigabytes, so almost none of the model can stay close to the arithmetic between passes. In a common path, each pass reads active weights from main memory and uses them for multiply-and-add operations; streaming or caching strategies change that traffic.

That ratio is the problem. Two operations for every weight fetched is far below what a modern accelerator is built to sustain, and arithmetic throughput has grown faster than memory bandwidth for decades, on every kind of hardware. The multiplier finishes and waits. Adding more multipliers does not help, because they were already idle.

The ceiling this sets can be calculated. Tokens per second cannot exceed memory bandwidth divided by the bytes of weights read per token. As an illustrative upper bound, dividing an assumed 50 GB/s bandwidth by 4 GB of weights gives about 12 full-weight reads per second before runtime overhead. Actual throughput depends on device, quantization, caching, and implementation.

Where processing power does matter

There is a phase where compute is the limit. Before the first word comes back, the model has to read the prompt, and it can read the whole prompt at once. Each weight fetched is then used across many tokens instead of one, the ratio of operations to bytes rises, and that phase is genuinely limited by arithmetic rather than by memory traffic.

This is why a long prompt costs a pause up front and the reply then arrives at a steady pace. Two different limits, in that order. It is also why a fast chip is not wasted on a phone. It shortens the pause. It does not raise the ceiling on model size, and it does not make the reply come out faster.

What actually moves the limit

Three things change the arithmetic, and none of them is a faster processor. Each acts on a different term in the same calculation: how many bytes a single weight costs, how many weights each token has to read, and how much of the memory budget is spent on something other than weights. Only the first two are properties of the model.

• Precision. Fewer bits per weight shrinks both the footprint and the bytes read per token, so it moves the ceiling and the speed together. What that costs in accuracy is a separate question with its own answer.
• Sparse or mixture-of-experts models may read only part of their weights for a token. Whether all experts remain loaded depends on the runtime; streaming or offloading can trade memory use against performance.
• Context. The attention cache grows with the length of the conversation and lives in the same memory as the weights, so a long thread takes room away from the model sitting beside it. What that cache is and why it grows the way it does is its own question.

The same wall shows up as heat

Moving a byte in from main memory costs far more energy than the arithmetic performed on it, by orders of magnitude. A memory-bound workload is therefore a power-bound one as well. Sustained memory traffic is what warms a phone, and a warm phone is slowed down on purpose, which lowers the effective bandwidth again.

So the largest model your phone can run is not one number. It depends on the memory the operating system leaves to a single app, the bits per weight, the length of context being held, and what else is running. Those limits vary by device and by the demands of the task.