What is quantization, and what does shrinking a model cost?

Fewer bits per number, not fewer numbers

A trained model is a long list of numbers, and each one is written down at some precision. Quantization rewrites the same parameter positions using fewer bits each: 8, or 4, sometimes fewer. Each value is approximated, which can lose information and affect performance.

The grid is built in groups. A block of weights, commonly 32 or 128 of them, shares one scale factor. Each weight is divided by that scale, rounded to the nearest available level and stored as a small integer. Four bits gives 16 levels. At inference the integers are multiplied back out, and what comes back is the weight plus a rounding error.

What shrinking the numbers buys

Weight memory is close to parameter count times bits per weight. At 16 bits that is 2 bytes each; at 4 bits it is half a byte, plus the scale factors, which is why practical 4-bit formats work out nearer 4.5 bits per weight. A model with seven billion parameters needs roughly 14 GB of weight memory at 16-bit precision and roughly 4 GB at 4-bit. That is the difference between a machine in a rack and a device in your pocket.

Speed can also depend on memory traffic. On some phone workloads, moving weights is a significant limit. See why memory often limits local models. Quantization can reduce weight storage, though actual speed gains depend on the model, format, device and runtime.

Does quantization make a model worse?

Quantization effects depend on the model, method, format and task. Lower bit widths can reduce storage, but may affect quality; calibration, mixed precision and other techniques change the tradeoffs. There is no universal bit-width threshold for every phone or model.

Effects can differ across tasks and capabilities. Quantization may change aggregate scores and particular behaviors by different amounts; evaluate the model on the task and outputs that matter rather than assuming a universal order of failure.

How much it costs depends on how it is done

Methods at the same bit width can have different quality and compute costs. Calibration-aware methods can reduce some quantization error; results depend on the model, method, format and task. Quantization-aware training is another approach and requires training.

Model size and quantization interact, and their effects vary. Choosing between a larger model at lower precision and a smaller model at higher precision requires measuring the candidates on the intended workload.

• Group size. Grouping and scale choices affect storage and quality; compare the specific formats being considered.
• What is left alone. Some formats retain selected parameters at higher precision; the quality and storage effects depend on the model and method.
• What else is quantized. Activations and the key-value cache have their own precision, and the cache is the part that grows with the length of a conversation.
• Model sensitivity. Quality and performance effects can differ between models at the same bit width; measure the models and task you intend to use.

What quantization does not change

Quantization reduces the precision used to store weights, usually lowering their memory cost without changing the parameter count. It can also change task performance, with effects that depend on the model, format and task. Whether a quantized model remains useful must be measured on the work you want it to do.

Quantization is one way to fit a model within available computing resources. The engineering above is general and can be evaluated on its own terms; its costs depend on the model, format and task.