What are a model's weights, and why is the file so large?

A weight is a number the model learned, not a rule someone wrote

A language model is a long chain of matrix multiplications. The numbers filling those matrices are its parameters: weights on the connections between units, plus a smaller set of biases added afterward. Training nudges them a fraction at a time against enormous amounts of text. When training stops, the numbers freeze. What the model knows is not stored anywhere as sentences or facts. It is stored as those coefficients.

Running the model does not change them. Every prompt reads the same numbers, multiplies your tokens through them and produces the next token. That is why a downloaded model is inert until you run it: it is a fixed table of numbers with a small program to walk them, not something that keeps learning from you in the background.

The file is the numbers, and almost nothing else

Open a model file and you find tensors, which are named blocks of numbers with their shapes recorded, laid end to end after a short header. Beside them sit the tokenizer vocabulary and a configuration listing the layer count, hidden size and attention layout. In a multi-gigabyte file that metadata is usually well under one percent. The weights are the file.

This is also why zipping a model buys you almost nothing. Trained weights look close to random noise to a general-purpose compressor, so there is little repetition to exploit. The way to make the file smaller is to spend fewer bits on each number, which is a different operation with a real cost attached.

The arithmetic: parameters times bytes per parameter

For an unchanged architecture, file size is close to parameter count multiplied by bytes per parameter. Quantization changes the bits used per parameter; other compression methods, such as structured pruning, can reduce parameter count after pretraining.

Two caveats keep the estimate honest. Storage is quoted in gigabytes of a billion bytes while an operating system may report gibibytes of 1,073,741,824 bytes, a gap of about seven percent you will see between a listed size and what your device shows. And a mixture-of-experts model keeps every expert in the file even though only a few of them run for any given token, so what you download tracks the total parameter count, not the count active at once.

• 32-bit float: 4 bytes per parameter. A 7-billion-parameter model comes to about 28 GB.
• 16-bit float, fp16 or bfloat16: 2 bytes. The same model is about 14 GB. This is the usual format for a freshly released model.
• 8-bit: 1 byte. About 7 GB.
• 4-bit: half a byte in principle, nearer 4.5 bits once you count the scale factors each block of weights needs. About 4 GB.

Why the parameter count runs to billions

Parameters concentrate in a handful of shapes repeated once per layer. Inside a transformer layer, the attention projections and the feed-forward network are large matrices sized by the model's hidden dimension. Because a matrix is two-dimensional, doubling that dimension roughly quadruples what it holds. A common rule of thumb puts one layer at around twelve times the hidden dimension squared.

Multiply by the layer count, then add the embedding matrix, which is the vocabulary size times the hidden dimension, and you land on the total. Thirty-two layers at a hidden dimension of 4,096 comes to about 6.4 billion parameters before embeddings, which is where the familiar seven-billion figure you will see on a model card comes from. Nobody picked billions as a target. It falls out of the shape.

Budget for weights and working memory

In a fully resident runtime, budget RAM for the weights, cache and working state. The key-value cache grows with conversation length, so leave headroom beyond the weights.

Many fast runtimes keep weights in RAM. If weights and working state exceed available memory, that execution path may fail; some runtimes can stream or offload weights from storage, with performance tradeoffs. For an unchanged architecture, quantization changes bits per parameter; methods such as structured pruning can also reduce parameter count.