How small can a model be and still be genuinely useful?

The size question has no answer until you name the job

There is no floor that holds across every task, because a model is not one capability that dims evenly as it shrinks. It is a bundle of separate abilities with very different costs. The question that does have an answer is narrower: what does this thing have to do, how wide is the range of inputs it will see, and what does a wrong answer cost?

Those answers move the floor by orders of magnitude. A model expected to answer any question about any subject has to carry a great deal of the world in its weights, and weights are what size buys. A model that reads one sentence you just spoke and decides it is a reminder for Thursday needs almost nothing it was not just handed.

Capability does not fall away evenly

Shrinking costs some things long before others. Breadth of factual recall goes first, along with competence in specialized domains and less common languages. Long chains of dependent reasoning go next: every step is a chance to drift, and a smaller model drifts sooner. Robustness goes too, so odd phrasing and unusual inputs are handled less gracefully at each reduction in size.

What survives longest is structure. Grammar, formatting, mapping a phrase onto one of a fixed set of slots, keeping fields in order, pulling the parts of a sentence that matter out of the rest. These are pattern jobs. The information is already in the input, so the model is rearranging rather than remembering, and rearranging is cheap in a way that remembering is not.

This is why a single benchmark average is a poor guide. It blends recall-heavy questions with structural ones and reports one number. A small model's score is usually dragged down by the part of the test that has nothing to do with the job you would actually give it.

What does a job that survives shrinking look like?

Four properties tend to travel together in work that keeps working as the model gets smaller. They are worth checking one at a time, because the more of them a task has, the further down in size it can go before the results stop being worth having.

Two ordinary engineering moves make that kind of work sturdier still. Generation can be constrained so that only text matching a fixed schema is ever produced, which removes malformed structure as a failure mode outright. And a small model trained on one narrow task closes much of the distance to a large general one, on that task alone. Both are well established. Neither does anything for open-ended reasoning.

• The information is in the input. Nothing has to be recalled that was not just said.
• The output space is small and known: one label from a fixed set, a date, a record with named fields.
• The task repeats in the same shape, which is the condition under which a model can be trained for it specifically.
• Errors are visible. A wrong field on the screen is corrected in a second. A wrong step buried inside a long answer is not.

Where does the floor actually sit?

Memory is the binding constraint, and it is arithmetic. Weights dominate the total. A weight stored at four bits costs half a byte in principle, but real four-bit files run nearer 4.5 bits per weight, because the block scales that keep the accuracy loss mild are themselves stored and some layers are held at higher precision. That puts a three-billion-parameter model at roughly 1.7 GB before the running context is counted. All of it has to fit alongside the operating system and whatever else is open, which is a tighter budget than the headline memory figure suggests.

Quantization is the lever that moves the floor, and it is not free. Storing weights at lower precision shrinks the model close to proportionally, with accuracy loss that stays mild down to around four bits per weight for most current methods and grows faster below that. Where exactly it turns depends on the method and the model, which is why that is a rule of thumb rather than a specification.

How would you know for your own task?

The only reliable answer is measured on your own task. Collect a few hundred real examples, the awkwardly phrased ones included, decide what error rate is acceptable and what a bad answer actually costs, then test candidates against that. It is unglamorous, and it is the whole method. A model that scores modestly in general can be entirely sufficient for one narrow job.

A narrow job lowers the odds of a wrong answer; it does not remove them. Push an input outside the shape the model was trained for, and a small model produces the same fluent, invented output a large one does when it runs out of what it actually knows.

For a narrow task, a small model can be useful, but no general rule can promise that result. Model size, training, input shape and evaluation all matter. The only sound way to establish a capability is to measure it against the task it is meant to serve.