What does it actually mean for AI to run on your device?
On-device AI means the model’s inference runs on the processor in the device, rather than on a remote server. The app may still use a network for updates, sync, or other features, so the label describes where the model runs, not every data path.
Where the computation physically happens
A language model is a large set of numbers called weights, and running one means loading those numbers into memory and doing arithmetic against them. The only question that matters is whose memory and whose chip. In the cloud arrangement, your words travel over a network to a data center, the arithmetic happens on machines you will never see, and the answer travels back.
On-device, the weights are a file in your own storage. The device reads them into its own memory and runs the same arithmetic on the processor in your hand. The model can produce its reply locally without a cloud inference request. The surrounding app may still send or receive data for other features, so check those paths separately.
What a phone can actually run
Memory is often an important constraint. Many runtimes keep weights in memory while generating, while some can stream parameters from storage with a speed tradeoff. Four-bit weights use roughly a quarter of the storage of sixteen-bit weights before format overhead; total runtime memory also includes working state and cache.
Speed is usually bound by memory bandwidth rather than arithmetic, because producing each word means reading the weights again. A modern phone chip carries a CPU, a GPU and a neural accelerator sharing one pool of memory with everything else running, and sustained work costs battery and raises heat. So what a given phone can run depends on its memory, its chip generation, how heavily the model was compressed, and how much conversation you ask it to hold at once.
On-device describes the model, not the whole app
This is where the phrase gets slippery. Inference can run entirely on the device while the app around it still opens a network connection for reasons that have nothing to do with the model: downloading its own updates, sending crash reports or usage analytics, checking a license, backing your data up, or fetching something a feature genuinely needs, like a map tile.
On-device is therefore a statement about where the model runs, not a guarantee that nothing leaves. The useful question about any product is narrower: which parts touch a network, when, and what happens when the network is gone. Running the model locally makes the strong version possible; it does not by itself prove it.
Most products are a mix, and the label hides it
Many assistants split the work. Speech recognition may run locally while the answer is composed on a server. A small local model may handle short requests and pass the hard ones to a larger remote one. A feature may work offline until it needs current information. All of that is reasonable engineering, and all of it is sold under the same phrase.
To see whether a feature needs a live connection, disconnect from every network and try it. If it still works, it may be using local processing or cached material. If it fails or changes, it may depend on a connection; the test alone does not show where model computation happens.
What changes when the model is local
Local inference avoids a cloud round trip for the model reply, but response time depends on hardware, model, and workload. It can work without a signal and avoids a remote service’s per-request inference charge. The app may still require an account or connection for other features, and long-term operation depends on compatible software and hardware.
That trade is one of the engineering questions behind useful, efficient technology: how capability can move closer to a person while respecting real limits. The principles apply across devices and uses; no single model size or product design answers them all.
On-device means the model runs on your hardware; which parts of the app still use a network is a separate question worth asking.