The push to run large language models directly on phones and tablets is driven by three hard requirements: latency, privacy, and offline availability. But the physics of mobile hardware creates a ceiling. NPUs and DSPs on flagship SoCs are powerful, yet thermal design power and battery capacity turn long-context inference or multi-turn reasoning into a rapid drain. The practical path forward is not all-edge or all-cloud. It is a tiered architecture where small, quantized models handle sensitive, frequent tasks locally, and a predictable cloud API handles everything else.
To keep power draw under control, the model must fit into device DRAM without constant swapping, and the working set must be small enough to avoid sustained high-frequency memory clocks. For most current mobile hardware, this means targeting models between 1B and 4B parameters, quantized to INT4 or INT8.
Strong candidates include Llama 3.2 1B and 3B, Qwen 2.5 0.5B through 3B, Phi-3 Mini 3.8B, and Gemma 2B and 4B. These architectures use grouped-query attention or multi-query attention, which shrinks the KV cache and reduces memory bandwidth, one of the largest contributors to energy consumption on mobile SoCs.
Use quantization formats that your runtime supports natively. GGUF via llama.cpp is the most common path for rapid prototyping. For production Android apps, ONNX Runtime with INT8 QDQ graphs and Qualcomm QNN delegates lets you execute on the Hexagon NPU. On iOS, Core ML Tools converts models to use the Neural Engine.
from llama_cpp import Llama
llm = Llama(
model_path="./qwen2.5-1.5b-q4_k_m.gguf",
n_ctx=2048,
n_threads=4,
verbose=False
)
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "Summarize this paragraph."}]
)
The choice of runtime determines whether you are burning watts on the CPU or executing efficiently on the NPU or GPU.