“On-device AI saves battery” and “cloud AI is more efficient” can both sound plausible. Neither is a measurement.
The placement decision crosses at least four different budgets:
user wait + network transfer + provider spend + device energy
Do not collapse them into one vague “cost” number. Measure each with its own unit and evidence boundary.
I reviewed MonkeyCode mobile code at commit c58bcd4. The
That reviewed path is not evidence of on-device model inference. So a fair current study would measure a mobile client using remote task and voice services. An on-device alternative would be a separate prototype with its model, runtime, and packaging declared.
The included CSV template begins with these fields:
sample_id,sample_kind,placement,device,os,framework,model,network,input_tokens,output_tokens,latency_ms,bytes_up,bytes_down,energy_joules,cost_usd
Why so many?
device
, os
, and framework
make thermal and runtime results interpretable;model
and token counts keep workload size visible;network
separates offline, Wi-Fi, and cellular behavior;sample_kind
prevents synthetic examples from masquerading as device measurements.Battery percentage is too coarse for short runs. It is affected by display, radio, background work, battery health, temperature, and OS estimation. If you cannot collect energy with an appropriate platform profiler or external power measurement, leave energy_joules
empty.
Compare the same tasks, not unrelated model demos:
| Flow | Cloud case | On-device case |
|---|---|---|
| Short prompt | Same input and output cap | Same semantic task and cap |
| Voice turn | Same audio fixture | Same audio fixture |
| Offline | Expected failure or queued action | Local completion if supported |
| Background/resume | Declared lifecycle | Declared lifecycle |
| Thermal loop | Repeated fixed workload | Repeated fixed workload |
Warm-up should be reported separately. Randomize case order, control display state and temperature, repeat enough times to show a distribution, and record failures rather than deleting them.
The companion synthetic-samples.csv
is explicitly labeled synthetic. Its numbers exist only to test parsing and summary behavior.
Run:
node analyze-costs.mjs synthetic-samples.csv
node test-analysis.mjs
The report includes:
{
"dataset": "synthetic",
"samples": 3,
"mean_latency_ms": 987,
"total_network_bytes": 66100,
"total_cost_usd": 0.0084,
"energy_conclusion": "REFUSED: energy requires measured joules for every row"
}
Those latency, transfer, and cost values are synthetic test output, not a MonkeyCode benchmark, a phone benchmark, or evidence that one placement wins.
The test then supplies measured-labeled rows with joule values and confirms that an energy summary becomes possible. In a real pipeline, provenance should be stronger than a label: ingest profiler exports, preserve raw files, and attach the collection command and timestamp.
A release decision can now be explicit:
An on-device model can remove provider calls while adding download size, RAM pressure, thermal load, and model-update work. A cloud model can reduce device compute while adding radio use, service dependency, and data transfer. Units keep those tradeoffs honest.
Disclosure: I contribute to the MonkeyCode project. The product observations are source-based and limited to the linked paths at commit
c58bcd4
. All included sample data is synthetic; no device energy, latency, or cost benchmark is claimed.