Do you need a big AI model? A one-hour test that tells you
The biggest model is a habit, not a requirement. How to tell which of your automation jobs a small, free, on-device model can handle, with a simple test on your own inputs that takes about an hour.
Most of us choose an AI model the same way: the strongest one we have access to. It is a reasonable habit when you chat with an assistant. In an automation that runs every day, it means paying for, and waiting on, far more model than the job needs.
The opposite mistake is just as easy: someone tells you small models handle “most” tasks, you switch, and your automation quietly gets things wrong. Nobody can give you a percentage for your work. What you can do is measure it, in about an hour. This post shows how.
The job decides, not the model
Start by looking at the shape of each AI step in your automation. Some jobs are narrow: the input is short, the possible answers are known, and a right answer is easy to recognise. Others are open: the model has to plan, reason over a long document or choose between tools.
Here is a starting point for common automation steps. It is our rule of thumb, not a measurement, and the test below is how you replace it with facts:
AI step
Shape
Start with
Sort messages into a few categories
Narrow
A small on-device model
Pull fields (name, price, date) out of a page
Narrow
A small on-device model
Summarise a page in a few bullets
Mostly narrow
A small model, check longer pages
Rewrite text in another tone
In between
Small first, step up if it reads badly
Plan several steps or choose between tools
Open
A larger or cloud model
Reason over a very long document
Open, and limited by context
A model with a long context window
The last row has a trap worth knowing. Small on-device models often come with a short context window, the amount of text they can read at once. WebLLM even lists variants with a 1k-token window, about a page of text. A long article does not fit, whatever the model’s quality.
If your answer is one of the first three, keep reading: those are the jobs where a small model has the best chance. The one-hour test
The test compares models on your inputs, against answers you decided in advance. Here is the loop:
Start small
Take the smallest model your device can run comfortably, and run your saved inputs through it.
Step up only if you must
If too many answers are wrong, try the next size up on exactly the same inputs. Nothing else changes. Keep the smallest that passes
The first model to pass is your answer. You now know, instead of hoping.
Three details make the test trustworthy:
Save ten real inputs. Ten emails, pages or rows from your actual work, including two or three awkward ones.
Write the right answer first. Decide what correct looks like before you run anything, so a confident wrong answer cannot convince you.
Score each answer. Right or wrong, nothing in between. Then count.
The scoring step looks like this. Your expected answers sit next to what the model said:
Email subject
Expected
Charged twice this month
Billing
Can I add a second user?
Sales
Password reset missing
Support
Invoice for May, please
Billing
Small modelAI
Email subject
Expected
Model said
Charged twice this month
Billing
Billing ✓
Can I add a second user?
Sales
Sales ✓
Password reset missing
Support
Billing ✗
Invoice for May, please
Billing
Billing ✓
Score them. Here: 3 of 4 right. Not yet good enough.Accuracy against your expected labels. Example data.Your inputs with the answers you expect.A labelled evaluation set, written before the run.The model answers each one.One call per item, same prompt every time.Score them. Here: 3 of 4 right. Not yet good enough.Accuracy against your expected labels. Example data.
Data through a nodeText version
Email subject | Expected: Charged twice this month | Billing · Can I add a second user? | Sales · Password reset missing | Support · Invoice for May, please | Billing
Small model
Email subject | Expected | Model said: Charged twice this month | Billing | Billing ✓ · Can I add a second user? | Sales | Sales ✓ · Password reset missing | Support | Billing ✗ · Invoice for May, please | Billing | Billing ✓
Correctness is half the story. A model that is right but slow may not suit a workflow that runs on every page you open. So time it too, on your own machine.
Run it in AWFlow
AWFlow’s Local AI page can put up to three models of the same task side by side and measure them on your device: how long each takes to be ready and, where the engine supports it, how many tokens per second it writes.
For correctness, put the model into a real workflow and run your ten inputs through it. The workflow below is a good harness: the prompt lives in one step and the model in a slot underneath it, so you swap models without touching anything else. Replace the prompt with your own job, such as “Answer with one word: Billing, Sales or Support.”
If even your largest on-device model fails the test, you have learned something useful too: that step deserves a cloud model. The same slot takes Chat OpenAI, Chat Anthropic or Chat Google with your own key. We cover the costs of that choice in Free AI vs API-key AI.