Do you need a big AI model? A one-hour test that tells you

A one-hour evaluation method can determine whether an automation job needs a large cloud AI model or a small on-device model, according to a guide that recommends testing the smallest model on ten saved real inputs with pre-written expected answers and stepping up in size only when answers are wrong. The guide's rule of thumb assigns narrow tasks such as sorting messages into categories and extracting fields like name, price and date to small on-device models, while open tasks such as multi-step planning or tool selection call for larger or cloud models. It warns that small on-device models often ship with short context windows — WebLLM lists variants with a 1k-token window, about a page of text — so long documents may not fit regardless of model quality.

Do you need a big AI model? A one-hour test that tells you The biggest model is a habit, not a requirement. How to tell which of your automation jobs a small, free, on-device model can handle, with a simple test on your own inputs that takes about an hour. Most of us choose an AI model the same way: the strongest one we have access to. It is a reasonable habit when you chat with an assistant. In an automation that runs every day, it means paying for, and waiting on, far more model than the job needs. The opposite mistake is just as easy: someone tells you small models handle “most” tasks, you switch, and your automation quietly gets things wrong. Nobody can give you a percentage for your work. What you can do is measure it, in about an hour. This post shows how. The job decides, not the model Start by looking at the shape of each AI step in your automation. Some jobs are narrow: the input is short, the possible answers are known, and a right answer is easy to recognise. Others are open: the model has to plan, reason over a long document or choose between tools. Here is a starting point for common automation steps. It is our rule of thumb, not a measurement, and the test below is how you replace it with facts: AI step Shape Start with Sort messages into a few categories Narrow A small on-device model Pull fields name, price, date out of a page Narrow A small on-device model Summarise a page in a few bullets Mostly narrow A small model, check longer pages Rewrite text in another tone In between Small first, step up if it reads badly Plan several steps or choose between tools Open A larger or cloud model Reason over a very long document Open, and limited by context A model with a long context window The last row has a trap worth knowing. Small on-device models often come with a short context window, the amount of text they can read at once. WebLLM even lists variants with a 1k-token window, about a page of text. A long article does not fit, whatever the model’s quality. If your answer is one of the first three, keep reading: those are the jobs where a small model has the best chance. The one-hour test The test compares models on your inputs, against answers you decided in advance. Here is the loop: Start small Take the smallest model your device can run comfortably, and run your saved inputs through it. Step up only if you must If too many answers are wrong, try the next size up on exactly the same inputs. Nothing else changes. Keep the smallest that passes The first model to pass is your answer. You now know, instead of hoping. Three details make the test trustworthy: Save ten real inputs. Ten emails, pages or rows from your actual work, including two or three awkward ones. Write the right answer first. Decide what correct looks like before you run anything, so a confident wrong answer cannot convince you. Score each answer. Right or wrong, nothing in between. Then count. The scoring step looks like this. Your expected answers sit next to what the model said: Email subject Expected Charged twice this month Billing Can I add a second user? Sales Password reset missing Support Invoice for May, please Billing Small modelAI Email subject Expected Model said Charged twice this month Billing Billing ✓ Can I add a second user? Sales Sales ✓ Password reset missing Support Billing ✗ Invoice for May, please Billing Billing ✓ Score them. Here: 3 of 4 right. Not yet good enough.Accuracy against your expected labels. Example data.Your inputs with the answers you expect.A labelled evaluation set, written before the run.The model answers each one.One call per item, same prompt every time.Score them. Here: 3 of 4 right. Not yet good enough.Accuracy against your expected labels. Example data. Data through a nodeText version Email subject | Expected: Charged twice this month | Billing · Can I add a second user? | Sales · Password reset missing | Support · Invoice for May, please | Billing Small model Email subject | Expected | Model said: Charged twice this month | Billing | Billing ✓ · Can I add a second user? | Sales | Sales ✓ · Password reset missing | Support | Billing ✗ · Invoice for May, please | Billing | Billing ✓ Correctness is half the story. A model that is right but slow may not suit a workflow that runs on every page you open. So time it too, on your own machine. Run it in AWFlow AWFlow’s Local AI page can put up to three models of the same task side by side and measure them on your device: how long each takes to be ready and, where the engine supports it, how many tokens per second it writes. For correctness, put the model into a real workflow and run your ten inputs through it. The workflow below is a good harness: the prompt lives in one step and the model in a slot underneath it, so you swap models without touching anything else. Replace the prompt with your own job, such as “Answer with one word: Billing, Sales or Support.” If even your largest on-device model fails the test, you have learned something useful too: that step deserves a cloud model. The same slot takes Chat OpenAI, Chat Anthropic or Chat Google with your own key. We cover the costs of that choice in Free AI vs API-key AI.