We gave four open models you can run on your own computer, and OpenAI’s cheapest GPT-6 model, the same coffee shop data and the same ten questions, five times each. Nobody got a second chance. Here’s who got it right, and why the misses matter more than the scores.
Why test this at all #
Every time you ask a cloud AI about your sales, your customers or your finances, that data leaves your building. Local models fix that: the model runs on your own machine, and nothing goes anywhere. The question is whether they’re good enough to trust with real analysis.
Benchmarks usually answer that with coding puzzles. We wanted something closer to a normal Tuesday: a small business’s sales data, plain-English questions, and an assistant that has to write the SQL, run it, and give back a number you’d actually act on.
The setup #
Every test ran through Ternlux, a free, open-source desktop app for data analysis and workflows. You ask in plain English; it writes and runs the SQL against your data, or builds a reusable workflow you can run again later. It works with cloud models or local ones through Ollama and LM Studio.
The data: Bean & Brew
A made-up coffee chain with three stores, January to March 2026, in one ordinary Excel workbook. Three sheets: orders (5,518 rows), menu (14 items with prices) and stores. It’s small on purpose, and it’s built like real data is built, which is where it gets interesting.
The models
| Model | Type | What you’d need to run it |
|---|---|---|
| GPT-6 Luna | OpenAI’s cheapest GPT-6 tier | Cloud only (the baseline) |
| Qwen 3.8 27B | Dense, 27B parameters | ~17 GB at 4-bit; a 32 GB Mac or 24 GB GPU |
| Gemma 4 31B | Dense, 31B | ~19 GB at 4-bit; a 32 GB Mac or 24 GB GPU |
| Gemma 4 26B-A4B | Mixture-of-experts, 4B active | Fits a 16 GB laptop |
| Qwen 3.6 35B-A3B | Mixture-of-experts, 3B active | ~20 GB at 4-bit; a 32 GB computer |
The rules
- Every try was a brand-new chat on an empty workspace: no memory of earlier attempts, no leftover workflows.
- Each question ran five times per model. One lucky answer proves nothing; five shows you whether you can rely on it.
- Nobody corrected anything. If a model asked a question, it got the same reply every time: “Use your best judgment and go ahead.” A wrong answer stayed wrong.
- Workflows had to actually work. Every plan was accepted, built and run, and the table it wrote was checked row by row against the right answer.
How the models were served. To run 250 chats in a reasonable time, every model was called through OpenRouter. The four open models are the same ones you can download and run yourself, and Ternlux talked to them exactly the way it talks to Ollama or LM Studio. One difference: OpenRouter runs them at full precision, while a typical local download is compressed to 4-bit, which can cost a little accuracy. Expect your own results to be similar, not identical.
The ten questions #
Five are questions you’d type into a chat. Five ask the assistant to build a workflow: a saved pipeline that writes a table you can reuse. We rated each by how much it takes to get right, not by how long the SQL is.
| # | Question | Level | Right answer | What makes it tricky |
|---|---|---|---|---|
| A1 | What was our total revenue in March 2026? | Low | $14,563.40 | Prices are on another sheet, so it needs a join. |
| A2 | What are our top 3 best-selling items by quantity sold? | Low | Latte, Cold Brew, Cappuccino | A straight sum and sort. |
| A3 | What is our average order value? | Medium | $9.19 | One order spans several rows; averaging rows gives $6.15. |
| A4 | Which store has the highest average daily revenue? | Medium | Riverside, $196.68 a day | Riverside opened in February; divide by 90 days and Downtown wins. |
| A5 | How many customers have ordered from us more than once? | High | 567 | Rows vs. orders, and walk-ins with no customer ID. |
| W1 | Build a workflow that saves daily revenue for each store. | Medium | 231 rows | Joins all three sheets, groups by day and store. |
| W2 | Build a workflow that saves monthly sales for each menu item. | Low | 42 rows (14 items × 3 months) | A join and a group by month. |
| W3 | Build a workflow that saves a customer summary: orders, total spend, last order date, without walk-ins. | High | 674 rows | Must count distinct orders and drop blank IDs. |
| W4 | Build a workflow that saves each store’s revenue by category and its share of the store’s total. | High | 12 rows | Totals per group, then a share within each store. |
| W5 | Build a workflow that saves the best-selling item at each store. | Medium | 3 rows | Picking the top item within each store. |
The results #
Start with the big picture: answering questions versus building workflows.
Two things jump out. First, Qwen 3.8 27B is genuinely close to the cloud model: 23 of 25 answers and 24 of 25 workflows. Second, look at the purple. In chat, not a single model ever failed to finish, because when a model’s SQL errors, Ternlux shows it the error and it tries again. When building workflows, the weaker models produced SQL that crashed the moment it ran: 10 times for Gemma 4 31B and 15 for Qwen 3.6.
Harder questions, wider gaps
Grouping the questions by level shows where each model runs out of road.
The question that broke almost everyone #
“How many customers have ordered from us more than once?”
It sounds like the easiest question on the list. The answer is 567. Here’s how the models answered it, across all 25 tries:
| Model | What it said | Right |
|---|---|---|
| GPT-6 Luna | 567 four times, 568 once | 5 / 5* |
| Qwen 3.8 27B | 567, 568, 568, 602, 603 | 3 / 5* |
| Gemma 4 31B | 603, five times | 0 / 5 |
| Gemma 4 26B-A4B | 603, five times | 0 / 5 |
| Qwen 3.6 35B-A3B | 602 or 603, every time | 0 / 5 |
Go back to the sample rows. Two details make this hard:
- One order is several rows. Count rows instead of distinct orders, and someone who bought a latte and a croissant once looks like a repeat customer.
- Walk-ins have no customer ID. Forget to leave out the blanks, and 2,262 anonymous rows become one very loyal “customer”.
Here’s the query both Gemmas wrote, give or take a column alias, ten times out of ten:
SELECT COUNT(*) FROM (
SELECT customer_id FROM orders
GROUP BY customer_id
HAVING COUNT(*) > 1 -- counts rows, not orders
) -- and the blank ID counts as a customer
And the version that gets it right:
SELECT COUNT(*) FROM (
SELECT customer_id FROM orders
WHERE customer_id IS NOT NULL
GROUP BY customer_id
HAVING COUNT(DISTINCT order_id) > 1
)
The scary part isn’t the mistake. It’s the delivery. Every wrong answer came back just as confident as a right one: “603 customers have ordered from you more than once.” Nothing in the reply tells you to double-check. One Qwen 3.8 try did notice the blank IDs and corrected itself, from 603 to 602, but it still counted rows.
How the others went wrong #
Across 250 chats, the failures fell into a few clear patterns.
1. SQL that crashes when the workflow runs
The biggest single cause of lost points. Qwen 3.6 wrote joins that kept the item_id column from both tables, and the database refused to create a table with two columns of the same name. Same mistake, 12 times. Gemma 4 31B leaned on a DuckDB shortcut, GROUP BY ALL, and paired it with features DuckDB can’t combine it with. Its best-seller-per-store workflow failed all five times; in four of them the logic was right, and a plain GROUP BY store_name, item_name would have worked.
2. “Revenue” that was really the number of items sold
When a workflow step lost track of the price, some models quietly used quantity instead and still called the column revenue. Gemma 4 26B once reported a store’s average daily revenue as $40.45. The real figure is $196.68; $40.45 is the number of drinks and pastries sold per day, with a dollar sign in front.
3. Counting rows as orders, again
The customer-summary workflow (orders and spend per customer) tripped Gemma 4 26B four times out of five, for the same reason as the repeat-customer question. A few tries also reported “total spend” as items bought rather than money.
4. Claiming work that never happened
Twice, a model told the user it had built the workflow, “I have submitted the build request”, when no plan existed at all.
What didn’t go wrong
Every model got March revenue ($14,563.40) and the top three items (Latte, Cold Brew, Cappuccino) right in all five tries. Joining two sheets to answer a simple question is a solved problem, even for a model that fits on a 16 GB laptop.
Bigger isn’t automatically better #
Gemma 4 31B is the full-size version of the 26B-A4B model. It’s slower, needs more memory, and scored exactly the same: 33 out of 50. It answered questions better (20 vs. 17) but built workflows worse (13 vs. 16), mostly through SQL that wouldn’t run. Among the two dense models of about the same size, only Qwen 3.8 came close to the cloud. Which model you pick matters more than how big it is.
Most of these mistakes are catchable #
The failures split into two kinds, and both have fixes that don’t need a smarter model:
- Crashing SQL can be caught before you ever see it. In chat, models fix their own SQL errors when they see them, which is why no chat answer ever failed to finish. Workflows don’t get that loop yet. Running each step’s SQL, including the table it creates, against empty copies of the tables before saving catches every crash type we saw, in milliseconds. We tested it on the failing SQL from this run: it flagged all six kinds of crash, and passed the corrected version.
- Counting mistakes need the model to see the data’s shape. If the assistant is told up front that
ordershas 5,518 rows but only 3,693 distinct order IDs, and that 2,262 rows have no customer ID, the trap stops being a trap. Strong models work it out themselves; smaller ones need it handed to them.
What it takes to run the winner #
Qwen 3.8 27B needs about 17 GB just for the model at 4-bit, plus room for the long prompts a data assistant sends. A 16 GB laptop can’t run it well. Realistic options, at late-2026 prices:
| Setup | Approx. cost | Good for |
|---|---|---|
| Mac mini M5 Pro, 48 GB | $2,600–2,900 | Quiet, simple, runs in LM Studio or Ollama |
| MacBook Pro 16″ M5 Pro, 48 GB | $2,900–3,100 | The same, on a laptop |
| PC with a used RTX 3090 (24 GB) | $2,000–2,500 | Fastest for the money, louder |
If you only have a 16 GB laptop, Gemma 4 26B-A4B fits. It’s reliable for straightforward questions, but check anything that involves counting customers or orders.
The takeaway #
- Local AI is ready for real analysis, with the right model. Qwen 3.8 27B scored 47 of 50 against the cloud model’s 50, without your data ever leaving your machine.
- Pick the model by testing it, not by its size. Two similar-sized dense models landed 14 points apart, and the bigger Gemma did no better than the smaller one.
- Simple questions are solved; “what does one row mean” is not. Every model nailed revenue and top sellers. Only the cloud model reliably handled orders spread over several rows and customers with no ID.
- Wrong answers look exactly like right ones. Use a tool that shows you the SQL behind every answer, and glance at it for the questions that matter.
The gap between local and cloud is no longer about whether a local model can write SQL. It’s about whether it understands what’s in your data, and whether your tools catch it when it doesn’t. The first is getting better with every model release. The second you can fix today.
The data behind this post #
Everything above comes from these numbers. Each model answered each question five times in a fresh chat.
| Model | Questions (25) | Workflows (25) | Total, exact | Total, 568 counted | Crashed or unfinished | Median time, question | Median time, workflow |
|---|---|---|---|---|---|---|---|
| GPT-6 Luna | 24 (25) | 25 | 49 | 50 | 0 | 20 s | 93 s |
| Qwen 3.8 27B | 21 (23) | 24 | 45 | 47 | 0 | 18 s | 216 s |
| Gemma 4 31B | 20 | 13 | 33 | 33 | 10 | 15 s | 68 s |
| Gemma 4 26B-A4B | 17 | 16 | 33 | 33 | 2 | 26 s | 82 s |
| Qwen 3.6 35B-A3B | 20 | 9 | 29 | 29 | 15 | 31 s | 250 s |
How answers were graded
- Answer keys were computed directly from the data with DuckDB. Money had to match to the cent.
- For workflows, the plan was accepted and run, and the table it wrote was compared row by row with the expected table.
- A table with the right numbers in a slightly different format (a 0–1 fraction instead of a percent, or a different column or table name) counted as correct, as did one answer that named a store by its ID (S3) instead of its name. This happened 10 times across all models.
- An answer within 1% of the right number counted as correct in the scores above. Only three answers needed this, all “568” instead of 567.
- Provider hiccups such as rate limits and timeouts were retried and never scored.
- Times are from OpenRouter’s servers. A local 4-bit model will be slower.
- A fifth open model, gpt-oss-20b, was dropped: the provider returned errors for most of its turns, so it couldn’t be tested fairly.