All posts Research / 001 November 9, 2026 · By Groovy
actually cost?
Particle vs SOL. The same Groovy tools. Ninety trials in coding and data science. A closer look at what each model could finish.
Explore the results passes · $2.76 provider cost
passes · $2.71 provider cost
passes · $9.64 provider cost
15 scenarios × 2 attempts × 3 configurations. Costs normalized to peak rates; failed trials included.
A model can be cheaper per token and still be expensive to use. If it breaks code that already worked, someone has to find the problem, fix it, and run the task again.
We wanted to know where that tradeoff shows up. So we put Particle-1 and GPT-6.1 SOL inside the same Groovy execution system, gave them the same tasks, and checked the work they produced.
Particle High and SOL each passed 28 of 30 trials. Particle Max passed 26. Both Particle configurations cost 71–72% less at peak provider rates. High and SOL also passed all four continued-coding trials; Max passed three. These results measure the behavior checked in this study, with the evaluation rules explained below.
This is a small, controlled study. It helps us decide what to test next. It is not a universal ranking of the models.
01 / Task performance
The task matters more than #
the headline score.
A pass means the runner recorded a final response and the submitted work passed all checks under our evaluation rules. Each count includes two attempts per scenario. See the method for accepted equivalent outputs and the storage contract.
High Particle-1
Max GPT-6.1 SOL
xhigh
The runner recorded a final response in all 90 trials; submitted work passed the evaluation in 82. A final response alone does not establish that the work is complete.
02 / Cost
Particle cost 71–72% less. #
Failed attempts included.
At normalized peak rates, each Particle configuration cost less than $2.76 to run all 30 trials. SOL cost $9.64. That comparison includes the attempts that failed. Every cost-per-pass and shared-pass comparison uses the same evaluation as the headline results.
We also compared only trials that both models passed. Particle High cost $2.68 versus SOL’s $9.44 across 28 shared passes. Particle Max cost $2.37 versus SOL’s $7.61 across 26 shared passes. The High and Max comparisons cover different sets of trials.
Particle’s runs happened during its half-price off-peak window. Estimated spend in that window was $1.38 for High and $1.36 for Max. We use peak rates in the charts so the time of day does not create an extra advantage.
These are provider-cost estimates from recorded usage and published rates on November 8–9, 2026, not reconciled invoices. They exclude Groovy’s platform fee, subscriptions, and local compute.
51% / 61% / 88% Cached share of input tokens
Particle High / Max / SOL
SOL reused more cached input and made fewer model calls. Particle still cost less at these rates. Token counts alone would have missed that.
03 / Data science
High and SOL passed 14 of 16. #
Max passed 13.
Every configuration passed our financial reconciliation and forecasting tasks. Forecast errors were very close: mean absolute error ranged from 0.522 to 0.530 on the same unseen synthetic data, against a pass threshold of 3.5.
For the public DS-1000 subset, we checked numeric results and cluster membership. Integer and floating-point counts with the same values are accepted, as are identical clusters with different label numbers. Particle High and SOL each passed 14 of 16 trials; Particle Max passed 13. One Max result produced different groups. All three configurations failed both attempts of a differential-equation task where the prompt and reference encouraged different interpretations.
These are results on eight selected public problems, not an official DS-1000 leaderboard score. Passing the checked fixtures does not prove correctness on every unseen input.
04 / Continued coding
High and SOL: four passes. #
Max: three.
We asked each model to extend two small projects over several stages. Every new stage had to preserve the earlier requirements. Four trials per model is a small sample, and one failed check makes an entire trial fail.
A persistent ledger
- Store transactions in SQLite
- Import CSV atomically
- Add categories and safe upgrades
- Add batch operations and a CLI
A durable job queue
- Claim work with leases and fencing
- Retry and cancel jobs
- Handle competing workers and batches
High and SOL passed all four continued-coding trials. Max passed three. The checks accept verified trimming for inputs described as trimmed strings, and exercise each queue worker with the storage implementation it supplies, including error handling. Compatibility with a different asynchronous storage adapter is outside this score.
One Max ledger run left real gaps. A stage-three edit changed a return value without updating its caller and removed a constant still in use. Stage four repaired some basic behavior but left category totals, batch operations, and the CLI unfinished.
In stages three and four of that run, Particle returned an unfinished plan with a stop signal and no tool call. The runner accepted it as the end of the stage. Both responses were below the output limit, and the recorded history was intact. We have not tested whether another continuation would recover the work.
These projects took minutes, with SOL’s successful continued-coding runs taking roughly 15–19 minutes each. This study does not establish reliability over days of work in a large codebase.
| Configuration | Median trial | All 30 trials | Model calls |
|---|---|---|---|
| Particle-1 · High | 89 sec | 54.1 min | 480 |
| Particle-1 · Max | 105 sec | 74.3 min | 458 |
| GPT-6.1 SOL · xhigh | 56 sec | 112.3 min | 323 |
The total is the sum of trial wall times. SOL had the fastest median, but spent longer on the continued-coding projects. Max used 37% more time overall and 31% more output tokens than High, with fewer passes. More effort did not guarantee a better result on these tasks.
05 / How Groovy runs the work
The model decides. #
Programs do the computation.
- 01Plan the next step The model reads the question and chooses a tool or writes a program.
- 02Execute the program Code reads files, computes results, or changes the project in an isolated environment.
- 03Keep the evidence Tool results and execution history survive between steps. Large results stay available by reference.
- 04Inspect and continue The model reads the relevant result, checks its work, and decides what to do next.
Both models had the same terminal, program-execution, and result-reading tools. No separate coding agent was delegated the work. This lets us compare models inside Groovy, while the provider adapters still differ: Particle used a compatible Chat Completions interface and SOL used Responses.
What we would do with these results
Choose a model for the work #
you can verify.
Particle High matched SOL’s pass count in this sample at substantially lower provider cost. Max took longer and passed two fewer trials than High. That makes High a useful candidate for further coding and data-analysis evaluations with clear checks; it does not establish a general ranking.
The next study should use more projects, fix the evaluation rules in advance, and measure recovery from unfinished final responses. Routing to a cheaper model and escalating when checks fail remains a separate experiment. We cannot claim its savings or accuracy from this study.
06 / Method & limits
What this benchmark can tell us. #
- Models
particle-1with explicit High and Max effort;gpt-6.1-solwith xhigh effort on Standard service. Effort names do not imply equal compute budgets.- Sample
- 15 scenarios, two attempts per configuration, 90 total trials and 120 stages. The attempts are repeated observations of the same scenarios, not 90 independent tasks.
- Checks
- Independent checks evaluated submitted artifacts. Models did not receive hidden grading feedback. Later coding stages checked earlier requirements. Data analysis included unseen fixtures. All charts, totals, and costs per pass use the same evaluation basis.
- Scoring basis
- This is an exploratory evaluation of saved program outputs, with no new model runs. The evaluation rules were selected after inspecting outputs and applied uniformly across configurations. Numerically equal counts, equivalent cluster membership, and verified trimming of inputs described as trimmed strings are accepted. Queue workers are evaluated with their own storage implementation.
- Storage scope
- Two Particle queue implementations do not support substituting asynchronous storage methods. That stronger interchangeability requirement is outside the reported score. Passing here does not establish support for every storage adapter.
- Public tasks
- Eight problems from DS-1000 at a pinned revision : NumPy 443/469, pandas 79/284, SciPy 774/789, and scikit-learn 890/885. Selected before inference at the 25th/75th percentiles of reference-code AST size per library. That measures code size, not calibrated difficulty. Dataset: CC BY-SA 4.0.
- Limits
- Public tasks may have appeared in training. The sample is too small for a general model ranking. These exploratory scores are neither a preregistered benchmark result nor an official DS-1000 score. They describe the checked fixtures, not correctness on all possible inputs.
- Execution
- Isolated local environments using public or synthetic data. No production customer data. The same runtime version was used within each matched comparison. Nine single-stage trials used a different runtime revision from the other 81; matched comparisons used the same revision. Limits were 30 minutes and 80 transitions per stage. No primary trial was stopped by those limits or the budget.
- Caching & cost
- Cache state was not reset; excluded attempts may have warmed it. This is not a cold-cache comparison. Recorded cached input and reasoning tokens were not double-counted. Estimated total provider spend, including excluded setup and invalidated attempts, was $17.15 against a $30 budget.
- Comparison scope
- This study compares models inside Groovy. Earlier single-task Codex and Claude Code pilot runs were not repeated across this suite and are not included in these charts.
Inspect the chart data
Exact aggregate values, model settings, and scoring notes.
Pricing sources and calculation #
Peak USD per million tokens: Particle fresh input $0.25, cached input $0.03, output $1.00; SOL Standard fresh input $2.00, cached input $0.10, cache writes $2.50, output $10.00. Particle’s off-peak rates were half its peak rates. No request reached SOL’s long-context pricing threshold. Rates were recorded for this study; they may change.
Back to the Groovy blogResearch, with the tradeoffs included.