50,000 problems. 10 mathematical and scientific categories. 100% correctly solved. Zero incorrect answers delivered.
We just completed a major milestone in the development of Pythos Oracle, and I wanted to share what we learned along the way.
But the interesting part isn't the final score.
It's how we got there.
Pythos is a mathematics and physics AI system built around an idea that I believe is important:
Language models should explain mathematics, not be trusted as the mathematical authority.
LLMs are excellent at explaining concepts, interpreting questions, and communicating solutions. But mathematical correctness requires something more dependable than a convincing explanation.
That's why Pythos combines language models with deterministic mathematical engines and independent verification.
We wanted a system that could solve problems, check its work, and refuse to confidently deliver an answer when it couldn't establish correctness.
So we built a testing campaign to put that architecture under pressure.
We call it the Pythos 50K Calibration Campaign.
The campaign consists of 50,000 independently generated problems distributed across ten categories.
| Category | Problems |
|---|---|
| Arithmetic | 5,000 |
| Fractions / Decimals / Percentages | 5,000 |
| Linear Equations | 5,000 |
| Systems of Equations | 5,000 |
| Quadratics / Polynomials | 5,000 |
| Functions / Algebra | 5,000 |
| Geometry / Trigonometry | 5,000 |
| Calculus | 5,000 |
| Probability / Statistics | 5,000 |
| Physics / Math Crossover | 5,000 |
| Total | 50,000 |
Each problem is evaluated against its expected result, with verification and delivery behavior included in the process.
We weren't just interested in whether an engine could produce an answer.
We wanted to know whether the answer was correct and whether the system could establish that correctness.
Our initial run completed the first 15,000 problems almost instantly.
Then things slowed down dramatically.
The culprit wasn't the mathematics. It was the verification infrastructure.
Pythos was mistakenly interpreting intermediate Cramer's Rule determinant expressions as standalone algebraic equations.
That triggered unnecessary Python subprocess creation, introducing enormous overhead.
While investigating that issue, we discovered another defect: an instructional prefix parser was stripping the leading variable from expressions such as:
Solve x^2 = 25
The resulting malformed expression caused unnecessary fallback verification.
We corrected the parsing behavior, improved determinant recognition, and strengthened the single-variable equation invariant.
The next run completed all 50,000 problems in 11.63 seconds.
But we weren't finished.
The initial optimized run reported:
Those 815 withheld problems were all in the physics category.
Rather than accepting the result, we extracted and individually examined every withheld case.
They turned out to be perfectly ordinary kinematics questions.
For example:
"A particle starts from rest with acceleration 19 m/s². What is its velocity after 20 seconds?"
The answer is straightforward:
[
v = at
]
[
v = 19 \times 20 = 380\text{ m/s}
]
The deterministic router simply didn't recognize one particular natural-language sentence structure.
Pythos did the right thing by withholding the answer rather than guessing. But it also revealed a gap in our language-to-mathematics routing.
We added the missing recognition pattern and ran the entire campaign again.
The final run produced the following results:
TOTAL PROBLEMS: 50,000 CORRECTLY SOLVED: 50,000 (100.00%) INCORRECT DELIVERIES: 0 UNKNOWN / WITHHELD: 0 VERIFICATION ESCAPES: 0 FALSE POSITIVES: 0
RUNTIME: 11.40 seconds PEAK NODE HEAP: 148.9 MB ACTIVE CHILD PROCESSES: 0
Every category completed with 5,000 out of 5,000 correct.
The entire campaign completed in approximately eleven seconds.
I'm careful about making claims with results like these.
This doesn't mean Pythos can solve every mathematical problem ever created. It doesn't establish a universal 100% accuracy rate, and a deterministic benchmark isn't the same thing as a live evaluation of an LLM handling arbitrary student questions.
What it does demonstrate is that, across this particular 50,000-problem corpus:
More importantly, the campaign helped us discover and repair actual defects.
That's the part I value most.
We aren't interested in running a benchmark just to produce an impressive number.
The purpose of the Pythos 50K Calibration Campaign is to continuously evaluate and improve the system's recognition, routing, solving, verification, and delivery behavior.
The process is iterative:
A successful run establishes a baseline. Future changes have to earn their place.
This is not the finish line.
Our next challenge is expanding the corpus with more varied natural-language constructions, unfamiliar problem structures, unit conversions, edge cases, and deliberately misleading inputs.
We also want to expand testing around initial velocity, negative acceleration, and more complicated physics scenarios.
The goal is to make Pythos increasingly dependable as an educational tool.
And that's ultimately why we're building it.
Pythos is part of a larger effort to make mathematics and science more accessible, while helping students understand not just what the answer is, but how to get there.
If you're interested in mathematics, physics, AI, or just want to see what we've been building, come give Pythos a try.
**Try Pythos:** [https://pythos.lanzar.me](https://pythos.lanzar.me)
**Explore the source code:** [https://github.com/JonScott79/Pythos](https://github.com/JonScott79/Pythos)
Built under **LANZAR | Launch Bigger.**
And the best part?
No subscription. No paywall to get started. Just mathematics, science, and a little curiosity.