How three OSLabs engineers built PennyWyze—and almost built something completely different.
Hi, I’m Maia. I’m one of three developers behind PennyWyze, an open-source CLI that audits which Claude tier—Opus, Sonnet, or Haiku—is the cheapest one that still passes your quality bar.
You point it at your production prompt and a handful of real examples where you already know the correct answer.
PennyWyze runs those examples against each Claude tier, measures accuracy, calculates the actual API cost, and tells you which tier clears your bar for the lowest cost.
In other words: instead of assuming you need the most expensive model, you can test it.
How PennyWyze Works
You give it two things:
Your prompt — the exact instructions you already send to Claude in production.
A golden dataset — real inputs paired with the answer you already know is correct.
For example:
{"input": "I was charged twice this month", "expected": "billing"}
{"input": "The app crashes on upload", "expected": "technical"}
Then run:
pennywyze audit --prompt prompt.md --dataset dataset.jsonl --pass-rate 90 PennyWyze sends every example to Opus, Sonnet, and Haiku using your real prompt and the real Anthropic API.
It grades each response against your expected answer and calculates cost using the token counts returned by the API—not an estimate.
It then produces a report showing each model's accuracy and projected monthly cost at your actual volume.
A real audit
Here's what one run looked like:
✓ opus audited — 50 questions
✓ sonnet audited — 50 questions
✓ haiku audited — 50 questions
PENNYWYZE AUDIT REPORT
┌───────────────────────────┬────────────┬────────────────┐
│ MODEL │ ACCURACY │ EST. COST / MO │
├───────────────────────────┼────────────┼────────────────┤
│ claude-opus-5 │ 49/50 PASS │ $205.94 / mo │
│ claude-sonnet-5 │ 48/50 PASS │ $77.30 / mo │
│ claude-haiku-4-5-20251001 │ 49/50 PASS │ $26.26 / mo │ └───────────────────────────┴────────────┴────────────────┘
VERDICT Switch to claude-haiku-4-5-20251001 — save ~$179.68/mo. ℹ Audit cost: $0.15
Opus and Haiku tied at 49/50.
Sonnet actually scored lower than both while costing nearly three times as much as Haiku.
On this particular task, there wasn't an accuracy difference to justify the additional cost.
And finding that out cost $0.15.
Why We Built It
There are already plenty of tools that try to answer which AI model is "best."
That's not the question we wanted to answer.
We wanted to answer:
Which model is cheapest for my prompt while still being good enough for my application?
Those are different questions.
A model that performs well on a general benchmark isn't necessarily the cheapest model that will reliably handle your specific production task.
PennyWyze uses your own examples to define what "correct" means.
That makes the verdict specific to your application instead of based on a generic benchmark.
The Problem With Always Using the Biggest Model
When you're building an AI feature for the first time, defaulting to the most capable model makes sense.
You want to know that your feature works.
The problem is that it's easy to never revisit that decision.
A lot of production AI work isn't open-ended reasoning. It's things like:
Classifying support tickets
Extracting fields
Routing messages
Detecting intent
Categorizing requests
For those tasks, you may not need the most expensive model. But checking requires building an evaluation harness: a dataset, a grading system, and a way to run the same task against multiple models.
Most teams don't want to build all of that just to answer one cost question.
So they don't check.
PennyWyze is that evaluation harness.
One Important Limitation
PennyWyze currently uses exact-match grading.
That means it works best for tasks where there is one correct answer:
Classification
Extraction
Routing
It isn't designed yet for open-ended generation such as drafting an email or summarizing a document.
LLM-as-a-judge grading is on the roadmap. Why Our Grading Is Strict
We wanted PennyWyze to catch a specific production problem: a model can technically know the right answer while still failing to follow the output format your application requires.
Before grading, both the model response and expected answer are normalized.
We remove things like:
Surrounding quotes
Code fences
Capitalization differences
Trailing punctuation
Then we compare the results for exact equality.
So:
Billing
passes when the expected answer is:
billing
But:
I think the answer is billing
doesn't.
That's intentional.
If your application expects a bare category name and the model wraps it in a sentence, that's a real production failure—not something we want to hide with a fuzzy grading system. Getting Started
Install PennyWyze globally:
npm install -g pennywyze
Then add your Anthropic API key to a .env file, create a prompt and golden dataset, and run your first audit.
You can start testing against real models in minutes.
The Part We Didn't Expect
The interesting thing about PennyWyze is that it wasn't our original idea.
Every OSLabs team pitches multiple product ideas before settling on a few.
We initially started with a completely different problem: conversation memory.
The idea was to stop AI chat applications from repeatedly sending their entire conversation history with every message.
What we actually cared about was the broader problem underneath it:
token efficiency.
But as we explored the idea, we kept running into projects that were already solving the problems we were trying to solve. Mem0, Zep, and Letta were already working on conversation memory, and Anthropic had started shipping automatic context compaction.
We eventually left one evening without an answer.
The next day, Olivia came back with a different idea:
Instead of trying to reduce tokens by changing how conversations were stored, what if we looked at which model you were using in the first place?
That became PennyWyze.
What We Actually Learned Building It
The most useful lessons weren't the ones we expected to learn.
When we first connected the real Anthropic provider, we assumed the model's text response would be the first item in the API response's content array.
That assumption worked for most models.
Then Opus's adaptive thinking broke it.
The response can contain a thinking block before the text block, which meant we were sometimes grading an internal reasoning fragment instead of the actual answer.
The fix was simple:
Instead of assuming position zero, we search for the response block where:
type === "text" The difficult part wasn't the fix.
It was noticing that our results were subtly wrong specifically for Opus and tracing the problem back to the response structure.
Our normalization logic removes code fences, lowercases the response, removes punctuation, and strips surrounding quotes.
At one point, those operations happened in the wrong order.
A response like:
"billing".
could leave a dangling quote after punctuation was removed.
The answer was semantically correct, but our grader marked it wrong.
We changed the order of operations and added a regression test for that exact case.
PennyWyze can stop auditing a model once it mathematically cannot reach the requested pass rate.
The basic calculation is:
allowedFailures = floor(datasetLength × (1 - passBar)) But at one point, we were converting the pass rate from a percentage to a fraction twice.
A 90% pass rate was effectively being divided by 100 again.
The feature existed.
The feature just almost never triggered.
It took several commits to notice that the math didn't add up and make sure the conversion happened exactly once.
This was probably the question that worried us most.
Adaptive thinking means the same prompt can produce different output token counts between runs.
That means the cost projection can move.
We ran the same real audit against the real API five consecutive times.
The dollar figures moved by a few percent, as expected from the variation in token counts.
But the accuracy scores and the resulting model choice stayed the same across all five runs.
That gave us confidence that PennyWyze's core result wasn't simply an artifact of one API run.
Under the Hood
For anyone interested in the architecture, the audit loop is built around two small, swappable contracts: ModelProvider
Anything that can take a prompt and question and return an answer plus the tokens it cost.
The real Anthropic provider implements this interface.
Adding another provider means implementing the contract rather than rewriting the audit loop.
Scorer
Anything that can grade a model's answer against the expected answer and return true or false.
Exact-match scoring is the only scorer currently included.
But the audit loop doesn't need to know how the scorer makes its decision.
That means adding a future LLM-as-a-judge scorer can happen without rewriting the core audit pipeline.
We also built a fake provider behind a --fake flag.
It runs the same pipeline using canned responses instead of the real API, which lets us develop and test without spending API money on every change.
What's Next?
PennyWyze currently audits Claude models.
Some things we're exploring next:
Cross-provider audits across OpenAI, Google, and Grok
LLM-as-a-judge grading for open-ended tasks pennywyze init to help users build their first golden dataset
A GitHub Action for running audits in CI
Structured JSON output
Shareable HTML reports
More flexible dataset formats
Prompt trimming
The goal is to eventually look beyond model selection.
The cheapest model is only half of the cost question.
The other half is how much you ask it to process.
If You Want to Try It PennyWyze is open source.
Then point it at your own production prompt and a handful of examples where you know the correct answer.
See what it tells you.
GitHub: [https://github.com/oslabs-beta/PennyWyze](https://github.com/oslabs-beta/PennyWyze)
If you try it, I'd genuinely like to know:
What prompt did you audit, and what did PennyWyze tell you?
PennyWyze was built at OSLabs.