{"slug": "we-built-a-cli-to-find-out-if-youre-overpaying-for-claude", "title": "We Built a CLI to Find Out If You’re Overpaying for Claude", "summary": "Three OSLabs engineers built PennyWyze, an open-source CLI that audits whether a production prompt can run on a cheaper Claude tier without dropping below a user-defined accuracy bar. The tool runs a golden dataset against Opus, Sonnet and Haiku via the real Anthropic API, grades responses and computes actual token-based costs; in one 50-question audit Opus and Haiku tied at 49/50 while Sonnet scored 48/50 at nearly three times Haiku's cost, yielding a recommendation to switch to Haiku and save about $179.68 per month. Developer Maia notes the tool currently relies on exact-match grading, limiting it to classification, extraction and routing tasks rather than open-ended generation.", "body_md": "How three OSLabs engineers built PennyWyze—and almost built something completely different.\n\nHi, I’m Maia. I’m one of three developers behind PennyWyze, an open-source CLI that audits which Claude tier—Opus, Sonnet, or Haiku—is the cheapest one that still passes your quality bar.\n\nYou point it at your production prompt and a handful of real examples where you already know the correct answer.\n\nPennyWyze runs those examples against each Claude tier, measures accuracy, calculates the actual API cost, and tells you which tier clears your bar for the lowest cost.\n\nIn other words: instead of assuming you need the most expensive model, you can test it.\n\nHow PennyWyze Works\n\nYou give it two things:\n\nYour prompt — the exact instructions you already send to Claude in production.\n\nA golden dataset — real inputs paired with the answer you already know is correct.\n\nFor example:\n\n{\"input\": \"I was charged twice this month\", \"expected\": \"billing\"}\n\n{\"input\": \"The app crashes on upload\", \"expected\": \"technical\"}\n\nThen run:\n\npennywyze audit --prompt prompt.md --dataset dataset.jsonl --pass-rate 90\n\nPennyWyze sends every example to Opus, Sonnet, and Haiku using your real prompt and the real Anthropic API.\n\nIt grades each response against your expected answer and calculates cost using the token counts returned by the API—not an estimate.\n\nIt then produces a report showing each model's accuracy and projected monthly cost at your actual volume.\n\nA real audit\n\nHere's what one run looked like:\n\n✓ opus audited — 50 questions\n\n✓ sonnet audited — 50 questions\n\n✓ haiku audited — 50 questions\n\nPENNYWYZE AUDIT REPORT\n\n┌───────────────────────────┬────────────┬────────────────┐\n\n│ MODEL                     │  ACCURACY  │ EST. COST / MO │\n\n├───────────────────────────┼────────────┼────────────────┤\n\n│ claude-opus-5             │ 49/50 PASS │   $205.94 / mo │\n\n│ claude-sonnet-5           │ 48/50 PASS │    $77.30 / mo │\n\n│ claude-haiku-4-5-20251001 │ 49/50 PASS │    $26.26 / mo │\n\n└───────────────────────────┴────────────┴────────────────┘\n\nVERDICT  Switch to claude-haiku-4-5-20251001 — save ~$179.68/mo.\n\nℹ Audit cost: $0.15\n\nOpus and Haiku tied at 49/50.\n\nSonnet actually scored lower than both while costing nearly three times as much as Haiku.\n\nOn this particular task, there wasn't an accuracy difference to justify the additional cost.\n\nAnd finding that out cost $0.15.\n\nWhy We Built It\n\nThere are already plenty of tools that try to answer which AI model is \"best.\"\n\nThat's not the question we wanted to answer.\n\nWe wanted to answer:\n\nWhich model is cheapest for my prompt while still being good enough for my application?\n\nThose are different questions.\n\nA model that performs well on a general benchmark isn't necessarily the cheapest model that will reliably handle your specific production task.\n\nPennyWyze uses your own examples to define what \"correct\" means.\n\nThat makes the verdict specific to your application instead of based on a generic benchmark.\n\nThe Problem With Always Using the Biggest Model\n\nWhen you're building an AI feature for the first time, defaulting to the most capable model makes sense.\n\nYou want to know that your feature works.\n\nThe problem is that it's easy to never revisit that decision.\n\nA lot of production AI work isn't open-ended reasoning. It's things like:\n\nClassifying support tickets\n\nExtracting fields\n\nRouting messages\n\nDetecting intent\n\nCategorizing requests\n\nFor those tasks, you may not need the most expensive model.\n\nBut checking requires building an evaluation harness: a dataset, a grading system, and a way to run the same task against multiple models.\n\nMost teams don't want to build all of that just to answer one cost question.\n\nSo they don't check.\n\nPennyWyze is that evaluation harness.\n\nOne Important Limitation\n\nPennyWyze currently uses exact-match grading.\n\nThat means it works best for tasks where there is one correct answer:\n\nClassification\n\nExtraction\n\nRouting\n\nIt isn't designed yet for open-ended generation such as drafting an email or summarizing a document.\n\nLLM-as-a-judge grading is on the roadmap.\n\nWhy Our Grading Is Strict\n\nWe wanted PennyWyze to catch a specific production problem: a model can technically know the right answer while still failing to follow the output format your application requires.\n\nBefore grading, both the model response and expected answer are normalized.\n\nWe remove things like:\n\nSurrounding quotes\n\nCode fences\n\nCapitalization differences\n\nTrailing punctuation\n\nThen we compare the results for exact equality.\n\nSo:\n\nBilling\n\npasses when the expected answer is:\n\nbilling\n\nBut:\n\nI think the answer is billing\n\ndoesn't.\n\nThat's intentional.\n\nIf your application expects a bare category name and the model wraps it in a sentence, that's a real production failure—not something we want to hide with a fuzzy grading system.\n\nGetting Started\n\nInstall PennyWyze globally:\n\nnpm install -g pennywyze\n\nThen add your Anthropic API key to a .env file, create a prompt and golden dataset, and run your first audit.\n\nYou can start testing against real models in minutes.\n\nThe Part We Didn't Expect\n\nThe interesting thing about PennyWyze is that it wasn't our original idea.\n\nEvery OSLabs team pitches multiple product ideas before settling on a few.\n\nWe initially started with a completely different problem: conversation memory.\n\nThe idea was to stop AI chat applications from repeatedly sending their entire conversation history with every message.\n\nWhat we actually cared about was the broader problem underneath it:\n\ntoken efficiency.\n\nBut as we explored the idea, we kept running into projects that were already solving the problems we were trying to solve. Mem0, Zep, and Letta were already working on conversation memory, and Anthropic had started shipping automatic context compaction.\n\nWe eventually left one evening without an answer.\n\nThe next day, Olivia came back with a different idea:\n\nInstead of trying to reduce tokens by changing how conversations were stored, what if we looked at which model you were using in the first place?\n\nThat became PennyWyze.\n\nWhat We Actually Learned Building It\n\nThe most useful lessons weren't the ones we expected to learn.\n\nWhen we first connected the real Anthropic provider, we assumed the model's text response would be the first item in the API response's content array.\n\nThat assumption worked for most models.\n\nThen Opus's adaptive thinking broke it.\n\nThe response can contain a thinking block before the text block, which meant we were sometimes grading an internal reasoning fragment instead of the actual answer.\n\nThe fix was simple:\n\nInstead of assuming position zero, we search for the response block where:\n\ntype === \"text\"\n\nThe difficult part wasn't the fix.\n\nIt was noticing that our results were subtly wrong specifically for Opus and tracing the problem back to the response structure.\n\nOur normalization logic removes code fences, lowercases the response, removes punctuation, and strips surrounding quotes.\n\nAt one point, those operations happened in the wrong order.\n\nA response like:\n\n\"billing\".\n\ncould leave a dangling quote after punctuation was removed.\n\nThe answer was semantically correct, but our grader marked it wrong.\n\nWe changed the order of operations and added a regression test for that exact case.\n\nPennyWyze can stop auditing a model once it mathematically cannot reach the requested pass rate.\n\nThe basic calculation is:\n\nallowedFailures = floor(datasetLength × (1 - passBar))\n\nBut at one point, we were converting the pass rate from a percentage to a fraction twice.\n\nA 90% pass rate was effectively being divided by 100 again.\n\nThe feature existed.\n\nThe feature just almost never triggered.\n\nIt took several commits to notice that the math didn't add up and make sure the conversion happened exactly once.\n\nThis was probably the question that worried us most.\n\nAdaptive thinking means the same prompt can produce different output token counts between runs.\n\nThat means the cost projection can move.\n\nWe ran the same real audit against the real API five consecutive times.\n\nThe dollar figures moved by a few percent, as expected from the variation in token counts.\n\nBut the accuracy scores and the resulting model choice stayed the same across all five runs.\n\nThat gave us confidence that PennyWyze's core result wasn't simply an artifact of one API run.\n\nUnder the Hood\n\nFor anyone interested in the architecture, the audit loop is built around two small, swappable contracts:\n\nModelProvider\n\nAnything that can take a prompt and question and return an answer plus the tokens it cost.\n\nThe real Anthropic provider implements this interface.\n\nAdding another provider means implementing the contract rather than rewriting the audit loop.\n\nScorer\n\nAnything that can grade a model's answer against the expected answer and return true or false.\n\nExact-match scoring is the only scorer currently included.\n\nBut the audit loop doesn't need to know how the scorer makes its decision.\n\nThat means adding a future LLM-as-a-judge scorer can happen without rewriting the core audit pipeline.\n\nWe also built a fake provider behind a --fake flag.\n\nIt runs the same pipeline using canned responses instead of the real API, which lets us develop and test without spending API money on every change.\n\nWhat's Next?\n\nPennyWyze currently audits Claude models.\n\nSome things we're exploring next:\n\nCross-provider audits across OpenAI, Google, and Grok\n\nLLM-as-a-judge grading for open-ended tasks\n\npennywyze init to help users build their first golden dataset\n\nA GitHub Action for running audits in CI\n\nStructured JSON output\n\nShareable HTML reports\n\nMore flexible dataset formats\n\nPrompt trimming\n\nThe goal is to eventually look beyond model selection.\n\nThe cheapest model is only half of the cost question.\n\nThe other half is how much you ask it to process.\n\nIf You Want to Try It\n\nPennyWyze is open source.\n\nThen point it at your own production prompt and a handful of examples where you know the correct answer.\n\nSee what it tells you.\n\nGitHub: [https://github.com/oslabs-beta/PennyWyze](https://github.com/oslabs-beta/PennyWyze)\n\nIf you try it, I'd genuinely like to know:\n\nWhat prompt did you audit, and what did PennyWyze tell you?\n\nPennyWyze was built at OSLabs.", "url": "https://wpnews.pro/news/we-built-a-cli-to-find-out-if-youre-overpaying-for-claude", "canonical_source": "https://dev.to/oliviamckelvey/we-built-a-cli-to-find-out-if-youre-overpaying-for-claude-1p26", "published_at": "2026-09-24 02:35:56+00:00", "updated_at": "2026-09-24 02:59:22.115318+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "mlops", "developer-tools"], "entities": ["PennyWyze", "OSLabs", "Anthropic", "Claude", "Opus", "Sonnet", "Haiku", "Maia"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-built-a-cli-to-find-out-if-youre-overpaying-for-claude", "markdown": "https://wpnews.pro/news/we-built-a-cli-to-find-out-if-youre-overpaying-for-claude.md", "text": "https://wpnews.pro/news/we-built-a-cli-to-find-out-if-youre-overpaying-for-claude.txt", "jsonld": "https://wpnews.pro/news/we-built-a-cli-to-find-out-if-youre-overpaying-for-claude.jsonld"}}