Home-made Evals Taste Better Developer Konstantin Gredeskoul released llm-evals 0.5.0 on RubyGems, a Ruby CLI with two commands, `evals` and `llm-evals`, for running multi-dataset, multi-case evaluation suites that compare models, reasoning efforts and temperatures and rank them by one weighted score balancing accuracy, hallucinations, errors, latency and cost. The gem grew out of the Qualified.At intake-form product, where an LLM extraction step set `filing_status` to `married_jointly` from a lead who said only that they had married in June and had not decided how to file — a guess the form then treated as settled. Gredeskoul argues evals are unit tests for probabilistic software: scoring the same representative cases every time is what makes a prompt tweak, temperature change or model swap measurable. // AI https://kig.re/categories/AI Home-made Evals Taste Better Here is what a real lead typed into a tax-preparation intake form: “Got married in June, haven’t figured out how we’re filing yet. There’s an IRS letter somewhere in a kitchen drawer I haven’t opened. I also deliver groceries on weekends.” Only one fact in that paragraph is clear: the weekend side gig. A model read it anyway and set filing status to married jointly . Helpful. Wrong. The form then skipped the filing-status question and treated a guess as settled. That is the failure I care about most, and it is invisible if you only eyeball a few outputs. I built llm-evals https://github.com/kigster/llm-evals so I would stop writing scripts/eval.rb for the fourth time. It is a Ruby CLI for multi-dataset, multi-case evaluation suites. Compare models, efforts, temperatures. Rank them by one weighted score that balances accuracy, hallucinations, errors, latency, and cost. On RubyGems https://rubygems.org/gems/llm-evals as of writing: 0.5.0. Two commands: evals and llm-evals . gem install llm-evals Home-made pasta tastes better because you chose the flour. Home-made evals taste better because you chose the weights, the cases, and the failure modes that matter to your product. Store-bought leaderboards cannot taste your soup. Evals are unit tests for probabilistic software A unit test asks: given this input, does my code return exactly this output? An eval asks a softer question: given representative inputs, how often and how well does the model produce what I expect? You cannot assert equality against a probability distribution. You can score it. Score the same cases every time, and you can finally measure what a prompt tweak, a temperature change, or a model swap actually did. The scorer decides what “good” means. The eval makes the comparison honest and repeatable. Background I keep recommending: - AI Evals: Everything You Need to Know https://hamel.dev/blog/posts/evals-faq/ by Hamel Husain - Demystifying Evals for AI Agents https://www.anthropic.com/research/demystifying-evals-for-ai-agents from Anthropic - Eval-Driven System Design https://developers.openai.com/cookbook/examples/partners/eval driven system design/receipt inspection from OpenAI The problem that started it: Qualified.At The gem grew out of Qualified.At https://qualified.at , which I wrote about when it launched https://kig.re/2026/07/29/qualified-at-is-live.html . A business builds an intake form, embeds it, and visitors answer questions before anyone picks up the phone. The forms use the Inquirex https://github.com/inquirex gems. Many start with a free-text box: tell us about your situation . An LLM reads that paragraph and extracts structured answers, so a 20- or 30-question form can shrink to a few follow-ups. Here is the extraction step from the tax preparation demo https://qualified.at/demos/tax-preparation , copied from the suite’s dsl.rb : extract :understanding do from :situation prompt "Extract the taxpayer's situation from their description. Only fill a " \ "field when the text genuinely supports it — leave anything the writer " \ "did not mention unset rather than guessing. ..." schema filing status: { type: :enum, values: %w single married jointly married separately head of household }, dependents: :integer, self employed: :boolean, rental properties: :integer, sold investments: :boolean, owns business entity: :boolean, multi state: :boolean, irs letter: :boolean transition to: :heard you end And here is what that lead typed the vague case : Honestly, this year was a blur and I’ve been avoiding thinking about taxes. I got married in June, so I assume that changes things, though we haven’t talked about how we’re filing. I also picked up a side gig doing grocery deliveries through one of those apps on weekends, which I’m told counts as being self-employed. Somewhere in a kitchen drawer there’s a letter from the IRS that came in August that I haven’t been brave enough to open. We rent our apartment, if that matters. I don’t really know what else you need from me, so please just tell me what to bring. The correct extraction is not “married filing jointly”. They got married. They said they have not decided how to file. The right answer is: { "filing status": null, "dependents": null, "self employed": true, "rental properties": null, "sold investments": null, "owns business entity": null, "multi state": null, "irs letter": true } Two answers, six honest null s. A model that “helpfully” fills in married jointly has invented a fact. On this case, openai/gpt-4o-mini scored 78.8% and the diagnostics named the hallucination: expected null, got married jointly . deepseek/deepseek-v4.1-flash-1017 at high effort scored 100% with empty diagnostics. So the product question became: which model captures the most answers, assigns them correctly, avoids hallucinations, and does not miss details , while staying cheap and fast? A lead who waits 30 seconds for the next question is a lead who closes the tab. At 1,000 leads, $0.0002 per call is twenty cents. $0.50 per call is five hundred dollars. The Qualified.At suite Qualified.At ships ten demo businesses: bookkeeping, couples therapy, home remodeling, immigration law, managed IT, personal injury, roofing, software development, tax preparation, and wedding photography. Each one became a dataset. Each dataset has four case variants: | Variant | What the lead wrote | |---|---| | complete | rambles, but answers every field | | vague | hedged and incomplete; about half the fields go unanswered | | partial | clear prose that answers some fields and explicitly defers the rest | | estimate | a complex description aimed at the fields that drive the price estimate | That is 10 datasets × 4 variants = 40 cases on disk: evals/ .llm-evals.config.yml eval helper.rb datasets/09-tax-preparation/ dsl.rb the demo flow, verbatim schema.rb its extract step's schema; evals verify fails on drift cases/vague/input.txt cases/vague/expected.json ... The dsl.rb and schema.rb files are copies of the production flow. A spec fails the moment they drift apart. An eval that quietly tests last month’s schema is worse than no eval. The adapter runs the same prompts production would. From evals/eval helper.rb : LLM::Evals.configure do |c| c.suite "qualified-at-extract", root: dir , env files: "../.env" , description: "Which model turns a lead's free text into the answers an extract step declares." c.cases "datasets/ /cases/ /input.txt" c.adapter QualifiedAt::Evals::Extract::Adapter.new c.scorer Inquirex::LLM::Evals::Scorer.new end Scoring: not all mistakes are equal Every schema field ends in one of five outcomes. Each is worth a fixed share of the point from Inquirex::LLM::Evals::FieldScore : | Outcome | Worth | When | |---|---|---| | correct | 1.0 | what the text says, as the schema spells it | | missed | 0.2 | the text answers it and the reply left it empty | | hallucinated | 0.15 | the text never answers it, and the reply answered anyway | | incorrect | 0.1 | an allowed value, the wrong one | | invalid | 0 | a value the schema does not allow: off the list, wrong type, not JSON | Missing a field is the least bad mistake, because the form just asks the question again. Hallucinating is worse, because the form now believes something the lead never said. Returning garbage is worst. On top of the per-field score , the scorer reports exact every field correct , hallucinated rate , and invalid rate . One weighted score, or: which one do I ship? Here is the thing that bit me. The report had a dozen sortable columns: accuracy, exact match, hallucination rate, errors, latency, cost, tokens. Sort by accuracy and an expensive, slow model wins. Sort by cost and a model that hallucinates wins. Sort by primary score and you crow a token hog. None of those columns answered the question I actually had, which was which one should I ship? So llm-evals lets you say what matters, in YAML, and folds it into one column called wscore . This is the weighted score section from Qualified.At’s .llm-evals.config.yml : weighted score: weights: Positive metrics higher is better exact: 10 Exact contributes high number score: 5 This is a score computed in other ways Negative metrics lower value is better latency: -5 Latency is important, but relatively less than others cost: -5 Same with the cost time: -3 latency contributes relatively low comparatively errors: -8 and any errors contribute quite a bit to the total hallucinated rate: -6 Hallucinations are bad, and are penalized invalid rate: -8 invalid rate is an error, so heavy penalty A positive weight means higher is better. A negative weight means lower is better. Raw weights are hard to reason about, though, so the llm-evals CLI ships a subcommand that does the math for you. I called it weights , which is not my most imaginative work. Run evals weights and it reads your .llm-evals.config.yml , then prints each column’s direction and its real share of the final score: NOTE Yes, that is a Starship https://starship.rs prompt, and yes, people always ask. Here is a small present: my which-starship gist https://gist.github.com/kigster/ee667e080423c24c2a92d2de53cd9a48 is a ZSH script that downloads and installs the starship binary for you. It then opens a two-step menu where you scroll through every Starship preset and pick a color palette for the presets that support one. Here is what it prints for the Qualified.At config: Read that table as a product brief. Exact alone is 20% of the score: I want every field right, not almost right. Errors and invalid together are 32%: a broken reply is worse than a slow one. Hallucinations are 12%. Cost is only 10%. Latency is 10%. Wall time is 6%. That is not a universal truth. That is what this intake product cares about. The formula is deliberately boring: wscore = Σ |weight| × goodness / Σ|weight| Each metric’s goodness is its value, or 1 - value under a negative weight. cost and time go on a log scale between the cheapest and most expensive contender in the table, so each doubling costs the same penalty, no matter how absurd the priciest model is. Errors come straight off the top: 20 points for the first, 25 for two, and so on. The weights are opinions, and that is the point. They force you to write down the trade-off you were going to make in your head anyway. The results Settings multiply into contenders. A model listed with effort: low, high becomes two experiments. Ten models, most at both efforts, temperature fixed at 0.1: 19 configurations , not ten models. evals compare made 760 calls. 751 were scored, 9 failed, and the whole run cost $1.11 . flowchart TB cfg ".llm-evals.config.yml