Note
This is a shortened version of the original writing on kig.re — my personal blog.
Here is what a real lead typed into a tax-preparation intake form:
“Got married in June, haven’t figured out how we’re filing yet. There’s an IRS letter somewhere in a kitchen drawer I haven’t opened. I also deliver groceries on weekends.”
Only one fact in that paragraph is clear: the weekend side gig. A model read it anyway and set filing_status to married_jointly.
Helpful. Wrong. The form then skipped the filing-status question and treated a guess as settled. That is the failure I care about most, and it is invisible if you only eyeball a few outputs.
The vague tax-preparation case: the lead's input, the expected JSON with filing_status null, gpt-4o-mini hallucinating married_jointly at 78.8%, and deepseek-v4.1-flash-1017 scoring 100% with empty diagnostics.
I built llm-evals so I would stop writing scripts/eval.rb for the fourth time. It is a Ruby CLI for multi-dataset, multi-case evaluation suites. Compare models, efforts, temperatures. Rank them by one weighted score that balances accuracy, hallucinations, errors, latency, and cost. On RubyGems as of writing: 0.5.0. Two commands: evals and llm-evals.
gem install llm-evals
Home-made pasta tastes better because you chose the flour. Home-made evals taste better because you chose the weights, the cases, and the failure modes that matter to your product. Store-bought leaderboards cannot taste your soup.
A unit test asks: given this input, does my code return exactly this output? An eval asks a softer question: given representative inputs, how often and how well does the model produce what I expect? You cannot assert equality against a probability distribution. You can score it. Score the same cases every time, and you can finally measure what a prompt tweak, a temperature change, or a model swap actually did.
The scorer decides what “good” means. The eval makes the comparison honest and repeatable.
The gem grew out of Qualified.At, which I wrote about when it launched. A business builds an intake form, embeds it, and visitors answer questions before anyone picks up the phone. The forms use the Inquirex gems. Many start with a free-text box: tell us about your situation. An LLM reads that paragraph and extracts structured answers, so a 20- or 30-question form can shrink to a few follow-ups.
Here is the extraction step from the tax preparation demo, copied from the suite’s dsl.rb:
extract :understanding do
from :situation
prompt "Extract the taxpayer's situation from their description. Only fill a " \
"field when the text genuinely supports it — leave anything the writer " \
"did not mention unset rather than guessing. ..."
schema filing_status: { type: :enum,
values: %w[single married_jointly married_separately head_of_household] },
dependents: :integer,
self_employed: :boolean,
rental_properties: :integer,
sold_investments: :boolean,
owns_business_entity: :boolean,
multi_state: :boolean,
irs_letter: :boolean
transition to: :heard_you
end
And here is what that lead typed (the vague case):
Honestly, this year was a blur and I’ve been avoiding thinking about taxes. I got married in June, so I assume that changes things, though we haven’t talked about how we’re filing. I also picked up a side gig doing grocery deliveries through one of those apps on weekends, which I’m told counts as being self-employed. Somewhere in a kitchen drawer there’s a letter from the IRS that came in August that I haven’t been brave enough to open. We rent our apartment, if that matters. I don’t really know what else you need from me, so please just tell me what to bring.
The correct extraction is not “married filing jointly”. They got married. They said they have not decided how to file. The right answer is:
{
"filing_status": null,
"dependents": null,
"self_employed": true,
"rental_properties": null,
"sold_investments": null,
"owns_business_entity": null,
"multi_state": null,
"irs_letter": true
}
Two answers, six honest nulls. A model that “helpfully” fills in married_jointly has invented a fact. On this case, openai/gpt-4o-mini scored 78.8% and the diagnostics named the hallucination: expected null, got married_jointly. deepseek/deepseek-v4.1-flash-1017 at high effort scored 100% with empty diagnostics.
So the product question became: which model captures the most answers, assigns them correctly, avoids hallucinations, and does not miss details, while staying cheap and fast?
A lead who waits 30 seconds for the next question is a lead who closes the tab. At 1,000 leads, $0.0002 per call is twenty cents. $0.50 per call is five hundred dollars.
Qualified.At ships ten demo businesses: bookkeeping, couples therapy, home remodeling, immigration law, managed IT, personal injury, roofing, software development, tax preparation, and wedding photography. Each one became a dataset. Each dataset has four case variants.
Here is the thing that bit me. The report had a dozen sortable columns: accuracy, exact match, hallucination rate, errors, latency, cost, tokens. Sort by accuracy and an expensive, slow model wins. Sort by cost and a model that hallucinates wins. Sort by primary score and you crow a token hog. None of those columns answered the question I actually had, which was which one should I ship?
So llm-evals lets you say what matters, in YAML, and folds it into one column called w wscore. This is the weighted_score section from Qualified.At’s .llm-evals.config.yml:
weighted_score:
weights:
exact: 10 # Exact contributes high number
score: 5 # This is a score computed in other ways
latency: -5 # Latency is important, but relatively less than others
cost: -5 # Same with the cost
time: -3 # latency contributes relatively low comparatively
errors: -8 # and any errors contribute quite a bit to the total
hallucinated_rate: -6 # Hallucinations are bad, and are penalized
invalid_rate: -8 # invalid_rate is an error, so heavy penalty
A positive weight means higher is better. A negative weight means lower is better. Raw weights are hard to reason about, though, so the llm-evals CLI ships a subcommand that does the math for you. I called it weights, which is not my most imaginative work. Run evals weights and it reads your .llm-evals.config.yml, then prints each column’s direction and its real share of the final score:
The help screen of evals weights -h: print how the config file's weights divide the weighted score, with -C to point at another lineup file.
Here is what it prints for the Qualified.At config:
Read that table as a product brief. Exact alone is 20% of the score: I want every field right, not almost right. Errors and invalid together are 32%: a broken reply is worse than a slow one. Hallucinations are 12%. Cost is only 10%. Latency is 10%. Wall time is 6%. That is not a universal truth. That is what this intake product cares about.
The formula is deliberately boring:
Each metric’s goodness is its value, or 1 - value under a negative weight. cost and time go on a log scale between the cheapest and most expensive contender in the table, so each doubling costs the same penalty, no matter how absurd the priciest model is. Errors come straight off the top: 20 points for the first, 25 for two, and so on.
The weights are opinions, and that is the point. They force you to write down the trade-off you were going to make in your head anyway.
Please see the original post for the full result set
You write two small classes. The gem does everything else.
The adapter runs one case against one model and returns a Reply. It owns everything the model sees and everything done to its answer. In Qualified.At, it calls the same prompt builder and the same parser production uses, so the scores reflect production rather than a lab version of it.
The scorer turns a case and a reply into a Score with whatever metrics you declare. It never calls anything.
The gem handles case discovery, the model lineup, providers (OpenRouter by default, through RubyLLM), rate limits, retries, thread pools, progress bars, the leaderboard, an HTML report, and optional reporting to Braintrust or Logfire. Some of the design came from Pydantic Evals.
A few decisions I am stubborn about:
A wrong answer is a 0. A network error is an error, not a 0. Your model did not answer wrongly because OpenRouter returned a 503.
A model the suite cannot ask is refused by name, not scored. If your suite sends chat prompts and the lineup lists a typed-question model, you get typesafe/jev-latest refused: needs typesafe; this suite produces prompt before anything runs.
Settings multiply. effort: [low, high] with temperature: [0.0, 0.7] becomes four contenders for that model. You are not comparing models; you are comparing configurations.
No lock-in. A local JSON log is always written. Any HTML report can be re-rendered from it later with evals render, with no model calls.
The repository ships two example suites that use fake models, so they need no API keys and no network:
git clone https://github.com/kigster/llm-evals.git
cd llm-evals
bundle install
just examples
For your own project, copy the toy suite’s evals/ folder, swap in your adapter and scorer, list real models under openrouter: in .llm-evals.config.yml, set OPENROUTER_API_KEY, and:
evals cases # list the cases; calls nothing
evals verify # offline checks of inputs, expected answers, adapter and scorer
evals compare -n # show the experiment matrix; calls nothing
evals compare # run it, and open the report
Run evals compare -n before every paid run. It prints the full experiment matrix, so you find out that you accidentally multiplied three efforts by three temperatures by twelve models before OpenRouter does.
The gem is MIT-licensed, and the README covers writing providers (there is a full Ollama example) and trackers (there is a W&B Weave sketch). Issues and pull requests are very welcome.
There is no best model. There is only the best model for a given task, at a given price, with a given tolerance for being confidently wrong. DeepSeek won my suite overall, and it still lost three datasets to gpt-6-luna and one to gpt-4o-mini. If I had picked one winner and wired it into every extract step, I would have shipped worse answers to a third of my customers and called it a decision.
The leaderboard is not the point. The point is that the question “which model should we use?” stops being an argument and becomes a command you can rerun next week, when the next model drops and the prices move again. Write the cases your product cannot afford to get wrong. Choose weights that say what you actually care about. Then let the numbers settle it.
And if you are about to write scripts/eval.rb for the fourth time, please don’t. I already did. Home-made tastes better when someone else rolled the dough.