I run evals for a few projects - yolop (a coding agent), bashkit (a sandboxed bash), Everruns, and a couple of private ones. Every time this comes up, someone asks which eval platform I use. LangSmith? Braintrust? MLflow Evals?
None. My evals live in the git repo. Datasets, the runner, the whole run history - all committed, right next to the code they measure.
And honestly, for most evals, that is all you need.
What an Eval Actually Is #
Strip away the dashboards and an eval is three things:
- a dataset- a list of cases with expected outcomes - a runner- something that feeds each case to a model and scores the result results- what happened, kept around so you can compare later
A dataset is a file. A runner is a script. Results are files. Git already stores files, versions them, and shares them with your team. So why rent a server for it?
The Feature List, Point by Point #
Eval platforms sell a feature list. Let me go through it with a repo in mind.
Available to the whole team→ same with a repo. Everyone already has it cloned.** Datasets versioned and maintained→ same with a repo. That is literally what git is for. A dataset change is a diff you can review. Runs the evals→ same with a repo. The runner is a script. It runs on your laptop and it runs in CI. Compare and review results**→ here is the fun part. The results are plain JSON in a folder. Point your coding agent at it and ask. More on this below.Runs large, long-running evals→ okay, this one is real. Servers win here. I will be honest about it at the end.
Four out of five, the repo just does. The fifth is a genuine trade-off, not a reason to move everything to a server.
What This Looks Like #
Here is one of these studies - bashkit’s eval crate. Datasets in data/
, the study in src/
, run history in results/
. And the history is not a figure of speech: that is 78 result files checked into git.
A dataset row is boring on purpose - one JSON object per line:
{
"id": "smoke_echo",
"prompt": "Print 'hello world' to stdout.",
"expectations": [
{ "check": "stdout_contains:hello world" },
{ "check": "exit_code:0" }
]
}
And a saved run stamps the git commit it ran against, so a number from three weeks ago is still interpretable today:
"environment": {
"git": { "commit": "1e187f3…", "branch": "…", "dirty": false },
"mira_version": "0.2.0"
},
"summary": { "scored": 174, "passed": 159, "failed": 15, "total_tokens": 1260246 }
No dashboard. Just files you can git log
.
The Fun Part: The Agent Reviews Results #
This is the point everyone underestimates. Once results are plain JSON in a folder, comparing and reviewing them stops being a product feature. It becomes a prompt.
Even plain jq
pulls a run’s headline numbers straight out of a committed file - pass rate, tokens, cost, tool calls:
The agent goes further. I open it in the repo and ask things like:
-
“Compare the last two runs in
evals/swebench_verified/results -
which tasks regressed?” - “Chart pass rate by category from the newest
report.json
.” - “The django cases fail more than the rest. Read the transcripts and tell me why.”
It reads the folder, does the diff, and gives me the answer - often with a chart. This works as a charm in agents, because the whole substrate is files and JSON, which is exactly what they are good at. No SDK, no query language, no export step.
The dashboards on eval platforms exist to answer these questions. The agent answers them from the raw files, and it answers the weird one-off questions too - the ones no dashboard has a widget for.
Mira: Evals as Runnable Scripts #
You could hand-roll all of this. It is genuinely easy - a loop, a scorer, write JSON to a folder. I did exactly that at first, and the flat eval-…-2026-02-07.md
files in bashkit’s history are the leftovers.
But then you re-invent the matrix, the retries, the resume, the reporting. So I moved everything onto Mira (from the Ukrainian word міра, meaning “measure”) - a small, code-first eval runner where an eval is a runnable script.
Here is the part I actually care about: I never write these studies by hand. I ask the agent. Mira ships an agent skill - npx skills add everruns/mira --global
- so my coding agent already knows the API, the CLI, and the conventions. I just describe what I want:
“evaluate yolop on SWE-bench Verified withhttps://github.com/everruns/mira”“eval the yolop system prompt - two variants, same 20 tasks”**“generate a Mira dataset from our yolop Braintrust traces”
and it writes the study, runs it, and hands me the report. So the eval is agent-driven on both ends - the agent authors it, and the agent reviews the results.
The study it writes stays small. Here is a complete one - a single file, no Cargo.toml
, dependencies right in the cargo-script header:
#!/usr/bin/env -S cargo +nightly -Zscript
---
[package]
edition = "2024"
[dependencies]
mira-eval = "0.4"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
---
use mira::scorer::{contains, succeeded};
use mira::subject::subject_fn;
use mira::{eval, Eval, Transcript};
#[eval]
fn greet() -> Eval {
Eval::new("greet")
.sample("hi", "Say hi and tell me the answer to life.")
.sample("formal", "Greet me formally and tell me the answer to life.")
.sample("casual", "Greet me casually and tell me the answer to life.")
.subject(subject_fn(|_sample, _cx| async move {
Transcript::response("Hi! The answer is 42.")
}))
.scorer(succeeded())
.scorer(contains("42"))
.build()
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
mira::Study::registered().serve().await
}
The study serves itself to a host CLI, discovered like cargo test
, and you run it:
mira run --study greet.rs
mira run --study greet.rs --tag smoke # just a slice
mira run --study greet.rs --format html --out report.html
Every run saves to ./results/<run_id>/
report.json
, a self-contained report.html
, one result.json
per case. That is the folder you commit.
And it is not a Rust thing. A study is just a program that speaks the protocol, so it can be Rust, Python, TypeScript, or any binary - pick whatever the thing you are testing already lives in. yolop’s SWE-bench study is one Python file; bashkit’s is Rust; Mira ships Python and TypeScript starters.
The one idea worth holding onto: a target is whatever is under test - not just a model. It can be a raw model, or a whole agent. Evaluating yolop means the target is yolop:
.targets([
Target::cli("yolop"), // the agent under test
Target::cli("claude-code"), // compared against another agent
Target::openai("gpt-5"), // or a raw model, same run
])
Mira takes the cross-product of samples × targets × any axes you add, runs it with bounded concurrency, and returns a non-zero exit when a case fails - so mira run …
drops straight into CI. No server to stand up, no keys to hand a third party.
Where a Server Actually Wins #
I said I would be honest, so: point 5 is real.
If you are running SWE-bench Verified end to end - 500 instances, each spinning a Docker container, for hours - a repo and your laptop are the wrong tool. That wants a fleet of workers, a queue, and shared storage. A platform earns its keep there. Same if non-engineers need to browse results without cloning anything, or you have compliance-grade retention requirements.
But that is the heavy tail. The daily eval - a couple hundred cases, run on a change, compared against last week - has none of those problems. For that, the server is overhead you carry to solve a problem you do not have.
Keep datasets in the repo. Keep results in the repo. Let the agent read them. Reach for a server the day you actually outgrow git - not before.
For comments or feedback, write atx.com/chaliy.