You Do Not Need a Server for Evals Evals for AI projects like coding agents and sandboxed bash do not require a dedicated server or platform, according to developer Everruns. Datasets, runners, and results can be stored and versioned directly in a git repository, with plain JSON files enabling agent-driven comparison and review. The approach covers team access, versioning, execution, and result comparison, though it lacks support for large, long-running evaluations that benefit from server infrastructure. I run evals for a few projects - yolop https://github.com/everruns/yolop a coding agent , bashkit https://github.com/everruns/bashkit a sandboxed bash , Everruns https://everruns.com/ , and a couple of private ones. Every time this comes up, someone asks which eval platform I use. LangSmith? Braintrust? MLflow Evals? None. My evals live in the git repo. Datasets, the runner, the whole run history - all committed, right next to the code they measure. And honestly, for most evals, that is all you need. What an Eval Actually Is Strip away the dashboards and an eval is three things: - a dataset - a list of cases with expected outcomes - a runner - something that feeds each case to a model and scores the result results - what happened, kept around so you can compare later A dataset is a file. A runner is a script. Results are files. Git already stores files, versions them, and shares them with your team. So why rent a server for it? The Feature List, Point by Point Eval platforms sell a feature list. Let me go through it with a repo in mind. Available to the whole team → same with a repo. Everyone already has it cloned. Datasets versioned and maintained → same with a repo. That is literally what git is for. A dataset change is a diff you can review. Runs the evals → same with a repo. The runner is a script. It runs on your laptop and it runs in CI. Compare and review results → here is the fun part. The results are plain JSON in a folder. Point your coding agent at it and ask. More on this below. Runs large, long-running evals → okay, this one is real. Servers win here. I will be honest about it at the end. Four out of five, the repo just does. The fifth is a genuine trade-off, not a reason to move everything to a server. What This Looks Like Here is one of these studies - bashkit’s eval crate. Datasets in data/ , the study in src/ , run history in results/ . And the history is not a figure of speech: that is 78 result files checked into git. A dataset row is boring on purpose - one JSON object per line: { "id": "smoke echo", "prompt": "Print 'hello world' to stdout.", "expectations": { "check": "stdout contains:hello world" }, { "check": "exit code:0" } } And a saved run stamps the git commit it ran against, so a number from three weeks ago is still interpretable today: "environment": { "git": { "commit": "1e187f3…", "branch": "…", "dirty": false }, "mira version": "0.2.0" }, "summary": { "scored": 174, "passed": 159, "failed": 15, "total tokens": 1260246 } No dashboard. Just files you can git log . The Fun Part: The Agent Reviews Results This is the point everyone underestimates. Once results are plain JSON in a folder, comparing and reviewing them stops being a product feature. It becomes a prompt. Even plain jq pulls a run’s headline numbers straight out of a committed file - pass rate, tokens, cost, tool calls: The agent goes further. I open it in the repo and ask things like: - “Compare the last two runs in evals/swebench verified/results - which tasks regressed?” - “Chart pass rate by category from the newest report.json .” - “The django cases fail more than the rest. Read the transcripts and tell me why.” It reads the folder, does the diff, and gives me the answer - often with a chart. This works as a charm in agents, because the whole substrate is files and JSON, which is exactly what they are good at. No SDK, no query language, no export step. The dashboards on eval platforms exist to answer these questions. The agent answers them from the raw files, and it answers the weird one-off questions too - the ones no dashboard has a widget for. Mira: Evals as Runnable Scripts You could hand-roll all of this. It is genuinely easy - a loop, a scorer, write JSON to a folder. I did exactly that at first, and the flat eval-…-2026-02-07.md files in bashkit’s history are the leftovers. But then you re-invent the matrix, the retries, the resume, the reporting. So I moved everything onto Mira https://github.com/everruns/mira from the Ukrainian word міра , meaning “measure” - a small, code-first eval runner where an eval is a runnable script. Here is the part I actually care about: I never write these studies by hand. I ask the agent. Mira ships an agent skill https://github.com/everruns/mira/tree/main/skills/mira - npx skills add everruns/mira --global - so my coding agent already knows the API, the CLI, and the conventions. I just describe what I want: “evaluate yolop on SWE-bench Verified with https://github.com/everruns/mira https://github.com/everruns/mira ” “eval the yolop system prompt - two variants, same 20 tasks” “generate a Mira dataset from our yolop Braintrust traces” and it writes the study, runs it, and hands me the report. So the eval is agent-driven on both ends - the agent authors it, and the agent reviews the results the-fun-part-the-agent-reviews-results . The study it writes stays small. Here is a complete one - a single file, no Cargo.toml , dependencies right in the cargo-script header: bash /usr/bin/env -S cargo +nightly -Zscript --- package edition = "2024" dependencies mira-eval = "0.4" tokio = { version = "1", features = "macros", "rt-multi-thread" } --- use mira::scorer::{contains, succeeded}; use mira::subject::subject fn; use mira::{eval, Eval, Transcript}; eval fn greet - Eval { Eval::new "greet" .sample "hi", "Say hi and tell me the answer to life." .sample "formal", "Greet me formally and tell me the answer to life." .sample "casual", "Greet me casually and tell me the answer to life." .subject subject fn | sample, cx| async move { Transcript::response "Hi The answer is 42." } .scorer succeeded .scorer contains "42" .build } tokio::main async fn main - std::io::Result< { mira::Study::registered .serve .await } The study serves itself to a host CLI, discovered like cargo test , and you run it: mira run --study greet.rs mira run --study greet.rs --tag smoke just a slice mira run --study greet.rs --format html --out report.html Every run saves to ./results/