{"slug": "you-do-not-need-a-server-for-evals", "title": "You Do Not Need a Server for Evals", "summary": "Evals for AI projects like coding agents and sandboxed bash do not require a dedicated server or platform, according to developer Everruns. Datasets, runners, and results can be stored and versioned directly in a git repository, with plain JSON files enabling agent-driven comparison and review. The approach covers team access, versioning, execution, and result comparison, though it lacks support for large, long-running evaluations that benefit from server infrastructure.", "body_md": "I run evals for a few projects - [yolop](https://github.com/everruns/yolop) (a coding agent), [bashkit](https://github.com/everruns/bashkit) (a sandboxed bash), [Everruns](https://everruns.com/), and a couple of private ones. Every time this comes up, someone asks which eval platform I use. LangSmith? Braintrust? MLflow Evals?\n\nNone. My evals live in the git repo. Datasets, the runner, the whole run history - all committed, right next to the code they measure.\n\nAnd honestly, for most evals, that is all you need.\n\n## What an Eval Actually Is\n\nStrip away the dashboards and an eval is three things:\n\n- a\n**dataset**- a list of cases with expected outcomes - a\n**runner**- something that feeds each case to a model and scores the result **results**- what happened, kept around so you can compare later\n\nA dataset is a file. A runner is a script. Results are files. Git already stores files, versions them, and shares them with your team. So why rent a server for it?\n\n## The Feature List, Point by Point\n\nEval platforms sell a feature list. Let me go through it with a repo in mind.\n\n**Available to the whole team**→ same with a repo. Everyone already has it cloned.** Datasets versioned and maintained**→ same with a repo. That is literally what git is for. A dataset change is a diff you can review.** Runs the evals**→ same with a repo. The runner is a script. It runs on your laptop and it runs in CI.** Compare and review results**→ here is the fun part. The results are plain JSON in a folder. Point your coding agent at it and ask. More on this below.**Runs large, long-running evals**→ okay, this one is real. Servers win here. I will be honest about it at the end.\n\nFour out of five, the repo just does. The fifth is a genuine trade-off, not a reason to move everything to a server.\n\n## What This Looks Like\n\nHere is one of these studies - bashkit’s eval crate. Datasets in `data/`\n\n, the study in `src/`\n\n, run history in `results/`\n\n. And the history is not a figure of speech: that is 78 result files checked into git.\n\nA dataset row is boring on purpose - one JSON object per line:\n\n```\n{\n  \"id\": \"smoke_echo\",\n  \"prompt\": \"Print 'hello world' to stdout.\",\n  \"expectations\": [\n    { \"check\": \"stdout_contains:hello world\" },\n    { \"check\": \"exit_code:0\" }\n  ]\n}\n```\n\nAnd a saved run stamps the git commit it ran against, so a number from three weeks ago is still interpretable today:\n\n```\n\"environment\": {\n  \"git\": { \"commit\": \"1e187f3…\", \"branch\": \"…\", \"dirty\": false },\n  \"mira_version\": \"0.2.0\"\n},\n\"summary\": { \"scored\": 174, \"passed\": 159, \"failed\": 15, \"total_tokens\": 1260246 }\n```\n\nNo dashboard. Just files you can `git log`\n\n.\n\n## The Fun Part: The Agent Reviews Results\n\nThis is the point everyone underestimates. Once results are plain JSON in a folder, comparing and reviewing them stops being a product feature. It becomes a prompt.\n\nEven plain `jq`\n\npulls a run’s headline numbers straight out of a committed file - pass rate, tokens, cost, tool calls:\n\nThe agent goes further. I open it in the repo and ask things like:\n\n- “Compare the last two runs in\n`evals/swebench_verified/results`\n\n- which tasks regressed?” - “Chart pass rate by category from the newest\n`report.json`\n\n.” - “The django cases fail more than the rest. Read the transcripts and tell me why.”\n\nIt reads the folder, does the diff, and gives me the answer - often with a chart. This works *as a charm* in agents, because the whole substrate is files and JSON, which is exactly what they are good at. No SDK, no query language, no export step.\n\nThe dashboards on eval platforms exist to answer these questions. The agent answers them from the raw files, and it answers the weird one-off questions too - the ones no dashboard has a widget for.\n\n## Mira: Evals as Runnable Scripts\n\nYou could hand-roll all of this. It is genuinely easy - a loop, a scorer, write JSON to a folder. I did exactly that at first, and the flat `eval-…-2026-02-07.md`\n\nfiles in bashkit’s history are the leftovers.\n\nBut then you re-invent the matrix, the retries, the resume, the reporting. So I moved everything onto [Mira](https://github.com/everruns/mira) (from the Ukrainian word *міра*, meaning “measure”) - a small, code-first eval runner where an eval *is* a runnable script.\n\nHere is the part I actually care about: **I never write these studies by hand.** I ask the agent. Mira ships an [agent skill](https://github.com/everruns/mira/tree/main/skills/mira) - `npx skills add everruns/mira --global`\n\n- so my coding agent already knows the API, the CLI, and the conventions. I just describe what I want:\n\n*“evaluate yolop on SWE-bench Verified with*[https://github.com/everruns/mira](https://github.com/everruns/mira)”*“eval the yolop system prompt - two variants, same 20 tasks”**“generate a Mira dataset from our yolop Braintrust traces”*\n\nand it writes the study, runs it, and hands me the report. So the eval is agent-driven on both ends - the agent authors it, and [the agent reviews the results](#the-fun-part-the-agent-reviews-results).\n\nThe study it writes stays small. Here is a complete one - a single file, no `Cargo.toml`\n\n, dependencies right in the cargo-script header:\n\n``` bash\n#!/usr/bin/env -S cargo +nightly -Zscript\n---\n[package]\nedition = \"2024\"\n\n[dependencies]\nmira-eval = \"0.4\"\ntokio = { version = \"1\", features = [\"macros\", \"rt-multi-thread\"] }\n---\nuse mira::scorer::{contains, succeeded};\nuse mira::subject::subject_fn;\nuse mira::{eval, Eval, Transcript};\n\n#[eval]\nfn greet() -> Eval {\n    Eval::new(\"greet\")\n        .sample(\"hi\", \"Say hi and tell me the answer to life.\")\n        .sample(\"formal\", \"Greet me formally and tell me the answer to life.\")\n        .sample(\"casual\", \"Greet me casually and tell me the answer to life.\")\n        .subject(subject_fn(|_sample, _cx| async move {\n            Transcript::response(\"Hi! The answer is 42.\")\n        }))\n        .scorer(succeeded())\n        .scorer(contains(\"42\"))\n        .build()\n}\n\n#[tokio::main]\nasync fn main() -> std::io::Result<()> {\n    mira::Study::registered().serve().await\n}\n```\n\nThe study serves itself to a host CLI, discovered like `cargo test`\n\n, and you run it:\n\n```\nmira run --study greet.rs\nmira run --study greet.rs --tag smoke        # just a slice\nmira run --study greet.rs --format html --out report.html\n```\n\nEvery run saves to `./results/<run_id>/`\n\n- `report.json`\n\n, a self-contained `report.html`\n\n, one `result.json`\n\nper case. That is the folder you commit.\n\nAnd it is not a Rust thing. A study is just a program that speaks the protocol, so it can be Rust, Python, TypeScript, or any binary - pick whatever the thing you are testing already lives in. yolop’s SWE-bench study is [one Python file](https://github.com/everruns/yolop/blob/main/evals/swebench_verified/swebench_verified.py); bashkit’s is [Rust](https://github.com/everruns/bashkit/tree/main/crates/bashkit-eval); Mira ships [Python](https://github.com/everruns/mira/tree/main/examples/greet-python) and [TypeScript](https://github.com/everruns/mira/tree/main/examples/greet-typescript) starters.\n\nThe one idea worth holding onto: a **target is whatever is under test** - not just a model. It can be a raw model, or a whole agent. Evaluating yolop means the target *is* yolop:\n\n```\n.targets([\n    Target::cli(\"yolop\"),           // the agent under test\n    Target::cli(\"claude-code\"),     // compared against another agent\n    Target::openai(\"gpt-5\"),        // or a raw model, same run\n])\n```\n\nMira takes the cross-product of samples × targets × any axes you add, runs it with bounded concurrency, and returns a non-zero exit when a case fails - so `mira run …`\n\ndrops straight into CI. No server to stand up, no keys to hand a third party.\n\n## Where a Server Actually Wins\n\nI said I would be honest, so: point 5 is real.\n\nIf you are running SWE-bench Verified end to end - 500 instances, each spinning a Docker container, for hours - a repo and your laptop are the wrong tool. That wants a fleet of workers, a queue, and shared storage. A platform earns its keep there. Same if non-engineers need to browse results without cloning anything, or you have compliance-grade retention requirements.\n\nBut that is the heavy tail. The daily eval - a couple hundred cases, run on a change, compared against last week - has none of those problems. For that, the server is overhead you carry to solve a problem you do not have.\n\nKeep datasets in the repo. Keep results in the repo. Let the agent read them. Reach for a server the day you actually outgrow git - not before.\n\nFor comments or feedback, write at[x.com/chaliy](https://x.com/chaliy).", "url": "https://wpnews.pro/news/you-do-not-need-a-server-for-evals", "canonical_source": "https://chaliy.name/blog/you-dont-need-a-server-for-evals/", "published_at": "2026-07-24 00:00:00+00:00", "updated_at": "2026-07-24 23:57:24.827597+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["Everruns", "yolop", "bashkit", "LangSmith", "Braintrust", "MLflow Evals", "Mira"], "alternates": {"html": "https://wpnews.pro/news/you-do-not-need-a-server-for-evals", "markdown": "https://wpnews.pro/news/you-do-not-need-a-server-for-evals.md", "text": "https://wpnews.pro/news/you-do-not-need-a-server-for-evals.txt", "jsonld": "https://wpnews.pro/news/you-do-not-need-a-server-for-evals.jsonld"}}