{"slug": "turn-your-coding-agent-sessions-into-private-evals", "title": "Turn your coding-agent sessions into private evals", "summary": "A developer released `evals-from-sessions`, an agent skill that converts a user's own coding-agent session history into verified Harbor eval tasks so agents, models and settings can be compared on real work by price and time. The skill follows a six-step workflow — finding candidate tasks in session copies scanned with gitleaks 8, writing a `provenance.md` task spec, building the Harbor task from a private repository snapshot or a shareable synthetic version, checking the verifier (nop fails, oracle passes 5 of 5, near-miss solutions fail), running a smoke test plus an optional calibration of two setups × three runs, and reviewing failures — stopping at three human checkpoints and writing `AWAITING_REVIEW` on unattended runs. It requires Docker, git, Python 3.11+, Harbor (installed via `uv tool install harbor`, examples using 0.24 syntax) and credentials for one agent and model, and installs through the skills CLI, Claude Code, Codex and Pi plugins or manual copy under an MIT license.", "body_md": "`evals-from-sessions` is an agent skill. It turns your own coding-agent sessions into\nverified [Harbor](https://github.com/harbor-framework/harbor) eval tasks, so you can\ncompare agents, models and settings on your own work by price and time.\n\nGuide: [Turn your agent sessions into evals. Mine found a 2× cheaper setup](https://ibragim.dev/writing/)\n· Leaderboard: [ibragim.dev/leaderboard](https://ibragim.dev/leaderboard/)\n\nThe skill follows the six steps of the guide:\n\n1. **Find tasks in your session history.** Copy the sessions, scan the copy with\ngitleaks, and list candidate task episodes in`candidates.md` .\n2. **Turn a session into a task spec.** Write`provenance.md` : requirements, failure\nmodes, and what the agent must discover by itself. Tasks with something to discover\nare labeled`discovery` ; tasks where you stated every rule are labeled`routine` and\nkept, because they still show which setup solves your everyday work cheaper and faster.\n3. **Build the Harbor task.** Use a snapshot of your own repository (private mode) or a\nre-authored synthetic task (shareable mode). Close the usual cheating paths: no\nfuture git history, no reference diff in the image, hidden tests.\n4. **Check the verifier.** Nop fails, oracle passes 5 of 5, near-miss solutions fail,\nan alternative correct solution passes. Then a leak probe and`harbor check` .\n5. **Run it.** One smoke run with a real agent, and an optional calibration with two\nsetups × three runs.\n6. **Review and repeat.** Classify every failure (model mistake, ambiguity, verifier\nfalse negative), fix the task, bump its version, and re-run.\n\nThe skill stops when a task is *ready for human review*. Comparing setups across many\nruns (`harbor run -p tasks/ ... -k 3`) is your next step, not part of the skill.\n\n- **Checkpoint 1:** approve the candidate list (`candidates.md` ).\n- **Checkpoint 2:** approve the task spec (`provenance.md` ) before the agent builds the task.\n- **Final review:** check each finished task and its`report.md` before it enters your set.\n\nUnattended runs write `AWAITING_REVIEW` at a checkpoint and stop.\n\n- Docker and git\n- Python 3.11+\n- Harbor: `uv tool install harbor` (any recent version; command examples use 0.24 syntax)\n- gitleaks 8\n- Credentials for one agent and model, for `harbor check` and the smoke run\n\nskills CLI (Claude Code, Codex, Pi and other agents):\n\n```\nnpx skills add ibragim-bad/evals-from-sessions\n```\n\nClaude Code plugin:\n\n```\n/plugin marketplace add ibragim-bad/evals-from-sessions\n/plugin install evals-from-sessions@evals-from-sessions\n```\n\nCodex plugin:\n\n```\ncodex plugin marketplace add ibragim-bad/evals-from-sessions\ncodex plugin add evals-from-sessions@evals-from-sessions\n```\n\nPi package:\n\n```\npi install git:github.com/ibragim-bad/evals-from-sessions\n```\n\nUpdate: `claude plugin update evals-from-sessions@evals-from-sessions`,\n`codex plugin marketplace upgrade evals-from-sessions`, or `pi update --extensions`.\n\nManual: copy `skills/evals-from-sessions/` into your agent's skills directory, for\nexample `~/.claude/skills/`, `~/.codex/skills/`, `~/.agents/skills/` or\n`~/.pi/agent/skills/`.\n\nStart your agent in an empty work directory and ask, for example:\n\n```\nUse evals-from-sessions to turn my Claude Code sessions from the last month into\neval tasks. Work directory: ./evals. Stop at each checkpoint.\nskills/evals-from-sessions/\n  SKILL.md                        # the workflow\n  references/harbor-contract.md   # Harbor layout, network, commands\n  assets/task-template/           # starting files for one task\n  assets/leak-probe.sh            # tries to recover the solution from the image\n  assets/please-hack.j2           # optional \"please game the verifier\" smoke prompt\n```\n\nSession transcripts can contain API keys and private code. The skill works on a copy, scans it with gitleaks, and never writes secrets. Private-mode tasks contain a snapshot of your repository: keep them private. Publish only shareable tasks.\n\nMIT", "url": "https://wpnews.pro/news/turn-your-coding-agent-sessions-into-private-evals", "canonical_source": "https://github.com/ibragim-bad/evals-from-sessions", "published_at": "2026-10-18 03:26:40+00:00", "updated_at": "2026-10-18 03:39:48.815114+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["evals-from-sessions", "Harbor", "gitleaks", "Claude Code", "Codex", "Pi", "ibragim.dev", "ibragim-bad"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turn-your-coding-agent-sessions-into-private-evals", "markdown": "https://wpnews.pro/news/turn-your-coding-agent-sessions-into-private-evals.md", "text": "https://wpnews.pro/news/turn-your-coding-agent-sessions-into-private-evals.txt", "jsonld": "https://wpnews.pro/news/turn-your-coding-agent-sessions-into-private-evals.jsonld"}}