cd /news/ai-agents/evalsift-100-hand-labeled-llm-agent-… · home › topics › ai-agents › article
[ARTICLE · art-173826] src=github.com ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

EvalSift – 100 hand-labeled LLM agent traces

EvalSift Lite, a free MIT-licensed Python tool for error analysis on LLM agent traces, ships with a hand-labeled case study of 100 traces from a tool-using support agent in which one labeler marked 88 pass and 12 fail. The 12 failures break down as 7 cases of asking for details instead of calling the tool, 2 misapplied policy rules at the "over $75" edge, 1 mishandled ambiguous product name, 1 date arithmetic error off by one, and 1 out-of-scope answer. The tool requires Python 3.10+, has no dependencies or API key, validates a traces.jsonl file, grades each trace with deterministic code checks, and writes Markdown/HTML reports; its demo grades 8 traces from a fake support agent with 4 planted failures and catches all 4.

key takeaway

EvalSift Lite grades agent traces with deterministic code checks and ships 100 hand-labeled support-agent traces (88 pass, 12 fail) that developers can rerun on their own agent to see if it reproduces the same failure modes.

read2 min views9 publishedNov 21, 2026
EvalSift – 100 hand-labeled LLM agent traces
Image: Michielbdejong (auto-discovered)

Find out how your LLM agent actually fails before you write a single eval.

EvalSift Lite is a free, MIT-licensed Python tool (3.10+, no dependencies, no API key) for error analysis on LLM agent traces: it validates a traces.jsonl file, grades each trace with deterministic code checks, and writes a Markdown/HTML report of what failed. It comes with a case study: 100 traces from a tool-using support agent, labeled by hand. Get the full Solo kit (100 labeled traces + data): https://merletownley.gumroad.com/l/fjppx

git clone <repo-url> && cd evalsift-lite
python -m venv .venv && . .venv/bin/activate
pip install -e .
python -m evalsift_lite.demo     # writes report.html, report.md, results.jsonl

The demo grades 8 traces from a fake support agent with 4 planted failures and catches all 4.

A support agent for a fictional outdoor store, with order, stock and policy tools. One labeler, 88 pass and 12 fail. Full breakdown: case-study/FAILURES.md

Failure mode Fails
Asked for details instead of calling the tool 7
Applied a policy rule wrong at the edge ("over $75") 2
Mishandled an ambiguous product name 1
Date arithmetic off by one 1
Answered an out-of-scope question 1

Examples worth reading:

The 100 questions are in case-study/questions.jsonl, so you can run them on your own agent and see whether it makes the same mistakes.

One JSON object per line: trace_id, input, output, optional steps[] (each with type set to llm or tool, plus name, args, result, error) and optional metadata{}. Then:

python -m evalsift_lite.demo --traces your_traces.jsonl --out-dir out/

This runs the built-in checks; add your own in evalsift_lite/graders.py.

EvalSift adds agent skills for Claude Code, Codex and Cursor that walk you through error analysis, building a failure taxonomy, writing an LLM judge, checking that judge against human labels, and gating regressions in CI. It also includes the full case-study data: all 100 traces, the labels with notes, and a note on how the labels were made.

MIT. See LICENSE.

── more in #ai-agents 4 stories · sorted by recency
── more on @evalsift lite 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/evalsift-100-hand-la…] indexed:0 read:2min 2026-11-21 · —