{"slug": "evalsift-100-hand-labeled-llm-agent-traces", "title": "EvalSift – 100 hand-labeled LLM agent traces", "summary": "EvalSift Lite, a free MIT-licensed Python tool for error analysis on LLM agent traces, ships with a hand-labeled case study of 100 traces from a tool-using support agent in which one labeler marked 88 pass and 12 fail. The 12 failures break down as 7 cases of asking for details instead of calling the tool, 2 misapplied policy rules at the \"over $75\" edge, 1 mishandled ambiguous product name, 1 date arithmetic error off by one, and 1 out-of-scope answer. The tool requires Python 3.10+, has no dependencies or API key, validates a traces.jsonl file, grades each trace with deterministic code checks, and writes Markdown/HTML reports; its demo grades 8 traces from a fake support agent with 4 planted failures and catches all 4.", "body_md": "Find out how your LLM agent actually fails before you write a single eval.\n\nEvalSift Lite is a free, MIT-licensed Python tool (3.10+, no dependencies, no API key) for\nerror analysis on LLM agent traces: it validates a `traces.jsonl` file, grades each trace with\ndeterministic code checks, and writes a Markdown/HTML report of what failed. It comes with a\ncase study: 100 traces from a tool-using support agent, labeled by hand.\n**Get the full Solo kit (100 labeled traces + data):** [https://merletownley.gumroad.com/l/fjppx](https://merletownley.gumroad.com/l/fjppx)\n\n```\ngit clone <repo-url> && cd evalsift-lite\npython -m venv .venv && . .venv/bin/activate\npip install -e .\npython -m evalsift_lite.demo     # writes report.html, report.md, results.jsonl\n```\n\nThe demo grades 8 traces from a fake support agent with 4 planted failures and catches all 4.\n\nA support agent for a fictional outdoor store, with order, stock and policy tools.\nOne labeler, 88 pass and 12 fail. Full breakdown: [case-study/FAILURES.md](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/FAILURES.md)\n\n| Failure mode | Fails | \n|---|---|\n| Asked for details instead of calling the tool | 7 | \n| Applied a policy rule wrong at the edge (\"over $75\") | 2 | \n| Mishandled an ambiguous product name | 1 | \n| Date arithmetic off by one | 1 | \n| Answered an out-of-scope question | 1 | \n\nExamples worth reading:\n\n- [q087: \"free over $75\" is not \"free at $75\"](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q087.md)\n- [q016: asking instead of looking](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q016.md)\n- [q046: a correct answer to the wrong question](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q046.md)\n- [q086: off by one day](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q086.md)\n\nThe 100 questions are in [case-study/questions.jsonl](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/questions.jsonl), so you can run them\non your own agent and see whether it makes the same mistakes.\n\nOne JSON object per line: `trace_id`, `input`, `output`, optional `steps[]` (each with `type` set to\n`llm` or `tool`, plus `name`, `args`, `result`, `error`) and optional `metadata{}`. Then:\n\n```\npython -m evalsift_lite.demo --traces your_traces.jsonl --out-dir out/\n```\n\nThis runs the built-in checks; add your own in `evalsift_lite/graders.py`.\n\nEvalSift adds agent skills for Claude Code, Codex and Cursor that walk you through error analysis, building a failure taxonomy, writing an LLM judge, checking that judge against human labels, and gating regressions in CI. It also includes the full case-study data: all 100 traces, the labels with notes, and a note on how the labels were made.\n\nMIT. See [LICENSE](https://github.com/Eddie-Town/evalsift-lite/blob/main/LICENSE).", "url": "https://wpnews.pro/news/evalsift-100-hand-labeled-llm-agent-traces", "canonical_source": "https://github.com/Eddie-Town/evalsift-lite", "published_at": "2026-11-21 16:00:08+00:00", "updated_at": "2026-11-21 16:13:08.121093+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models"], "entities": ["EvalSift Lite", "EvalSift", "Claude Code", "Codex", "Cursor", "Python"], "also_reported_by": [], "takeaway": "EvalSift Lite grades agent traces with deterministic code checks and ships 100 hand-labeled support-agent traces (88 pass, 12 fail) that developers can rerun on their own agent to see if it reproduces the same failure modes.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/evalsift-100-hand-labeled-llm-agent-traces", "markdown": "https://wpnews.pro/news/evalsift-100-hand-labeled-llm-agent-traces.md", "text": "https://wpnews.pro/news/evalsift-100-hand-labeled-llm-agent-traces.txt", "jsonld": "https://wpnews.pro/news/evalsift-100-hand-labeled-llm-agent-traces.jsonld"}}