# EvalSift – 100 hand-labeled LLM agent traces

> Source: <https://github.com/Eddie-Town/evalsift-lite>
> Published: 2026-11-21 16:00:08+00:00

**Key takeaway:** EvalSift Lite grades agent traces with deterministic code checks and ships 100 hand-labeled support-agent traces (88 pass, 12 fail) that developers can rerun on their own agent to see if it reproduces the same failure modes.

Find out how your LLM agent actually fails before you write a single eval.

EvalSift Lite is a free, MIT-licensed Python tool (3.10+, no dependencies, no API key) for
error analysis on LLM agent traces: it validates a `traces.jsonl` file, grades each trace with
deterministic code checks, and writes a Markdown/HTML report of what failed. It comes with a
case study: 100 traces from a tool-using support agent, labeled by hand.
**Get the full Solo kit (100 labeled traces + data):** [https://merletownley.gumroad.com/l/fjppx](https://merletownley.gumroad.com/l/fjppx)

```
git clone <repo-url> && cd evalsift-lite
python -m venv .venv && . .venv/bin/activate
pip install -e .
python -m evalsift_lite.demo     # writes report.html, report.md, results.jsonl
```

The demo grades 8 traces from a fake support agent with 4 planted failures and catches all 4.

A support agent for a fictional outdoor store, with order, stock and policy tools.
One labeler, 88 pass and 12 fail. Full breakdown: [case-study/FAILURES.md](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/FAILURES.md)

| Failure mode | Fails | 
|---|---|
| Asked for details instead of calling the tool | 7 | 
| Applied a policy rule wrong at the edge ("over $75") | 2 | 
| Mishandled an ambiguous product name | 1 | 
| Date arithmetic off by one | 1 | 
| Answered an out-of-scope question | 1 | 

Examples worth reading:

- [q087: "free over $75" is not "free at $75"](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q087.md)
- [q016: asking instead of looking](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q016.md)
- [q046: a correct answer to the wrong question](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q046.md)
- [q086: off by one day](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/examples/q086.md)

The 100 questions are in [case-study/questions.jsonl](https://github.com/Eddie-Town/evalsift-lite/blob/main/case-study/questions.jsonl), so you can run them
on your own agent and see whether it makes the same mistakes.

One JSON object per line: `trace_id`, `input`, `output`, optional `steps[]` (each with `type` set to
`llm` or `tool`, plus `name`, `args`, `result`, `error`) and optional `metadata{}`. Then:

```
python -m evalsift_lite.demo --traces your_traces.jsonl --out-dir out/
```

This runs the built-in checks; add your own in `evalsift_lite/graders.py`.

EvalSift adds agent skills for Claude Code, Codex and Cursor that walk you through error analysis, building a failure taxonomy, writing an LLM judge, checking that judge against human labels, and gating regressions in CI. It also includes the full case-study data: all 100 traces, the labels with notes, and a note on how the labels were made.

MIT. See [LICENSE](https://github.com/Eddie-Town/evalsift-lite/blob/main/LICENSE).
