Hi everyone, I am Rishabh. I work on fraud detection in production ML, and I kept hitting the same wall: everyone claims their LLM catches fraud, but nobody agrees on how to test that claim. Benchmarks built for chat or coding do not cover the world I live in: account takeovers, synthetic identities, refund scams, and the cases that hurt most in production, the legitimate users your model wrongly flags.
So I built fraud-eval, an open benchmark for evaluating LLM-based fraud and abuse detectors.
What is in v0.1.0:
The honest part: this is a smoke test, not a statistically powered benchmark. One case moves a pack rate by 20 to 33 points, and a simple keyword baseline averages just 5% detection across the packs, so there is real headroom here. I would rather ship a small honest benchmark than a big unverified one, and the full case set follows once it is verified.
Try it and break it:
If you run a detector against it, I genuinely want your numbers, especially where it fails. Tell me which packs break and why, and I will fold that into the next version.