Fraud-eval: an open benchmark for LLM fraud detectors, starting with a 20-case smoke test

Rishabh, a production ML engineer working on fraud detection, released fraud-eval v0.1.0, an open benchmark for evaluating LLM-based fraud and abuse detectors, starting with a 20-case smoke test. The release covers account takeovers, synthetic identities, refund scams and false positives on legitimate users, and Rishabh notes that one case moves a pack rate by 20 to 33 points while a simple keyword baseline averages just 5% detection across the packs. Rishabh is soliciting detector results, especially failures, to fold into the next version, with the full case set to follow once verified.

Hi everyone, I am Rishabh. I work on fraud detection in production ML, and I kept hitting the same wall: everyone claims their LLM catches fraud, but nobody agrees on how to test that claim. Benchmarks built for chat or coding do not cover the world I live in: account takeovers, synthetic identities, refund scams, and the cases that hurt most in production, the legitimate users your model wrongly flags. So I built fraud-eval, an open benchmark for evaluating LLM-based fraud and abuse detectors. What is in v0.1.0: The honest part: this is a smoke test, not a statistically powered benchmark. One case moves a pack rate by 20 to 33 points, and a simple keyword baseline averages just 5% detection across the packs, so there is real headroom here. I would rather ship a small honest benchmark than a big unverified one, and the full case set follows once it is verified. Try it and break it: If you run a detector against it, I genuinely want your numbers, especially where it fails. Tell me which packs break and why, and I will fold that into the next version.