# Fraud-eval: an open benchmark for LLM fraud detectors, starting with a 20-case smoke test

> Source: <https://discuss.huggingface.co/t/fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case-smoke-test/191159#post_1>
> Published: 2026-10-28 20:37:30+00:00

Hi everyone, I am Rishabh. I work on fraud detection in production ML, and I kept hitting the same wall: everyone claims their LLM catches fraud, but nobody agrees on how to test that claim. Benchmarks built for chat or coding do not cover the world I live in: account takeovers, synthetic identities, refund scams, and the cases that hurt most in production, the legitimate users your model wrongly flags.

So I built fraud-eval, an open benchmark for evaluating LLM-based fraud and abuse detectors.

What is in v0.1.0:

The honest part: this is a smoke test, not a statistically powered benchmark. One case moves a pack rate by 20 to 33 points, and a simple keyword baseline averages just 5% detection across the packs, so there is real headroom here. I would rather ship a small honest benchmark than a big unverified one, and the full case set follows once it is verified.

Try it and break it:

If you run a detector against it, I genuinely want your numbers, especially where it fails. Tell me which packs break and why, and I will fold that into the next version.