{"slug": "fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case", "title": "Fraud-eval: an open benchmark for LLM fraud detectors, starting with a 20-case smoke test", "summary": "Rishabh, a production ML engineer working on fraud detection, released fraud-eval v0.1.0, an open benchmark for evaluating LLM-based fraud and abuse detectors, starting with a 20-case smoke test. The release covers account takeovers, synthetic identities, refund scams and false positives on legitimate users, and Rishabh notes that one case moves a pack rate by 20 to 33 points while a simple keyword baseline averages just 5% detection across the packs. Rishabh is soliciting detector results, especially failures, to fold into the next version, with the full case set to follow once verified.", "body_md": "Hi everyone, I am Rishabh. I work on fraud detection in production ML, and I kept hitting the same wall: everyone claims their LLM catches fraud, but nobody agrees on how to test that claim. Benchmarks built for chat or coding do not cover the world I live in: account takeovers, synthetic identities, refund scams, and the cases that hurt most in production, the legitimate users your model wrongly flags.\n\nSo I built fraud-eval, an open benchmark for evaluating LLM-based fraud and abuse detectors.\n\nWhat is in v0.1.0:\n\nThe honest part: this is a smoke test, not a statistically powered benchmark. One case moves a pack rate by 20 to 33 points, and a simple keyword baseline averages just 5% detection across the packs, so there is real headroom here. I would rather ship a small honest benchmark than a big unverified one, and the full case set follows once it is verified.\n\nTry it and break it:\n\nIf you run a detector against it, I genuinely want your numbers, especially where it fails. Tell me which packs break and why, and I will fold that into the next version.", "url": "https://wpnews.pro/news/fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case", "canonical_source": "https://discuss.huggingface.co/t/fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case-smoke-test/191159#post_1", "published_at": "2026-10-28 20:37:30+00:00", "updated_at": "2026-10-28 20:44:52.411318+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-safety"], "entities": ["Rishabh", "fraud-eval"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case", "markdown": "https://wpnews.pro/news/fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case.md", "text": "https://wpnews.pro/news/fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case.txt", "jsonld": "https://wpnews.pro/news/fraud-eval-an-open-benchmark-for-llm-fraud-detectors-starting-with-a-20-case.jsonld"}}