cd /news/ai-safety/rsb-1000-1000-record-synthetic-bench… · home › topics › ai-safety › article
[ARTICLE · art-155852] src=discuss.huggingface.co ↗ pub= topic=ai-safety verified=true sentiment=· neutral

RSB-1000: 1,000-record synthetic benchmark for phishing detection (CC-BY-4.0)

Reveal Wire released RSB-1000, a frozen, SHA256-pinned, CC-BY-4.0 benchmark of 1,000 fully synthetic phishing records (780 scam, 220 legitimate, including 60 AI-written adversary-in-the-middle lures), after finding no published detection scores for open-weight models. In zero-shot tests, Qwen 3.8-27B scored 98.9% recall at an 11.6% false-positive rate, while GPT-OSS 20B scored 94.0% recall at a 23.1% false-positive rate, with the AI-written lures the hardest subset for both models. The dataset and MIT-licensed evaluation code are published on Hugging Face and GitHub, and Reveal Wire is asking others to run additional models against it.

read1 min views3 publishedOct 23, 2026

Hi — I run Reveal Wire (youtube.com/@RevealWire), a channel that tests AI claims with real experiments. For an episode on phishing detection I looked for published detection scores for open-weight models and found zero — for any model. So we built RSB-1000: 1,000 fully synthetic records (780 scam / 220 legitimate, including 60 AI-written ‘adversary-in-the-middle’ lures with polished language and lookalike domains), frozen October 2026, SHA256-pinned, CC-BY-4.0. Dataset: RevealWire/rsb-1000 · Datasets at Hugging Face — code and eval script: GitHub - RevealWire/rsb-1000: RSB-1000: a frozen 1,000-record synthetic benchmark for scam/phishing detection (780 scam / 220 legit, incl. 60 AiTM cases). Data: CC-BY-4.0, code: MIT. · GitHub. Tested zero-shot: Qwen 3.8-27B — 98.9% recall, 11.6% FPR; GPT-OSS 20B — 94.0% recall, 23.1% FPR. The AI-written lures were the hardest subset for both models. If you run other models against it, I’d genuinely love to see the numbers.

── more in #ai-safety 4 stories · sorted by recency
── more on @reveal wire 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/rsb-1000-1000-record…] indexed:0 read:1min 2026-10-23 · —