RSB-1000: 1,000-record synthetic benchmark for phishing detection (CC-BY-4.0)

Reveal Wire released RSB-1000, a frozen, SHA256-pinned, CC-BY-4.0 benchmark of 1,000 fully synthetic phishing records (780 scam, 220 legitimate, including 60 AI-written adversary-in-the-middle lures), after finding no published detection scores for open-weight models. In zero-shot tests, Qwen 3.8-27B scored 98.9% recall at an 11.6% false-positive rate, while GPT-OSS 20B scored 94.0% recall at a 23.1% false-positive rate, with the AI-written lures the hardest subset for both models. The dataset and MIT-licensed evaluation code are published on Hugging Face and GitHub, and Reveal Wire is asking others to run additional models against it.

Hi — I run Reveal Wire youtube.com/@RevealWire http://youtube.com/@RevealWire , a channel that tests AI claims with real experiments. For an episode on phishing detection I looked for published detection scores for open-weight models and found zero — for any model. So we built RSB-1000: 1,000 fully synthetic records 780 scam / 220 legitimate, including 60 AI-written ‘adversary-in-the-middle’ lures with polished language and lookalike domains , frozen October 2026, SHA256-pinned, CC-BY-4.0. Dataset: RevealWire/rsb-1000 · Datasets at Hugging Face https://huggingface.co/datasets/RevealWire/rsb-1000 — code and eval script: GitHub - RevealWire/rsb-1000: RSB-1000: a frozen 1,000-record synthetic benchmark for scam/phishing detection 780 scam / 220 legit, incl. 60 AiTM cases . Data: CC-BY-4.0, code: MIT. · GitHub https://github.com/RevealWire/rsb-1000 . Tested zero-shot: Qwen 3.8-27B — 98.9% recall, 11.6% FPR; GPT-OSS 20B — 94.0% recall, 23.1% FPR. The AI-written lures were the hardest subset for both models. If you run other models against it, I’d genuinely love to see the numbers.