{"slug": "rsb-1000-1000-record-synthetic-benchmark-for-phishing-detection-cc-by-4-0", "title": "RSB-1000: 1,000-record synthetic benchmark for phishing detection (CC-BY-4.0)", "summary": "Reveal Wire released RSB-1000, a frozen, SHA256-pinned, CC-BY-4.0 benchmark of 1,000 fully synthetic phishing records (780 scam, 220 legitimate, including 60 AI-written adversary-in-the-middle lures), after finding no published detection scores for open-weight models. In zero-shot tests, Qwen 3.8-27B scored 98.9% recall at an 11.6% false-positive rate, while GPT-OSS 20B scored 94.0% recall at a 23.1% false-positive rate, with the AI-written lures the hardest subset for both models. The dataset and MIT-licensed evaluation code are published on Hugging Face and GitHub, and Reveal Wire is asking others to run additional models against it.", "body_md": "Hi — I run Reveal Wire ([youtube.com/@RevealWire](http://youtube.com/@RevealWire)), a channel that tests AI claims with real experiments. For an episode on phishing detection I looked for published detection scores for open-weight models and found zero — for any model. So we built RSB-1000: 1,000 fully synthetic records (780 scam / 220 legitimate, including 60 AI-written ‘adversary-in-the-middle’ lures with polished language and lookalike domains), frozen October 2026, SHA256-pinned, CC-BY-4.0. Dataset: [RevealWire/rsb-1000 · Datasets at Hugging Face](https://huggingface.co/datasets/RevealWire/rsb-1000) — code and eval script: [GitHub - RevealWire/rsb-1000: RSB-1000: a frozen 1,000-record synthetic benchmark for scam/phishing detection (780 scam / 220 legit, incl. 60 AiTM cases). Data: CC-BY-4.0, code: MIT. · GitHub](https://github.com/RevealWire/rsb-1000). Tested zero-shot: Qwen 3.8-27B — 98.9% recall, 11.6% FPR; GPT-OSS 20B — 94.0% recall, 23.1% FPR. The AI-written lures were the hardest subset for both models. If you run other models against it, I’d genuinely love to see the numbers.", "url": "https://wpnews.pro/news/rsb-1000-1000-record-synthetic-benchmark-for-phishing-detection-cc-by-4-0", "canonical_source": "https://discuss.huggingface.co/t/rsb-1000-1-000-record-synthetic-benchmark-for-phishing-detection-cc-by-4-0/190933#post_1", "published_at": "2026-10-23 04:33:13+00:00", "updated_at": "2026-10-23 04:40:17.953684+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "machine-learning", "ai-research"], "entities": ["Reveal Wire", "RSB-1000", "Qwen 3.8-27B", "GPT-OSS 20B", "Hugging Face", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rsb-1000-1000-record-synthetic-benchmark-for-phishing-detection-cc-by-4-0", "markdown": "https://wpnews.pro/news/rsb-1000-1000-record-synthetic-benchmark-for-phishing-detection-cc-by-4-0.md", "text": "https://wpnews.pro/news/rsb-1000-1000-record-synthetic-benchmark-for-phishing-detection-cc-by-4-0.txt", "jsonld": "https://wpnews.pro/news/rsb-1000-1000-record-synthetic-benchmark-for-phishing-detection-cc-by-4-0.jsonld"}}