{"slug": "bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox", "title": "Bulwark: a Turkish-first scam/spam classifier + a benchmark built from a real inbox (false alarms 53% → 14%)", "summary": "A developer publishing as SavienWalker released Bulwark, a Turkish-first scam and spam classifier fine-tuned from the 307M-parameter mmBERT-base model Horizon-Labs/phishing-scam-detector-base, cutting false alarms on legitimate messages from 53% (54/102) to 14% (14/102) and raising 3-class accuracy from 64% to 87% on a sealed 434-message test split drawn from a real inbox. The base model labelled 26% of legitimate messages as fraud versus 9% for Bulwark, while unwanted spam-plus-fraud catch fell from 100% to 98%, and on a 13-pair human-reviewed contrast set Bulwark got 12/13 pairs fully correct against 9/13 for the base. The model (Apache-2.0), benchmark (CC BY 4.0) and collection/training code are open, with the author noting fraud recall relies on LLM-assisted synthetic hard sets and that phone calls, long chats and fake websites are out of scope.", "body_md": "**TL;DR:** I tested a popular multilingual scam detector (Horizon-Labs/phishing-scam-detector-base) on real SMS from my own phone. It flagged **53%** of genuine Turkish bank, cargo and e-Government notifications as unwanted. I fine-tuned it into **Bulwark** and built a small, honest Turkish benchmark around the problem. Everything is open (model: Apache-2.0, benchmark: CC BY 4.0, code: Apache-2.0).\n\nModel: [SavienWalker/bulwark-scam-detector](https://huggingface.co/SavienWalker/bulwark-scam-detector)\n\n## \n\n|  | Base model (t=0.5) | Bulwark (t=0.6) | \n| False alarm on legitimate messages | 53% (54/102) | **14%** (14/102) | \n| Legitimate message labelled *fraud* | 26% | **9%** | \n| Unwanted (spam+fraud) caught | 100% | 98% | \n| 3-class accuracy | 64% | **87%** | \n\n On a human-reviewed hard set built from **contrast pairs** (a fake message and the genuine one it imitates, e.g. fake vs. real toll-violation notice), both items correct: base 9/13, Bulwark **12/13**. A quick 6-language probe (de/fr/es/pt/ru/ar) shows no regression.\n\n## \n\n- **A sealed test split.** 434 real SMS from my inbox, PII-masked, split by*sender* into dev/test so one institution never appears in both halves. Model selection and threshold sweep on dev only.\n- **Leakage check.** My first synthetic training set had 1,303 messages too similar (token Jaccard ≥ 0.5) to evaluation messages, because I had written templates after seeing them. Rewrote 31 templates and dropped the rest.\n- **Leaked personal data is a shortcut trap.** Scammers now include your name, plate, card’s last 4 digits, old address. In my generated data those details appeared almost only in legitimate messages, so the model was about to learn “it knows my name, so it’s real”. I injected them into legitimate*and* fraud messages at equal rates.\n- **Real-world patterns from public complaints.** Escalating prize fees, fake car rentals that “can’t take cards”, job offers that end in money-mule transfers or screen-sharing into e-Government. No complaint text was copied; patterns were rewritten as templates.\n\n## \n\nBase: mmBERT-base (307M). Embeddings frozen (110M trained), bf16, 2 epochs, ~9k messages (~8k Turkish synthetic from 190 templates + ~1k UCI SMS Spam as English replay, spam/fraud distilled from the base model). About 6 minutes on a laptop RTX 4050 (6 GB).\n\n## \n\nSingle messages only: phone calls, long chats and fake websites are out of scope. Real non-betting fraud in one inbox is rare, so fraud recall relies on the synthetic hard sets, which were written by the same LLM-assisted pipeline as the training templates. That is stated in the cards and results are reported separately.\n\n**Next up:** multi-turn chat context, and scams that arrive via fake websites & ads.\n\nCode (collection from iPhone/Android backups with immediate PII masking, labelling tool, leakage check, training, eval): [github.com/SavienWalker/bulwark](https://github.com/SavienWalker/bulwark)\n\nBenchmark on the Hub: SavienWalker/bulwark-bench-tr\n\nFeedback, and real scam samples (especially non-Turkish ones), are very welcome!", "url": "https://wpnews.pro/news/bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox", "canonical_source": "https://discuss.huggingface.co/t/bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox-false-alarms-53-14/191357#post_3", "published_at": "2026-11-03 07:08:45+00:00", "updated_at": "2026-11-03 07:19:28.783299+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-safety"], "entities": ["Bulwark", "SavienWalker", "Horizon-Labs/phishing-scam-detector-base", "mmBERT-base", "Hugging Face", "UCI SMS Spam"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox", "markdown": "https://wpnews.pro/news/bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox.md", "text": "https://wpnews.pro/news/bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox.txt", "jsonld": "https://wpnews.pro/news/bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox.jsonld"}}