Bulwark: a Turkish-first scam/spam classifier + a benchmark built from a real inbox (false alarms 53% → 14%)

A developer publishing as SavienWalker released Bulwark, a Turkish-first scam and spam classifier fine-tuned from the 307M-parameter mmBERT-base model Horizon-Labs/phishing-scam-detector-base, cutting false alarms on legitimate messages from 53% (54/102) to 14% (14/102) and raising 3-class accuracy from 64% to 87% on a sealed 434-message test split drawn from a real inbox. The base model labelled 26% of legitimate messages as fraud versus 9% for Bulwark, while unwanted spam-plus-fraud catch fell from 100% to 98%, and on a 13-pair human-reviewed contrast set Bulwark got 12/13 pairs fully correct against 9/13 for the base. The model (Apache-2.0), benchmark (CC BY 4.0) and collection/training code are open, with the author noting fraud recall relies on LLM-assisted synthetic hard sets and that phone calls, long chats and fake websites are out of scope.

TL;DR: I tested a popular multilingual scam detector Horizon-Labs/phishing-scam-detector-base on real SMS from my own phone. It flagged 53% of genuine Turkish bank, cargo and e-Government notifications as unwanted. I fine-tuned it into Bulwark and built a small, honest Turkish benchmark around the problem. Everything is open model: Apache-2.0, benchmark: CC BY 4.0, code: Apache-2.0 . Model: SavienWalker/bulwark-scam-detector https://huggingface.co/SavienWalker/bulwark-scam-detector | | Base model t=0.5 | Bulwark t=0.6 | | False alarm on legitimate messages | 53% 54/102 | 14% 14/102 | | Legitimate message labelled fraud | 26% | 9% | | Unwanted spam+fraud caught | 100% | 98% | | 3-class accuracy | 64% | 87% | On a human-reviewed hard set built from contrast pairs a fake message and the genuine one it imitates, e.g. fake vs. real toll-violation notice , both items correct: base 9/13, Bulwark 12/13 . A quick 6-language probe de/fr/es/pt/ru/ar shows no regression. - A sealed test split. 434 real SMS from my inbox, PII-masked, split by sender into dev/test so one institution never appears in both halves. Model selection and threshold sweep on dev only. - Leakage check. My first synthetic training set had 1,303 messages too similar token Jaccard ≥ 0.5 to evaluation messages, because I had written templates after seeing them. Rewrote 31 templates and dropped the rest. - Leaked personal data is a shortcut trap. Scammers now include your name, plate, card’s last 4 digits, old address. In my generated data those details appeared almost only in legitimate messages, so the model was about to learn “it knows my name, so it’s real”. I injected them into legitimate and fraud messages at equal rates. - Real-world patterns from public complaints. Escalating prize fees, fake car rentals that “can’t take cards”, job offers that end in money-mule transfers or screen-sharing into e-Government. No complaint text was copied; patterns were rewritten as templates. Base: mmBERT-base 307M . Embeddings frozen 110M trained , bf16, 2 epochs, ~9k messages ~8k Turkish synthetic from 190 templates + ~1k UCI SMS Spam as English replay, spam/fraud distilled from the base model . About 6 minutes on a laptop RTX 4050 6 GB . Single messages only: phone calls, long chats and fake websites are out of scope. Real non-betting fraud in one inbox is rare, so fraud recall relies on the synthetic hard sets, which were written by the same LLM-assisted pipeline as the training templates. That is stated in the cards and results are reported separately. Next up: multi-turn chat context, and scams that arrive via fake websites & ads. Code collection from iPhone/Android backups with immediate PII masking, labelling tool, leakage check, training, eval : github.com/SavienWalker/bulwark https://github.com/SavienWalker/bulwark Benchmark on the Hub: SavienWalker/bulwark-bench-tr Feedback, and real scam samples especially non-Turkish ones , are very welcome