TL;DR: I tested a popular multilingual scam detector (Horizon-Labs/phishing-scam-detector-base) on real SMS from my own phone. It flagged 53% of genuine Turkish bank, cargo and e-Government notifications as unwanted. I fine-tuned it into Bulwark and built a small, honest Turkish benchmark around the problem. Everything is open (model: Apache-2.0, benchmark: CC BY 4.0, code: Apache-2.0).
Model: SavienWalker/bulwark-scam-detector
#
| | Base model (t=0.5) | Bulwark (t=0.6) |
| False alarm on legitimate messages | 53% (54/102) | **14%** (14/102) |
| Legitimate message labelled fraud | 26% | 9% | | Unwanted (spam+fraud) caught | 100% | 98% | | 3-class accuracy | 64% | 87% |
On a human-reviewed hard set built from contrast pairs (a fake message and the genuine one it imitates, e.g. fake vs. real toll-violation notice), both items correct: base 9/13, Bulwark 12/13. A quick 6-language probe (de/fr/es/pt/ru/ar) shows no regression.
#
- A sealed test split. 434 real SMS from my inbox, PII-masked, split bysender into dev/test so one institution never appears in both halves. Model selection and threshold sweep on dev only.
- Leakage check. My first synthetic training set had 1,303 messages too similar (token Jaccard ≥ 0.5) to evaluation messages, because I had written templates after seeing them. Rewrote 31 templates and dropped the rest.
- Leaked personal data is a shortcut trap. Scammers now include your name, plate, card’s last 4 digits, old address. In my generated data those details appeared almost only in legitimate messages, so the model was about to learn “it knows my name, so it’s real”. I injected them into legitimateand fraud messages at equal rates.
- Real-world patterns from public complaints. Escalating prize fees, fake car rentals that “can’t take cards”, job offers that end in money-mule transfers or screen-sharing into e-Government. No complaint text was copied; patterns were rewritten as templates.
#
Base: mmBERT-base (307M). Embeddings frozen (110M trained), bf16, 2 epochs, ~9k messages (~8k Turkish synthetic from 190 templates + ~1k UCI SMS Spam as English replay, spam/fraud distilled from the base model). About 6 minutes on a laptop RTX 4050 (6 GB).
#
Single messages only: phone calls, long chats and fake websites are out of scope. Real non-betting fraud in one inbox is rare, so fraud recall relies on the synthetic hard sets, which were written by the same LLM-assisted pipeline as the training templates. That is stated in the cards and results are reported separately.
Next up: multi-turn chat context, and scams that arrive via fake websites & ads.
Code (collection from iPhone/Android backups with immediate PII masking, labelling tool, leakage check, training, eval): github.com/SavienWalker/bulwark
Benchmark on the Hub: SavienWalker/bulwark-bench-tr Feedback, and real scam samples (especially non-Turkish ones), are very welcome!