# Bulwark: a Turkish-first scam/spam classifier + a benchmark built from a real inbox (false alarms 53% → 14%)

> Source: <https://discuss.huggingface.co/t/bulwark-a-turkish-first-scam-spam-classifier-a-benchmark-built-from-a-real-inbox-false-alarms-53-14/191357#post_3>
> Published: 2026-11-03 07:08:45+00:00

**TL;DR:** I tested a popular multilingual scam detector (Horizon-Labs/phishing-scam-detector-base) on real SMS from my own phone. It flagged **53%** of genuine Turkish bank, cargo and e-Government notifications as unwanted. I fine-tuned it into **Bulwark** and built a small, honest Turkish benchmark around the problem. Everything is open (model: Apache-2.0, benchmark: CC BY 4.0, code: Apache-2.0).

Model: [SavienWalker/bulwark-scam-detector](https://huggingface.co/SavienWalker/bulwark-scam-detector)

## 

|  | Base model (t=0.5) | Bulwark (t=0.6) | 
| False alarm on legitimate messages | 53% (54/102) | **14%** (14/102) | 
| Legitimate message labelled *fraud* | 26% | **9%** | 
| Unwanted (spam+fraud) caught | 100% | 98% | 
| 3-class accuracy | 64% | **87%** | 

 On a human-reviewed hard set built from **contrast pairs** (a fake message and the genuine one it imitates, e.g. fake vs. real toll-violation notice), both items correct: base 9/13, Bulwark **12/13**. A quick 6-language probe (de/fr/es/pt/ru/ar) shows no regression.

## 

- **A sealed test split.** 434 real SMS from my inbox, PII-masked, split by*sender* into dev/test so one institution never appears in both halves. Model selection and threshold sweep on dev only.
- **Leakage check.** My first synthetic training set had 1,303 messages too similar (token Jaccard ≥ 0.5) to evaluation messages, because I had written templates after seeing them. Rewrote 31 templates and dropped the rest.
- **Leaked personal data is a shortcut trap.** Scammers now include your name, plate, card’s last 4 digits, old address. In my generated data those details appeared almost only in legitimate messages, so the model was about to learn “it knows my name, so it’s real”. I injected them into legitimate*and* fraud messages at equal rates.
- **Real-world patterns from public complaints.** Escalating prize fees, fake car rentals that “can’t take cards”, job offers that end in money-mule transfers or screen-sharing into e-Government. No complaint text was copied; patterns were rewritten as templates.

## 

Base: mmBERT-base (307M). Embeddings frozen (110M trained), bf16, 2 epochs, ~9k messages (~8k Turkish synthetic from 190 templates + ~1k UCI SMS Spam as English replay, spam/fraud distilled from the base model). About 6 minutes on a laptop RTX 4050 (6 GB).

## 

Single messages only: phone calls, long chats and fake websites are out of scope. Real non-betting fraud in one inbox is rare, so fraud recall relies on the synthetic hard sets, which were written by the same LLM-assisted pipeline as the training templates. That is stated in the cards and results are reported separately.

**Next up:** multi-turn chat context, and scams that arrive via fake websites & ads.

Code (collection from iPhone/Android backups with immediate PII masking, labelling tool, leakage check, training, eval): [github.com/SavienWalker/bulwark](https://github.com/SavienWalker/bulwark)

Benchmark on the Hub: SavienWalker/bulwark-bench-tr

Feedback, and real scam samples (especially non-Turkish ones), are very welcome!