{"slug": "i-benchmarked-4-lightweight-transformers-for-fault-detection-here-s-what", "title": "I Benchmarked 4 Lightweight Transformers for Fault Detection. Here's What Survived.", "summary": "A developer benchmarked four lightweight transformer models—DistilBERT, MobileBERT, TinyBERT-6L, and TinyBERT-4L—against traditional ML baselines for fault detection, finding that TinyBERT-4L achieved 87.8% F1 with 55 MB size and 18 ms CPU latency, nearly matching XGBoost's 87.9% F1 at 0.5 MB. MobileBERT, designed for mobile deployment, scored 0% F1 on every dataset by predicting only the majority class. The most promising result came from combining models, with all code and results published on GitHub.", "body_md": "Everyone talks about deploying ML on edge devices. Very few people show what happens when you actually try.\n\nI ran a full benchmark of four lightweight transformer models - **DistilBERT, MobileBERT, TinyBERT-6L, and TinyBERT-4L** — against traditional ML baselines on three real-world fault detection datasets.\n\nAll experiments ran on a T4 GPU with consistent hyperparameters.\n\n| Model | F1 | Size | CPU Latency |\n|---|---|---|---|\n| XGBoost | 87.9% |\n0.5 MB |\n0.002 ms |\n| TinyBERT-4L | 87.8% | 55 MB | 18 ms |\n| DistilBERT | 87.6% | 255 MB | 138 ms |\n\nMobileBERT — specifically designed for mobile deployment — scored **0% F1 on every dataset**. It predicted the majority class for every sample across all configurations.\n\n“Designed for mobile” does not mean “works for your use case.”\n\nThe most promising result came from combining models:\n\nAll code and results:\n\n[https://github.com/disha8611/edge-fault-detection-benchmark](https://github.com/disha8611/edge-fault-detection-benchmark)\n\nPrevious research on LLM-based anomaly detection:\n\n[https://arxiv.org/abs/2604.12218](https://arxiv.org/abs/2604.12218)\n\n*Disha Patel — Software Engineer & ML Researcher. I write about engineering, on-device ML, and building systems that work in the real world.*", "url": "https://wpnews.pro/news/i-benchmarked-4-lightweight-transformers-for-fault-detection-here-s-what", "canonical_source": "https://dev.to/dishapatel8/i-benchmarked-4-lightweight-transformers-for-fault-detection-heres-what-survived-n0g", "published_at": "2026-05-31 03:36:34+00:00", "updated_at": "2026-05-31 04:12:05.390269+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "neural-networks", "ai-research", "mlops"], "entities": ["DistilBERT", "MobileBERT", "TinyBERT-6L", "TinyBERT-4L", "XGBoost", "Disha Patel", "T4 GPU"], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-4-lightweight-transformers-for-fault-detection-here-s-what", "markdown": "https://wpnews.pro/news/i-benchmarked-4-lightweight-transformers-for-fault-detection-here-s-what.md", "text": "https://wpnews.pro/news/i-benchmarked-4-lightweight-transformers-for-fault-detection-here-s-what.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-4-lightweight-transformers-for-fault-detection-here-s-what.jsonld"}}