Deploying ML-IDS: An Open-Source, CatBoost-Powered Network Classifier

An open-source network intrusion detection system called ML-IDS, powered by CatBoost, has been released on GitHub under the MIT License. The tool uses only Layer 3 and Layer 4 traffic features to avoid overfitting and class imbalance that plague traditional ML-based IDS. The project aims to provide a deployable, behavior-based classifier for real-world network security.

When building defensive cybersecurity tools, there is a massive, frustrating divide between theoretical machine learning models and real-world utility. If you have ever tried to train a classifier on raw network traffic, you have likely run into the classic textbook traps. You parse some packet captures, split your dataset, run a standard classifier, and watch your terminal light up with a stellar 99.9% validation accuracy . You feel like an absolute genius who has solved network security with fifty lines of Python. Then, you deploy the model against a live, hostile attack stream. Absolute, deafening silence. The model lets every single malicious exploit slip straight through your boundary controls. The actual detection rate?Exactly zero. What went wrong? Your model didn’t actually learn how an attack behaves. It fell victim to the systemic traps of network telemetry: severe class imbalance and massive overfitting on transient features . To solve this, I built ML-IDS — a lightweight, command-line Network Intrusion Detection System powered by CatBoost. It operates strictly on the behavioral heuristics of Layer 3 and Layer 4 traffic to spot threats without relying on fragile, static signatures or volatile routing data. The project is fully open-source on GitHub under the MIT License, and I’m actively looking for contributors to help scale its capabilities: Repo:github.com/pop123-ux/pcap-ml-traffic-classifier Here is exactly how the model works, why traditional classifiers fail in network environments, and how we solved the zero-detection problem. To understand why traditional classifiers fail on network streams, we have to look at what happens inside a raw network socket. When you capture traffic with Scapy, a single packet yields dozens of potential features. In my early, failed iterations, I made the mistake of feeding the model almost everything I could extract: On paper, this looks highly descriptive. In reality, it is a recipe for catastrophic overfitting . php The Overfitting Loop: Attacker IP: 10.0.0.15 - Launches exploit targeting Ephemeral Port: 49152 Your ML Model Learns - "Ah Anything involving IP 10.0.0.15 or Port 49152 is malicious " In production networks, ephemeral ports rotate instantly and IP addresses change constantly. By training on these variables, the model never learns the actual structural behavior of an attack. Instead, it merely memorizes a static set of coordinates. The moment you expose the model to a different subnet or a new attacker IP, it goes completely blind. To build an IDS that actually generalizes across different networks, we have to strip away high-cardinality routing variables. We must force the model to evaluate how the data is moving, rather than who is sending it. ML-IDS restricts its inputs to four core Layer 3 and Layer 4 telemetry headers: Feature Vector = { packet size, ttl, protocol, tcp flags } Once we sanitized our feature space, traditional algorithms like Random Forest or standard Gradient Boosting still struggled. They kept falling victim to extreme class imbalance . In any production network, benign baseline traffic is a massive, raging ocean. Malicious packets are tiny, quiet drops in that ocean. If 99.9% of your training data is benign, a standard classifier will quickly learn that it can achieve a near-perfect loss score by simply predicting “0” Benign for every single packet it ever sees. This is where CatBoost changes the game. Instead of manually manipulating our dataset using synthetic oversampling algorithms like SMOTE — which can distort the delicate timing sequences of network packets — we handle the imbalance directly inside the model’s objective function during initialization: self.model = CatBoostClassifier iterations=300, learning rate=0.05, depth=3, l2 leaf reg=5, loss function='Logloss', eval metric='F1', cat features= 'protocol', 'tcp flags' , auto class weights='Balanced', The exact parameter that fixed the 0-detection bug early stopping rounds=50 By specifying auto class weights='Balanced', CatBoost automatically calculates the ratio between benign and malicious frames. It heavily penalizes the model during training if it misclassifies a malicious packet. If it misses a benign packet, it gets a tiny slap on the wrist; if it misses a single exploit packet, the loss penalties skyrocket. This single parameter completely eliminated our zero-detection failure. CatBoost’s architectural advantage lies in its use of Symmetric Decision Trees . Unlike traditional deep trees that split unbalanced paths, CatBoost builds balanced trees where the exact same splitting criterion is used at all nodes of a given depth. This acts as a powerful regularizer, dramatically reducing training time and preventing overfitting on high-cardinality categorical features like tcp flags. It handles categories natively, mapping complex structural relationships without blowing up our feature space through clumsy, sparse One-Hot Encoding OHE arrays. A model that simply prints “Danger Detected ” is functionally useless to an incident responder. You need to know where the threat is originating so you can stop it. Even though we explicitly omit IP addresses from our ML training features to prevent overfitting, we don’t throw them away. The ML-IDS ingestion pipeline carries the IP metadata out-of-band in a parallel data structure. When CatBoost flags a specific packet vector as malicious, our engine maps that prediction back to its original raw src ip. This allows ML-IDS to output an aggregated, real-time threat matrix ranking the top malicious hosts: === Inference Summary ===Total Packets Scanned : 32,450Malicious Detections : 714 2.20% 🚨 WARNING: Malicious traffic detected Aggregating threat actors... --- TOP SUSPECTED ATTACKING IPs FOR FIREWALL BLOCKLIST --- ▸ IP: 10.0.0.89 | Incident Packets: 612 ▸ IP: 192.168.1.110 | Incident Packets: 102 This transforms a raw machine learning pipeline into a practical, actionable blue-teaming asset . The repository is now live at pop123-ux/pcap-ml-traffic-classifier https://github.com/pop123-ux/pcap-ml-traffic-classifier . I have structured this project to be highly modular because I want the security and machine learning communities to break, build, and extend it. The foundation is solid, but there is so much more we can build. Here is where the project is heading next, and where your contributions can make an immediate impact: We want to integrate Shannon Entropy calculations of packet payloads to identify encrypted command-and-control C2 beacons or hidden reverse shells. The calculation is: H X = −Σ P xᵢ log₂ P xᵢ If a protocol that typically carries plain text like HTTP suddenly shows highly random payload entropy, our model can immediately flag it as encrypted data exfiltration or a tunnel. Currently, the tool processes offline PCAP files. The next major milestone is extending Scapy’s ingestion engine to sniff live packets directly from a network interface card NIC using raw asynchronous sockets, enabling real-time edge inference . We want to write robust benchmark scripts to compare CatBoost’s processing speed and F1-scores against LightGBM and custom PyTorch architectures, specifically focusing on low-power hardware like a Raspberry Pi running at a network gateway . To keep this project highly collaborative yet legally protected for everyone, the entire codebase is distributed under the MIT License . This means you are completely free to use, modify, copy, and distribute the software — whether for personal research, academic thesis work, or private commercial enterprise applications — provided that the original copyright notice and permission notice are included in all copies of the software. Get the tool running locally in less than two minutes: 1. Clone the repositorygit clone https://github.com/pop123-ux/pcap-ml-traffic-classifier.gitcd pcap-ml-traffic-classifier 2. Install dependenciespip install -r requirements.txt 3. Train the engine with your capture files clean vs. attack baseline python3 app.py --train benign baseline.pcap simulated attack.pcap 4. Analyze suspicious traffic and extract top threat actorspython3 app.py --analyze active capture.pcap Theoretical machine learning is comfortable. It lives in clean Jupyter Notebooks, evaluates static CSV files, and thrives on perfect validation scores that collapse under the slightest bit of real-world noise. But defensive security isn’t clean. It is chaotic, highly imbalanced, and constantly changing. If you want to build defensive tools that actually protect systems, you have to build systems that generalize . You have to write code that deals with the physical reality of the data. Whether you are a blue teamer looking to upgrade your defensive stack, an ML engineer interested in high-speed tabular classification, or a security researcher who loves getting their hands dirty with packet headers — head over to the repo, star it, fork it, and let’s build this together. The code is ready. What are you going to build with it? If this post helped you, a clap or fifty 👏 goes a long way. Follow me for more posts on defensive ML, packet-level analytics, and building open-source security tools. GitHub: pop123-ux/pcap-ml-traffic-classifier https://github.com/pop123-ux/pcap-ml-traffic-classifier Deploying ML-IDS: An Open-Source, CatBoost-Powered Network Classifier https://pub.towardsai.net/deploying-ml-ids-an-open-source-catboost-powered-network-classifier-2a0558363c50 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.