{"slug": "deploying-ml-ids-an-open-source-catboost-powered-network-classifier", "title": "Deploying ML-IDS: An Open-Source, CatBoost-Powered Network Classifier", "summary": "An open-source network intrusion detection system called ML-IDS, powered by CatBoost, has been released on GitHub under the MIT License. The tool uses only Layer 3 and Layer 4 traffic features to avoid overfitting and class imbalance that plague traditional ML-based IDS. The project aims to provide a deployable, behavior-based classifier for real-world network security.", "body_md": "When building defensive cybersecurity tools, there is a massive, frustrating divide between theoretical machine learning models and real-world utility.\n\nIf you have ever tried to train a classifier on raw network traffic, you have likely run into the classic textbook traps. You parse some packet captures, split your dataset, run a standard classifier, and watch your terminal light up with a stellar **99.9% validation accuracy**. You feel like an absolute genius who has solved network security with fifty lines of Python.\n\nThen, you deploy the model against a live, hostile attack stream.\n\n**Absolute, deafening silence.**\n\nThe model lets every single malicious exploit slip straight through your boundary controls. The actual detection rate?Exactly zero.\n\nWhat went wrong? Your model didn’t actually learn how an attack behaves. It fell victim to the systemic traps of network telemetry: **severe class imbalance** and **massive overfitting on transient features**.\n\nTo solve this, I built **ML-IDS** — a lightweight, command-line Network Intrusion Detection System powered by CatBoost. It operates strictly on the behavioral heuristics of Layer 3 and Layer 4 traffic to spot threats without relying on fragile, static signatures or volatile routing data.\n\nThe project is fully open-source on GitHub under the MIT License, and I’m actively looking for contributors to help scale its capabilities:\n\nRepo:github.com/pop123-ux/pcap-ml-traffic-classifier\n\nHere is exactly how the model works, why traditional classifiers fail in network environments, and how we solved the zero-detection problem.\n\nTo understand why traditional classifiers fail on network streams, we have to look at what happens inside a raw network socket. When you capture traffic with Scapy, a single packet yields dozens of potential features. In my early, failed iterations, I made the mistake of feeding the model almost everything I could extract:\n\nOn paper, this looks highly descriptive. In reality, it is a **recipe for catastrophic overfitting**.\n\n``` php\nThe Overfitting Loop:[Attacker IP: 10.0.0.15] -> Launches exploit targeting Ephemeral Port: 49152[Your ML Model Learns]   -> \"Ah! Anything involving IP 10.0.0.15                             or Port 49152 is malicious!\"\n```\n\nIn production networks, ephemeral ports rotate instantly and IP addresses change constantly. By training on these variables, the model never learns the actual structural behavior of an attack. Instead, it merely memorizes a static set of coordinates. The moment you expose the model to a different subnet or a new attacker IP, it goes completely blind.\n\nTo build an IDS that actually generalizes across different networks, we have to strip away high-cardinality routing variables. We must force the model to evaluate **how** the data is moving, rather than **who** is sending it.\n\nML-IDS restricts its inputs to four core Layer 3 and Layer 4 telemetry headers:\n\n```\nFeature Vector = { packet_size, ttl, protocol, tcp_flags }\n```\n\nOnce we sanitized our feature space, traditional algorithms like Random Forest or standard Gradient Boosting still struggled. They kept falling victim to **extreme class imbalance**.\n\nIn any production network, benign baseline traffic is a massive, raging ocean. Malicious packets are tiny, quiet drops in that ocean. If 99.9% of your training data is benign, a standard classifier will quickly learn that it can achieve a near-perfect loss score by simply predicting “0” (Benign) for every single packet it ever sees.\n\nThis is where **CatBoost changes the game.**\n\nInstead of manually manipulating our dataset using synthetic oversampling algorithms (like SMOTE) — which can distort the delicate timing sequences of network packets — we handle the imbalance directly inside the model’s objective function during initialization:\n\n```\nself.model = CatBoostClassifier(    iterations=300,    learning_rate=0.05,    depth=3,    l2_leaf_reg=5,    loss_function='Logloss',    eval_metric='F1',    cat_features=['protocol', 'tcp_flags'],    auto_class_weights='Balanced',  # The exact parameter that fixed                                    # the 0-detection bug    early_stopping_rounds=50)\n```\n\nBy specifying auto_class_weights='Balanced', CatBoost automatically calculates the ratio between benign and malicious frames. It heavily penalizes the model during training if it misclassifies a malicious packet. If it misses a benign packet, it gets a tiny slap on the wrist; if it misses a single exploit packet, the loss penalties skyrocket. **This single parameter completely eliminated our zero-detection failure.**\n\nCatBoost’s architectural advantage lies in its use of **Symmetric Decision Trees**. Unlike traditional deep trees that split unbalanced paths, CatBoost builds balanced trees where the exact same splitting criterion is used at all nodes of a given depth.\n\nThis acts as a powerful regularizer, dramatically reducing training time and preventing overfitting on high-cardinality categorical features like tcp_flags. It handles categories natively, mapping complex structural relationships without blowing up our feature space through clumsy, sparse One-Hot Encoding (OHE) arrays.\n\nA model that simply prints *“Danger Detected!”* is functionally useless to an incident responder. You need to know **where** the threat is originating so you can stop it.\n\nEven though we explicitly omit IP addresses from our ML training features to prevent overfitting, we don’t throw them away. The ML-IDS ingestion pipeline carries the IP metadata **out-of-band** in a parallel data structure.\n\nWhen CatBoost flags a specific packet vector as malicious, our engine maps that prediction back to its original raw src_ip. This allows ML-IDS to output an aggregated, real-time threat matrix ranking the top malicious hosts:\n\n```\n=== Inference Summary ===Total Packets Scanned : 32,450Malicious Detections  : 714 (2.20%)\n🚨 WARNING: Malicious traffic detected! Aggregating threat actors...\n--- TOP SUSPECTED ATTACKING IPs (FOR FIREWALL BLOCKLIST) --- ▸ IP: 10.0.0.89       | Incident Packets: 612 ▸ IP: 192.168.1.110   | Incident Packets: 102\n```\n\nThis transforms a raw machine learning pipeline into a practical, actionable **blue-teaming asset**.\n\nThe repository is now live at [ pop123-ux/pcap-ml-traffic-classifier](https://github.com/pop123-ux/pcap-ml-traffic-classifier).\n\nI have structured this project to be highly modular because I want the security and machine learning communities to break, build, and extend it.\n\nThe foundation is solid, but there is so much more we can build. Here is where the project is heading next, and where your contributions can make an immediate impact:\n\nWe want to integrate **Shannon Entropy** calculations of packet payloads to identify encrypted command-and-control (C2) beacons or hidden reverse shells. The calculation is:\n\nH(X) = −Σ P(xᵢ) log₂ P(xᵢ)\n\nIf a protocol that typically carries plain text (like HTTP) suddenly shows highly random payload entropy, our model can immediately flag it as encrypted data exfiltration or a tunnel.\n\nCurrently, the tool processes offline PCAP files. The next major milestone is extending Scapy’s ingestion engine to sniff live packets directly from a network interface card (NIC) using raw asynchronous sockets, enabling **real-time edge inference**.\n\nWe want to write robust benchmark scripts to compare CatBoost’s processing speed and F1-scores against **LightGBM** and custom **PyTorch** architectures, specifically focusing on low-power hardware (like a Raspberry Pi running at a network gateway).\n\nTo keep this project highly collaborative yet legally protected for everyone, the entire codebase is distributed under the **MIT License**.\n\nThis means you are completely free to use, modify, copy, and distribute the software — whether for personal research, academic thesis work, or private commercial enterprise applications — provided that the original copyright notice and permission notice are included in all copies of the software.\n\nGet the tool running locally in less than two minutes:\n\n```\n# 1. Clone the repositorygit clone https://github.com/pop123-ux/pcap-ml-traffic-classifier.gitcd pcap-ml-traffic-classifier\n# 2. Install dependenciespip install -r requirements.txt\n# 3. Train the engine with your capture files (clean vs. attack baseline)python3 app.py --train benign_baseline.pcap simulated_attack.pcap\n# 4. Analyze suspicious traffic and extract top threat actorspython3 app.py --analyze active_capture.pcap\n```\n\nTheoretical machine learning is comfortable. It lives in clean Jupyter Notebooks, evaluates static CSV files, and thrives on perfect validation scores that collapse under the slightest bit of real-world noise.\n\nBut defensive security isn’t clean. It is **chaotic, highly imbalanced, and constantly changing.** If you want to build defensive tools that actually protect systems, you have to build systems that **generalize**. You have to write code that deals with the physical reality of the data.\n\nWhether you are a blue teamer looking to upgrade your defensive stack, an ML engineer interested in high-speed tabular classification, or a security researcher who loves getting their hands dirty with packet headers — head over to the repo, star it, fork it, and let’s build this together.\n\nThe code is ready. **What are you going to build with it?**\n\n*If this post helped you, a clap (or fifty 👏) goes a long way. Follow me for more posts on defensive ML, packet-level analytics, and building open-source security tools.*\n\n**GitHub:** [pop123-ux/pcap-ml-traffic-classifier](https://github.com/pop123-ux/pcap-ml-traffic-classifier)\n\n[Deploying ML-IDS: An Open-Source, CatBoost-Powered Network Classifier](https://pub.towardsai.net/deploying-ml-ids-an-open-source-catboost-powered-network-classifier-2a0558363c50) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/deploying-ml-ids-an-open-source-catboost-powered-network-classifier", "canonical_source": "https://pub.towardsai.net/deploying-ml-ids-an-open-source-catboost-powered-network-classifier-2a0558363c50?source=rss----98111c9905da---4", "published_at": "2026-07-17 06:05:06+00:00", "updated_at": "2026-07-17 06:26:28.754087+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-tools"], "entities": ["CatBoost", "ML-IDS", "GitHub", "Scapy", "Random Forest", "SMOTE"], "alternates": {"html": "https://wpnews.pro/news/deploying-ml-ids-an-open-source-catboost-powered-network-classifier", "markdown": "https://wpnews.pro/news/deploying-ml-ids-an-open-source-catboost-powered-network-classifier.md", "text": "https://wpnews.pro/news/deploying-ml-ids-an-open-source-catboost-powered-network-classifier.txt", "jsonld": "https://wpnews.pro/news/deploying-ml-ids-an-open-source-catboost-powered-network-classifier.jsonld"}}