{"slug": "real-time-network-telemetry-for-ai-building-an-asynchronous-netflow-sflow-in", "title": "Real-Time Network Telemetry for AI: Building an Asynchronous NetFlow/sFlow Ingestion Pipeline in Python", "summary": "A developer built a production-grade asynchronous NetFlow/sFlow ingestion pipeline in Python using Scapy to capture live network traffic and stream structured JSON events for AI-driven anomaly detection and network observability. The pipeline decouples packet capture from processing using a producer-consumer multi-threaded pattern with a thread-safe queue to handle traffic bursts without dropping packets.", "body_md": "As AI architectures transition from static data processing to real-time agentic monitoring, the data engineering pipelines feeding them must adapt. If you are building an AI-driven anomaly detection model, a security agent, or an automated network observability tool, you can't rely on historical batch logs. You need live, structural telemetry straight from the network edge.\n\nWhen monitoring network traffic at scale, two industry standards dominate: **NetFlow** (stateful conversation aggregation) and **sFlow** (stateless packet-level sampling).\n\nIn this post, we will build a production-grade, asynchronous network ingestion engine using **Python** and **Scapy** that captures live wireless traffic, extracts 5-tuple flow metrics, and streams structured JSON events ready to be swallowed by a message broker (like Kafka) or an AI vector database.\n\nThe biggest pitfall when writing a packet sniffer in Python is blocking the execution loop. If your script captures a packet, processes it, formats a JSON string, and logs it sequentially, your script will choke and drop thousands of packets during a sudden burst of network traffic.\n\nTo feed an AI infrastructure safely, we must decouple the **Capture Phase** from the **Processing Phase** using a Producer-Consumer multi-threaded pattern.\n\n```\n[ Wireless Adapter ] \n       │\n       ▼ (Raw Packets)\n┌────────────────────────────────────────────────────────┐\n│ PHASE 1: Capture (Scapy AsyncSniffer Thread)          │\n│ - Sniffs raw frames in a non-blocking background loop   │\n└────────────────────────────────────────────────────────┘\n       │\n       ▼ (Fast Handoff: .put_nowait)\n┌────────────────────────────────────────────────────────┐\n│ PHASE 2: Buffer (Thread-Safe bounded Queue)           │\n│ - Acts as a shock absorber for network traffic bursts  │\n└────────────────────────────────────────────────────────┘\n       │\n       ▼ (Distributed to Worker Pool)\n┌────────────────────────────────────────────────────────┐\n│ PHASE 3: Process & Emit (Concurrent Thread Workers)    │\n│ - Worker 1  │  Worker 2  │  Worker 3  │  Worker 4      │\n│ - Decodes Layers (IP, TCP, UDP)                        │\n│ - Updates Stateful NetFlow Cache (Aggregates)          │\n│ - Outputs Structured JSON Streaming Event              │\n└────────────────────────────────────────────────────────┘\n```\n\nBelow is the complete, modern Python implementation utilizing Scapy's explicit layer APIs, asynchronous execution, and thread-safe buffering.\n\nEnsure you have the native packet capture library installed on your host system (`libpcap`\n\nfor Linux/macOS, `Npcap`\n\nfor Windows) and install Scapy:\n\n```\npip install scapy\npython\nimport os\nimport sys\nimport queue\nimport time\nimport json\nfrom dataclasses import dataclass, asdict\nfrom concurrent.futures import ThreadPoolExecutor\nfrom typing import Dict, Tuple, Optional\n\n# Explicitly target specific submodules for deterministic enterprise scopes\nfrom scapy.all import AsyncSniffer, Packet, conf\nfrom scapy.layers.inet import IP, TCP, UDP\n\n# --- CONFIGURATION ---\nMAX_QUEUE_SIZE = 10000\nNUM_WORKERS = 4\n# Replace with your specific active network interface index or string name\nINTERFACE: Optional[str | int] = 24  \n\n# --- DATA MODELS ---\n@dataclass(frozen=True)\nclass FlowKey:\n    src_ip: str\n    dst_ip: str\n    src_port: int\n    dst_port: int\n    protocol: int\n\n@dataclass\nclass FlowMetrics:\n    packet_count: int = 0\n    byte_count: int = 0\n    first_seen: float = 0.0\n    last_seen: float = 0.0\n\n# --- INGESTION PIPELINE ENGINE ---\nclass AIStreamIngestionPipeline:\n    def __init__(self, interface: Optional[str | int] = None):\n        self.interface = interface\n        self.packet_queue: queue.Queue[Packet] = queue.Queue(maxsize=MAX_QUEUE_SIZE)\n        self.flow_cache: Dict[FlowKey, FlowMetrics] = {}\n        self.executor = ThreadPoolExecutor(max_workers=NUM_WORKERS)\n        self.is_running = False\n        self.sniffer: Optional[AsyncSniffer] = None\n\n    def _extract_packet_meta(self, packet: Packet) -> Optional[Tuple[FlowKey, int]]:\n        \"\"\"Parses raw layers using container-optimized syntax.\"\"\"\n        if IP not in packet:\n            return None\n\n        ip_layer = packet[IP]\n        src_port, dst_port = 0, 0\n\n        # Safe inspection of Layer 4 using direct class indexing\n        if TCP in packet:\n            src_port = packet[TCP].sport\n            dst_port = packet[TCP].dport\n        elif UDP in packet:\n            src_port = packet[UDP].sport\n            dst_port = packet[UDP].dport\n\n        key = FlowKey(\n            src_ip=ip_layer.src,\n            dst_ip=ip_layer.dst,\n            src_port=src_port,\n            dst_port=dst_port,\n            protocol=ip_layer.proto\n        )\n        return key, len(packet)\n\n    def _worker_process_queue(self) -> None:\n        \"\"\"Isolated consumer thread pool extracting flow features.\"\"\"\n        while self.is_running or not self.packet_queue.empty():\n            try:\n                packet = self.packet_queue.get(timeout=0.5)\n            except queue.Empty:\n                continue\n\n            meta = self._extract_packet_meta(packet)\n            if not meta:\n                self.packet_queue.task_done()\n                continue\n\n            key, packet_len = meta\n            current_time = time.time()\n\n            # 1. Stateful Aggregation (NetFlow Telemetry Layer)\n            if key not in self.flow_cache:\n                self.flow_cache[key] = FlowMetrics(first_seen=current_time)\n\n            metrics = self.flow_cache[key]\n            metrics.packet_count += 1\n            metrics.byte_count += packet_len\n            metrics.last_seen = current_time\n\n            # 2. Structured Streaming Payload (sFlow Event Layer)\n            ingest_payload = {\n                \"event_type\": \"network_flow_sample\",\n                \"timestamp\": current_time,\n                \"flow_key\": asdict(key),\n                \"packet_size_bytes\": packet_len,\n                \"aggregate_metrics\": asdict(metrics)\n            }\n\n            # Emit clean JSON to stdout (pipe into a stream wrapper, Kafka, or vector db)\n            print(json.dumps(ingest_payload))\n            self.packet_queue.task_done()\n\n    def _enqueue_packet(self, packet: Packet) -> None:\n        \"\"\"High-speed producer callback. Hands off raw socket frames immediately.\"\"\"\n        try:\n            self.packet_queue.put_nowait(packet)\n        except queue.Full:\n            # Backpressure handling: Drops edge frames gracefully if queue overflows\n            pass\n\n    def start(self) -> None:\n        \"\"\"Spawns workers and executes the non-blocking packet sniffer thread.\"\"\"\n        print(f\"[*] Initializing pipeline on interface: {self.interface or 'Default Active OS Interface'}\",\n              file=sys.stderr)\n        self.is_running = True\n\n        for _ in range(NUM_WORKERS):\n            self.executor.submit(self._worker_process_queue)\n\n        self.sniffer = AsyncSniffer(\n            iface=self.interface,\n            prn=self._enqueue_packet,\n            store=False  # Crucial: Drops raw historical packets out of memory instantly\n        )\n        self.sniffer.start()\n        print(\"[*] Ingestion pipeline is live and streaming...\", file=sys.stderr)\n\n    def stop(self) -> None:\n        \"\"\"Gracefully signs off threads and closes open sockets.\"\"\"\n        print(\"\\n[*] Shutting down ingestion pipeline gracefully...\", file=sys.stderr)\n        if self.sniffer:\n            self.sniffer.stop()\n\n        self.is_running = False\n        self.executor.shutdown(wait=True)\n        print(\"[*] Pipeline stopped cleanly.\", file=sys.stderr)\n\nif __name__ == \"__main__\":\n    # Administrative privilege guard for raw socket access\n    if os.name != 'nt' and os.getuid() != 0:\n        print(\"[-] Critical Error: Root privileges required for raw network socket access.\", file=sys.stderr)\n        sys.exit(1)\n\n    # Windows Device GUID Resolution block\n    target_interface = INTERFACE\n    if os.name == 'nt' and isinstance(INTERFACE, int):\n        try:\n            target_interface = conf.ifaces.dev_from_index(INTERFACE)\n            print(f\"[*] Resolved index {INTERFACE} to device: {target_interface.description}\", file=sys.stderr)\n        except Exception as e:\n            print(f\"[-] Warning: Could not resolve interface index {INTERFACE}: {e}. Falling back.\", file=sys.stderr)\n\n    pipeline = AIStreamIngestionPipeline(interface=target_interface)\n    try:\n        pipeline.start()\n        while True:\n            time.sleep(1)\n    except KeyboardInterrupt:\n        pipeline.stop()\n```\n\nInstead of executing high-overhead inspection functions like `packet.haslayer(IP)`\n\n, this script checks containment explicitly using `IP in packet`\n\n. Scapy overloads Python’s internal `__contains__`\n\nmagic methods, ensuring layer validation occurs at the compiler layer.\n\n`store=False`\n\n)\nBy default, Scapy stores every single packet captured in an internal history list inside RAM. Running a default sniffer on an enterprise pipeline for an hour will result in a quick memory leak crash. By assigning `store=False`\n\nin our `AsyncSniffer`\n\n, we consume the packet from the adapter, pass it to our local queue, and free its memory instantly.\n\nOur ingestion loop forces a strict `maxsize=10000`\n\nrule and uses a non-blocking `put_nowait()`\n\nexecution. If the downstream data processor (e.g., your LLM evaluator or vector engine) experiences a latency hiccup, the pipeline intentionally drops incoming packets at the edge rather than letting the memory buffer balloon out of control.\n\nThe pipeline aggregates connection metadata into a stateful 5-tuple format, while instantly streaming stateless structural JSON tokens. This gives your downstream AI immediate access to critical parameters:\n\n```\n{\n  \"event_type\": \"network_flow_sample\", \n  \"timestamp\": 1719409375.42, \n  \"flow_key\": {\n    \"src_ip\": \"10.17.245.72\", \n    \"dst_ip\": \"142.250.180.142\", \n    \"src_port\": 54211, \n    \"dst_port\": 443, \n    \"protocol\": 6\n  }, \n  \"packet_size_bytes\": 1420, \n  \"aggregate_metrics\": {\n    \"packet_count\": 42, \n    \"byte_count\": 59640, \n    \"first_seen\": 1719409371.11, \n    \"last_seen\": 1719409375.42\n  }\n}\n```\n\nThis structural metadata provides your AI system with a comprehensive macro-view map of network behavior, allowing agents to accurately cross-reference anomalies without drowning your entire feature store in costly, raw packet captures.", "url": "https://wpnews.pro/news/real-time-network-telemetry-for-ai-building-an-asynchronous-netflow-sflow-in", "canonical_source": "https://dev.to/ussdlover/real-time-network-telemetry-for-ai-building-an-asynchronous-netflowsflow-ingestion-pipeline-in-35op", "published_at": "2026-06-26 10:56:50+00:00", "updated_at": "2026-06-26 11:04:04.250684+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "machine-learning", "ai-infrastructure"], "entities": ["Python", "Scapy", "NetFlow", "sFlow", "Kafka"], "alternates": {"html": "https://wpnews.pro/news/real-time-network-telemetry-for-ai-building-an-asynchronous-netflow-sflow-in", "markdown": "https://wpnews.pro/news/real-time-network-telemetry-for-ai-building-an-asynchronous-netflow-sflow-in.md", "text": "https://wpnews.pro/news/real-time-network-telemetry-for-ai-building-an-asynchronous-netflow-sflow-in.txt", "jsonld": "https://wpnews.pro/news/real-time-network-telemetry-for-ai-building-an-asynchronous-netflow-sflow-in.jsonld"}}