PCAP-LM: Helping AI assistants read network traffic

Researchers introduced PCAP-LM, a packet-capture notation that rewrites network traffic as compact symbol lines called PacketGlyphs, shrinking real 5G/4G captures about 800 times smaller than Wireshark's tshark detailed text output and fitting every capture inside a single model context window. In tests on 30 unseen captures, a state-of-the-art AI model answered ten basic forensic questions 99.3% correctly from PCAP-LM documents versus 51% from conventional tshark output truncated to the same length, though the authors note the truncated baseline saw only the first few dozen packets. The work, detailed in an arXiv paper, targets the mismatch that turns a few megabytes of traffic into nearly 20 million tokens under standard text conversion.

Packet captures are the ground truth of network operations. When a customer complains about slow downloads or a security team investigates an incident, someone eventually opens a capture file and reads what actually crossed the wire. That work is slow, needs rare expertise and does not scale. Large language models LLMs seem like an obvious helper. They already summarize logs and draft troubleshooting steps. But hand one a packet capture, and it chokes. When converted to text using standard tools, a few megabytes of traffic can expand to nearly 20 million tokens, far more than any model can read at once. We set out to fix that mismatch. The result is PCAP-LM, a way of rewriting a capture so that an AI assistant can read the whole thing in one go. The full details are in our research paper https://arxiv.org/abs/2607.28100 . | Symbol | Layer | Meaning | | / < | Direction | Client → server / reverse | | S A F R P U | TCP | SYN, ACK, FIN, RST, PSH, URG | | ~ / h | TLS | Application data / handshake | | ? / | DNS | Query / response | | +0 – +9 | Size | Log₂ bucket: +0 ≤ 64 B, +3 ≈ 512 B, +6 ≈ 4 KB | | :u :m :s :S | Delay | Gap < 1 ms, < 1 s, < 10 s, ≥ 10 s | | <BOF / <EOF | Boundary | Begin / end of flow | Translate, don’t just shrink The obvious idea, compressing the file, barely helps gzip shrinks a raw capture only about a factor of two . Most traffic today is encrypted, and encrypted data is already as dense as data gets. What an analyst usually needs is the story of each conversation: who talked to whom, when, for how long, how much data flowed in each direction, whether the connection was encrypted, and whether anything went wrong along the way. PCAP-LM keeps that story and deliberately drops the rest. It rewrites each conversation as a compact line of symbols, a little like musical notation, where each packet becomes a short mark for its direction, type, size, and timing. We call these symbols ‘PacketGlyphs’. A short legend at the top explains them, so a general-purpose model can read the document without any special training. Repetitive stretches, such as the thousands of near-identical packets in a large download, are folded into a single ‘this pattern, repeated N times’ run-length encoding . Frequent symbol combinations are merged into composite tokens a constrained variant of Byte-Pair Encoding, or BPE . In the ideal case, a 14,000-packet download collapses to a single line. On real captures, where sizes and timing vary, a typical document is around 100 KB of text. What we found We tested PCAP-LM on real captures from a 5G/4G measurement campaign. On average, the rewritten files were about 800 times smaller than the standard detailed text output of Wireshark’s command-line tool, and every capture fit comfortably inside a single model’s context window. Smaller is only useful if the information remains accurate. We asked a state-of-the-art AI model ten basic forensic questions about each of 30 captures it had never seen, including how many conversations, how long, how many packets, whether there were resets or retransmissions, whether the traffic was encrypted, and which service was contacted. Reading PCAP-LM, it answered 99.3% correctly. Given the conventional text output instead, truncated to the same length as the PCAP-LM document, it managed 51%. That gap needs an honest reading. Because the input was truncated, the model only saw the first few dozen packets of each capture, so questions such as the total duration or packet count were simply unanswerable from it. The lesson is not that our notation is magic. Rather, a summary of the whole capture can be more useful than a detailed view of only a small part. Most questions could in fact be answered from the header and per-conversation summaries alone; testing what the packet-level symbols add is still to be done. | Representation | Mean tokens | Versus PCAP-LM+RLE+BPE | | tshark -T json | 22,050,813 | 955× larger | | tshark -V | 18,738,238 | 812× larger | | Raw PCAP bytes ÷ 4 | 852,750 | 37× larger | | PCAP + gzip-9 | 415,780 | 18× larger | | PCAP-LM raw glyphs | 44,591 | 1.9× larger | | PCAP-LM + BPE no RLE | 39,745 | 1.7× larger | | PCAP-LM + RLE | 25,424 | 1.1× larger | | PCAP-LM + RLE + BPE | 23,091 | 1× | Knowing the limits This has a price. PCAP-LM does not keep payload contents, sequence numbers or exact timings, and our simple checks missed retransmissions in about a quarter of the affected captures, all taken behind a proxy. To manage that trade-off, each PCAP-LM document can retain a pointer back to its original packets, allowing an LLM or human analyst to retrieve the full details when something looks suspicious. Our tests also covered only one kind of traffic: Large encrypted downloads from a single test server. Other protocols and a broader mix of applications may require additional symbols and retraining the compression vocabulary and potentially fine-tuning the model if the legend grows too large . PCAP-LM never decrypts anything; encrypted content stays opaque by design. Why this matters for operators The bigger picture is about how network operators can practically deploy AI on network data without compromising security, budgets, or accuracy: - Cost and context limits: Many AI services charge according to the amount of text they process. By collapsing standard text exports from over 20 million tokens down to roughly 25,000, PCAP-LM allows an LLM to holistically read an entire capture in a single prompt. - Pragmatic privacy : Operators should not send raw customer traffic to the cloud. PCAP-LM drops all payloads by design and locally swaps IP addresses and hostnames for safe placeholders before the data leaves the building. - Augmenting expertise : Packet-analysis expertise is scarce. An AI can use PCAP-LM for a high-speed ‘first reading’ to flag suspicious flows, relying on a side-index to point back to the original raw frames. This lets human engineers bypass routine packet counting and focus exclusively on hard edge cases, such as retransmissions masked by upstream proxies. - Vendor independence : A plain-text format prevents AI vendor lock-in. The exact same file can be processed by a commercial cloud model, a local open-weight model, or even a human analyst referencing the embedded legend. What’s next PCAP-LM opens a broader question: How should AI read network data? Standard CLI-driven agents excel at targeted queries but suffer from tunnel vision. PCAP-LM addresses this by exposing a conversation’s overall behaviour at a glance, although its lossy design can occasionally miss fine-grained details. We challenge network operators to investigate this boundary in their own networks. Try comparing PCAP-LM’s holistic summaries against your standard agentic workflows to see which approach catches ‘unknown unknowns’ faster. Better yet, help us bridge the gap. Because every PCAP-LM document can include a side-index pointing back to the original frame numbers, you can experiment with hybrid AI assistants. An LLM can read the summary to grasp the big picture, then use the index to trigger local tools for raw packet retrieval when it spots an anomaly. We invite you to look beyond standard HTTP/S and test this hybrid approach on entirely different protocols and share your findings with the community. Xavier Marjou is a telecommunications and deep learning researcher at Orange in France. With 30 years of industry experience since 1996 and a decade attending the IETF, his research focuses on integrating artificial intelligence with network architecture. The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct https://blog.apnic.net/?p=395 applies to this blog.