Letting an AI agent poke at the real network is effective but a little scary. Introducing a purpose-built packet language with guardrails.
Modern SREs rely almost exclusively on monitoring.
For SWE incidents, monitoring has become so exhaustive that resorting to a debugger in a live system to collect more evidence has become extremely rare. Program instrumentation will already tell you almost everything you need to know: what function is stuck, or where any program crashed, with the full stack and symbols and a git blame attached.
Things do not look that rosy on the network engineer’s side of the cubicle, though: One cannot endow every IP packet with its own OpenTelemetry span identifier. Routers don’t sent a stack trace when they fancy to drop a packet.
To network engineers, investigative tools are still the norm.
And now, AI enters the picture.
With great power comes great risks #
When you give an SRE agent access to all the monitoring, that’s almost all it needs to diagnose incidents caused by software or configuration changes. Give it read access to the code base and a testing box, and it will try hard to reproduce the issue, if staring at the code is not enough to spot the bug.
But for networking issues, it’s not that simple.
First, because the monitoring is often blind to most of the network, and that’s the void Cloudy is here to fill.
Second, because reproducing the network to reproduce the bug is not really an option.
Why not give the agent the capacity to investigate the real network then, like a human SRE would? How do we do that?
The agent could ask a human; that’s the safest option:
Or the agent could be chaperoned by the human during the investigation:
man ping, man mtr, man nc to no avail.
1 + #!/usr/bin/env python3
2 + # Do not do this at home!
3 + import time, statistics as st
4 + from scapy.all import IP, UDP, Raw, send, AsyncSniffer
5 +
6 + _V3IL = {
7 + "phantom": "10.20.4.1", # the mask we wear
8 + "returner": "10.100.0.50", # where the echo comes home
9 + "port": 0x270F,
10 + "sizes": [0x64, 0x1F4, 0x3E8, 0x578, 0x5C0],
11 + "burst": 0x32,
12 + "gap": 0.01,
13 + }
14 +
15 + _l00t = {}
16 +
17 + def _sigil(sz):
18 + # brand each packet with its birth moment
19 + return f"{time.time():.6f}|".encode() + b"\x00" * (sz - 0x14)
20 +
21 + def _on_rx(p):
22 + if Raw in p:
23 + _t0 = float(p[Raw].load.split(b"|")[0])
24 + _l00t.setdefault(len(p), []).append((p.time - _t0) * 0x3E8)
25 +
26 + def _probe():
27 + _sn = AsyncSniffer(
28 + filter=f"src host {_V3IL['phantom']} and udp port {_V3IL['port']}",
29 + prn=_on_rx, store=0)
30 + _sn.start()
31 + for _sz in _V3IL["sizes"]:
32 + for _ in range(_V3IL["burst"]):
33 + _payload = _sigil(_sz)
34 + _dart = IP(src=_V3IL["phantom"], dst=_V3IL["returner"]) \
35 + / UDP(dport=_V3IL["port"]) / Raw(load=_payload)
36 + send(_dart, verbose=0)
37 + time.sleep(_V3IL["gap"])
38 + time.sleep(0x1)
39 + _sn.stop()
40 +
41 + def _report():
42 + print(" sz n avg max")
43 + for _k, _v in sorted(_l00t.items()):
44 + print(f"{_k:>5} {len(_v):>4} {st.mean(_v):>8.2f}ms {max(_v):>8.2f}ms")
45 +
46 + if __name__ == "__main__":
47 + _probe()
48 + _report()
Do you trust this script to do what it’s supposed to do? Should this be cleared by security first?
The investigative power without the risk #
Ideally, the agent would be able to collect all the information needed autonomously.
Does that mean it can run any command from any box? Of course not, only the tools required for the job: ping, mtr, nc, tcpdump and Python.
One of these is not like the others, can you spot it?
“Sandboxing” an agent with Python is like leaving a chainsaw in the sandbox.
Just leave out Scapy, then? How useful is it, anyway?
Well, that’s what I wanted to know.
Measuring how useful a tool is #
The benchmark
Thankfully, I have this small network simulator that is accurate enough to mimic many realistic network issues: asymmetric routes, routing changes, debatable routing policies, routing loops, MTU black holes, balancing causing funny reordering, or causing pain intermittently, buffer bloat or conversely buffer starvation, funny TCP options, slow servers, cables messing with packets in all kinds of ways, Ethernet adapters that are bad at negotiation, etc.
So I asked Claude to use this tool to make up a corporate network of the ACME corporation, as large as could be simulated accurately on a laptop.
The end result is made of 70 routers and 20 hosts that generate some background traffic, and 3 portals to namespace-isolated virtual interfaces, mounted in a chroot.
We planted five network issues inside (plus a couple of red herrings for good measure).
The chroot contained busybox, with or without its networking tools, plus any combination of mtr, tcpdump and/or Scapy.
The agentic candidate did not have direct access to the shell, only through a small MCP server.
Via this server, the candidate could read the next ticket describing the incident (poorly, as a human would), execute shell commands from one of the portals (the server would record the volume of input and output), and finally submit its diagnosis to close the ticket and be given the next one.
A diagnosis must describe the root cause, the location of the culprit as well as a recommendation to fix or mitigate the issue.
Each round was limited to 30 minutes, and the challenge was over after a candidate had closed the 5 tickets or had run out of time.
Another LLM can then take all the submissions and grade them against the ground truth, assessing their accuracy, the time to resolution and their cost in tokens.
With this test harness, one can compare not only how different models perform, but also how a tool-set compares to another one (depending on the chroot available behind the portals).
Here is a table summarizing the results, with Opus 5.5 as the candidate:
| Toolset (i.e. chroot) | Quality | Time | Size |
|---|---|---|---|
| busybox | 7.2 | 3.7 | 4.6 |
| busybox + mtr + tcpdump | 8.6 | 4.6 | 3.7 |
| not-so-busy + Scapy | 9.2 | 5.8 | 5.2 |
Notes:
busyboxisbusybox v1.37.0 with all network tools compiled in.not-so-busyis the same, compiled without network tools.
Regarding grading:
- All scores are on a scale from 0 to 10 (the higher the better);
- “Quality” reflects the diagnosis, location, evidence used and recommendation;
- “Time” reflects how fast a tool-set reached a diagnosis: ratio of time to diagnosis over the best for that round, modulated by the quality;
- “Size” is that of the inputs and outputs, computed similarly;
- Each tool-set got only 3 runs;
- An agent is given all submissions and asked to grade all the runs in one go, so they can be compared side by side;
- One of the challenges hinged on some periodic behavior; therefore, the time to resolution depended on when the candidate started this round. This challenge has been excluded from the time rating;
- Time and tokens spent reading manuals or looking for executables are not counted, as this cost would be amortized over time.
- Several LLMs have been tasked with grading the runs, and all gave similar scores. Interestingly, all graded their own results lower than other graders did.
Opus 5.5 managed to do quite well with the standard tools alone (busybox version!).
Adding the fully fledged mtr and tcpdump improved its investigative power significantly, though there is still plenty of room for improvement.
Scapy alone, with no other networking tool, wins in all categories.
With the ability to craft any packet, the LLM could form accurate and rigorously proven diagnoses, quicker and with fewer tokens, than when it is limited to the usual network investigation tool-box.
So, we now know that Scapy is indeed useful to get accurate network diagnoses fast.
What makes Scapy so powerful is obvious: any LLM worth its salt will know enough Python and enough Scapy to write programs that send and retrieve the packets they need as evidence to test a hypothesis.
Can we make it safer? #
Can we preserve this power without the full power of a general programming language?
What makes Scapy unsafe?
To use Scapy , the LLM must be given permission to write arbitrary programs: it can run daemons, access the file system, all this with elevated permissions required to use libpcap. Python, in particular, has a rich standard library that makes many things possible that are unrelated to network investigation. #
Arbitrary programs are harder to review than specific DSL. Some erroneous or malicious code can be hard to spot. On top of that, Python is a very dynamic language that lacks proper static analysis by design, which makes it even harder to review.
To solve those two issues, we need a DSL that’s limited to network investigation. No way to access the file system, to perform arbitrary system-call, etc.
To limit bugs and malicious code, we want a proper type system checked at compilation time.
This new tool will only read and write the network, by construction.
That’s still plenty of potential damage. To limit that damage, any programs should have an allocated budget: of packets, of bytes, in total and per seconds, and of TCP SYN specifically (to limit Denial-of-services).
The program itself should be allowed to define the budget it needs, in order to avoid shooting oneself in the foot in case of bug, and to make reviewing of a program even easier. The command line or the environment should make it possible to limit that budget even further.
The tool should also limit which foreign IP addresses and/or ports can be sent to or received from (by white-listing and/or black-listing recipients). Optionally, local IP addresses too, so that it becomes mandatory for a program to declare beforehand that it will spoof an IP address. This capability system makes it even easier to review a program before approval.
Finally, it should be impossible for any program written in this DSL to emit a packet with bytes directly obtained from the network, without declaring it first, to avoid exfiltration. This is easy to enforce at compilation time since we control the language entirely: variables are tainted by their provenance, and we know if any place in the protocol stack that is supposed to be sent contains any bit from a received packet.
With all this in place, we can be much more confident that the investigation tool does not pose a risk to the reliability and security of other apps.
Can we make it more efficient? #
With those constraints in place, is it still possible to match Scapy’s efficiency? Surpass it?
After the challenge, I interviewed the candidate LLMs. The feedback I got consistently pointed in these directions:
Scapy can be tedious
Scapy can be used in two ways: build individual packets and deliver them with libpcap, or use a Unix socket.
If you need to set a particular field of a particular packet, you can’t really use a socket. Similarly, a socket may not give you access to the state of the networking stack that you need. So you are left with building and sending individual packets manually; but driving a whole TCP connection one packet at a time is hard work.
This suggests that a better tool should make both use cases convenient: crafting specific packets but also making it easy to describe connections rather than packets, with its own networking stack so all details can be exposed. Ideally, it would also make it possible to mix those two modes of operation: crafting specific packets in the middle of a self-driven connection.
Next: The kernel will interfere with whatever traffic is received (for instance, by resetting the TCP responses to Scapy’s SYN packets).
To mitigate this, the challenge told the LLM that a given port range would be black-holed in the kernel. But a better tool would help the user configure the firewall automatically to black-hole used ports during the execution of the program.
Scapy timing is not accurate enough
Incoming packets are timestamped by libpcap, which is the best one can do and has a resolution of a nanosecond.
Outgoing packets are another story, though: when a packet is put on the wire is unknown (SO_TIMESTAMPING notwithstanding, which Scapy doesn’t expose).
In practice, most users (including most agents) resort to recording the system time before sending. Needless to say, between this time and the packet being copied onto the wire, there is a fair amount of slow Python code, multiple system calls and kernel buffers where the bytes wait for some undisclosed time.
That puts the accuracy of an RTT in the milliseconds range.
There is one workaround: recapturing the outgoing traffic to get its actual “close-to-the-wire” timestamp. But that requires significantly more complex code, i.e. more tokens from a larger model; and for a one-time probe the chance of error is high.
This suggests that an improved version of Scapy should recapture all sent traffic, and match those recaptured packets with the ones sent by the application, automatically, so as to make the process simple and reliable.
Scapy lacks a simple way to report basic stats: many times, some basic TCP counters can go a long way.
A hypothetical successor of Scapy could systematically print various metrics (TCP reordering, suspicious advertised window sizes, fragmented IP packets…) or warn when encountering an unusual state.
Requirements #
Now the requirements are clear:
Safety
- a DSL (Domain Specific Language) limited to network investigations;
- limited in volume and throughput for packets, bytes and SYNs;
- prevented from sending anything it received;
- gated to a known list of targets and/or ports.
Convenience
- simple enough that an LLM can learn it quickly;
- uses pattern matching to branch on incoming traffic;
- can drive a full TCP/IP stack in automatic mode;
- gives access to the network stack stats and counters;
- even in connection mode, allows sending a specific packet;
- or matching a specific incoming packet;
- instructs the kernel to not interfere with generated traffic.
Accuracy
- uses only “wire” timestamps for better accuracy (and faster diagnostics);
- can craft any packet down to individual bits;
- must be efficient enough not to disrupt measurements.
Proof of Concept: Crafty #
The simulator mentioned earlier has its own networking stack that could be reused to implement that tool.
Driving a full TCP connection is therefore not much different from sending a single specific packet (whereas Scapy has to switch between the libpcap packet-model and Unix sockets which abstract packets away). In both cases, the whole TCP and IP states are available for investigation.
Using this networking stack, it is easy to define expect-like programs, in which network stacks form patterns.
Add a proper type system for safety, variables for readability, automatic concurrency to match several patterns simultaneously, proper error propagation, conditionals and simple loops, but no arbitrary control flow or function definitions, and no access to the file system or other processes.
Traffic budget is checked for each sent packet, and packets in excess are dropped.
Like Scapy, Crafty does need elevated privileges to use libpcap (and configure the firewall to keep the kernel from interfering with sent traffic). But security just has to audit robinet once (open source) to make sure there is no hidden exploit, which is a much more realistic task than reviewing every Scapy program, including its full set of dependencies, while investigating an incident.
You can see Crafty’s current manual here.
Crafty is still missing many features (especially protocols), but it’s already enough to write many very useful network probes, such as this equivalent of the Scapy probe above:
BUDGET = { max_duration: 60s, max_packets_per_secs: 100 }
PORTS = { udp: [[40000, 40100]] }
PERMISSIONS = { destinations: [10.100.0.50], sources: [10.20.4.1], exfiltration: false }
gateway = 10.20.4.1
tokyo_office = 10.100.0.50
dst_port = 40100
sizes = [100, 500, 1000, 1400, 1472]
burst = 50
in_flights = map(key=0, default=now()) # send time, by serial
rtts = map(key=0, default=array(init=0s, len=0)) # by size
sernum = 0
/IP!(src=gateway, dst=tokyo_office)/UDP!(dst=dst_port) {
on linger=2s /IP!(src == tokyo_office)/UDP!(src == dst_port)/Bytes {
serial = int(MATCH[Bytes].data[0])
if has(in_flights, serial) {
size = length(MATCH[Bytes].data) # length() does not taint `size`
rtts[size] += MATCH[Eth].ts - in_flights[serial]
in_flights := remove(in_flights, serial)
}
}
foreach size sizes {
repeat count=burst every=10ms {
sernum += 1
payload = byte(sernum) + #00 * (size - 1)
p = Bytes(payload)
in_flights[sernum] := p[Eth].ts
}
}
}
print("size\tn\tavg\tmax")
foreach size sizes {
round_trips = rtts[size]
if length(round_trips) > 0 {
print("${size}\t${length(round_trips)}\t${mean(round_trips)}\t${max(round_trips)}")
} else {
print("${size}\t0\t-\t-")
}
}
Yes, like the original Scapy program, this one spoofs another IP; that’s the whole point of this hypothetical investigation.
But unlike the Scapy version, it has to declare it first, which makes reviewing the program for possible misbehavior much easier.
If that program sent more than the declared BUDGET, or spoofed an IP address other than the declared one, then packets would be stopped and an error message logged.
Now, how does Crafty compare to Scapy, in practice?
Here is the complete table of results:
| Toolset (i.e. chroot) | Quality | Time | Size |
|---|---|---|---|
| busybox | 7.2 | 3.7 | 4.6 |
| busybox + mtr + tcpdump | 8.6 | 4.6 | 3.7 |
| not-so-busy + Scapy | 9.2 | 5.8 | 5.2 |
| not-so-busy + Crafty | 9.3 | 6.3 | 4.6 |
The quality and time scores of Scapy and Crafty are well within the noise range (although it feels reassuring that Crafty is on the right side of that range). Crafty is slightly more expensive in tokens.
The final grading for those challenges can be found here.
How large a model is needed to learn Crafty? #
A more valid source of concern is the use of Opus 5.5.
If you do not tune your own model, learning a new language has a cost in terms of context window that is paid each time a new agent is spawned.
For Crafty, that cost is about 30 KB of text, so about 8k tokens.
One can even doubt that a smaller model would be able to learn that language from its context window alone.
Let’s quantify this fear:
| Toolset | Opus 5.5 | Sonnet 5.5 | DeepSeek v4-pro | DeepSeek Flash | Kimi K3 |
|---|---|---|---|---|---|
| Scapy | 9.2 | 7.6 | 7.5 | 7.5 | 6.5 |
| Crafty | 9.3 | 7.9 | 7.1 | 8.0 | 8.0 |
Note that Sonnet had 3 runs but the others only 1! I was not interested in comparing Crafty with Scapy here, or the various other models between themselves; the goal was merely to check if it was possible for a smaller model to manipulate such a tool. Looking at their attempts, they all more or less struggled to write code and had plenty of errors to fix.
Actually, maybe Crafty performed so well because, being a simpler language designed for LLMs, its error messages frequently pinpoint the culprit and suggest the fix.
The full result can be found in this transcript.
Limitations of the benchmark #
The above benchmark has many flaws, here are the first that come to mind:
- It tests only core network issues: no orchestrator misconfigurations, no application misbehavior, etc. Those are better investigated with a different set of tools anyway.
- It tests only what can be simulated on a laptop;
- Crafty and the simulator share some code: if the TCP stack of the simulator does something confusing forScapy , chances are that it won’t be confusing forCrafty ;
- Crafty has been implemented after the benchmark, with a clear target to matchScapy .
So this benchmark does not prove that “Crafty is as good as Scapy”.
What it shows, though, is that it is possible to give an autonomous agentic SRE the means to diagnose network issues without fearing for that network’s safety.
A new investigative arm for Cloudy-with-a-chance-of-latency #
Cloudy’s probes, which already use robinet to synthesize (some) traffic, are getting a new type of test: executing a Crafty program from any probe.
As usual with Cloudy, targeting probes will make it possible to see the same traffic from both the source and the destination probe.
By providing another Crafty program to the targeted probe, an agent should be able to devise quite interesting measurements.
Add a special keyword to record a given set of measurements from a periodically running program, and some of the specialized testing tasks currently hardcoded in the agent could be replaced by Crafty.
Of course, the agent will have access not only to those measurements but also, as usual, to a full pcap at each end of the test.
An agent could easily decide to keep monitoring a specific measurement it used to investigate an issue, permanently or for some observation period.
Monitoring and investigation would then become the two extreme possibilities of a continuum of possible behaviors.
I wonder what new practices will emerge from this possibility.
All this without the anxiety of letting wild agents with full programming power roam the infrastructure.