I'm a Dev Who Barely Knows the Kernel. Here's How I'm Learning How to Track a Packet with pwru

Developer Maneshwar explains how he learned to use Cilium's pwru, an eBPF-powered tool that traces packets through the Linux kernel by instrumenting every function that touches a `sk_buff` struct, discovered automatically via BTF. He highlights pwru's use of familiar pcap-filter expressions and its ability to trace thousands of kernel functions without manual selection.

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback. Hey, so here's a confession. My idea of "networking debugging" used to top out at iptables -L -v -n , squinting at counters, and occasionally yelling "WHY IS IT DROPPING" at a terminal that does not care about my feelings. I know just enough kernel to be dangerous: I know packets go in, sometimes they don't come out, and somewhere in between there's a black box that I was told is "the kernel networking stack." Then I found cilium/pwru https://github.com/cilium/pwru sitting in a repo I was poking around, and it broke my brain a little. So I read the source, dragged my devops brain through it, and now I'm going to explain it to you the way I wish someone had explained it to me. No kernel PhD required. Some puns required. console.log As a web dev, when something's broken, I add a console.log , or I set a breakpoint, or worst case I print my way to enlightenment. The request comes in, flows through my middleware, my handlers, my ORM, and I can trace every step because it's all just... functions calling functions, in my process, in my language. A packet inside the Linux kernel does not work like that. It gets handed from function to function across the network stack ip rcv , tcp v4 rcv , netfilter hooks, qdiscs, the works , it can get cloned, NAT'd, encapsulated in a tunnel, handed to a virtual ethernet pair, sent through an eBPF program of its own, and dropped at literally any of a few thousand points, often silently. There's no stack trace. There's no packet.log . The tool you reach for as a network person is tcpdump , which is great for seeing a packet arrive and a packet leave, but useless for seeing what happened to it in between , inside the kernel, especially when it never arrives at all. That's the gap pwru packet, where are you? fills. It's basically an eBPF-powered console.log for the kernel's packet path. Here's the first thing that made my jaw drop reading this codebase. Instead of the pwru authors hand-picking a list of "interesting" kernel functions to trace which is what I assumed, naively , they let BTF BPF Type Format, basically the kernel's own type metadata do the work for them. In internal/pwru/utils.go https://internal/pwru/utils.go , GetFuncs walks the struct sk buff ?" If yes, that function touches a packet, and it gets added to the hit list. if ptr, ok := p.Type. btf.Pointer ; ok { if strct, ok := ptr.Target. btf.Struct ; ok { if strct.Name == "sk buff" && i <= 5 { funcs name = i continue } } } That's it. That's the whole discovery mechanism. On a typical kernel this turns up a few thousand functions, and pwru then kprobes or kprobe-multi attaches, in batches, concurrently all of them, at once. Every one of those probes runs the same handler, checks your filter, and if it matches, ships an event to userspace. Coming from web dev, this is the equivalent of somebody saying "instead of adding logging to the 12 functions we think matter, let's just instrument literally every function in the entire codebase that touches a Request object" and then actually shipping it in a way that doesn't melt the server. It's completely unhinged in the best way, and it works because eBPF probes are cheap enough and the filtering happens in-kernel , before the event ever reaches userspace that "just trace everything" is a viable strategy. Here's roughly what that discovery-and-attach pipeline looks like: The second thing that made me feel at home: you don't write eBPF filter logic by hand, you write a normal pcap-filter expression. The same syntax from tcpdump . The stuff every devops person already has muscle memory for, like tcp and port 443 or host 10.0.0.5 . pwru --output-tuple 'host 1.1.1.1 and tcp' How does that actually work under the hood? This is the part where I had to reread the code three times. pwru takes your pcap expression, compiles it to classic BPF cBPF, the old-school packet filter bytecode, same lineage as tcpdump -d using cloudflare/cbpfc https://github.com/cloudflare/cbpfc to translate cBPF into modern eBPF instructions, and then for idx, inst := range program.Instructions { if inst.Symbol == "filter pcap ebpf"+suffix { injectIdx = idx break } } In bpf/kprobe pwru.c there's a deliberately empty, noinline placeholder function called filter pcap ebpf l3 / l2 whose only job is to exist as an injection point: static noinline bool filter pcap ebpf l3 void skb, void skb, void skb, void data, void data end { return data = data end && skb == skb && skb == skb; } pwru finds that function in the compiled instruction stream and literally performs bytecode surgery, cutting it out and stitching in your compiled filter instead, before the program is even loaded into the kernel. It's like if your reverse proxy let you write location blocks in nginx syntax, but under the hood it was actually recompiling and hot-patching the C binary per request. Deranged. Also very cool. Okay here's the thing that actually made me appreciate why this tool needed to exist and isn't just "kprobe everything and print." A packet does not keep a fixed identity as it flows through the kernel. host 1.1.1.1 can stop matching halfway through the journey, right after the thing you were trying to debug happens. skb clone or copied skb copy for things like local delivery plus forwarding. sk buff might legitimately get freed and a So pwru maintains a small state machine to keep following a packet even after your original filter would technically stop matching. Once a packet matches your filter once, its pointer gets remembered in a skb addresses BPF hash map bpf/kprobe pwru.c , and every subsequent kprobe hit checks that map first, before even bothering with your filter again: php if cfg- track skb && bpf map lookup elem &skb addresses, &skb addr { tracked by = stackid ? TRACKED BY STACKID : TRACKED BY SKB; goto cont; } When the skb gets cloned, an fexit probe on skb clone / skb copy propagates the "yes, we're tracking this one" flag from the old pointer to the new one. When the kernel finally frees it kfree skbmem , pwru cleans up the tracking entry so the map doesn't grow forever. For the gnarlier case, like bridging, where the pointer identity is genuinely lost, there's a --filter-track-skb-by-stackid mode that fingerprints by call stack instead of pointer, so it can pick the "same" packet back up under a new address. This is basically the eBPF equivalent of trying to correlate a request across microservices without a trace ID: you're stitching identity back together from context clues because nobody handed you a clean handle to hold onto. The best part for someone coming from iptables-land: when a packet dies, pwru doesn't just say "the trace stopped here, good luck." It hooks kfree skb reason and decodes the kernel's skb drop reason enum, so you get an actual human string like SKB DROP REASON NETFILTER DROP instead of silence. That's the difference between "curl hung and I have no idea why" and "curl hung, and here is the literal function and the literal drop reason, with a timestamp." The README's example https://README.md is exactly the workflow I'd have wanted a year ago: run a curl , install an iptables DROP rule, run pwru, and watch it print the exact function where the packet gets killed, instead of you bisecting iptables -L rules by hand at 2am. If you've ever debugged distributed systems, this whole tool is basically: That's a debugging methodology, not just a tool. It's the "attach a debugger to the whole system, but keep it filtered so it doesn't drown you, and correlate identity across boundaries" pattern, and it happens to be implemented with kprobes and BTF instead of trace IDs and Jaeger. Here's the shape of what happens end to end, from the kprobe firing to a line on your screen: I still don't "know the kernel." But I don't need to anymore, at least not for this. pwru turned "the kernel is a black box that eats packets" into "the kernel is a big, well-typed program I can attach cheap breakpoints to, filter with syntax I already know, and follow even when it tries to shapeshift on me." If you're a web/devops person who's only ever fought the network stack from the outside with iptables and tcpdump , go read bpf/kprobe pwru.c https://github.com/cilium/pwru/tree/main/bpf/kprobe pwru.c and internal/pwru/utils.go It reads less like kernel wizardry and more like a really well-designed observability tool that happens to live one layer lower than you're used to. Packet, where are you? Increasingly, I actually know how to ask. AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production. git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free. Any feedback or contributors are welcome It's online, source-available, and ready for anyone to use. ⭐ Star it on GitHub: | 🇩🇰 Dansk https://github.com/HexmosTech/git-lrc/readme/README.da.md | 🇪🇸 Español https://github.com/HexmosTech/git-lrc/readme/README.es.md | 🇮🇷 Farsi https://github.com/HexmosTech/git-lrc/readme/README.fa.md | 🇫🇮 Suomi https://github.com/HexmosTech/git-lrc/readme/README.fi.md | 🇯🇵 日本語 https://github.com/HexmosTech/git-lrc/readme/README.ja.md | 🇳🇴 Norsk https://github.com/HexmosTech/git-lrc/readme/README.nn.md | 🇵🇹 Português https://github.com/HexmosTech/git-lrc/readme/README.pt.md | 🇷🇺 Русский https://github.com/HexmosTech/git-lrc/readme/README.ru.md | 🇦🇱 Shqip https://github.com/HexmosTech/git-lrc/readme/README.sq.md | 🇨🇳 中文 https://github.com/HexmosTech/git-lrc/readme/README.zh.md | 🇮🇳 हिन्दी https://github.com/HexmosTech/git-lrc/readme/README.hi.md | GenAI today is a race car without brakes . It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things : they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production. git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen At a glance: 10 risk categories https://github.com/HexmosTech/git-lrc what-git-lrc-checks-for · 100+ failure patterns tracked https://github.com/HexmosTech/git-lrc what-git-lrc-checks-for · every commit…