{"slug": "i-m-a-dev-who-barely-knows-the-kernel-here-s-how-i-m-learning-how-to-track-a", "title": "I'm a Dev Who Barely Knows the Kernel. Here's How I'm Learning How to Track a Packet with pwru", "summary": "Developer Maneshwar explains how he learned to use Cilium's pwru, an eBPF-powered tool that traces packets through the Linux kernel by instrumenting every function that touches a `sk_buff` struct, discovered automatically via BTF. He highlights pwru's use of familiar pcap-filter expressions and its ability to trace thousands of kernel functions without manual selection.", "body_md": "*Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.*\n\nHey, so here's a confession.\n\nMy idea of \"networking debugging\" used to top out at `iptables -L -v -n`\n\n, squinting at counters, and occasionally yelling \"WHY IS IT DROPPING\" at a terminal that does not care about my feelings.\n\nI know just enough kernel to be dangerous: I know packets go in, sometimes they don't come out, and somewhere in between there's a black box that I was told is \"the kernel networking stack.\"\n\nThen I found [ cilium/pwru](https://github.com/cilium/pwru) sitting in a repo I was poking around, and it broke my brain a little.\n\nSo I read the source, dragged my devops brain through it, and now I'm going to explain it to you the way I wish someone had explained it to me.\n\nNo kernel PhD required.\n\nSome puns required.\n\n`console.log`\n\nAs a web dev, when something's broken, I add a `console.log`\n\n, or I set a breakpoint, or worst case I `print`\n\nmy way to enlightenment.\n\nThe request comes in, flows through my middleware, my handlers, my ORM, and I can trace every step because it's all just... functions calling functions, in my process, in my language.\n\nA packet inside the Linux kernel does not work like that.\n\nIt gets handed from function to function across the network stack (`ip_rcv`\n\n, `tcp_v4_rcv`\n\n, netfilter hooks, qdiscs, the works), it can get cloned, NAT'd, encapsulated in a tunnel, handed to a virtual ethernet pair, sent through an eBPF program of its own, and dropped at literally any of a few thousand points, often silently.\n\nThere's no stack trace.\n\nThere's no `packet.log()`\n\n.\n\nThe tool you reach for as a network person is `tcpdump`\n\n, which is great for seeing a packet arrive and a packet leave, but useless for seeing what happened to it *in between*, inside the kernel, especially when it never arrives at all.\n\nThat's the gap `pwru`\n\n(packet, where are you?) fills.\n\nIt's basically an eBPF-powered `console.log`\n\nfor the kernel's packet path.\n\nHere's the first thing that made my jaw drop reading this codebase.\n\nInstead of the pwru authors hand-picking a list of \"interesting\" kernel functions to trace (which is what I assumed, naively), they let BTF (BPF Type Format, basically the kernel's own type metadata) do the work for them.\n\nIn [ internal/pwru/utils.go](https://internal/pwru/utils.go),\n\n`GetFuncs`\n\nwalks the `struct sk_buff`\n\n?\" If yes, that function touches a packet, and it gets added to the hit list.\n\n```\nif ptr, ok := p.Type.(*btf.Pointer); ok {\n    if strct, ok := ptr.Target.(*btf.Struct); ok {\n        if strct.Name == \"sk_buff\" && i <= 5 {\n            funcs[name] = i\n            continue\n        }\n    }\n}\n```\n\nThat's it. That's the whole discovery mechanism.\n\nOn a typical kernel this turns up a few thousand functions, and pwru then kprobes (or `kprobe-multi`\n\nattaches, in batches, concurrently) all of them, at once.\n\nEvery one of those probes runs the same handler, checks your filter, and if it matches, ships an event to userspace.\n\nComing from web dev, this is the equivalent of somebody saying \"instead of adding logging to the 12 functions we think matter, let's just instrument literally every function in the entire codebase that touches a `Request`\n\nobject\" and then actually shipping it in a way that doesn't melt the server.\n\nIt's completely unhinged in the best way, and it works because eBPF probes are cheap enough (and the filtering happens *in-kernel*, before the event ever reaches userspace) that \"just trace everything\" is a viable strategy.\n\nHere's roughly what that discovery-and-attach pipeline looks like:\n\nThe second thing that made me feel at home: you don't write eBPF filter logic by hand, you write a normal `pcap-filter`\n\nexpression.\n\nThe same syntax from `tcpdump`\n\n.\n\nThe stuff every devops person already has muscle memory for, like `tcp and port 443`\n\nor `host 10.0.0.5`\n\n.\n\n```\npwru --output-tuple 'host 1.1.1.1 and tcp'\n```\n\nHow does that actually work under the hood? This is the part where I had to reread the code three times.\n\npwru takes your pcap expression, compiles it to *classic* BPF (cBPF, the old-school packet filter bytecode, same lineage as `tcpdump -d`\n\n) using [ cloudflare/cbpfc](https://github.com/cloudflare/cbpfc) to translate cBPF into modern eBPF instructions, and then\n\n```\nfor idx, inst := range program.Instructions {\n    if inst.Symbol() == \"filter_pcap_ebpf\"+suffix {\n        injectIdx = idx\n        break\n    }\n}\n```\n\nIn `bpf/kprobe_pwru.c`\n\nthere's a deliberately empty, `__noinline`\n\nplaceholder function called `filter_pcap_ebpf_l3`\n\n/ `_l2`\n\nwhose only job is to exist as an injection point:\n\n```\nstatic __noinline bool\nfilter_pcap_ebpf_l3(void *_skb, void *__skb, void *___skb, void *data, void* data_end)\n{\n    return data != data_end && _skb == __skb && __skb == ___skb;\n}\n```\n\npwru finds that function in the compiled instruction stream and literally performs bytecode surgery, cutting it out and stitching in your compiled filter instead, before the program is even loaded into the kernel.\n\nIt's like if your reverse proxy let you write `location`\n\nblocks in nginx syntax, but under the hood it was actually recompiling and hot-patching the C binary per request. Deranged. Also very cool.\n\nOkay here's the thing that actually made me appreciate why this tool needed to exist and isn't just \"kprobe everything and print.\"\n\nA packet does not keep a fixed identity as it flows through the kernel.\n\n`host 1.1.1.1`\n\n) can stop matching halfway through the journey, right after the thing you were trying to debug happens.`skb_clone`\n\n) or copied (`skb_copy`\n\n) for things like local delivery plus forwarding.`sk_buff`\n\nmight legitimately get freed and a So pwru maintains a small state machine to keep following a packet even after your original filter would technically stop matching.\n\nOnce a packet matches your filter once, its pointer gets remembered in a `skb_addresses`\n\nBPF hash map (`bpf/kprobe_pwru.c`\n\n), and every subsequent kprobe hit checks that map first, before even bothering with your filter again:\n\n``` php\nif (cfg->track_skb && bpf_map_lookup_elem(&skb_addresses, &skb_addr)) {\n    tracked_by = _stackid ? TRACKED_BY_STACKID : TRACKED_BY_SKB;\n    goto cont;\n}\n```\n\nWhen the skb gets cloned, an `fexit`\n\nprobe on `skb_clone`\n\n/`skb_copy`\n\npropagates the \"yes, we're tracking this one\" flag from the old pointer to the new one.\n\nWhen the kernel finally frees it (`kfree_skbmem`\n\n), pwru cleans up the tracking entry so the map doesn't grow forever.\n\nFor the gnarlier case, like bridging, where the pointer identity is genuinely lost, there's a `--filter-track-skb-by-stackid`\n\nmode that fingerprints by call stack instead of pointer, so it can pick the \"same\" packet back up under a new address.\n\nThis is basically the eBPF equivalent of trying to correlate a request across microservices without a trace ID: you're stitching identity back together from context clues because nobody handed you a clean handle to hold onto.\n\nThe best part for someone coming from iptables-land: when a packet dies, pwru doesn't just say \"the trace stopped here, good luck.\"\n\nIt hooks `kfree_skb_reason`\n\nand decodes the kernel's `skb_drop_reason`\n\nenum, so you get an actual human string like `SKB_DROP_REASON_NETFILTER_DROP`\n\ninstead of silence.\n\nThat's the difference between \"curl hung and I have no idea why\" and \"curl hung, and here is the literal function and the literal drop reason, with a timestamp.\"\n\nThe [README's example](https://README.md) is exactly the workflow I'd have wanted a year ago: run a `curl`\n\n, install an iptables DROP rule, run pwru, and watch it print the exact function where the packet gets killed, instead of you bisecting `iptables -L`\n\nrules by hand at 2am.\n\nIf you've ever debugged distributed systems, this whole tool is basically:\n\nThat's a debugging methodology, not just a tool.\n\nIt's the \"attach a debugger to the whole system, but keep it filtered so it doesn't drown you, and correlate identity across boundaries\" pattern, and it happens to be implemented with kprobes and BTF instead of trace IDs and Jaeger.\n\nHere's the shape of what happens end to end, from the kprobe firing to a line on your screen:\n\nI still don't \"know the kernel.\"\n\nBut I don't need to anymore, at least not for this. pwru turned \"the kernel is a black box that eats packets\" into \"the kernel is a big, well-typed program I can attach cheap breakpoints to, filter with syntax I already know, and follow even when it tries to shapeshift on me.\"\n\nIf you're a web/devops person who's only ever fought the network stack from the outside with `iptables`\n\nand `tcpdump`\n\n, go read [ bpf/kprobe_pwru.c](https://github.com/cilium/pwru/tree/main/bpf/kprobe_pwru.c) and\n\n`internal/pwru/utils.go`\n\nIt reads less like kernel wizardry and more like a really well-designed observability tool that happens to live one layer lower than you're used to.\n\nPacket, where are you? Increasingly, I actually know how to ask.\n\nAI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.\n\ngit-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.\n\nAny feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.\n\n⭐ Star it on GitHub:\n\n| [🇩🇰 Dansk](https://github.com/HexmosTech/git-lrc/readme/README.da.md) | [🇪🇸 Español](https://github.com/HexmosTech/git-lrc/readme/README.es.md) | [🇮🇷 Farsi](https://github.com/HexmosTech/git-lrc/readme/README.fa.md) | [🇫🇮 Suomi](https://github.com/HexmosTech/git-lrc/readme/README.fi.md) | [🇯🇵 日本語](https://github.com/HexmosTech/git-lrc/readme/README.ja.md) | [🇳🇴 Norsk](https://github.com/HexmosTech/git-lrc/readme/README.nn.md) | [🇵🇹 Português](https://github.com/HexmosTech/git-lrc/readme/README.pt.md) | [🇷🇺 Русский](https://github.com/HexmosTech/git-lrc/readme/README.ru.md) | [🇦🇱 Shqip](https://github.com/HexmosTech/git-lrc/readme/README.sq.md) | [🇨🇳 中文](https://github.com/HexmosTech/git-lrc/readme/README.zh.md) | [🇮🇳 हिन्दी](https://github.com/HexmosTech/git-lrc/readme/README.hi.md) |\n\nGenAI today is a **race car without brakes**. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents *silently break things*: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.\n\n** git-lrc is your braking system.** It hooks into\n\n`git commit`\n\nand runs an AI review on every diff In short, git-lrc helps **Prevent Outages, Breaches, and Technical Debt Before They Happen**\n\n**At a glance:** [10 risk categories](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · [100+ failure patterns tracked](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · every commit…", "url": "https://wpnews.pro/news/i-m-a-dev-who-barely-knows-the-kernel-here-s-how-i-m-learning-how-to-track-a", "canonical_source": "https://dev.to/lovestaco/im-a-dev-who-barely-knows-the-kernel-heres-how-im-learning-how-to-track-a-packet-with-pwru-5937", "published_at": "2026-07-25 13:01:01+00:00", "updated_at": "2026-07-25 13:33:12.331592+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["Maneshwar", "Cilium", "pwru", "git-lrc", "Linux"], "alternates": {"html": "https://wpnews.pro/news/i-m-a-dev-who-barely-knows-the-kernel-here-s-how-i-m-learning-how-to-track-a", "markdown": "https://wpnews.pro/news/i-m-a-dev-who-barely-knows-the-kernel-here-s-how-i-m-learning-how-to-track-a.md", "text": "https://wpnews.pro/news/i-m-a-dev-who-barely-knows-the-kernel-here-s-how-i-m-learning-how-to-track-a.txt", "jsonld": "https://wpnews.pro/news/i-m-a-dev-who-barely-knows-the-kernel-here-s-how-i-m-learning-how-to-track-a.jsonld"}}