Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.
Hey, so here's a confession.
My idea of "networking debugging" used to top out at iptables -L -v -n
, squinting at counters, and occasionally yelling "WHY IS IT DROPPING" at a terminal that does not care about my feelings.
I know just enough kernel to be dangerous: I know packets go in, sometimes they don't come out, and somewhere in between there's a black box that I was told is "the kernel networking stack."
Then I found cilium/pwru sitting in a repo I was poking around, and it broke my brain a little.
So I read the source, dragged my devops brain through it, and now I'm going to explain it to you the way I wish someone had explained it to me.
No kernel PhD required.
Some puns required.
console.log
As a web dev, when something's broken, I add a console.log
, or I set a breakpoint, or worst case I print
my way to enlightenment.
The request comes in, flows through my middleware, my handlers, my ORM, and I can trace every step because it's all just... functions calling functions, in my process, in my language.
A packet inside the Linux kernel does not work like that.
It gets handed from function to function across the network stack (ip_rcv
, tcp_v4_rcv
, netfilter hooks, qdiscs, the works), it can get cloned, NAT'd, encapsulated in a tunnel, handed to a virtual ethernet pair, sent through an eBPF program of its own, and dropped at literally any of a few thousand points, often silently.
There's no stack trace.
There's no packet.log()
.
The tool you reach for as a network person is tcpdump
, which is great for seeing a packet arrive and a packet leave, but useless for seeing what happened to it in between, inside the kernel, especially when it never arrives at all.
That's the gap pwru
(packet, where are you?) fills.
It's basically an eBPF-powered console.log
for the kernel's packet path.
Here's the first thing that made my jaw drop reading this codebase.
Instead of the pwru authors hand-picking a list of "interesting" kernel functions to trace (which is what I assumed, naively), they let BTF (BPF Type Format, basically the kernel's own type metadata) do the work for them.
GetFuncs
walks the struct sk_buff
?" If yes, that function touches a packet, and it gets added to the hit list.
if ptr, ok := p.Type.(*btf.Pointer); ok {
if strct, ok := ptr.Target.(*btf.Struct); ok {
if strct.Name == "sk_buff" && i <= 5 {
funcs[name] = i
continue
}
}
}
That's it. That's the whole discovery mechanism.
On a typical kernel this turns up a few thousand functions, and pwru then kprobes (or kprobe-multi
attaches, in batches, concurrently) all of them, at once.
Every one of those probes runs the same handler, checks your filter, and if it matches, ships an event to userspace.
Coming from web dev, this is the equivalent of somebody saying "instead of adding logging to the 12 functions we think matter, let's just instrument literally every function in the entire codebase that touches a Request
object" and then actually shipping it in a way that doesn't melt the server.
It's completely unhinged in the best way, and it works because eBPF probes are cheap enough (and the filtering happens in-kernel, before the event ever reaches userspace) that "just trace everything" is a viable strategy.
Here's roughly what that discovery-and-attach pipeline looks like:
The second thing that made me feel at home: you don't write eBPF filter logic by hand, you write a normal pcap-filter
expression.
The same syntax from tcpdump
.
The stuff every devops person already has muscle memory for, like tcp and port 443
or host 10.0.0.5
.
pwru --output-tuple 'host 1.1.1.1 and tcp'
How does that actually work under the hood? This is the part where I had to reread the code three times.
pwru takes your pcap expression, compiles it to classic BPF (cBPF, the old-school packet filter bytecode, same lineage as tcpdump -d
) using cloudflare/cbpfc to translate cBPF into modern eBPF instructions, and then
for idx, inst := range program.Instructions {
if inst.Symbol() == "filter_pcap_ebpf"+suffix {
injectIdx = idx
break
}
}
In bpf/kprobe_pwru.c
there's a deliberately empty, __noinline
placeholder function called filter_pcap_ebpf_l3
/ _l2
whose only job is to exist as an injection point:
static __noinline bool
filter_pcap_ebpf_l3(void *_skb, void *__skb, void *___skb, void *data, void* data_end)
{
return data != data_end && _skb == __skb && __skb == ___skb;
}
pwru finds that function in the compiled instruction stream and literally performs bytecode surgery, cutting it out and stitching in your compiled filter instead, before the program is even loaded into the kernel.
It's like if your reverse proxy let you write location
blocks in nginx syntax, but under the hood it was actually recompiling and hot-patching the C binary per request. Deranged. Also very cool.
Okay here's the thing that actually made me appreciate why this tool needed to exist and isn't just "kprobe everything and print."
A packet does not keep a fixed identity as it flows through the kernel.
host 1.1.1.1
) can stop matching halfway through the journey, right after the thing you were trying to debug happens.skb_clone
) or copied (skb_copy
) for things like local delivery plus forwarding.sk_buff
might legitimately get freed and a So pwru maintains a small state machine to keep following a packet even after your original filter would technically stop matching.
Once a packet matches your filter once, its pointer gets remembered in a skb_addresses
BPF hash map (bpf/kprobe_pwru.c
), and every subsequent kprobe hit checks that map first, before even bothering with your filter again:
if (cfg->track_skb && bpf_map_lookup_elem(&skb_addresses, &skb_addr)) {
tracked_by = _stackid ? TRACKED_BY_STACKID : TRACKED_BY_SKB;
goto cont;
}
When the skb gets cloned, an fexit
probe on skb_clone
/skb_copy
propagates the "yes, we're tracking this one" flag from the old pointer to the new one.
When the kernel finally frees it (kfree_skbmem
), pwru cleans up the tracking entry so the map doesn't grow forever.
For the gnarlier case, like bridging, where the pointer identity is genuinely lost, there's a --filter-track-skb-by-stackid
mode that fingerprints by call stack instead of pointer, so it can pick the "same" packet back up under a new address.
This is basically the eBPF equivalent of trying to correlate a request across microservices without a trace ID: you're stitching identity back together from context clues because nobody handed you a clean handle to hold onto.
The best part for someone coming from iptables-land: when a packet dies, pwru doesn't just say "the trace stopped here, good luck."
It hooks kfree_skb_reason
and decodes the kernel's skb_drop_reason
enum, so you get an actual human string like SKB_DROP_REASON_NETFILTER_DROP
instead of silence.
That's the difference between "curl hung and I have no idea why" and "curl hung, and here is the literal function and the literal drop reason, with a timestamp."
The README's example is exactly the workflow I'd have wanted a year ago: run a curl
, install an iptables DROP rule, run pwru, and watch it print the exact function where the packet gets killed, instead of you bisecting iptables -L
rules by hand at 2am.
If you've ever debugged distributed systems, this whole tool is basically:
That's a debugging methodology, not just a tool.
It's the "attach a debugger to the whole system, but keep it filtered so it doesn't drown you, and correlate identity across boundaries" pattern, and it happens to be implemented with kprobes and BTF instead of trace IDs and Jaeger.
Here's the shape of what happens end to end, from the kprobe firing to a line on your screen:
I still don't "know the kernel."
But I don't need to anymore, at least not for this. pwru turned "the kernel is a black box that eats packets" into "the kernel is a big, well-typed program I can attach cheap breakpoints to, filter with syntax I already know, and follow even when it tries to shapeshift on me."
If you're a web/devops person who's only ever fought the network stack from the outside with iptables
and tcpdump
, go read bpf/kprobe_pwru.c and
internal/pwru/utils.go
It reads less like kernel wizardry and more like a really well-designed observability tool that happens to live one layer lower than you're used to.
Packet, where are you? Increasingly, I actually know how to ask.
AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
⭐ Star it on GitHub:
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
** git-lrc is your braking system.** It hooks into
git commit
and runs an AI review on every diff In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…