{"slug": "the-agent-said-it-worked-i-asked-the-kernel", "title": "The Agent Said It Worked. I Asked the Kernel.", "summary": "A developer built a small native backup client with eight deliberately seeded behaviors to test whether independent evidence—execution, network activity, and resulting file state—can verify an AI agent's claim that its code works. The experiment, inspired by Hemapriya Kanagala's question \"92% according to what?\", found that some variants report success without backing anything up, while others produce correct files through questionable means. The developer argues that independently collected evidence, such as packet captures and file comparisons, offers a way to challenge plausible but unverified implementations.", "body_md": "*Before evaluating an agent’s code, I built a backup client with eight known behaviors to check the instrument itself.*\n\nMy response to “it works” is becoming: “Let me see the packet capture.”\n\nThis may become a personality problem. For now, it is an experiment.\n\nThis experiment was inspired by [Hemapriya Kanagala (@hemapriya_kanagala)](https://dev.to/hemapriya_kanagala) and her article, [What Happens When AI Outgrows the Tests We Use to Measure It?](https://dev.to/hemapriya_kanagala/what-happens-when-ai-outgrows-the-tests-we-use-to-measure-it-30al). Her question—“92% according to what?”—stayed with me. She examines how evaluation can become less informative as benchmarks saturate, reference answers become complicated, and test conditions change.\n\nI wanted to take one piece of that discussion to the workbench: **what independent evidence supports a program’s claim that it succeeded?**\n\nSo, with AI assistance, I built a small native backup client, gave it eight deliberately chosen behaviors, and observed it from outside its own logging. Some variants produce the right file while doing questionable things along the way. One cheerfully reports success without backing anything up.\n\nThe kernel has very little appreciation for cheerful reporting.\n\nMy first instinct was to declare that, until someone redefines how computers work, we have two sources of truth: the CPU and the network.\n\nIt sounded excellent in my head. Then the engineering questions arrived.\n\nA CPU can faithfully execute the wrong algorithm. A packet capture can faithfully record the wrong bytes reaching the wrong destination. The saved file matters too. And none of those observations knows what the user actually requested.\n\nThe more defensible position is that **execution, network activity, and resulting state provide evidence we can judge against a requirement**. They have different coverage and different blind spots.\n\nFor this experiment, the requirement begins with something wonderfully unromantic: the saved backup must match the source file.\n\nNow we have something to measure.\n\nFrameworks and libraries have helped us build software without personally supervising every system call. That remains a useful division of labor.\n\nBut familiar frameworks do not automatically validate unfamiliar code assembled on top of them.\n\nAn agent can help produce an implementation, tests, and an explanation of the passing tests. If the implementation and its tests share a misunderstanding, agreement between them can be misleading.\n\nHumans can do this too. We have been writing tests that flatter our implementations since before the current generation of autocomplete had electricity.\n\nMy concern is the growing volume of plausible implementations we can produce and the time available to examine them. Independently collected evidence gives us another way to challenge the result.\n\nSometimes that evidence is simply a file comparison. Sometimes the output is correct and we need to investigate how the program got there.\n\nThe fixture is intentionally modest:\n\nEach scenario runs the client twice against the same receiver state. The baseline uploads on the first invocation and recognizes unchanged content on the second.\n\nThe contract we want the baseline to satisfy is explicit: save matching content, send no upload payload on an unchanged second invocation, contact only the configured receiver, and recover from the injected first-transfer interruption within three upload attempts. An upload whose payload does not match its advertised digest must be rejected before it replaces the saved backup.\n\nRepeated hashing and spinning are diagnostic cases for unnecessary work. We have not set a performance threshold or measured a speedup here.\n\nA second local endpoint lets us demonstrate an unexpected connection without contacting an outside service.\n\n**The eight behaviors are deliberately seeded demonstrations.** They check whether our observers detect known behaviors. They do not measure how often an AI agent introduces these defects, and they are not evidence about a particular model’s coding ability.\n\nThe utility collects several kinds of evidence. Their distinctions matter more than the sophistication of their names.\n\n`strace` records system calls: operations through which a process requests services such as reading files or opening connections. In this lab, it records timestamps, call durations, decoded file descriptors, and return values.\n\nThat lets us count bytes returned by reads of the source file. These are **logical reads**, not physical disk traffic; the operating system may satisfy them from memory.\n\neBPF supports observation at hooks such as system calls, kernel tracepoints, and function entry or exit. The lab uses `bpftrace` to count selected events for the native client. [eBPF’s introduction](https://ebpf.io/what-is-ebpf/) explains the underlying mechanism.\n\nA userspace probe, or *uprobe*, observes a point in an executable. Here is the probe that counts entries into our hashing function:\n\n```\nuprobe:__BINARY__:hash_file /pid == cpid/ {\n    printf(\"%llu pid=%d hash_file\\n\", nsecs, pid);\n    @hash_calls = count();\n}\n```\n\nThe runner replaces `__BINARY__` with the executable’s path. `cpid` identifies the child launched through `bpftrace -c`, so the filter restricts these observations to that client. [bpftrace documents this built-in here](https://bpftrace.org/docs/release_025/stdlib).\n\nOther probes observe connection calls, successful read/send byte counts, scheduling events, and entries into the payload-send and busy-wait functions.\n\nAn entry counter tells us that control reached a function. It does not establish that the function returned the correct answer. That is why the outcome check remains separate.\n\nThe host used for this demonstration has an AMD Threadripper PRO 5975WX. Linux exposed AMD Instruction-Based Sampling through `ibs_op`, and the utility successfully collected it through `perf`.\n\nThe saved artifacts include CPU call-stack profiles and assembly annotated with sample weights. These let us inspect execution inside the client, its libraries, and sampled kernel paths.\n\nThis is sampling. It is not a recording of every instruction, every register, or every intermediate value.\n\n`tcpdump` captures loopback traffic filtered to the two fixture ports. We preserve the packet file, a readable packet summary, and the capture tool’s drop counters.\n\nReceiver payload counts are a different measurement: bytes the application consumed. A successful send can hand data to the local kernel before the receiver consumes it. Headers, retries, and buffering also affect what each observer sees.\n\nKeep those quantities separate. Otherwise, the instrument starts manufacturing the confusion it was built to investigate.\n\nThe `false-success` variant prints a completion event and exits with status zero. It does not upload the file.\n\nThe independent verifier finds no saved backup. In the eBPF run, there are no observed client connection calls, and the filtered packet capture contains zero packets.\n\nThis is the bluntest example in the collection. A test that checks only the exit status would accept it. A test that verifies the resulting file would reject it immediately.\n\nWe did not need a CPU probe to discover that the file was missing. The low-level observations help establish what accompanied that failure. **Use the simplest independent check that answers the question.**\n\nThe `hash-storm` variant saves a backup whose digest matches the source. It also hashes the entire source 256 times per invocation.\n\nOn the second invocation, the system-call trace records **134,217,728 logical source bytes read** for a **524,288-byte file**:\n\n```\n134,217,728 / 524,288 = 256\n```\n\nThe receiver consumes no upload payload on that invocation. The content is unchanged; the unnecessary work happens before that decision.\n\nA separate eBPF run observes 256 entries into `hash_file` on each invocation. Here are the final counters, copied verbatim from its second invocation’s [raw probe output](//evidence/historical/ebpf/hash-storm/pass-2/kernel-events.txt):\n\n```\n@connects: 1\n@hash_calls: 256\n@read_bytes: 134233569\n@sent_bytes: 75\n@switches_out: 4\n```\n\nThe read counter is slightly larger than the source-file total above: this probe counts successful `read` bytes across the client’s descriptors, while the system-call summary selects reads of the source file. The send counter includes the digest-check request; zero upload payload does not mean zero network activity.\n\nA separate AMD IBS run collects 233 CPU samples on its first invocation, reports zero lost samples, and places most inclusive sample weight beneath `hash_file` in the [call-stack profile](//evidence/historical/ibs/hash-storm/pass-1/cpu-profile.txt). “Inclusive” includes work in functions called by `hash_file`, such as the hashing library and file-read paths.\n\nThe samples help locate the work. The syscall and function-entry counts establish the repetition. This run does not quantify how much faster a corrected implementation would be.\n\nThese are observations from separate runs of the same seeded behavior. They corroborate the explanation; they are not one combined trace.\n\nThe output is correct. The computer has simply been asked to check the same pocket for its keys 256 times.\n\nThe `unexpected-egress` variant makes a harmless connection to our second local endpoint before performing the backup.\n\nThe file still verifies correctly.\n\nIn the eBPF showcase, baseline connection counts are two on the first invocation and one on the second: a digest check plus an upload, followed by a digest check alone. The extra-connection variant records three and two.\n\nCounts tell us there is more connection activity. The socket trace, packet addresses, and second endpoint’s records establish where it goes.\n\nThat distinction matters. Three connections do not inherently mean something is wrong. A destination requirement gives the observation its meaning.\n\nHere is the compact view of the captured demonstrations:\n\n| Behavior | Saved content matches? | Additional observation | \n|---|---|---|\n| Baseline | Yes | Second invocation sends no upload payload | \n| Redundant upload | Yes | Unchanged file is uploaded again | \n| Hash storm | Yes | 256 hashing-function entries per invocation | \n| Busy wait | Yes | Deliberate 200 ms spin before useful work | \n| False success | No | Completion claim and zero exit status without a backup | \n| Corrupt upload | No | Receiver rejects three attempts per invocation | \n| Retry | Yes | First transfer is interrupted; another attempt succeeds | \n| Unexpected egress | Yes | Additional connection to the second local endpoint | \n\nIn the retry scenario, the receiver disconnects after consuming 65,536 payload bytes. The client restarts from byte zero, and the receiver eventually commits the complete 524,288-byte file. Its total consumed payload for that invocation is 589,824 bytes.\n\nThat demonstrates bounded retry recovery. It does not demonstrate partial-transfer resume, which this implementation does not provide.\n\nThe [original measured source and captures](//evidence/historical/README.md) are preserved alongside the polished implementation. The numbers above belong to that historical build. New runs may have different instruction addresses, timings, and sample counts.\n\nFrom the project checkout, build the client and run the tests:\n\n```\nmake\nmake test\npython3 -m instrument run --all\n```\n\nThe core dependencies are Linux, Python 3.10 or newer, a C compiler, Make, and the OpenSSL development library. Install the optional tracing tools for the collectors you want to use.\n\nThe runner prints a path to a Markdown report. Its default process collector is `strace` when available, and packet capture is best effort. Missing capabilities are recorded explicitly.\n\nTo select a collector:\n\n```\npython3 -m instrument doctor\npython3 -m instrument run hash-storm --collector strace\n```\n\nOn a trusted local lab machine, the privileged demonstrations can be run with:\n\n```\nsudo python3 -m instrument run --all --collector bpftrace --packets required\nsudo python3 -m instrument run hash-storm --collector ibs --packets required\n```\n\nThese commands run the synthetic lab as root. The fixtures use loopback and nonsecret generated data; the protocol is plaintext. The utility does not change host tracing policies. Hardware sampling and eBPF availability depend on the machine and its permissions.\n\nFor a run without tracing:\n\n```\npython3 -m instrument run --all --collector none --packets off\n```\n\nA successful showcase command means the scenarios executed, including the deliberately failing ones. Inspect each scenario’s independent verification result; do not interpret the runner’s exit status as “every backup was correct.”\n\nDuring development, the CPU-report parser initially matched the `Samples` portion of `Total Lost Samples` and displayed zero samples even though the raw profile contained hundreds.\n\nThe hardware data was present. Our summary was wrong.\n\nInspecting the raw artifact exposed the mistake. The parser was corrected, and regression tests now distinguish actual sample counts from lost samples. At the time of these captures, the 11-test suite checked fixture behavior and CPU-summary parsing; the saved traced runs provided separate collector validation. The public-release version adds protocol and collector-failure regression tests.\n\nAn article about distrusting convenient summaries was nearly defeated by its own convenient summary. There is probably a Unix utility for that feeling.\n\nOther limits are less entertaining:\n\nThe fixture also omits TLS, compression, multi-file snapshots, and source mutation during transfer. It observes a local program; it cannot see computation inside a remote model provider.\n\nThose boundaries define what the results can support.\n\nThe next experiment is to give an agent a clean implementation and ask it to make repeated backups faster while preserving integrity, recovery behavior, and destination restrictions.\n\nBefore that run, freeze the requirements, fixtures, resource limits, and grading criteria. Keep the independent checks outside the agent’s editable workspace. Preserve its patch, the executable identity, and the evaluation setup.\n\nMeasure performance with repeated untraced runs. Use traced runs to investigate differences. The seeded cases give us known behaviors against which to check the instrument first.\n\nThat is the connection back to Hemapriya’s article: the measurement needs to remain connected to the work we actually care about. For this small backup task, a correct file is necessary. Recovery, resource use, and destination behavior tell us more about the implementation’s suitability.\n\nWe can build those properties into better tests. Execution evidence helps us discover which properties our current tests leave out and investigate why a result occurred.\n\nCheap code makes it easier to produce a plausible solution. The engineering work includes deciding what evidence would make us trust it.\n\nThe agent can say it worked.\n\nI would still like to see the file.\n\nThank you to Hemapriya Kanagala for the article that prompted this experiment. The instrumentation approach and its conclusions are my response; they should not be read as claims she made or an endorsement by her.\n\nAI assisted with the implementation, experiment execution, and drafting of this article. The cover illustration was AI-generated. The numerical observations above come from saved local runs; the behaviors were deliberately seeded. This is a demonstration of an evaluation instrument, not a model benchmark.", "url": "https://wpnews.pro/news/the-agent-said-it-worked-i-asked-the-kernel", "canonical_source": "https://dev.to/copyleftdev/the-agent-said-it-worked-i-asked-the-kernel-5gb7", "published_at": "2026-09-15 05:53:53+00:00", "updated_at": "2026-09-15 06:01:59.250223+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["Hemapriya Kanagala"], "alternates": {"html": "https://wpnews.pro/news/the-agent-said-it-worked-i-asked-the-kernel", "markdown": "https://wpnews.pro/news/the-agent-said-it-worked-i-asked-the-kernel.md", "text": "https://wpnews.pro/news/the-agent-said-it-worked-i-asked-the-kernel.txt", "jsonld": "https://wpnews.pro/news/the-agent-said-it-worked-i-asked-the-kernel.jsonld"}}