The 3% Drop That Only Showed Up on a Clean Server: A Debugging Retrospective

A developer traced a 3% message loss in a TCP forwarder to a read loop that replied once per read instead of per line, a bug that only surfaced under burst load. The fix was a twelve-line change, and the developer recommends burst-mode testing to expose pipelining issues that polite tests miss.

My forwarder lost 3% of its messages. My laptop reported zero. CI reported zero. Staging lost thousands every hour. I blamed the network. I blamed the VM. I blamed everything except my read loop. This is the retrospective: one wrong assumption, three counters, a twelve-line fix. The service was a small TCP forwarder. Accept a connection, read text lines, forward them. Nothing exotic. Staging showed the loss: 97,004 lines in, 94,183 lines out. No errors. No exceptions. Just missing messages. Replays on my laptop always passed. 100,000 lines in, 100,000 lines out. That mismatch is the first clue. I was not testing what production actually did. When a bug only lives in one environment, do you fix the environment first? Or the assumption your tests never stressed? Before touching code, I changed only the machine. Same binary, same load script, same counters. I started a fresh server from the free server option of MonkeyCode, an open source AI coding project. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The loss reappeared. 100,000 lines in, 97,128 lines out. The environment changed the probability, not the existence of the bug. A second machine is a microscope for timing bugs. A fresh one has no history and no bias. Logs tell stories. Counters add up. I placed a counter at every layer. Run once, compare four numbers. The gap between them names the losing layer. | Counter gap | Suspect | |---|---| | parsed < sent | read loop or buffer size | | replies < parsed | reply logic this bug | | received < replies | client read logic | The clean-server run was loud: sent 100,000, parsed 100,000, replies 99,987, received 99,987. The server parsed everything. It just did not reply enough. That gap is the fingerprint. Smallest code that fails. One socket, one buffer, no frameworks. load.py - burst mode, no waiting between sends: python load.py - burst mode. No waiting between sends. import socket N = 100 000 s = socket.create connection "127.0.0.1", 9000 s.settimeout 5 acks = 0 try: for in range N : s.sendall b"PING\n" while acks < N: data = s.recv 65536 if not data: break acks += data.count b"\n" except TimeoutError: pass print f"sent={N} acks={acks} lost={N - acks}" bad server.cpp - one reply per read, not per line: // bad server.cpp - one reply per read, not per line. include <arpa/inet.h include <netinet/in.h include <sys/socket.h include <unistd.h include <cstdio int main { int listen fd = socket AF INET, SOCK STREAM, 0 ; int one = 1; setsockopt listen fd, SOL SOCKET, SO REUSEADDR, &one, sizeof one ; sockaddr in addr{}; addr.sin family = AF INET; addr.sin port = htons 9000 ; addr.sin addr.s addr = INADDR ANY; bind listen fd, sockaddr &addr, sizeof addr ; listen listen fd, 64 ; char buf 256 ; long parsed = 0; long replies = 0; for ;; { int c = accept listen fd, nullptr, nullptr ; if c < 0 continue; for ;; { ssize t r = read c, buf, sizeof buf ; if r <= 0 break; for ssize t i = 0; i < r; ++i if buf i == '\n' parsed++; write c, "ok\n", 3 ; // one reply per read, not per line replies++; } close c ; fprintf stderr, "parsed=%ld replies=%ld\n", parsed, replies ; } } Look at the reply line. One read can return many lines. TCP is a byte stream, not a message queue. read has no idea what a PING is. The kernel may merge ten messages into one segment, and one read consumes all ten. My polite local test sent one PING, waited for the ack, sent the next. One message per segment. The bug stayed asleep. The staging load was a burst. Thousands of messages per second, batching inside the kernel. One read swallowed a hundred lines, the server answered once. Test shape decides which bugs survive. Polite tests hide pipelining bugs. Burst tests expose them. The small fix replies per newline instead of per read. for ssize t i = 0; i < r; ++i { if buf i == '\n' { parsed++; write c, "ok\n", 3 ; } } The robust fix also survives a line split across two reads. That is the other half of the same bug family. std::string acc; ssize t r; while r = read c, buf, sizeof buf 0 { acc.append buf, static cast<size t r ; size t pos; while pos = acc.find '\n' = std::string::npos { acc.erase 0, pos + 1 ; parsed++; write c, "ok\n", 3 ; } } After the fix, both environments produced identical numbers. sent == parsed == replied == received. man 2 read . The man page says: "read attempts to read up to count bytes." It never promises one message. Verify that claim at This workflow targets timing and batching bugs. Data bugs need property-based tests, not a second machine. Free servers are run-to-failure tools, not persistent hosts. Do not store state there, do not run compliance workloads, and do not benchmark the server itself. Use one for a clean slate and one reproduction. The fix changed reply counts, not throughput. Measure both after touching a read loop. Your local test is not your load test. Your laptop is not your server. One read is not one message. Start your next local-only bug with a burst client and a second machine. Then count at every layer. The counters know the truth. Listen to them.