{"slug": "the-3-drop-that-only-showed-up-on-a-clean-server-a-debugging-retrospective", "title": "The 3% Drop That Only Showed Up on a Clean Server: A Debugging Retrospective", "summary": "A developer traced a 3% message loss in a TCP forwarder to a read loop that replied once per read instead of per line, a bug that only surfaced under burst load. The fix was a twelve-line change, and the developer recommends burst-mode testing to expose pipelining issues that polite tests miss.", "body_md": "My forwarder lost 3% of its messages. My laptop reported zero. CI reported zero. Staging lost thousands every hour.\n\nI blamed the network. I blamed the VM. I blamed everything except my read loop.\n\nThis is the retrospective: one wrong assumption, three counters, a twelve-line fix.\n\nThe service was a small TCP forwarder. Accept a connection, read text lines, forward them. Nothing exotic.\n\nStaging showed the loss: 97,004 lines in, 94,183 lines out. No errors. No exceptions. Just missing messages.\n\nReplays on my laptop always passed. 100,000 lines in, 100,000 lines out.\n\nThat mismatch is the first clue. I was not testing what production actually did.\n\nWhen a bug only lives in one environment, do you fix the environment first? Or the assumption your tests never stressed?\n\nBefore touching code, I changed only the machine. Same binary, same load script, same counters.\n\nI started a fresh server from the free server option of MonkeyCode, an open source AI coding project.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nThe loss reappeared. 100,000 lines in, 97,128 lines out. The environment changed the probability, not the existence of the bug.\n\nA second machine is a microscope for timing bugs. A fresh one has no history and no bias.\n\nLogs tell stories. Counters add up. I placed a counter at every layer.\n\nRun once, compare four numbers. The gap between them names the losing layer.\n\n| Counter gap | Suspect |\n|---|---|\n| parsed < sent | read loop or buffer size |\n| replies < parsed | reply logic (this bug) |\n| received < replies | client read logic |\n\nThe clean-server run was loud: sent 100,000, parsed 100,000, replies 99,987, received 99,987.\n\nThe server parsed everything. It just did not reply enough. That gap is the fingerprint.\n\nSmallest code that fails. One socket, one buffer, no frameworks.\n\n`load.py`\n\n- burst mode, no waiting between sends:\n\n``` python\n# load.py - burst mode. No waiting between sends.\nimport socket\n\nN = 100_000\ns = socket.create_connection((\"127.0.0.1\", 9000))\ns.settimeout(5)\nacks = 0\ntry:\n    for _ in range(N):\n        s.sendall(b\"PING\\n\")\n    while acks < N:\n        data = s.recv(65536)\n        if not data:\n            break\n        acks += data.count(b\"\\n\")\nexcept TimeoutError:\n    pass\nprint(f\"sent={N} acks={acks} lost={N - acks}\")\n```\n\n`bad_server.cpp`\n\n- one reply per read, not per line:\n\n```\n// bad_server.cpp - one reply per read, not per line.\n#include <arpa/inet.h>\n#include <netinet/in.h>\n#include <sys/socket.h>\n#include <unistd.h>\n\n#include <cstdio>\n\nint main() {\n  int listen_fd = socket(AF_INET, SOCK_STREAM, 0);\n  int one = 1;\n  setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);\n\n  sockaddr_in addr{};\n  addr.sin_family = AF_INET;\n  addr.sin_port = htons(9000);\n  addr.sin_addr.s_addr = INADDR_ANY;\n  bind(listen_fd, (sockaddr*)&addr, sizeof addr);\n  listen(listen_fd, 64);\n\n  char buf[256];\n  long parsed = 0;\n  long replies = 0;\n  for (;;) {\n    int c = accept(listen_fd, nullptr, nullptr);\n    if (c < 0) continue;\n    for (;;) {\n      ssize_t r = read(c, buf, sizeof buf);\n      if (r <= 0) break;\n      for (ssize_t i = 0; i < r; ++i)\n        if (buf[i] == '\\n') parsed++;\n      write(c, \"ok\\n\", 3);  // one reply per read, not per line\n      replies++;\n    }\n    close(c);\n    fprintf(stderr, \"parsed=%ld replies=%ld\\n\", parsed, replies);\n  }\n}\n```\n\nLook at the reply line. One read can return many lines.\n\nTCP is a byte stream, not a message queue. `read()`\n\nhas no idea what a PING is. The kernel may merge ten messages into one segment, and one read consumes all ten.\n\nMy polite local test sent one PING, waited for the ack, sent the next. One message per segment. The bug stayed asleep.\n\nThe staging load was a burst. Thousands of messages per second, batching inside the kernel. One read swallowed a hundred lines, the server answered once.\n\nTest shape decides which bugs survive. Polite tests hide pipelining bugs. Burst tests expose them.\n\nThe small fix replies per newline instead of per read.\n\n```\nfor (ssize_t i = 0; i < r; ++i) {\n  if (buf[i] == '\\n') {\n    parsed++;\n    write(c, \"ok\\n\", 3);\n  }\n}\n```\n\nThe robust fix also survives a line split across two reads. That is the other half of the same bug family.\n\n```\nstd::string acc;\nssize_t r;\nwhile ((r = read(c, buf, sizeof buf)) > 0) {\n  acc.append(buf, static_cast<size_t>(r));\n  size_t pos;\n  while ((pos = acc.find('\\n')) != std::string::npos) {\n    acc.erase(0, pos + 1);\n    parsed++;\n    write(c, \"ok\\n\", 3);\n  }\n}\n```\n\nAfter the fix, both environments produced identical numbers. sent == parsed == replied == received.\n\n`man 2 read`\n\n. The man page says: \"read() attempts to read up to count bytes.\" It never promises one message. Verify that claim at This workflow targets timing and batching bugs. Data bugs need property-based tests, not a second machine.\n\nFree servers are run-to-failure tools, not persistent hosts. Do not store state there, do not run compliance workloads, and do not benchmark the server itself. Use one for a clean slate and one reproduction.\n\nThe fix changed reply counts, not throughput. Measure both after touching a read loop.\n\nYour local test is not your load test. Your laptop is not your server. One read is not one message.\n\nStart your next local-only bug with a burst client and a second machine. Then count at every layer.\n\nThe counters know the truth. Listen to them.", "url": "https://wpnews.pro/news/the-3-drop-that-only-showed-up-on-a-clean-server-a-debugging-retrospective", "canonical_source": "https://dev.to/datacpp_3670/the-3-drop-that-only-showed-up-on-a-clean-server-a-debugging-retrospective-1g3c", "published_at": "2026-08-28 03:51:57+00:00", "updated_at": "2026-08-28 04:19:31.548509+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/the-3-drop-that-only-showed-up-on-a-clean-server-a-debugging-retrospective", "markdown": "https://wpnews.pro/news/the-3-drop-that-only-showed-up-on-a-clean-server-a-debugging-retrospective.md", "text": "https://wpnews.pro/news/the-3-drop-that-only-showed-up-on-a-clean-server-a-debugging-retrospective.txt", "jsonld": "https://wpnews.pro/news/the-3-drop-that-only-showed-up-on-a-clean-server-a-debugging-retrospective.jsonld"}}