{"slug": "system-design-the-agentic-way-phase-1-the-single-machine-ceiling", "title": "System Design the Agentic Way — Phase 1: The Single Machine Ceiling", "summary": "A developer argues that understanding single-machine resource ceilings is critical before designing distributed systems. The post details how file descriptor limits, TCP handshake overhead, and TIME_WAIT port exhaustion cause total failure modes distinct from gradual CPU/memory degradation, illustrated with a war story of a startup killed by idle keep-alive connections.", "body_md": "Before reasoning about distributed systems, you need a concrete model of what a single machine actually runs out of. Phase 1 builds that model from the ground up — not from diagrams, but from pushing a server until it stops accepting connections, and understanding exactly what gave out and why.\n\nA server has CPU, memory, disk, and network capacity. When it fails under load, which one ran out — and does the answer change what you do about it?\n\nPhase 1 is an argument that it matters enormously, and that most engineers are watching the wrong gauges.\n\nBefore getting to failure modes, it helps to have a concrete model of what happens when a user connects to a server.\n\nThe OS tracks every open resource — files, network connections, pipes — using a small integer called a **file descriptor (FD)**. The first three are always reserved:\n\n```\nFD 0 → stdin  (keyboard input)\nFD 1 → stdout (console output)\nFD 2 → stderr (error output)\nFD 3 → your log file\nFD 4 → user A's connection\nFD 5 → user B's connection\n...\nFD 1024 → OS says \"EMFILE — no more\" ❌\n```\n\nEvery TCP connection your server accepts consumes one FD. The default per-process limit on Linux is **1,024**. When that fills, the OS returns `EMFILE`\n\n(too many open files) and refuses new connections — regardless of how much CPU or memory you have left.\n\n``` bash\n$ ulimit -n        # check current limit → probably 1024\n$ ulimit -n 65535  # raise it (current session only)\n# Permanent: edit /etc/security/limits.conf\n```\n\nEvery HTTP connection starts with a TCP three-way handshake before any data flows:\n\n```\nClient → Server: SYN      (\"I'd like to connect\")\nServer → Client: SYN-ACK  (\"Got it, I'm ready\")\nClient → Server: ACK       (\"Confirmed — send your request\")\nClient → Server: [data]    (the actual HTTP request)\n```\n\nEach step takes a network round-trip. For a client in the same data center that's ~0.1ms; across continents it's 100–300ms. This overhead motivated **HTTP Keep-Alive**.\n\nHTTP Keep-Alive reuses a single TCP connection for multiple requests instead of handshaking for each one:\n\n```\nWithout: handshake → request → close,  handshake → request → close  (2 handshakes)\nWith:    handshake → request → request → request → close             (1 handshake)\n```\n\nFaster for the client. But idle keep-alive connections still hold an FD open on the server doing nothing. 5,000 users sitting on an idle browser tab = 5,000 FDs consumed. This is a trap that ended the startup in the war story below.\n\nWhen a TCP connection closes, the port on the *client* side enters **TIME_WAIT** for 60–120 seconds. During this window, that port can't be reused for new outbound connections.\n\n```\n1,000 outbound connections/sec × 60 sec TIME_WAIT = 60,000 ports locked\nTotal available ephemeral ports: ~65,000\n→ Port exhaustion, even with healthy CPU and memory\n```\n\nThis matters most on servers that make many outbound connections (proxies, services that call many APIs). You'll see `connect: Cannot assign requested address`\n\nin logs.\n\nA server has five independent ceilings, each with its own failure signature:\n\n| Resource | What fills it | Failure signature | Default limit |\n|---|---|---|---|\nFile descriptors |\nTCP connections, open files |\n`EMFILE` — new connections refused |\n1,024/process |\nMemory |\nHeap, thread stacks, buffers | OOM Killer silently kills process | Depends on RAM |\nCPU |\nComputation, context switching | All requests slow down together | 100% across cores |\nDisk IOPS |\nRead/write operations | Write latency spikes | SSD: ~100K–500K/s |\nNetwork ports |\nOutbound connections (TIME_WAIT) | New outbound connections fail | ~65K ephemeral ports |\n\nThe critical distinction: **CPU and memory saturation cause gradual degradation** — everything slows down proportionally. **FD and port exhaustion cause total failure** — new connections are refused while existing ones and dashboard metrics look completely healthy. The second kind is the one that produces a 47-minute outage where nobody can figure out why the server is \"down.\"\n\nHow a server handles multiple simultaneous connections determines which ceiling it hits first. There are three main approaches.\n\nA new OS thread is created for each incoming connection. That thread handles everything for that connection — reading the request, processing it, writing the response — then terminates.\n\n```\nUser A connects → Thread 1 created (1MB RAM)\nUser B connects → Thread 2 created (1MB RAM)\nUser C connects → Thread 3 created (1MB RAM)\n...\nUser 32,000 → Thread 32,000 → OOM\n```\n\nSimple to code: each thread runs straight-line blocking code. But each thread costs ~1MB of stack space, and **context switching** between thousands of threads burns CPU. At 10,000 threads, 10–30% of CPU goes just to saving and restoring thread state. At 100,000 — the system is unusable.\n\n**First ceiling:** Memory (32GB ÷ 1MB/thread ≈ 32,000 max threads), then CPU from context switching.\n\nOne thread handles all connections by processing callbacks as they become ready. Instead of blocking while waiting for I/O, it registers a callback and moves on.\n\n```\nHotel receptionist analogy:\n  Loop forever:\n    1. Any new guests?        → start check-in\n    2. Any responses ready?   → send them\n    3. Any room service done? → notify the room (non-blocking)\n    4. Back to step 1\n\nONE receptionist. MANY guests. Never stands idle waiting.\n```\n\nEach connection costs only a few KB (just a file descriptor and some socket state), not 1MB. The same 32GB server can hold 100,000+ connections.\n\n**First ceiling:** File descriptors (OS limit), or CPU if any one request does synchronous computation.\n\n**Weakness:** Head-of-line blocking. One slow synchronous task holds the single thread, freezing every other connection behind it.\n\n```\nDrive-through analogy:\n  Car A: \"Custom birthday cake\" (10 min)\n  Car B: \"Just a coffee\"  ← waits 10 minutes behind Car A\n  Car C: \"A muffin\"       ← waits 10:30\n```\n\nA fixed number of pre-created threads pull work from a shared queue. Balances the two extremes: bounded memory cost (N threads, not one per connection), and true parallelism across CPU cores.\n\n```\nHire 20 waiters. All new tables wait in the lobby.\nA free waiter → grabs next table → serves → returns to lobby.\nIf all 20 are busy → table waits in queue.\nIf queue is full → reject with 503.\n```\n\nThe pool size formula:\n\n```\npool_size = num_cores × (1 + wait_time / compute_time)\nExample: 8 cores, 90ms DB wait, 10ms compute → 8 × 10 = 80 threads\n```\n\n**Failure mode:** If a slow downstream dependency holds all threads (e.g., 200 threads × 30-second timeout = all blocked), the queue fills and the service rejects everything — even though your code is fine.\n\n| Thread-per-Connection | Event Loop | Thread Pool | |\n|---|---|---|---|\n| Memory @ 10K connections | ~10 GB | ~50 MB | ~500 MB |\n| Max practical connections | 1K–10K | 10K–100K+ | 10K–50K |\n| Failure mode | OOM / context-switch death spiral | Head-of-line blocking | Queue overflow |\n| Race conditions | Every CPU instruction boundary | Only at `await` boundaries |\nEvery CPU instruction boundary |\n| Who uses it | Apache httpd | Node.js, nginx, Redis | Java Tomcat, Go/Netty |\n\nNode.js is described as single-threaded. It isn't, exactly. The event loop handles network I/O directly (non-blocking), but **file I/O and DNS lookups are blocking operations** that can't be done asynchronously by the OS. Node offloads these to a small internal thread pool via libuv:\n\n```\nEvent loop handles:          libuv thread pool (default: 4 threads) handles:\n  TCP connections              fs.readFile / writeFile\n  HTTP parsing                 DNS lookups (dns.lookup)\n  DB queries over network      crypto (pbkdf2, scrypt)\n  setTimeout / setInterval     zlib compression\n```\n\nThe 4-thread default is dangerously low in production. If 4 file reads are in progress, a DNS lookup queues behind them. HTTP calls to external APIs start timing out for no obvious reason. The fix: `UV_THREADPOOL_SIZE=64`\n\nset before the process starts.\n\nEvery \"event loop\" runtime is actually a hybrid. Knowing which operations go to the thread pool and which stay on the event loop is essential for debugging latency spikes.\n\nThe prototype: a Node.js HTTP key-value store, raw `http`\n\nmodule, zero npm packages. Four routes: `/data/:key`\n\n(read/write), `/stats`\n\n(live metrics — memory, FDs, event loop lag), and `/heavy`\n\n(a deliberate CPU blocker). Simple enough to break in predictable ways.\n\nOpening TCP connections without closing them, stepping up in batches. The server stayed alive until:\n\n```\n15,965 connections → ENOBUFS (no buffer space available)\nCPU: ~0%    Node heap: ~480 MB (healthy)\n```\n\nThe event loop was not blocked. Node's memory was fine. What ran out was **OS kernel network buffer space** — roughly 250 MB of non-paged pool memory consumed by socket receive/send buffers (~16 KB each). The ceiling was in the kernel, not in the application.\n\n```\nCeiling hierarchy hit in order:\n  1. OS kernel network buffers  ← actual ceiling on Windows\n  2. FD limit (~16K on Windows) ← would have been next\n  3. Node.js heap               ← never reached\n  4. CPU                        ← never reached\n```\n\nThis is the war story that opens the phase: a social media API during a viral spike. CPU at 5%, memory at 40%, every dashboard looked healthy — but the server was refusing every new connection. 47-minute outage. The fix:\n\n```\nulimit -n 65535   # raises the per-process FD limit from the default 1,024\n```\n\nThe point isn't to memorize `ulimit`\n\n. It's that the server had massive headroom on four of its five ceilings, and failed on the fifth one nobody was measuring.\n\nA single request to `/heavy`\n\n:\n\n``` js\n// This holds the event loop thread — nothing else can run\nconst start = Date.now();\nwhile (Date.now() - start < 5000) {}\n```\n\nDuring those 5 seconds:\n\n```\n/stats response time:  5,000 ms  (normally 1 ms)\nEvent loop lag:        12,000 ms\nCPU:                   98%\nAll other requests:    frozen\n```\n\nThe `/stats`\n\nroute does trivial work — read process memory, count handles, respond. Under normal conditions it takes 1ms. The while loop on `/heavy`\n\nblocked it for 5 full seconds because **the event loop is a single thread with cooperative scheduling**. There is no OS scheduler interrupting a running synchronous task to let another request proceed. The CPU-bound route held the thread until it finished.\n\nThis surfaces in production from: `JSON.parse()`\n\non a 50 MB payload, an unanchored regex on user input (ReDoS), or a forgotten `fs.readFileSync()`\n\nin a request handler. All of them stall every connected user simultaneously.\n\n100 concurrent PUT requests, each reading the current counter, incrementing it, and writing it back. Expected final value: 100. Actual: 60-something.\n\nThe interleaving that causes this:\n\n```\nRequest A: await readStore()   → gets { counter: 1 }  ← yields here\nRequest B: await readStore()   → gets { counter: 1 }  ← stale; A hasn't written\nRequest A: await writeStore({ counter: 2 })\nRequest B: await writeStore({ counter: 2 })            ← overwrites; should be 3\n```\n\nThe assumption going in: single-threaded event loop means no interleaving. That's true *between* `await`\n\ncalls — code between two awaits is atomic. But the full async function is not. Every `await`\n\nis a yield point where the event loop can run another callback. In a read-modify-write pattern, that's enough.\n\nThe same lost-update problem exists in multi-threaded code, just at every CPU instruction instead of only at `await`\n\nboundaries. The event loop makes races less frequent, not impossible.\n\nEach experiment has a distinct fix:\n\n`ulimit -n`\n\n/ set `nofile`\n\nin systemd unit or container config`worker_threads`\n\n; never call blocking APIs in a hot pathAll three experiments produced similar surface symptoms — slow or unresponsive server. The root causes are completely different. Applying the wrong intervention (or adding a second server) wouldn't have helped. The resource that fails determines both the symptom and the remedy.\n\n**Not all resource ceilings behave like slowness.** FD and port exhaustion produce hard refusals while CPU and memory metrics look healthy. This class of failure is disproportionately misdiagnosed because teams are watching the wrong gauges.\n\n**The concurrency model determines which ceiling you hit first.** Thread-per-connection hits memory first (~1MB/thread → ~32K threads on 32GB RAM). Event-loop hits file descriptors or CPU-bound blocking first. The model isn't just a performance choice — it changes the failure mode.\n\n**1. A server has five independent resource ceilings, and the smallest one is the actual limit.** CPU saturation causes gradual degradation. FD exhaustion and port exhaustion cause total failure while all other metrics look fine. Know which ceiling you're approaching before an incident.\n\n**2. The thread-per-connection model kills itself on memory and context switching.** At 10,000 threads: ~10 GB of stack space, and 10–30% of CPU wasted on saving/restoring thread state. This is why Node.js, nginx, and Redis all chose the event loop.\n\n**3. The event loop trades memory efficiency for CPU vulnerability.** A single synchronous task holds the thread. Head-of-line blocking isn't a bug to fix — it's a fundamental property of cooperative concurrency. The architecture choice is: which failure mode can I tolerate? OOM death spiral, or one hot request freezing everyone?\n\n**4. \"Single-threaded\" is a simplification.** Node.js has 4 libuv worker threads by default handling file I/O, DNS, and crypto. DNS lookups queue behind file reads. In production, set `UV_THREADPOOL_SIZE=64`\n\n.\n\n**5. Async code has race conditions at await boundaries.** Code between two awaits is atomic. A full async function is not. Every read-modify-write pattern across awaits needs explicit serialization.\n\n**When to use which model:**\n\n```\n< 5K connections, simple code needed        → Thread-per-connection\n> 10K connections, mostly I/O-bound         → Event loop\nMixed CPU + I/O, need parallelism           → Thread pool\nProduction at scale                         → Hybrid (event loop + worker pool)\n```\n\n** ENOBUFS vs EMFILE** — I hit\n\n`ENOBUFS`\n\nat ~16K connections on Windows, which maps to OS kernel buffer exhaustion, not the FD limit. On Linux with default `ulimit -n 1024`\n\n, `EMFILE`\n\nwould appear much earlier. Are these always separable in production?**TIME_WAIT at high connection churn** — I understand port exhaustion in theory (1K conns/sec × 60s = 60K locked ports), but I haven't built something that actually hits it. That would require a high-outbound-connection scenario — probably relevant in Phase 9 (service decomposition) when services call each other.\n\n**libuv thread pool saturation in practice** — the 4-thread default is clearly too low, but what does the performance curve look like as you raise it? Is there a point where more threads hurt?\n\nFour experiments worth running:\n\n`GET /stats`\n\n— see live FD count, heap usage, and event loop lag`/heavy`\n\nin one tab, then hit `/stats`\n\nfrom another — watch the lag numberThe stats endpoint measures event loop lag by scheduling a `setTimeout(0)`\n\ncallback and measuring how long before it actually runs. Under normal load: < 5ms. During a CPU-bound request: in the thousands.\n\nPhase 2 moves the problem to multiple machines — which immediately surfaces a new class of failures that single-machine thinking doesn't predict.\n\n*Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.*", "url": "https://wpnews.pro/news/system-design-the-agentic-way-phase-1-the-single-machine-ceiling", "canonical_source": "https://dev.to/ayrawas/system-design-the-agentic-way-phase-1-the-single-machine-ceiling-2i8n", "published_at": "2026-07-13 13:34:05+00:00", "updated_at": "2026-07-13 13:46:39.947177+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Linux"], "alternates": {"html": "https://wpnews.pro/news/system-design-the-agentic-way-phase-1-the-single-machine-ceiling", "markdown": "https://wpnews.pro/news/system-design-the-agentic-way-phase-1-the-single-machine-ceiling.md", "text": "https://wpnews.pro/news/system-design-the-agentic-way-phase-1-the-single-machine-ceiling.txt", "jsonld": "https://wpnews.pro/news/system-design-the-agentic-way-phase-1-the-single-machine-ceiling.jsonld"}}