# System Design the Agentic Way — Phase 1: The Single Machine Ceiling > Source: > Published: 2026-07-13 13:34:05+00:00 Before reasoning about distributed systems, you need a concrete model of what a single machine actually runs out of. Phase 1 builds that model from the ground up — not from diagrams, but from pushing a server until it stops accepting connections, and understanding exactly what gave out and why. A server has CPU, memory, disk, and network capacity. When it fails under load, which one ran out — and does the answer change what you do about it? Phase 1 is an argument that it matters enormously, and that most engineers are watching the wrong gauges. Before getting to failure modes, it helps to have a concrete model of what happens when a user connects to a server. The OS tracks every open resource — files, network connections, pipes — using a small integer called a **file descriptor (FD)**. The first three are always reserved: ``` FD 0 → stdin (keyboard input) FD 1 → stdout (console output) FD 2 → stderr (error output) FD 3 → your log file FD 4 → user A's connection FD 5 → user B's connection ... FD 1024 → OS says "EMFILE — no more" ❌ ``` Every TCP connection your server accepts consumes one FD. The default per-process limit on Linux is **1,024**. When that fills, the OS returns `EMFILE` (too many open files) and refuses new connections — regardless of how much CPU or memory you have left. ``` bash $ ulimit -n # check current limit → probably 1024 $ ulimit -n 65535 # raise it (current session only) # Permanent: edit /etc/security/limits.conf ``` Every HTTP connection starts with a TCP three-way handshake before any data flows: ``` Client → Server: SYN ("I'd like to connect") Server → Client: SYN-ACK ("Got it, I'm ready") Client → Server: ACK ("Confirmed — send your request") Client → Server: [data] (the actual HTTP request) ``` Each step takes a network round-trip. For a client in the same data center that's ~0.1ms; across continents it's 100–300ms. This overhead motivated **HTTP Keep-Alive**. HTTP Keep-Alive reuses a single TCP connection for multiple requests instead of handshaking for each one: ``` Without: handshake → request → close, handshake → request → close (2 handshakes) With: handshake → request → request → request → close (1 handshake) ``` Faster for the client. But idle keep-alive connections still hold an FD open on the server doing nothing. 5,000 users sitting on an idle browser tab = 5,000 FDs consumed. This is a trap that ended the startup in the war story below. When a TCP connection closes, the port on the *client* side enters **TIME_WAIT** for 60–120 seconds. During this window, that port can't be reused for new outbound connections. ``` 1,000 outbound connections/sec × 60 sec TIME_WAIT = 60,000 ports locked Total available ephemeral ports: ~65,000 → Port exhaustion, even with healthy CPU and memory ``` This matters most on servers that make many outbound connections (proxies, services that call many APIs). You'll see `connect: Cannot assign requested address` in logs. A server has five independent ceilings, each with its own failure signature: | Resource | What fills it | Failure signature | Default limit | |---|---|---|---| File descriptors | TCP connections, open files | `EMFILE` — new connections refused | 1,024/process | Memory | Heap, thread stacks, buffers | OOM Killer silently kills process | Depends on RAM | CPU | Computation, context switching | All requests slow down together | 100% across cores | Disk IOPS | Read/write operations | Write latency spikes | SSD: ~100K–500K/s | Network ports | Outbound connections (TIME_WAIT) | New outbound connections fail | ~65K ephemeral ports | The critical distinction: **CPU and memory saturation cause gradual degradation** — everything slows down proportionally. **FD and port exhaustion cause total failure** — new connections are refused while existing ones and dashboard metrics look completely healthy. The second kind is the one that produces a 47-minute outage where nobody can figure out why the server is "down." How a server handles multiple simultaneous connections determines which ceiling it hits first. There are three main approaches. A new OS thread is created for each incoming connection. That thread handles everything for that connection — reading the request, processing it, writing the response — then terminates. ``` User A connects → Thread 1 created (1MB RAM) User B connects → Thread 2 created (1MB RAM) User C connects → Thread 3 created (1MB RAM) ... User 32,000 → Thread 32,000 → OOM ``` Simple to code: each thread runs straight-line blocking code. But each thread costs ~1MB of stack space, and **context switching** between thousands of threads burns CPU. At 10,000 threads, 10–30% of CPU goes just to saving and restoring thread state. At 100,000 — the system is unusable. **First ceiling:** Memory (32GB ÷ 1MB/thread ≈ 32,000 max threads), then CPU from context switching. One thread handles all connections by processing callbacks as they become ready. Instead of blocking while waiting for I/O, it registers a callback and moves on. ``` Hotel receptionist analogy: Loop forever: 1. Any new guests? → start check-in 2. Any responses ready? → send them 3. Any room service done? → notify the room (non-blocking) 4. Back to step 1 ONE receptionist. MANY guests. Never stands idle waiting. ``` Each connection costs only a few KB (just a file descriptor and some socket state), not 1MB. The same 32GB server can hold 100,000+ connections. **First ceiling:** File descriptors (OS limit), or CPU if any one request does synchronous computation. **Weakness:** Head-of-line blocking. One slow synchronous task holds the single thread, freezing every other connection behind it. ``` Drive-through analogy: Car A: "Custom birthday cake" (10 min) Car B: "Just a coffee" ← waits 10 minutes behind Car A Car C: "A muffin" ← waits 10:30 ``` A fixed number of pre-created threads pull work from a shared queue. Balances the two extremes: bounded memory cost (N threads, not one per connection), and true parallelism across CPU cores. ``` Hire 20 waiters. All new tables wait in the lobby. A free waiter → grabs next table → serves → returns to lobby. If all 20 are busy → table waits in queue. If queue is full → reject with 503. ``` The pool size formula: ``` pool_size = num_cores × (1 + wait_time / compute_time) Example: 8 cores, 90ms DB wait, 10ms compute → 8 × 10 = 80 threads ``` **Failure mode:** If a slow downstream dependency holds all threads (e.g., 200 threads × 30-second timeout = all blocked), the queue fills and the service rejects everything — even though your code is fine. | Thread-per-Connection | Event Loop | Thread Pool | | |---|---|---|---| | Memory @ 10K connections | ~10 GB | ~50 MB | ~500 MB | | Max practical connections | 1K–10K | 10K–100K+ | 10K–50K | | Failure mode | OOM / context-switch death spiral | Head-of-line blocking | Queue overflow | | Race conditions | Every CPU instruction boundary | Only at `await` boundaries | Every CPU instruction boundary | | Who uses it | Apache httpd | Node.js, nginx, Redis | Java Tomcat, Go/Netty | Node.js is described as single-threaded. It isn't, exactly. The event loop handles network I/O directly (non-blocking), but **file I/O and DNS lookups are blocking operations** that can't be done asynchronously by the OS. Node offloads these to a small internal thread pool via libuv: ``` Event loop handles: libuv thread pool (default: 4 threads) handles: TCP connections fs.readFile / writeFile HTTP parsing DNS lookups (dns.lookup) DB queries over network crypto (pbkdf2, scrypt) setTimeout / setInterval zlib compression ``` The 4-thread default is dangerously low in production. If 4 file reads are in progress, a DNS lookup queues behind them. HTTP calls to external APIs start timing out for no obvious reason. The fix: `UV_THREADPOOL_SIZE=64` set before the process starts. Every "event loop" runtime is actually a hybrid. Knowing which operations go to the thread pool and which stay on the event loop is essential for debugging latency spikes. The prototype: a Node.js HTTP key-value store, raw `http` module, zero npm packages. Four routes: `/data/:key` (read/write), `/stats` (live metrics — memory, FDs, event loop lag), and `/heavy` (a deliberate CPU blocker). Simple enough to break in predictable ways. Opening TCP connections without closing them, stepping up in batches. The server stayed alive until: ``` 15,965 connections → ENOBUFS (no buffer space available) CPU: ~0% Node heap: ~480 MB (healthy) ``` The event loop was not blocked. Node's memory was fine. What ran out was **OS kernel network buffer space** — roughly 250 MB of non-paged pool memory consumed by socket receive/send buffers (~16 KB each). The ceiling was in the kernel, not in the application. ``` Ceiling hierarchy hit in order: 1. OS kernel network buffers ← actual ceiling on Windows 2. FD limit (~16K on Windows) ← would have been next 3. Node.js heap ← never reached 4. CPU ← never reached ``` This is the war story that opens the phase: a social media API during a viral spike. CPU at 5%, memory at 40%, every dashboard looked healthy — but the server was refusing every new connection. 47-minute outage. The fix: ``` ulimit -n 65535 # raises the per-process FD limit from the default 1,024 ``` The point isn't to memorize `ulimit` . It's that the server had massive headroom on four of its five ceilings, and failed on the fifth one nobody was measuring. A single request to `/heavy` : ``` js // This holds the event loop thread — nothing else can run const start = Date.now(); while (Date.now() - start < 5000) {} ``` During those 5 seconds: ``` /stats response time: 5,000 ms (normally 1 ms) Event loop lag: 12,000 ms CPU: 98% All other requests: frozen ``` The `/stats` route does trivial work — read process memory, count handles, respond. Under normal conditions it takes 1ms. The while loop on `/heavy` blocked it for 5 full seconds because **the event loop is a single thread with cooperative scheduling**. There is no OS scheduler interrupting a running synchronous task to let another request proceed. The CPU-bound route held the thread until it finished. This surfaces in production from: `JSON.parse()` on a 50 MB payload, an unanchored regex on user input (ReDoS), or a forgotten `fs.readFileSync()` in a request handler. All of them stall every connected user simultaneously. 100 concurrent PUT requests, each reading the current counter, incrementing it, and writing it back. Expected final value: 100. Actual: 60-something. The interleaving that causes this: ``` Request A: await readStore() → gets { counter: 1 } ← yields here Request B: await readStore() → gets { counter: 1 } ← stale; A hasn't written Request A: await writeStore({ counter: 2 }) Request B: await writeStore({ counter: 2 }) ← overwrites; should be 3 ``` The assumption going in: single-threaded event loop means no interleaving. That's true *between* `await` calls — code between two awaits is atomic. But the full async function is not. Every `await` is a yield point where the event loop can run another callback. In a read-modify-write pattern, that's enough. The same lost-update problem exists in multi-threaded code, just at every CPU instruction instead of only at `await` boundaries. The event loop makes races less frequent, not impossible. Each experiment has a distinct fix: `ulimit -n` / set `nofile` in systemd unit or container config`worker_threads` ; never call blocking APIs in a hot pathAll three experiments produced similar surface symptoms — slow or unresponsive server. The root causes are completely different. Applying the wrong intervention (or adding a second server) wouldn't have helped. The resource that fails determines both the symptom and the remedy. **Not all resource ceilings behave like slowness.** FD and port exhaustion produce hard refusals while CPU and memory metrics look healthy. This class of failure is disproportionately misdiagnosed because teams are watching the wrong gauges. **The concurrency model determines which ceiling you hit first.** Thread-per-connection hits memory first (~1MB/thread → ~32K threads on 32GB RAM). Event-loop hits file descriptors or CPU-bound blocking first. The model isn't just a performance choice — it changes the failure mode. **1. A server has five independent resource ceilings, and the smallest one is the actual limit.** CPU saturation causes gradual degradation. FD exhaustion and port exhaustion cause total failure while all other metrics look fine. Know which ceiling you're approaching before an incident. **2. The thread-per-connection model kills itself on memory and context switching.** At 10,000 threads: ~10 GB of stack space, and 10–30% of CPU wasted on saving/restoring thread state. This is why Node.js, nginx, and Redis all chose the event loop. **3. The event loop trades memory efficiency for CPU vulnerability.** A single synchronous task holds the thread. Head-of-line blocking isn't a bug to fix — it's a fundamental property of cooperative concurrency. The architecture choice is: which failure mode can I tolerate? OOM death spiral, or one hot request freezing everyone? **4. "Single-threaded" is a simplification.** Node.js has 4 libuv worker threads by default handling file I/O, DNS, and crypto. DNS lookups queue behind file reads. In production, set `UV_THREADPOOL_SIZE=64` . **5. Async code has race conditions at await boundaries.** Code between two awaits is atomic. A full async function is not. Every read-modify-write pattern across awaits needs explicit serialization. **When to use which model:** ``` < 5K connections, simple code needed → Thread-per-connection > 10K connections, mostly I/O-bound → Event loop Mixed CPU + I/O, need parallelism → Thread pool Production at scale → Hybrid (event loop + worker pool) ``` ** ENOBUFS vs EMFILE** — I hit `ENOBUFS` at ~16K connections on Windows, which maps to OS kernel buffer exhaustion, not the FD limit. On Linux with default `ulimit -n 1024` , `EMFILE` would appear much earlier. Are these always separable in production?**TIME_WAIT at high connection churn** — I understand port exhaustion in theory (1K conns/sec × 60s = 60K locked ports), but I haven't built something that actually hits it. That would require a high-outbound-connection scenario — probably relevant in Phase 9 (service decomposition) when services call each other. **libuv thread pool saturation in practice** — the 4-thread default is clearly too low, but what does the performance curve look like as you raise it? Is there a point where more threads hurt? Four experiments worth running: `GET /stats` — see live FD count, heap usage, and event loop lag`/heavy` in one tab, then hit `/stats` from another — watch the lag numberThe stats endpoint measures event loop lag by scheduling a `setTimeout(0)` callback and measuring how long before it actually runs. Under normal load: < 5ms. During a CPU-bound request: in the thousands. Phase 2 moves the problem to multiple machines — which immediately surfaces a new class of failures that single-machine thinking doesn't predict. *Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.*