# I Ran One Local Agent for 22 Hours on a Single DGX Spark. It Timed Out 1,330 Times. > Source: > Published: 2026-10-28 05:15:31+00:00 In [the previous article](https://pub.towardsai.net/one-harness-one-agent-three-gpus-running-qwen-on-a-dgx-spark-an-rtx-5090-and-a-radeon-r9700-d23585ec6a2f), I ran Qwen3.8 27B through DeepSeek Harness (DSH) on three machines: the DGX Spark, an RTX 5090, and a Radeon R9700. In the [follow-up](https://medium.com/towards-artificial-intelligence/my-rtx-5090-fell-from-77-to-7-tokens-per-second-vram-pressure-was-the-culprit-38ecf388772c), I isolated the 5090’s VRAM pressure problem. Both were built from real agent sessions, not synthetic benchmarks. This one is the smallest in the series: one machine, one long session, one workload. The Spark served vLLM with the configuration from the first article: the Unsloth NVFP4 checkpoint, FP8 KV cache, a 0.80 memory budget, a 262,144-token model length, four concurrent sequences, and MTP speculative decoding with three draft tokens. DSH ran on the client side against the OpenAI-compatible endpoint. Nothing was retuned for this run. I let it go and then read the export. One sourcing note: the serving flags come from the first article, not from this export. The export records the agent side (model, context window, steps, attempts, retries) but not the server process. The task was company research at scale: take a list of 93 private equity groups, find every portfolio company, score each one against a written profile, write research notes, and produce a shortlist of the strongest matches. It produced: How much of that output is right is covered in its own section below. The session structure is what made it a stress test: Human input for the entire run consisted of three messages: the task at 18:32 UTC, a one-line scoping correction at 18:50, and a single “continue” at 00:45 UTC. Everything after that, more than 15 hours, ran unattended. Context compaction fired three times in the parent and seven times across the subagents, and the session absorbed it all. Definitions first, because they matter for everything below. A **step** is one agent step (one model response), bounded by `step/start` and `step/end`. An **attempt** is one HTTP request to the model endpoint; a step can contain several. A **retry** is a reissued attempt after a timeout. A **timeout** is an attempt that the client aborted after 300 seconds of inactivity without receiving any bytes. Token counts are cumulative usage from the agent logs, not unique context. The input count includes conversation history resubmitted on every step. The 48.5 tokens/sec is the output tokens divided by the full wall clock time, so it includes orchestration, tool execution, waiting, and retry overhead. It is not a vLLM decode benchmark. For scale: at roughly 0.75 words per token, 3.77M output tokens is about 2.8M words, the **length of 35 novels, produced in under a day by one small machine** across thousands of separate requests rather than one long generation. At a hypothetical constant 240 W draw, 21.6 hours would consume about 5.2 kWh. I did not measure actual power consumption. Summing the client-observed duration of all 2,874 steps gives 384.9 hours. The session ran 21.6. That gap made me re-check the analysis before I trusted any of it. The explanation is concurrency. 384.9 step-hours over 21.6 wall-clock hours means an average of **17.8 steps outstanding at any moment**, peaking at 30, against a server configured for four active sequences. `max-num-seqs` caps how many sequences vLLM executes at once. It does not cap the number of requests that can wait. Put plainly, the client had no limit on how many model requests could be in flight. The parent dispatched subagents in waves; those subagents dispatched their own, and nothing between the agents and the endpoint held them back. The hourly pattern is consistent with that. Timeouts rose and fell with the number of outstanding steps: 107 to 127 per hour during the first subagent waves, when 26 to 27 steps were outstanding on average; 134 to 173 per hour at the second peak; and near zero once fewer than 12 steps were outstanding in the final hours. Across the 22 hourly bins, the **correlation is r = 0.86**. Outstanding steps are client-side concurrency, not server queue depth: some of those steps were actively generating rather than waiting. Two caveats. The relationship runs both ways: a step that times out stays outstanding for at least five more minutes, so timeouts raise the concurrency count as well as following it. And concurrency isn’t the whole story. Hour 7 averaged 20 outstanding steps but had only 34 timeouts. The timeouts were expensive. Measured from send to abort, the 1,330 timed-out attempts total **114.4 cumulative client request-hours, about 30% of all client step-time**. That is client-side time spent across overlapping requests, not idle server time, and none of it produced usable output: whatever the attempt received was discarded when the client aborted it. Long steps were not the problem. Of the 2,114 steps that never timed out, 1,304 ran longer than 300 seconds. Those requests ran long but never went for more than 300 seconds without data. A timeout is something narrower: 300 seconds without a byte. The export has no server-side timing, so it can’t show queue, prefill, or decode directly. But its timestamps bound the silence where it started. Assuming the client’s idle timer resets whenever a byte arrives, a timed-out attempt ends 300 seconds after the last byte it received, so the logged duration minus 300 seconds is the latest point at which it could have heard anything. For all 1,330 timeouts, that point falls within 28 seconds of the attempt starting; the median is about 8 seconds, and for 454 attempts, it is under one second. By that measure, none of the timed-out attempts received data for more than about 28 seconds before going silent. The export doesn’t say what arrived in those first seconds, whether response headers or an initial chunk. My leading hypothesis is that these requests were waiting to start: either queued behind roughly 16 other requests for one of four slots, or prefilling a prompt averaging about 70,000 tokens. Server-side metrics would be needed to separate those, and I am not claiming vLLM was the bottleneck. What the retry data does show: The retry budget is 5 retries after the initial attempt, for a maximum of 6 attempts. The 732 recovered steps completed after one to five timeouts; the 15 worst timed out five times and succeeded on the sixth. The 28 lost steps each show retries 1 through 5, six timeouts, then no response. Those account for the gap between timeouts and retries: 1,302 timeouts were followed by a retry, and 28 were final. I did not trace what the agents did after each of the 28 lost steps. The two firms missing from the workbook (next section) were not lost to these steps: their research files are complete. Throughput and retries say nothing about whether the work is right, so I checked the deliverables directly. **Against the source list.** The source PDF lists the 93 firms, not their portfolio companies, so it can confirm coverage but not whether a company exists. Coverage came up short: only 91 of the 93 firms reached the workbook. The other two were fully researched, 32 current portfolio companies, with three scored as strong matches, but never made it in. The aggregation script the agent wrote matched the section heading `## Portfolio (current)` exactly, while the two research files used slight variants. The workbook’s firm summary sheet shows both firms with zero companies reviewed, and nothing is flagged. That is a pipeline bug, not a model failure, but it is exactly the kind of silent loss a long unattended run makes easy to miss. **Across the workbook.** 302 of the 1,819 rows have no numeric score: 279 were excluded outright as out of scope (financial services, pure services, and the like), 21 have no verdict at all, and 2 carry a verdict with no score behind it. Fifteen companies appear more than once. The full company sheet carries 130 strong-match verdicts and 19 CRM hits, compared with 122 and 17 on the shortlist. **By hand.** The first 25 rows of the full company sheet got a desk review against their research notes and the rubric. They are the first 25 rows, not a random sample: the sheet sorts strong matches first, so all 25 are companies rated 7 or higher, which is where an error costs the most. The five errors: three companies the notes themselves label as exited, still scored 8/10 with every other field empty; one row whose business description and revenue don’t match the company; and one company listed under two sponsors with conflicting descriptions. The three exited companies did not carry over to the shortlist sheet, which accounts for part of the 130 versus 122 gap. None of the 25 looked invented, though several are 2026 formations that could not be independently verified. Five clear errors in 25 rows is a 20% error rate for this subset. Because the subset is the top of a sorted list, not a random draw, that is not an estimate of the error rate across the workbook. So the strong-match list is a useful starting point, not a finished answer. I have been working from it since the run, and find its overall usefulness higher than the error rate would suggest. Two experiments, in order: First, rerun a comparable workload with vLLM’s Prometheus metrics scraped throughout. The `/metrics` endpoint exposes queue time and time-to-first-token, which would separate queue wait from prefill for the attempts that went silent. Second, add the backpressure the client was missing: a semaphore on in-flight model requests, capped at two to four, roughly matching the server’s four active sequences, while agents keep doing tool work in between. Record three numbers: timeout count, median step duration, and completed-work rate. If timeouts fall and completed work holds, the cap helped. If both fall, the agents were using the concurrency they were given. Neither result would make 48.5 tokens/sec a hardware ceiling, since prompt length, caching, tool latency, and workload mix all move that number. I have not run either test, so I am not claiming an optimization result. In the three-GPU comparison, I argued that local inference should be judged on sustained agent work, not peak tokens per second. This session is the stress test for that argument. The timeout overhead was real: about 30% of client step time was spent on aborted attempts. But the retry policy recovered 732 of 760 affected steps, and the workflow finished. On a configuration I had already published and never re-tuned, one Spark ran a 91-subagent workload for almost a full day. The output needed checking, and the check found a silent pipeline bug and a handful of scoring errors. Long unattended runs need a verification pass as much as they need a retry policy. This was not a demonstration of reliable inference. It was a demonstration that imperfect local inference, paired with resilient orchestration, can still complete a substantial agentic workload. **Every model token was generated on the Spark. The agents still accessed external websites and search services, but no cloud inference API was involved.** [I Ran One Local Agent for 22 Hours on a Single DGX Spark. It Timed Out 1,330 Times.](https://pub.towardsai.net/i-ran-one-local-agent-for-22-hours-on-a-single-dgx-spark-it-timed-out-1-330-times-27972d91088c) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.