From Goroutines to Agents: Lessons from 1M Concurrent Threads and the New Wave of AI Engineering

A developer recounts how a Go-based infrastructure service hit 1 million concurrent goroutines in 2021, and how the lessons from that incident—structured concurrency, cancellation propagation, and resource budgeting—are now being applied to AI agent systems. The post draws parallels between unbounded goroutine fan-out and unbounded LLM call fan-out, advocating for semaphore-based concurrency limits and context propagation in agent frameworks.

Originally published on tamiz.pro. In 2021, a team running a Go-based infrastructure service hit 1 million concurrent goroutines under load. The lessons they pulled from that scale—structured concurrency, cancellation propagation, resource budgeting, observable failure—landed differently this time around. Today, the same patterns are surfacing as engineers build production AI agent systems, except instead of goroutines racing on an event loop, we're managing LLM calls, tool executions, and streaming responses across distributed services. The parallel isn't coincidental. Both domains share a core tension: unbounded fan-out looks elegant in code and catastrophic in production. Understanding how the Go community solved this at massive scale gives AI engineers a head start on the problems that are now hitting agent platforms. The 1M-goroutine incident started innocently. A request handler spawned a worker goroutine per downstream call: func HandleRequest ctx context.Context, req Request Result, error { var wg sync.WaitGroup results := make Result, len req.Subtasks for i, task := range req.Subtasks { wg.Add 1 go func idx int, t Task { defer wg.Done results idx = process ctx, t } i, task } wg.Wait return results, nil } Under normal load this was fine. But when the system saw a burst of requests—each with 50–200 subtasks—the goroutine count spiked. Without cancellation on ctx , every in-flight goroutine survived until its upstream request timed out or the process was killed. The Go runtime didn't crash it's designed for this , but the scheduler overhead became significant, and memory from pending operations accumulated. The fix wasn't removing goroutines—it was adding boundaries : func HandleRequest ctx context.Context, req Request Result, error { // Bound concurrency, not just spawn freely sem := make chan struct{}, 500 // max 500 concurrent workers var wg sync.WaitGroup results := make Result, len req.Subtasks for i, task := range req.Subtasks { wg.Add 1 go func idx int, t Task { defer wg.Done select { case sem <- struct{}{}: defer func { <-sem } results idx = process ctx, t case <-ctx.Done : return } } i, task } wg.Wait return results, nil } AI agent frameworks exhibit the exact same pattern today. Consider a typical "plan-and-execute" agent: Common anti-pattern in agent frameworks for subtask in plan.subtasks: result = await execute agent subtask Unbounded fan-out results.append result A single user request can fan out to 20–50 parallel LLM calls, each with its own context window, API latency, and error surface. Without concurrency limits, you're hitting rate limits, blowing your token budget, and degrading response quality for all concurrent users. The goroutine leak becomes a token leak and a latency cascade . The parallel solution is identical: semaphore-based bounded concurrency, context propagation, and structured cleanup: python from asyncio import Semaphore async def run plan plan: Plan, max concurrent: int = 10 - list Result : sem = Semaphore max concurrent async def bounded execute subtask: Subtask - Result: async with sem: return await execute agent subtask tasks = bounded execute t for t in plan.subtasks results, = await asyncio.wait tasks, timeout=plan.timeout return r.result for r in results The lesson scales: whether it's goroutines, HTTP connections, or LLM invocations, unconstrained parallelism is a design smell. The Go community learned this the hard way at scale. AI engineers are learning it now—often before the hard part. Go's context package is a masterclass in cooperative cancellation. When a parent context is cancelled, all descendants must observe that signal and stop work: func deepProcess ctx context.Context, work Work error { // Every internal call must respect ctx step1Data, err := stepOne ctx, work.Input if err = nil { return fmt.Errorf "step one failed: %w", err } childCtx, cancel := context.WithCancel ctx defer cancel // Always clean up go func { select { case <-childCtx.Done : return default: step2Data := stepTwo childCtx, step1Data writeResult step2Data } } return nil } The critical invariant: every goroutine in the call tree inherits a context that can be cancelled. If any link in the chain drops the context and spawns an uncontrolled goroutine, you have a leak. The Go runtime doesn't enforce this—you do. n AI agent pipelines face the same cancellation problem, but it's harder to see. When a user cancels a long-running agent, you need to propagate that signal through: python from dataclasses import dataclass from typing import Optional import asyncio @dataclass class AgentContext: cancelled: asyncio.Event budget: TokenBudget steps: list StepResult async def agent step ctx: AgentContext, plan: AgentPlan, llm client: LLMClient, - AgentResult: """Each step must observe cancellation and budget.""" for subtask in plan.sequence: Check cancellation before each step if ctx.cancelled.is set : return AgentResult status="cancelled", steps=ctx.steps Check token budget cost = await llm client.estimate cost subtask.prompt if not ctx.budget.can spend cost : return AgentResult status="budget exceeded", steps=ctx.steps result = await llm client.generate prompt=subtask.prompt, cancel event=ctx.cancelled, Pass cancellation down ctx.steps.append result ctx.budget.spend result.usage.tokens return AgentResult status="completed", steps=ctx.steps The insight from the Go community: cancellation isn't an error handling concern—it's a correctness concern. An agent that continues producing results after the user has moved on isn't just wasteful; it's actively harmful if those results feed back into state that another request reads. Go's guiding principle for concurrency is well-known: "Do not communicate by sharing memory; share memory by communicating." Channels enforce ordering, prevent races, and make the control flow explicit. A goroutine that receives on a channel will block until data arrives or the channel closes—it doesn't spin, poll, or guess. // Clean pipeline: producer → processor → consumer func pipeline ctx context.Context error { tasks := make chan Task results := make chan Result go producer ctx, tasks go processor ctx, tasks, results go consumer ctx, results <-ctx.Done return ctx.Err } Each stage communicates through typed channels. There are no shared mutable variables between stages. The lifecycle of each goroutine is tied to a channel close or a context cancellation. The new wave of production agent frameworks—CrewAI, AutoGen, LangGraph, custom builds—are converging on the same structured patterns. Instead of arbitrary function calls between agents, they use tool graphs where each node's inputs and outputs are explicitly declared: python from langgraph.graph import StateGraph, END Define the graph structure explicitly workflow = StateGraph AgentState Each node is a named function with typed inputs/outputs workflow.add node "researcher", researcher agent workflow.add node "analyst", analyst agent workflow.add node "writer", writer agent workflow.add node "reviewer", reviewer agent Edges define the control flow—no mystery workflow.add conditional edges "researcher", should analyze, conditional routing {"analyze": "analyst", "done": END} workflow.add edge "analyst", "writer" workflow.add conditional edges "writer", needs review, {"review": "reviewer", "done": END} app = workflow.compile This is the agent equivalent of Go channels: data flows through explicit conduits, and control is visible in the graph structure, not hidden in side effects. The difference is that Go channels are synchronous by default or explicitly made async , while agent graphs often mix synchronous and asynchronous I/O in ways that require careful coordination. At 1M goroutines, the team didn't hit an OOM error from goroutine memory alone each goroutine starts at 2KB . They hit it from stack growth under load combined with buffered channels that accumulated unprocessed messages. The runtime could schedule them, but the heap couldn't keep up with allocation velocity. The real fix wasn't reducing goroutine count—it was reducing the work each goroutine did and ensuring every goroutine had a bounded scope: // Bounded work per goroutine: one task, done, exit func worker ctx context.Context, tasks <-chan Task, results chan<- Result { for { select { case task, ok := <-tasks: if ok { return // Channel closed, exit gracefully } result := doOneTask ctx, task select { case results <- result: case <-ctx.Done : return } case <-ctx.Done : return } } } Each worker has a clear lifecycle: enter, process one item, exit. No accumulation. No state carrying over between calls. AI agents face analogous resource constraints—token budgets, API rate limits, and reasoning depth budgets how many steps before you decide you're stuck : python class AgentBudget: def init self, max tokens: int, max steps: int, max time seconds: float : self.max tokens = max tokens self.max steps = max steps self.max time = max time seconds self.tokens spent = 0 self.steps taken = 0 self.start time = time.monotonic def check self - BudgetStatus: elapsed = time.monotonic - self.start time if elapsed self.max time: return BudgetStatus.TIMEOUT if self.steps taken = self.max steps: return BudgetStatus.MAX STEPS if self.tokens spent = self.max tokens: return BudgetStatus.TOKEN EXHAUSTED return BudgetStatus.OK def spend self, tokens: int - bool: self.tokens spent += tokens self.steps taken += 1 return self.check == BudgetStatus.OK Every agent step checks the budget before proceeding. This isn't paranoia—it's the difference between an agent that produces a useful answer in 8 steps and one that loops 47 times burning $12 in API costs. The parallel is direct: goroutine budgeting prevented scheduler thrash; token/step budgeting prevents reasoning loops. Both are about constraining per-unit resource consumption, not just total concurrency. One of Go's greatest gifts to production engineering is profiling . The standard library includes net/http/pprof out of the box, giving you goroutine dumps, heap profiles, blocking profiles, and mutex contention data with a single import: python import "net/http/pprof" // That's it. Now /debug/pprof/goroutine gives you // a full snapshot of every goroutine, its stack trace, // and what it's waiting on. http.ListenAndServe ":6060", nil At 1M goroutines, the team used blocking profiles to discover that 30% of goroutines were blocked on channel sends to a single congested output channel. The fix was fan-out: replacing one shared channel with N per-worker channels that a single aggregator goroutine collected. AI agent systems need equivalent visibility, but the profiling surface is different: | Go Concern | Agent Equivilent | |---|---| | Goroutine count | Active agent invocations | | Block profile | Waiting-on-LLM profile | | Heap profile | Token cost accumulation | | Mutex contention | Tool call serialization bottlenecks | | Stack trace | Agent step trace with LLM prompts/responses | python Example: OpenTelemetry tracing for an agent pipeline from opentelemetry import trace from opentelemetry.trace import Status, StatusCode tracer = trace.get tracer "agent.pipeline" async def run agent user request: str - AgentOutput: with tracer.start as current span "agent.execution" as span: span.set attribute "user.id", user request.metadata.user id span.set attribute "agent.type", "research-agent" Trace each step plan = await generate plan user request span.set attribute "plan.steps", len plan.subtasks results = for i, subtask in enumerate plan.subtasks : with tracer.start as current span f"agent.step.{i}" as step span: step span.set attribute "step.index", i step span.set attribute "step.type", subtask.type result = await execute with budget subtask, budget if result.error: step span.set status Status StatusCode.ERROR step span.record exception result.error results.append result return AgentOutput results=results, total tokens=sum r.tokens for r in results With this trace data, you can answer: which step is consuming the most tokens? Which is blocked waiting for LLM responses? How many agent invocations are in-flight concurrently? These are the agent-equivalent of "how many goroutines are blocked on which channel." When the 1M-goroutine system encountered failures, the engineers learned to fail closed: if one worker failed, the whole pipeline shouldn't silently continue producing garbage results. Instead, the pipeline should either complete with partial results or fail fast with a clear error. func processAll ctx context.Context, tasks Task Result, error { type taskResult struct { index int result Result err error } resultsCh := make chan taskResult, len tasks var wg sync.WaitGroup for i, task := range tasks { wg.Add 1 go func idx int, t Task { defer wg.Done result, err := doWork ctx, t resultsCh <- taskResult{idx, result, err} } i, task } go func { wg.Wait close resultsCh } outputs := make Result, len tasks var firstErr error for r := range resultsCh { if r.err = nil && firstErr == nil { firstErr = fmt.Errorf "task %d failed: %w", r.index, r.err } outputs r.index = r.result } if firstErr = nil { return nil, firstErr // Fail closed: no partial results on error } return outputs, nil } AI agents need equally principled failure handling. The stakes are different—agents don't crash your program, but they can generate plausible-sounding but incorrect outputs hallucinations or waste resources on unproductive loops. Common degradation patterns: class AgentFallback: """Strategies for when an agent pipeline fails.""" @classmethod async def handle step failure cls, step: Step, error: Exception - StepResult: match error: case LLMRateLimitError : Exponential backoff, don't fail the whole pipeline return await cls.retry with backoff step, error case LLMTimeoutError : Use a cheaper/faster model as fallback return await cls.fallback to quick model step case BudgetExceededError : Return partial results collected so far return StepResult status="partial", output=cls.summarize partial step.context , warning="token budget exhausted mid-pipeline" case HallucinationDetectionError : Re-prompt with stronger constraints return await cls.retry with constraints step, strict=True case MaxRetriesExceeded : Escalate: return an error the user can act on return StepResult status="failed", error=AgentPipelineError f"step {step.id} failed after retries" , recovery hint="try simplifying the prompt or reducing subtask complexity" The key principle, learned from systems that ran at massive concurrency: failure modes should be observable, bounded, and recoverable. A goroutine that panics without recovery brings down the process. An agent that hallucinates without guardrails brings down trust in the system. Running a million concurrent units of work doesn't make you a concurrency expert. It makes you learn the hard way that: The same principles apply to agents, which are themselves concurrent units of work that happen to call LLMs: | Principle | Goroutine Era | Agent Era | |---|---|---| | Bounded concurrency | Semaphore per service | Token/step budget per agent | | Cancellation propagation | context.Context | Cancellation events through agent graph | | Structured composition | Channels + goroutines | DAGs + tool calling | | Observability | pprof + expvar | Traces + token accounting | | Failure handling | Panic recovery + graceful shutdown | Fallback chains + partial result policies | | Cost management | CPU/memory budgets | Token/time budgets | The shift isn't that the problems changed—it's that the abstraction surface grew. A goroutine is a thread of execution. An agent is a thread of execution that can reason, call tools, and invoke other agents. The concurrency principles are the same; the failure modes are just richer. Q: Are AI agents really comparable to goroutines, or is the analogy overstated? A: The analogy holds at the systems level—both are lightweight concurrent units whose lifecycle must be managed. The difference is that agents have semantic state and can make decisions, while goroutines don't. But the resource management problems cancellation, bounding, observability, failure handling are structurally identical. Treating agents as "just another concurrent thing" rather than something qualitatively different is often the right engineering instinct. Q: Do I need to implement all of these patterns from scratch for my agent system? A: No. Modern agent frameworks like LangGraph, CrewAI, and AutoGen already encode many of these patterns. The value is in understanding why they exist—so when a framework doesn't cover your edge case and it won't , you know which principle to reach for. For example, if your framework doesn't support step-level token budgeting, you add it using the AgentBudget pattern above. Q: What's the single most important lesson from the 1M-goroutine experience for agent builders? A: Measure before you optimize, and bound before you scale. The teams that survived 1M goroutines didn't start with 1M goroutines—they started with 1,000, measured everything, and added concurrency only where the data showed it helped. The same applies to agents: build with bounded concurrency and rich observability from day one. Don't assume you'll remember to add budgets and traces later. For more on building production-grade AI systems, explore the engineering insights at tamiz.pro https://tamiz.pro , which covers agent architectures, concurrency patterns, and real-world lessons from shipping AI systems at scale.