{"slug": "solid-queue-1-6-0-fiber-workers-how-rails-turned-concurrency-primitives-into-job", "title": "Solid Queue 1.6.0 Fiber Workers: How Rails Turned Concurrency Primitives into Agent Job Orchestration", "summary": "Solid Queue 1.6.0 introduces fiber-based workers that replace the thread-per-job model with cooperative multitasking built on the Async gem, letting a single OS thread run hundreds of concurrent jobs with kilobyte-sized stacks instead of megabyte-sized ones. The release makes database queries fiber-aware so waiting fibers yield rather than block, and adds configuration options such as mode: fiber and max_fibers to cap concurrency per worker process. The trade-offs are documented as new failure modes, including fiber leaks, connection starvation, and blocking I/O from gems that bypass Active Record.", "body_md": "Solid Queue 1.6.0 ships with fiber-based workers, replacing the traditional thread-per-job model with cooperative multitasking. This is not a cosmetic change. Fibers let you run hundreds of concurrent jobs in a single OS thread, cutting memory overhead and context-switching costs. For agent workflows that spawn many lightweight tasks (API calls, LLM token streaming, webhook dispatches), this changes the economics of job orchestration.\n\nRails already had Active Job and Solid Queue for database-backed job queues. The new fiber mode integrates the Async gem, which provides non-blocking I/O primitives. Instead of blocking a thread while waiting for HTTP responses or database queries, a fiber yields control back to the scheduler. The scheduler picks another ready fiber, keeping the CPU busy without spawning new threads.\n\nAgent systems often need to coordinate many small tasks: fetching context from vector stores, calling LLM APIs, updating state in a database, sending notifications. Each task involves I/O waits. Traditional thread pools handle this by allocating one thread per job. If you have 100 concurrent jobs, you need 100 threads. Each thread consumes 1-2 MB of stack memory, and the OS scheduler juggles them with preemptive context switches.\n\nFibers flip the model. You can run 1,000 fibers on a handful of threads. Each fiber has a small stack (kilobytes, not megabytes). Context switches happen cooperatively when a fiber explicitly yields, not when the OS preempts it. This reduces memory pressure and eliminates the overhead of kernel-level thread scheduling.\n\nThe trade-off: fibers only work if your I/O operations are non-blocking. If a fiber calls a blocking system call, it stalls the entire thread. Solid Queue 1.6.0 solves this by integrating Async, which wraps network and file I/O in non-blocking primitives.\n\nSolid Queue's fiber mode runs on top of the Async gem. Here's the execution flow:\n\n`perform` method. When it hits I/O (HTTP request, database query), it yields control.\nThe supervisor process itself runs in a fiber. This means polling the job queue is non-blocking. If no jobs are ready, the supervisor fiber yields, and other fibers (like health checks or metrics exporters) can run.\n\nYou enable fiber mode in `config/queue.yml`:\n\n```\nproduction:\n  dispatchers:\n    - polling_interval: 1\n      batch_size: 50\n  workers:\n    - queues: default\n      threads: 4\n      processes: 2\n      mode: fiber\n      max_fibers: 100\n```\n\nKey parameters:\n\n`mode: fiber` activates the fiber scheduler.`max_fibers` caps the number of concurrent fibers per worker process.`threads` still matters: each thread runs its own fiber scheduler. More threads mean more parallel I/O operations.\nThis is where fiber concurrency gets tricky. Active Record's connection pool expects one connection per thread. If you have 4 threads and 100 fibers, you still only have 4 database connections. If 10 fibers try to query the database simultaneously on the same thread, 6 of them will block waiting for a connection.\n\nSolid Queue handles this by making database queries fiber-aware. When a fiber requests a connection and none are available, it yields. The scheduler moves to another fiber. When a connection is released, the waiting fiber resumes.\n\nThis works because Async patches Active Record's connection adapter to use non-blocking I/O. If you use raw SQL or a gem that bypasses Active Record, you lose this benefit. The fiber will block, stalling the thread.\n\nFiber-based concurrency introduces new failure modes:\n\n| Failure Mode | Cause | Mitigation | \n|---|---|---|\n| **Fiber leak** | Job never yields or completes | Set `max_execution_time` per job class | \n| **Connection starvation** | Too many fibers, too few DB connections | Increase connection pool size or reduce `max_fibers` | \n| **Blocking I/O** | Gem uses blocking system calls | Audit dependencies, use Async-compatible libraries | \n| **Memory bloat** | Fibers accumulate state | Explicitly clear large objects after use | \n\nBackpressure is critical. If jobs arrive faster than fibers can process them, the queue grows unbounded. Solid Queue's `max_fibers` setting acts as a concurrency limiter. Once you hit the cap, new jobs wait in the database until a fiber finishes.\n\nFor agent workflows, this means you need to tune `max_fibers` based on your bottleneck. If your agents spend most of their time waiting for LLM API responses, you can run many fibers. If they do CPU-heavy parsing, fewer fibers per thread makes sense.\n\nAgents often maintain state across multiple job executions: conversation history, tool call results, partial outputs. In a thread-per-job model, you can store state in thread-local variables. In fiber mode, thread-locals are shared across all fibers on that thread. You need fiber-local storage.\n\nRuby provides `Fiber[]` for this:\n\n``` python\nclass AgentJob < ApplicationJob\n  def perform(agent_id, task)\n    Fiber[:agent_context] = load_context(agent_id)\n\n    result = execute_task(task)\n\n    save_context(agent_id, Fiber[:agent_context])\n    result\n  ensure\n    Fiber[:agent_context] = nil\n  end\nend\n```\n\nFiber-local variables are isolated per fiber. When the scheduler switches fibers, each fiber sees its own `Fiber[:agent_context]`.\n\nThe risk: if you forget to clear fiber-locals, they persist until the fiber is garbage-collected. In a long-running worker process with a fiber pool, this leaks memory.\n\nTraditional thread-based observability tools break in fiber mode. Thread IDs no longer map one-to-one with job executions. You need fiber-aware instrumentation.\n\nSolid Queue emits Active Support notifications for job lifecycle events:\n\n`enqueue.active_job`` perform_start.active_job``perform.active_job`\nThese events include job metadata but not fiber IDs. To trace fiber execution, you need to inject fiber context into your logs:\n\n```\nclass AgentJob < ApplicationJob\n  around_perform do |job, block|\n    fiber_id = Fiber.current.object_id\n\n    Rails.logger.tagged(\"fiber:#{fiber_id}\") do\n      block.call\n    end\n  end\nend\n```\n\nThis tags all log lines from a fiber with its object ID. You can correlate logs across yields and resumes.\n\nFor distributed tracing (OpenTelemetry, Datadog), you need to propagate trace context through fiber yields. The Async gem provides hooks for this, but you must configure them explicitly.\n\nFiber mode shines when your jobs are I/O-bound and you need high concurrency. Typical agent workloads that benefit:\n\nFiber mode is a poor fit when:\n\n`IO.read` or `Net::HTTP` without Async wrappers.\nSolid Queue runs as a standalone process, separate from your web server. In production, you typically deploy:\n\nFor agent systems, you might split workers by job type:\n\nThis prevents slow jobs from starving fast ones.\n\nUse Solid Queue with fiber workers when you need to orchestrate hundreds of lightweight, I/O-bound agent tasks on a single Rails instance. The memory savings and reduced context-switching overhead make it viable to run high-concurrency workloads without scaling horizontally.\n\nAvoid fiber mode if your jobs are CPU-intensive, your dependencies use blocking I/O, or you need strict thread isolation. The cooperative multitasking model requires careful auditing of your job code and its dependencies. One blocking call can stall an entire thread's worth of fibers.\n\nFor agent workflows that mix LLM API calls, database queries, and webhook dispatches, fiber workers are a practical upgrade. You get better resource utilization without rewriting your job logic. Just watch your connection pool size and tune `max_fibers` based on your actual I/O wait times.", "url": "https://wpnews.pro/news/solid-queue-1-6-0-fiber-workers-how-rails-turned-concurrency-primitives-into-job", "canonical_source": "https://dev.to/mech_app_ai/solid-queue-160-fiber-workers-how-rails-turned-concurrency-primitives-into-agent-job-2dp7", "published_at": "2026-10-18 20:06:13+00:00", "updated_at": "2026-10-18 20:19:56.533502+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "ai-tools"], "entities": ["Solid Queue", "Rails", "Active Job", "Async", "Active Record"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/solid-queue-1-6-0-fiber-workers-how-rails-turned-concurrency-primitives-into-job", "markdown": "https://wpnews.pro/news/solid-queue-1-6-0-fiber-workers-how-rails-turned-concurrency-primitives-into-job.md", "text": "https://wpnews.pro/news/solid-queue-1-6-0-fiber-workers-how-rails-turned-concurrency-primitives-into-job.txt", "jsonld": "https://wpnews.pro/news/solid-queue-1-6-0-fiber-workers-how-rails-turned-concurrency-primitives-into-job.jsonld"}}