Solid Queue 1.6.0 Fiber Workers: How Rails Turned Concurrency Primitives into Agent Job Orchestration

Solid Queue 1.6.0 introduces fiber-based workers that replace the thread-per-job model with cooperative multitasking built on the Async gem, letting a single OS thread run hundreds of concurrent jobs with kilobyte-sized stacks instead of megabyte-sized ones. The release makes database queries fiber-aware so waiting fibers yield rather than block, and adds configuration options such as mode: fiber and max_fibers to cap concurrency per worker process. The trade-offs are documented as new failure modes, including fiber leaks, connection starvation, and blocking I/O from gems that bypass Active Record.

Solid Queue 1.6.0 ships with fiber-based workers, replacing the traditional thread-per-job model with cooperative multitasking. This is not a cosmetic change. Fibers let you run hundreds of concurrent jobs in a single OS thread, cutting memory overhead and context-switching costs. For agent workflows that spawn many lightweight tasks API calls, LLM token streaming, webhook dispatches , this changes the economics of job orchestration. Rails already had Active Job and Solid Queue for database-backed job queues. The new fiber mode integrates the Async gem, which provides non-blocking I/O primitives. Instead of blocking a thread while waiting for HTTP responses or database queries, a fiber yields control back to the scheduler. The scheduler picks another ready fiber, keeping the CPU busy without spawning new threads. Agent systems often need to coordinate many small tasks: fetching context from vector stores, calling LLM APIs, updating state in a database, sending notifications. Each task involves I/O waits. Traditional thread pools handle this by allocating one thread per job. If you have 100 concurrent jobs, you need 100 threads. Each thread consumes 1-2 MB of stack memory, and the OS scheduler juggles them with preemptive context switches. Fibers flip the model. You can run 1,000 fibers on a handful of threads. Each fiber has a small stack kilobytes, not megabytes . Context switches happen cooperatively when a fiber explicitly yields, not when the OS preempts it. This reduces memory pressure and eliminates the overhead of kernel-level thread scheduling. The trade-off: fibers only work if your I/O operations are non-blocking. If a fiber calls a blocking system call, it stalls the entire thread. Solid Queue 1.6.0 solves this by integrating Async, which wraps network and file I/O in non-blocking primitives. Solid Queue's fiber mode runs on top of the Async gem. Here's the execution flow: perform method. When it hits I/O HTTP request, database query , it yields control. The supervisor process itself runs in a fiber. This means polling the job queue is non-blocking. If no jobs are ready, the supervisor fiber yields, and other fibers like health checks or metrics exporters can run. You enable fiber mode in config/queue.yml : production: dispatchers: - polling interval: 1 batch size: 50 workers: - queues: default threads: 4 processes: 2 mode: fiber max fibers: 100 Key parameters: mode: fiber activates the fiber scheduler. max fibers caps the number of concurrent fibers per worker process. threads still matters: each thread runs its own fiber scheduler. More threads mean more parallel I/O operations. This is where fiber concurrency gets tricky. Active Record's connection pool expects one connection per thread. If you have 4 threads and 100 fibers, you still only have 4 database connections. If 10 fibers try to query the database simultaneously on the same thread, 6 of them will block waiting for a connection. Solid Queue handles this by making database queries fiber-aware. When a fiber requests a connection and none are available, it yields. The scheduler moves to another fiber. When a connection is released, the waiting fiber resumes. This works because Async patches Active Record's connection adapter to use non-blocking I/O. If you use raw SQL or a gem that bypasses Active Record, you lose this benefit. The fiber will block, stalling the thread. Fiber-based concurrency introduces new failure modes: | Failure Mode | Cause | Mitigation | |---|---|---| | Fiber leak | Job never yields or completes | Set max execution time per job class | | Connection starvation | Too many fibers, too few DB connections | Increase connection pool size or reduce max fibers | | Blocking I/O | Gem uses blocking system calls | Audit dependencies, use Async-compatible libraries | | Memory bloat | Fibers accumulate state | Explicitly clear large objects after use | Backpressure is critical. If jobs arrive faster than fibers can process them, the queue grows unbounded. Solid Queue's max fibers setting acts as a concurrency limiter. Once you hit the cap, new jobs wait in the database until a fiber finishes. For agent workflows, this means you need to tune max fibers based on your bottleneck. If your agents spend most of their time waiting for LLM API responses, you can run many fibers. If they do CPU-heavy parsing, fewer fibers per thread makes sense. Agents often maintain state across multiple job executions: conversation history, tool call results, partial outputs. In a thread-per-job model, you can store state in thread-local variables. In fiber mode, thread-locals are shared across all fibers on that thread. You need fiber-local storage. Ruby provides Fiber for this: python class AgentJob < ApplicationJob def perform agent id, task Fiber :agent context = load context agent id result = execute task task save context agent id, Fiber :agent context result ensure Fiber :agent context = nil end end Fiber-local variables are isolated per fiber. When the scheduler switches fibers, each fiber sees its own Fiber :agent context . The risk: if you forget to clear fiber-locals, they persist until the fiber is garbage-collected. In a long-running worker process with a fiber pool, this leaks memory. Traditional thread-based observability tools break in fiber mode. Thread IDs no longer map one-to-one with job executions. You need fiber-aware instrumentation. Solid Queue emits Active Support notifications for job lifecycle events: enqueue.active job perform start.active job perform.active job These events include job metadata but not fiber IDs. To trace fiber execution, you need to inject fiber context into your logs: class AgentJob < ApplicationJob around perform do |job, block| fiber id = Fiber.current.object id Rails.logger.tagged "fiber: {fiber id}" do block.call end end end This tags all log lines from a fiber with its object ID. You can correlate logs across yields and resumes. For distributed tracing OpenTelemetry, Datadog , you need to propagate trace context through fiber yields. The Async gem provides hooks for this, but you must configure them explicitly. Fiber mode shines when your jobs are I/O-bound and you need high concurrency. Typical agent workloads that benefit: Fiber mode is a poor fit when: IO.read or Net::HTTP without Async wrappers. Solid Queue runs as a standalone process, separate from your web server. In production, you typically deploy: For agent systems, you might split workers by job type: This prevents slow jobs from starving fast ones. Use Solid Queue with fiber workers when you need to orchestrate hundreds of lightweight, I/O-bound agent tasks on a single Rails instance. The memory savings and reduced context-switching overhead make it viable to run high-concurrency workloads without scaling horizontally. Avoid fiber mode if your jobs are CPU-intensive, your dependencies use blocking I/O, or you need strict thread isolation. The cooperative multitasking model requires careful auditing of your job code and its dependencies. One blocking call can stall an entire thread's worth of fibers. For agent workflows that mix LLM API calls, database queries, and webhook dispatches, fiber workers are a practical upgrade. You get better resource utilization without rewriting your job logic. Just watch your connection pool size and tune max fibers based on your actual I/O wait times.