{"slug": "running-ai-agents-as-background-jobs-with-solid-queue", "title": "Running AI Agents as Background Jobs with Solid Queue", "summary": "AI agents running as background jobs with Solid Queue require dedicated queues, concurrency limits, and controlled retries to prevent starving other jobs and exceeding budgets, as agent jobs are slow, costly, and non-deterministic unlike typical background jobs.", "body_md": "# Running AI Agents as Background Jobs with Solid Queue\n\nRun AI agents as background jobs with Solid Queue: queue isolation, concurrency limits, safe retries, resumable runs, and cost control for production.\n\nAn AI agent is the worst-behaved background job you will ever run. It is slow, often tens of seconds across a multi-tool loop. It costs real money every time it executes, because every step is tokens on a provider bill. And worst of all, retrying it does not replay. A re-run is a fresh roll of the dice that can take a different path and call different tools than the attempt that failed.\n\nThat combination quietly breaks assumptions baked into how most teams configure their queue. The defaults assume jobs are fast, free, and deterministic, so retrying one is harmless. None of that is true for an agent. Solid Queue has the right primitives to handle it - dedicated workers, concurrency limits, controlled retries - but only if you set them up on purpose.\n\nI have written about building the agent itself, with the [Anthropic SDK](/rails/ai-agents/architecture/2026/06/09/building-ai-agents-ruby-anthropic-sdk/) and the [Gemini Interactions API](/rails/ai-agents/architecture/2026/06/26/building-ai-agents-ruby-gemini-interactions-api/). This post is about the other half, the part nobody talks about: once you have a working agent, how do you run it as a background job without it starving your app or torching your budget. It assumes Solid Queue is already running; if not, start with the [practical guide](/rails/background-jobs/performance/2025/10/07/solid-queue-rails-practical-guide/).\n\n## How an agent job differs from a normal job\n\nAn agent run violates the three things a queue's defaults assume about a job: that it is fast, that it is free, and that it is safe to retry. Every operational decision below follows from this table, so it is worth fixing in your head before any code:\n\n| Property | Typical background job | AI agent run |\n|---|---|---|\n| Duration | Milliseconds to a few seconds | Tens of seconds across a tool loop |\n| Cost per run | Effectively free | Real money, every step is metered tokens |\n| Determinism | Same input, same path | Non-deterministic, a retry can take a new path |\n| Safe to blind-retry? | Usually yes | No, replays side effects and spend |\n| Right home | Shared `default` queue |\nIsolated queue and worker pool |\n\nRead the right column as a list of problems to solve. Isolation, throttling, narrow retries, resumability, and cost tracking each exist to neutralize one row.\n\n## Why it has to be a job\n\nYou cannot run a multi-tool agent loop inside a web request. The loop calls the model, runs a tool, calls the model again, and keeps going until it has an answer, which routinely outlives both your Puma worker's timeout and the user's patience. So the controller creates a record, kicks off a job, and returns immediately. The job owns the loop.\n\n``` python\nclass AgentRunJob < ApplicationJob\n  queue_as :agents\n\n  def perform(agent_run_id)\n    agent_run = AgentRun.find(agent_run_id)\n\n    result = AgentRunner.new(\n      client: LlmClient.build,\n      tool_registry: ToolRegistry.new(agent_run.user)\n    ).run(\n      input: agent_run.input,\n      previous_interaction_id: agent_run.previous_interaction_id\n    )\n\n    agent_run.update!(status: \"completed\", output: result[:output])\n  end\nend\n```\n\nThe loop internals - tool execution, state, parsing - are their own topic, covered in the agent-building posts. The only line that matters yet is `queue_as :agents`\n\n. That one choice is what makes everything below possible.\n\n## Isolate agent work on its own queue and workers\n\nThis is the single highest-leverage move, so do it first. Agent runs are slow, and a slow job on a shared queue blocks every fast job behind it. Picture a 90-second agent run sitting on your `default`\n\nqueue while an order-confirmation email waits in line behind it. The email is now 90 seconds late because an unrelated AI task happened to be ahead of it.\n\nGive agent work a dedicated queue and a dedicated worker pool, separate from the workers that run your fast transactional jobs:\n\n```\n# config/queue.yml\nproduction:\n  workers:\n    - queues: [real_time, default, mailers]\n      threads: 5\n      polling_interval: 0.1\n      processes: 2\n\n    - queues: [agents]\n      threads: 2\n      polling_interval: 1\n      processes: 1\n```\n\nNow a long agent run can only ever occupy a thread in the agent pool. Your transactional jobs run on their own workers and never queue behind an AI task. The two workloads stop competing for the same threads.\n\n## Throttle with worker count, not just concurrency limits\n\nYou have two different throttling needs, and they want two different tools.\n\n| Throttling need | Right tool | Example setting | Why this tool |\n|---|---|---|---|\n| Global rate-limit and cost ceiling | Worker pool size | 2 threads on the `agents` queue |\nConcurrency controls carry per-job overhead when the cap is above 1 |\n| Per-account fairness (one run at a time) | `limits_concurrency to: 1` |\nkeyed on `account_id` |\nCheap at a limit of 1; extra runs wait in `blocked_executions` |\n\nThe first is a global cap: respect the provider's rate limit, and do not let a flood of runs hammer the model API or run up the bill all at once. The instinct is to reach for `limits_concurrency`\n\nwith a high limit, but Solid Queue's own docs steer you away from that. For throttling where the limit is meaningfully larger than 1, a small worker pool is the right lever, because concurrency controls carry real per-job overhead. You already have that lever from the previous section: two threads on the `agents`\n\nqueue means at most two agent runs in flight at once, full stop. Want five? Set five threads. The worker count is your global throttle.\n\nThe second need is fairness: one user should not be able to launch fifty concurrent runs and starve everyone else, and for many workflows a single user should only have one run going at a time. That is exactly what `limits_concurrency`\n\nis for, with a limit of 1, keyed per account:\n\n```\nclass AgentRunJob < ApplicationJob\n  queue_as :agents\n\n  limits_concurrency(\n    to: 1,\n    key: ->(agent_run_id) { \"agent_run_account_#{AgentRun.find(agent_run_id).account_id}\" },\n    duration: 10.minutes\n  )\n\n  # ...\nend\n```\n\nA second run for the same account is held in `blocked_executions`\n\nand promoted to ready when the first finishes. The `duration`\n\nis a failsafe, not a timeout: if a worker dies without releasing the semaphore, the lock expires after `duration`\n\nso the account is not blocked forever. Set it comfortably above your longest expected run. If a run outlives `duration`\n\n, the semaphore is cleaned up while the job keeps going, and a second run can start and overlap, which is the exact thing you were trying to prevent.\n\nIf duplicate runs should be dropped rather than queued, like a \"refresh this summary\" button a user can mash, switch the conflict behavior to discard:\n\n``` php\nlimits_concurrency to: 1, key: ->(id) { ... }, duration: 10.minutes, on_conflict: :discard\n```\n\nUse the worker pool for the global ceiling, and `limits_concurrency`\n\nfor per-account fairness. They are not interchangeable.\n\n## Retries cost money and do not replay\n\nHere is the default that will hurt you:\n\n```\nretry_on StandardError, attempts: 5\n```\n\nOn a normal job that is sensible. On an agent it is a liability. If the run already sent an email or charged a card on attempt one and then hit an error, you do not get a clean replay. You get the side effects again, the token spend again, and a retried run that may now take a different path and make different decisions than the one that failed. Retrying a non-deterministic, side-effecting, metered process is neither free nor idempotent unless you make it so.\n\nTwo things keep this safe. First, retry narrowly. Only retry the transient, transport-level failures that genuinely benefit from a second attempt, and discard the permanent ones instead of burning five attempts on an error that will never succeed:\n\n```\nclass AgentRunJob < ApplicationJob\n  queue_as :agents\n\n  retry_on Faraday::TimeoutError, wait: :polynomially_longer, attempts: 3\n  retry_on ProviderRateLimited,   wait: :polynomially_longer, attempts: 5\n\n  discard_on AgentRun::InvalidInput\n  discard_on ActiveRecord::RecordNotFound\n\n  # ...\nend\n```\n\nSecond, make the side effects idempotent and the run resumable. Persist the interaction id and the steps you have already executed, so a retry continues from where it stopped instead of re-charging a card it already charged. The provider's `429`\n\nshould trigger a backoff, but the real throttle on how hard you hit the API is the worker count from earlier, not the retry policy. This is the same at-least-once delivery model that makes [recurring and cron jobs in Solid Queue](/rails/background-jobs/scheduling/2026/06/29/solid-queue-recurring-cron-jobs-guide/) demand idempotent bodies; an agent just raises the stakes because a double-run costs money.\n\n## Deploys will interrupt long runs\n\nLong jobs collide with deploys, and agents are long jobs. When Solid Queue shuts a worker down, in-flight jobs get a grace period - `shutdown_timeout`\n\n, around 25 seconds by default - to finish before they are terminated. A two-minute agent run does not finish in 25 seconds, so a deploy in the middle of it kills the run partway through the loop.\n\nYou can raise `shutdown_timeout`\n\nto give runs more room, within reason, but the durable fix is the resumability you already built for retries. If a run persists its interaction id and completed steps as it goes, an interrupted run is just another run to continue: re-enqueue it, pass the stored `previous_interaction_id`\n\n, and it picks up rather than starting over. Treat interruption as normal, not exceptional, because on any actively deployed app it is.\n\n## Human-in-the-loop without holding a thread hostage\n\nGood agents pause before doing something irreversible. The write-tool rule - draft the action, let a human confirm, then execute - is covered in the agent-building posts. The question here is how to model that pause as a job, and the wrong answer is tempting: have the job sleep, polling for confirmation.\n\nDo not do that. A sleeping job holds a worker thread for as long as the human takes, which could be minutes or hours, doing nothing but occupying a slot in your small agent pool. And the deploy from the last section will kill it anyway.\n\nThe Solid-Queue-native answer is to end the job and start a new one when the human acts. When the agent reaches a step that needs confirmation, persist the proposed action and the interaction id, move the run to a `waiting_for_confirmation`\n\nstate, and return. The job is done. No thread is held.\n\n``` python\ndef perform(agent_run_id)\n  agent_run = AgentRun.find(agent_run_id)\n\n  result = AgentRunner.new(...).run(\n    input: agent_run.input,\n    previous_interaction_id: agent_run.previous_interaction_id\n  )\n\n  if result[:needs_confirmation]\n    agent_run.update!(\n      status: \"waiting_for_confirmation\",\n      previous_interaction_id: result[:interaction_id],\n      pending_action: result[:pending_action]\n    )\n    return\n  end\n\n  agent_run.update!(status: \"completed\", output: result[:output])\nend\n```\n\nWhen the user confirms in the UI, the controller enqueues a continuation that resumes from the stored interaction id:\n\n``` python\nclass ResumeAgentRunJob < ApplicationJob\n  queue_as :agents\n\n  def perform(agent_run_id)\n    agent_run = AgentRun.find(agent_run_id)\n    return unless agent_run.status == \"waiting_for_confirmation\"\n\n    result = AgentRunner.new(...).run(\n      input: \"User confirmed the pending action.\",\n      previous_interaction_id: agent_run.previous_interaction_id\n    )\n\n    agent_run.update!(status: \"completed\", output: result[:output])\n  end\nend\n```\n\nThe pause now costs zero worker time and survives any number of deploys, because between confirmation steps there is no running job at all, only a row in the database waiting for a human.\n\n## Persist runs and track cost\n\nThere are two reasons to write every run and its token usage to your own tables, beyond whatever the provider stores. The first is debugging: an agent will eventually do something strange in production, and without the step log you are guessing about a probabilistic system. [Mission Control Jobs](/rails/background-jobs/monitoring/2026/01/14/mission-control-jobs-rails-production-monitoring/) gives you the queue-level view of which runs failed and retried, but the per-step trace has to come from your own tables. The second reason is money. Every run is a line item on your model bill, and if you cannot attribute spend to an account or a feature, you cannot see which customer or which workflow is quietly costing you. Capture the usage metadata each run returns and roll it up per account, so cost is a number you can query rather than a surprise at the end of the month.\n\n## When you do not need any of this\n\nIf your \"agent\" is a single cheap model call - classify this ticket, extract these fields, one shot, no tool loop - skip the machinery. It is fast, it is cheap, retrying it is harmless, and a normal job on `default`\n\nis completely fine. The queue isolation, concurrency limits, resumability, and cost tracking all exist to tame the slow-expensive-non-deterministic trio, and a small one-shot call simply does not have it.\n\nOne sharp edge to know before you lean hard on concurrency limits: under large spikes of enqueued jobs, Solid Queue can be slow to promote blocked executions back to ready, so do not put tight `limits_concurrency`\n\non a latency-critical path. For the per-account fairness case it is the right tool; for anything where a few seconds of delay matters, prefer the worker-count approach. If you are weighing the backend itself for this kind of workload, the [Solid Queue vs Sidekiq vs GoodJob comparison](/rails/background-jobs/architecture/2026/02/17/solid-queue-vs-sidekiq-vs-goodjob-rails/) covers where each one earns its keep.\n\nThe operational layer is the half nobody demos: the queue isolation that keeps a slow run from starving the app, the right throttle for your provider limits, retry semantics that do not double-charge a customer, runs that survive deploys, and per-account cost controls so the bill never surprises you. The agent itself is the easy half. Need a hand getting agents safe to run in production? I help Rails teams with the queue isolation, throttling, and cost controls that keep agent workloads from starving an app or blowing a budget.", "url": "https://wpnews.pro/news/running-ai-agents-as-background-jobs-with-solid-queue", "canonical_source": "https://nsinenko.com/rails/ai-agents/background-jobs/2026/06/15/solid-queue-ai-agents-background-jobs/", "published_at": "2026-06-15 07:20:00+00:00", "updated_at": "2026-06-29 16:25:50.517138+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure"], "entities": ["Solid Queue", "Anthropic", "Gemini", "Puma"], "alternates": {"html": "https://wpnews.pro/news/running-ai-agents-as-background-jobs-with-solid-queue", "markdown": "https://wpnews.pro/news/running-ai-agents-as-background-jobs-with-solid-queue.md", "text": "https://wpnews.pro/news/running-ai-agents-as-background-jobs-with-solid-queue.txt", "jsonld": "https://wpnews.pro/news/running-ai-agents-as-background-jobs-with-solid-queue.jsonld"}}