{"slug": "simplify-ai-agent-orchestration-with-lakebase-postgres", "title": "Simplify AI agent orchestration with Lakebase Postgres", "summary": "CLA (CliftonLarsonAllen LLP) and Databricks built a Databricks-native agentic auditing solution using Lakebase Postgres that reduces document extraction time from hours to minutes. The orchestration layer, built entirely on Databricks services including Lakebase, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes, eliminates the need for separate infrastructure for queueing, orchestration, and observability by using Lakebase as a task queue backed by Postgres tables.", "body_md": "How CLA built a Databricks-native solution for long-running tasks, observability and cost attribution\n\nby [Li Yu](/blog/author/li-yu), [Michelle JanneyCoyle](/blog/author/michelle-janneycoyle), [Jon Cormack](/blog/author/jon-cormack), [Yarri Bryn](/blog/author/yarri-bryn), [Alec Sorensen](/blog/author/alec-sorensen) and [Darshana Nair](/blog/author/darshana-nair)\n\nTraditionally, auditing is a tedious process that often requires detailed document review and information extraction. To accelerate this process, CLA (CliftonLarsonAllen LLP), a leading professional services firm with a growing global presence, worked with Databricks Forward Deployed Engineering team to build and productionize an agentic auditing solution. Jointly, we developed a document processing application that reduces extraction time from hours to minutes with no compromise in quality. The application is built entirely on Databricks, using [Lakebase Postgres](https://docs.databricks.com/aws/en/oltp/projects/), [Databricks Apps](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/), [Lakeflow Jobs](https://docs.databricks.com/aws/en/jobs/), [MLflow](https://docs.databricks.com/aws/en/mlflow/), and [Unity Catalog Volumes](https://docs.databricks.com/aws/en/volumes/). In this blog, we focus on one key component of that system, the Lakebase-powered orchestration layer.\n\nThe orchestration layer is responsible for coordinating long-running tasks, managing retries, attributing cost, and providing real-time visibility. Lakebase and Databricks Apps, we eliminate the need for separate infrastructure for queueing, orchestration, and observability.\n\nLakebase also makes this architecture practical at scale by separating storage from compute. Unlike traditional Postgres deployments, compute can scale with demand while storage remains durable and independent. Together, these capabilities make Lakebase a practical foundation for a simpler, scalable orchestration pattern for long-running agentic workloads on Databricks.\n\nDocument parsing is a very common, high-volume agentic workload. Companies across industries need to convert large volumes of contracts, invoices, financial filings, and other documents into structured data. Running this at scale surfaces five distinct distributed-systems problems:\n\nMany organizations combine several specialized systems for orchestration and observability. Each system brings its own infrastructure, authentication, monitoring, and operational requirements, along with the work required to integrate them. For long-running, independent agentic tasks, that overhead is disproportionate to the actual scheduling complexity.\n\nThe Databricks-native solution we developed meets all of the above requirements with Lakebase as the foundation.\n\nThe full application stack consists exclusively of Databricks services:\n\nData flows between components as follows. The Web App writes PDFs to Unity Catalog Volumes and parses requests to Lakebase. The Orchestrator dequeues tasks from Lakebase and dispatches Databricks Jobs to the AI Agents layer. The AI Agents process documents, write results back to Lakebase, and call back to the Orchestrator with status updates.\n\nDue to these built-in capabilities on Databricks, we did not need to rely on external message brokers (Kafka, Redis), separate schedulers (Airflow, Temporal), or dedicated caching layers.\n\nThe task queue is backed by two Postgres tables in Lakebase. The tasks table holds one row per logical unit of work, recording the task's current status, lease information, agent assignment, parent extraction, and final result. The task_attempts table holds one row per execution attempt, capturing the Databricks Job run ID, MLflow trace ID, and per-attempt cost metadata. The parent-child relationship supports retries (a single task may have multiple attempts) and preserves attempt-level observability for cost attribution and debugging.\n\nA pair of Postgres tables on their own are not yet a task queue. Four Postgres-native patterns transform them into a robust, concurrent, crash-resilient, rate-limit-aware queue suitable for long-running agentic workloads.\n\nA basic dequeue query might select the next available task using WHERE status = 'enqueued' and LIMIT batch_size. While that query correctly identifies an enqueued task, it is not sufficient when multiple workers are dequeuing concurrently. Without row locking several workers may select the same task before the status is updated.\n\nAdding FOR UPDATE SKIP LOCKED makes the dequeue concurrency-safe. Each worker locks the row it selects, while other workers skip that row and continue to the next available task. Additionally, an ORDER BY priority DESC, created_at clause ensures that higher-priority tasks are selected first while preserving FIFO ordering within each priority level.\n\nThe complete concurrency-safe and priority-stable statement is:\n\nWorkers can terminate mid-task due to VM eviction, out-of-memory conditions, or deployment events. If terminated tasks remain marked as processing, they would be held indefinitely. The solution is to record an expiring lease at dequeue time:\n\nA periodic sweeper re-enqueues any task whose lease_expires_at has passed. Tasks held by terminated workers are recovered automatically within minutes, without requiring an external coordination service.\n\nLLM and vision model endpoints typically enforce two distinct quotas: a request-per-second cap and a tokens-per-minute (TPM) cap. A single throttling strategy rarely fits both. The orchestrator supports three modes, selected per agent through configuration.\n\n**Concurrency cap.** A MAX_CONCURRENT_TASKS parameter bounds the number of tasks the orchestrator dispatches concurrently. The cap is enforced at dequeue time by counting the current PROCESSING rows in the tasks table:\n\nIf the count is at or above the cap, no new task is dequeued. Anchoring the check on the database row count, rather than on the local executor's queue size, keeps the cap accurate across worker restarts, lease recoveries, and multi-replica deployments. This mode is well suited for endpoints constrained by request-per-second limits where each task's token usage is roughly uniform.\n\n**Token budget.** A MAX_TPM parameter bounds the projected token rate across in-flight tasks. The orchestrator estimates a task's token count, and sums the projected token rate across all PROCESSING tasks. A new task is dequeued only if the sum plus the new task's projected tokens fits within the budget.\n\n**Combined cap.** When both MAX_CONCURRENT_TASKS and MAX_TPM are configured, the orchestrator applies whichever constraint is tighter. This mode handles workloads that are concurrency-bound under one regime (many short, cheap tasks) and token-bound under another (a single very long document saturating the per-minute quota).\n\nIn all three modes, the throttling decision is made at dequeue time within the same transaction as FOR UPDATE SKIP LOCKED. A task that cannot fit within the current quota stays in the queue and is reconsidered on the next dequeue cycle - no separate scheduling state, no in-memory waiting queue, no coordination layer between worker replicas.\n\nWhen the AI Agents layer completes a task, it posts a callback to the orchestrator with the result. Callback delivery is not exactly-once: Databricks may retry, networks may interrupt, and proxies may redeliver. The callback handler is designed to be idempotent: it accepts both PROCESSING and ENQUEUED states, and treats already-terminal tasks as no-ops. Identical payloads produce identical results, eliminating the risk of double-billing or duplicate processing.\n\nThese four patterns combined yield a task queue that is correct under concurrency, durable under crashes, rate-limit-aware under load, and idempotent under retry. Real-time visibility into the running system is provided by a separate mechanism described in the next section.\n\nWhen many documents are in flight, operators need a clear view of agent performance, task status, and workload cost. They should not have to continuously poll the orchestrator or rely on a separate metrics platform. The orchestrator integrates this functionality directly into a single dashboard surfaced by the same Databricks App that runs the worker daemon.\n\nThe operator-facing dashboard surfaces a set of operational metrics that together characterize the running system. All metrics support filtering by date range, task status, and agent.\n\nState changes in the tasks table fire Postgres LISTEN/NOTIFY events. The backend maintains a single LISTEN connection and fans events out over **Server-Sent Events (SSE)** to connected dashboard clients. Browsers open an EventSource connection and receive live updates within approximately one second of each meaningful state change. The implementation requires no Redis, no WebSocket server, and no message bus.\n\nPolling is retained as a permanent fallback at a default ten-second interval. Streaming connections through cloud ingress proxies can silently drop bytes without firing client-side error events; permanent polling ensures the dashboard remains current even in such cases. A UI indicator distinguishes between live (SSE active) and polling (SSE unavailable) channels.\n\nThe dashboard's data spans three sources of differing latency characteristics: Postgres (instant), MLflow's trace API (sub-second), and warehouse queries against system billing tables (occasionally tens of seconds). The fast queries feed every refresh cycle; the slow queries execute only on user action and return optimistically with a loading state until results are available.\n\nDatabricks system billing tables are account-scoped: every job, every model call, and every other application contributes to the same system.billing.usage rows. Without scoping, an application-level \"OCR cost\" tile would aggregate usage across every model call in the workspace.\n\nThe solution is to record which Databricks Job runs the orchestrator submitted (tracked in tasks.locked_by and task_attempts.run_id) and filter the billing query to that set. A single SQL warehouse can back multiple applications, and each dashboard surfaces only its own spend.\n\nThe same query architecture composes naturally with operator-driven filters. Cost figures, along with every other dashboard metric, can be further narrowed by date range, task status, or agent, supporting questions such as \"what did failed tasks cost over the last seven days?\" or \"what was the median spend per task for agent X this month?\" without leaving the dashboard.\n\nThis makes cost figures easy to monitor, allocate and report.\n\nPostgres-as-queue patterns are well established in the data engineering community. Lakebase provides the additional operational characteristics that make the pattern viable as a production architecture on Databricks:\n\nThese capabilities eliminate the operational drag that typically motivates teams to adopt managed message brokers in place of self-hosted Postgres for task queueing.\n\nAt CLA, the orchestration pattern described here supports a production document-processing workflow that reduces extraction time from hours to minutes. The architecture uses Databricks-native services with Lakebase Postgres at the center to manage queueing, scheduling, and observability without requiring external systems. This reduces integration overhead while taking full advantage of a unified platform build to scale.\n\nIn production this pattern provides durable task management, priority control, rate-limit aware scheduling, real-time visibility, and per-task cost tracking. Together, these capabilities offer a practical way to orchestrate agentic workloads while keeping the surrounding infrastructure simple.\n\nReady to simplify AI agent orchestration on a single platform?[ Try Databricks Free Edition](https://docs.databricks.com/aws/en/getting-started/free-edition), create your first[ Lakebase Postgres](https://docs.databricks.com/aws/en/oltp/projects/) project and[ Databricks App](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/), then follow the[ MLflow Tracing 10-minute demo](https://docs.databricks.com/aws/en/mlflow3/genai/getting-started/tracing/tracing-notebook) to add end-to-end observability to your agent workflow.\n\nSubscribe to our blog and get the latest posts delivered to your inbox.", "url": "https://wpnews.pro/news/simplify-ai-agent-orchestration-with-lakebase-postgres", "canonical_source": "https://www.databricks.com/blog/simplify-ai-agent-orchestration-lakebase-postgres", "published_at": "2026-07-22 23:00:00+00:00", "updated_at": "2026-07-22 23:26:37.670190+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["CLA (CliftonLarsonAllen LLP)", "Databricks", "Lakebase Postgres", "Databricks Apps", "Lakeflow Jobs", "MLflow", "Unity Catalog Volumes"], "alternates": {"html": "https://wpnews.pro/news/simplify-ai-agent-orchestration-with-lakebase-postgres", "markdown": "https://wpnews.pro/news/simplify-ai-agent-orchestration-with-lakebase-postgres.md", "text": "https://wpnews.pro/news/simplify-ai-agent-orchestration-with-lakebase-postgres.txt", "jsonld": "https://wpnews.pro/news/simplify-ai-agent-orchestration-with-lakebase-postgres.jsonld"}}