Durable Execution with DBOS and CockroachDB

Cockroach Labs partner solutions architect Amine El Kouhen explains how DBOS embeds durable execution directly into applications as a library, using CockroachDB to persist workflow state and recover from failures without additional infrastructure. The approach eliminates the need for separate orchestration clusters, reducing complexity and cost for long-running AI agent workflows.

This article originally appeared on the personal blog of Amine El Kouhen. Amine is Senior Partner Solutions Architect for Cockroach Labs. Modern AI applications are no longer single-shot inference calls. They are long-running agents https://www.cockroachlabs.com/blog/agentic-ai-database-architecture/ that plan, act, observe, and retry across time. An AI agent loop that retrieves context from a , calls an LLM, writes results to a database, waits for human approval, and then triggers downstream actions can run for minutes, hours, or even days. https://www.cockroachlabs.com/glossary/distributed-db/vector-database/ vector store As these workflows become longer-lived and more autonomous, infrastructure reliability becomes a business concern as much as a technical one. Restarting an interrupted workflow can increase cloud costs https://www.cockroachlabs.com/blog/agentic-ai-costs-at-scale/ , delay downstream processes, and produce duplicate actions that require manual cleanup. Durable execution helps organizations keep AI agent workflows progressing safely without sacrificing operational efficiency. Without a durable orchestration layer , any transient infrastructure failure https://www.cockroachlabs.com/blog/surviving-application-database-failures/ restarts the entire loop from scratch: re-billing expensive LLM calls, duplicating side effects, and losing all accumulated context. Platforms like Temporal https://temporal.io/ solve this by deploying a dedicated orchestration cluster a separate server process with its own persistence backend that your application workers connect to over gRPC. This is powerful, but it means an extra service to provision, monitor, scale, and keep available before your first workflow can run. How DBOS embeds durable execution into your application Traditional workflow orchestration platforms often introduce an additional operational layer that teams must deploy, monitor, secure, and scale alongside their applications. For organizations building AI systems, reducing infrastructure complexity https://www.cockroachlabs.com/blog/database-for-ai-applications/ can be just as valuable as improving workflow reliability. DBOS https://dbos.dev/ takes a fundamentally different approach: it embeds durable execution directly into your application as a Python or TypeScript library, using the database you already have. There is no orchestration server, no task queue, no sidecar process. Your application writes workflow state to tables in its own database as a natural side effect of execution, and recovers from those tables on restart. Pair DBOS with CockroachDB https://www.cockroachlabs.com/product/overview/ and you get a globally distributed, self-healing execution platform with no additional infrastructure to manage. A durable workflow is a function whose execution state which steps have completed, what they returned, what inputs were given is persisted to the database after every step. If the process crashes mid-run, it restarts and replays from the last committed step: no work is lost, no step is re-executed, no external side effect is duplicated. Built for AI-driven scale Unify operational data, vector search, and durable agent state in one resilient, distributed SQL database. Start with $400 in free credits. Trusted by Fortune 50 financial institutions and teams in 40+ countries. What Is DBOS? DBOS is a Python and TypeScript library that decorates ordinary functions with durable execution guarantees. There is no server to deploy, no task queue to operate, or separate persistence cluster to manage. DBOS writes workflow state to tables in your application database as a side effect of normal execution, and recovers from those tables on restart. Core Concepts Architecture DBOS is implemented entirely as an open-source library embedded in your application: There is no orchestration server and no external dependencies except a PostgreSQL-compatible database. While your application runs, DBOS checkpoints workflow and step state to that database. On failure, it uses those checkpoints to resume each workflow from the last completed step. DBOS architecture: the durable execution library lives inside your application process; the only external dependency is a Postgres-compatible database How does DBOS checkpoint workflow execution? Every workflow execution produces a fixed number of database writes regardless of complexity. Understanding this write pattern helps explain both DBOS's recovery model and the benchmark results discussed later. Specifically, each workflow generates: One write at workflow start : inputs are persisted before any step runs One write per completed step : the step’s return value is stored so replay can skip it One write at workflow end : the final status is committed Write sizes are proportional to your inputs and outputs. For large payloads files, embeddings , the recommended pattern is to store them externally e.g., S3 and have steps return pointers only. How does DBOS scale across multiple servers? DBOS scales naturally to a fleet of servers. All application servers connect to the same system database , the only coordination point. By default, each workflow runs on a single server; durable queues distribute work across the fleet with configurable rate and concurrency limits. For multi-application setups e.g., an API server, a data-ingestion service, and an AI agent loop , each application connects to its own isolated system database. A single physical database host can serve multiple system databases. The DBOS Client lets external code enqueue jobs and monitor results across application boundaries. How does DBOS recover interrupted workflows? When a process crashes, DBOS detects incomplete workflows and replays them in three steps: Detection – at startup, DBOS scans for pending workflows. In distributed deployments,coordinates detection across the fleet. Conductor Restart – each interrupted workflow is called again with its original checkpointed inputs. Resume – as the workflow re-executes, every step whose output is already checkpointed is skipped instantly. Execution resumes from the first un-checkpointed step. To resume execution safely without repeating completed work, DBOS relies on two principles : Determinism – the workflow function must produce the same steps in the same order given the same inputs. Non-deterministic operations DB access, API calls, random numbers, timestamps must live inside @DBOS.step decorators, never directly in the workflow body. Idempotency – steps may be retried on recovery and must be safe to re-execute. For engineering teams, durable recovery reduces the operational burden of managing long-running workflows. Instead of building custom retry logic, recovery scripts, or compensating transactions for every failure scenario, teams can rely on workflow state that survives process restarts and infrastructure interruptions. What is DBOS Conductor? For production deployments, DBOS recommends connecting to Conductor , a management service that adds distributed recovery coordination, workflow dashboards, and queue observability. Conductor is architecturally off the critical path: Each server opens an outbound websocket connection to it, and if the connection drops the application continues operating normally. Conductor has no direct access to your database and is never involved in workflow execution itself. Conductor is out of band: application servers open outbound websocket connections to it for observability and recovery, never for workflow execution Why use CockroachDB for DBOS? DBOS uses the PostgreSQL wire protocol https://www.cockroachlabs.com/glossary/distributed-db/postgresql-wire-protocol/ , so it connects to CockroachDB directly without any driver changes. What CockroachDB adds over a single-node PostgreSQL is the persistence tier you always wanted, but couldn’t justify operating separately. For long-running AI workflows https://www.cockroachlabs.com/docs/stable/cockroachdb-and-ai , the database becomes more than a place to store application data; it also becomes the system of record for workflow progress. As workflow volume grows, organizations need that execution state to remain available during node, zone, or regional failures without introducing another distributed system to manage. CockroachDB provides those capabilities through these architectural features: – concurrent workflow executions never produce lost updates or phantom reads Serializable isolation Multi-region active-active replication – workflow state is durable across data-center failures without manual intervention– the system database scales with your application without re-sharding Horizontal scalability – CockroachDB node failures are transparent to DBOS, which simply retries on the next available node Automatic failover When DBOS connects to CockroachDB, it provides three categories of tables in the system database. Together, these tables persist workflow progress, completed steps, and application events that enable durable execution and recovery. They include: Tables created by DBOS in CockroachDB: workflow state, step outputs, and events, all in your existing database Workflow status table – one row per execution, tracking ID, status, and function inputs Operation outputs table – one row per completed step, storing the serialized return value for replay Events table – named key-value pairs published within workflows and consumed via get event For teams that want globally resilient agentic workflows https://www.cockroachlabs.com/blog/agentic-ai-production-infrastructure/ without the complexity of a Temporal cluster, DBOS + CockroachDB is the lowest-overhead path. How to deploy DBOS on CockroachDB There are two required configuration changes when using CockroachDB instead of PostgreSQL. Prerequisites pip install "dbos otel ==2.15.0" "fastapi standard " psycopg2-binary sqlalchemy-cockroachdb export DBOS COCKROACHDB URL="postgresql://<user :<password @<crdb-host :26257/dbos system?sslmode=verify-full&sslrootcert=/certs/ca.crt" 1. Disable LISTEN/NOTIFY LISTEN/NOTIFY PostgreSQL’s LISTEN/NOTIFY mechanism is used by DBOS to wake up waiting workflows without polling. CockroachDB does not implement this mechanism, so it must be disabled explicitly; DBOS falls back to polling automatically: 2. Set the system database URL In dbos-config.yaml , point the system database at CockroachDB using the standard PostgreSQL connection string format: name: my-agent-app language: python runtimeConfig: start: - python3 app/main.py system database url: ${DBOS COCKROACHDB URL} Set the environment variable with your CockroachDB connection string: export DBOS COCKROACHDB URL="postgresql://dbos user:password@<crdb-host :26257/dbos system?sslmode=verify-full&sslrootcert=/certs/ca.crt" Build a durable AI agent workflow with DBOS and CockroachDB The following example implements a three-step durable agent workflow backed by CockroachDB. The workflow publishes progress events after each step that a frontend can poll in real time. If the process crashes mid-execution, restarting it resumes from the last completed step: no re-billing, no duplicate writes, no lost context. Install dependencies and run: pip install "dbos otel ==2.15.0" "fastapi standard " psycopg2-binary sqlalchemy-cockroachdb export DBOS COCKROACHDB URL="postgresql://dbos user:pass@localhost:26257/dbos system?sslmode=disable" python3 app/main.py Benchmarking Durable execution introduces additional coordination because workflow progress must be safely recorded after every completed step. The important architectural question isn’t simply how quickly a database can write rows, but whether it can continue supporting growing numbers of durable workflows while preserving correctness, availability, and operational simplicity. The following benchmarks compare PostgreSQL and CockroachDB under identical conditions to measure both raw database throughput and end-to-end DBOS workflow execution. Test environment All benchmarks ran from an EC2 instance co-located in AWS us-east-1 , eliminating WAN overhead: PostgreSQL RDS 17 : db.m7i.24xlarge , 96 vCPU, 384 GB RAM, gp3 500 GiB, 16,000 IOPS, 1,000 MB/s throughput CockroachDB 3 nodes : 3× m7i.8xlarge , 96 vCPU total, nodes spread across multiple us-east-1 AZs genuine zone-redundant deployment Benchmark artifacts: all scripts and raw JSON results are published in the repository under assets/bench/dbos-cockroachdb/ : raw write bench.py · bench direct.py · results raw pg.json · results raw crdb.json · results pg.json · results coloc.json Step 1: Establishing the raw database baseline The DBOS engineering team published a benchmark https://dbos.dev/blog/benchmarking-workflow-execution-scalability-on-postgres claiming 144K database writes per second on PostgreSQL db.m7i.24xlarge , 96 vCPU, 384 GB RAM . Reading the methodology carefully reveals an important distinction: that figure measures raw INSERT throughput into a simple 3-column table, not end-to-end DBOS workflow completions. The benchmark client was co-located on the same host as the database, and the test performed bare INSERT statements with autocommit no workflow orchestration, no step sequencing, no durability checkpointing . We replicated this exact methodology on both PostgreSQL and CockroachDB to establish an honest baseline before comparing real workflow performance. Raw write benchmark : 3-column table id , val , ts , single-row INSERT per operation, autocommit, concurrent writers: Raw INSERT peak: PostgreSQL 62,990 writes/s · CockroachDB 54,740 writes/s. PG is faster on raw writes; its local WAL flush ~1.9 ms p50 beats CockroachDB’s cross-AZ Raft quorum ~4–8 ms p50 . Both are far below the DBOS blog’s 144K claim, which used a higher-IOPS storage configuration co-located with the benchmark client. Our raw write numbers are lower than the DBOS blog’s 144K because their storage configuration had significantly higher provisioned IOPS and the benchmark client ran on the same host as the database zero network round-trip . Our setup EC2 client to RDS over the us-east-1 network, gp3 at 16K IOPS reflects real-world deployment conditions, not a co-located best-case. Step 2: Why don’t raw writes equal workflow throughput? A DBOS 2-step workflow is not a single INSERT . It produces four sequential, acknowledged database writes : Workflow start: inputs persisted before any step runs Step 1 output committed: return value stored for replay Step 2 output committed: return value stored for replay Workflow completion: final status updated Critically, each write must be fully acknowledged before the next step begins. This is the durability guarantee: If the process crashes after step 1, step 2 is never re-executed. The sequential commit chain means workflow latency ≈ 4 × single-write latency; throughput does not scale with raw write capacity. Raw writes vs actual DBOS workflow completions at peak throughput. The ~500× gap between raw writes and workflow throughput is not a bug; it is the cost of durable, exactly-once execution guarantees. The ratio is the overhead of durable orchestration: Every workflow completion serialises four round-trips through the database, each one waiting for acknowledgement before the next begins. Step 3: Benchmarking complete DBOS workflows While raw write performance establishes a useful baseline, complete DBOS workflows better reflect production behavior. The following results compare end-to-end workflow throughput and latency on PostgreSQL and CockroachDB under the same workload. Workflow throughput comparison Both databases plateau at ~117 wf/s. PostgreSQL peaks faster 122 wf/s at c=4 ; CockroachDB reaches its 3-node ceiling at c=32 117.5 wf/s . The bottleneck is the sequential step-commit pattern in DBOS, not the database engine . Workflow latency comparison PostgreSQL is faster at low concurrency 19 ms p50 at c=1 local WAL flush . At c=8 both databases converge to identical p50: 69 ms. Above c=32 both plateau at ~250 ms; the sequential commit pattern dominates completely. Both benchmarks run at Read Committed isolation. Why CockroachDB scales beyond a single node Both databases saturate at ~117 wf/s workflows per second under this DBOS workflow load: The bottleneck is DBOS’s sequential step-commit pattern, not the database. The difference is what happens when you need more than 117 wf/s . PostgreSQL’s ceiling is measured at 122 wf/s, a hard limit of its single-node WAL. CockroachDB surpasses that ceiling at just ~3.1 nodes and keeps scaling linearly. Each node adds ~39 wf/s. PostgreSQL’s Write-Ahead Log serialises every write through a single flush path . Once that path is saturated, no additional hardware increases write throughput; You can scale reads with replicas, but writes are bounded by one node forever. CockroachDB replaces the single WAL with a distributed Raft log : Each node flushes its own log, and writes are spread across the cluster. The throughput ceiling rises with every node you add. With just four nodes , CockroachDB ~156 wf/s projected already exceeds PostgreSQL’s measured ceiling. And it keeps scaling: 10 nodes means ~390 wf/s, with zone-redundant durability throughout. The differences can be summarized as follows: Summary Related CockroachDB vs. PostgreSQL — Full Comparison A side-by-side breakdown of every meaningful architectural and feature difference between CockroachDB and PostgreSQL, including multi-active availability, serializable isolation, horizontal scale-out, and PostgreSQL wire compatibility. Both benchmarks ran at Read Committed isolation. PostgreSQL’s advantage at low concurrency 19 ms vs 65 ms p50 at c=1 is purely the cost of Raft cross-AZ quorum; it disappears at c=8 where both databases land at identical 69 ms p50. At saturation both converge to ~250 ms. The decisive difference is what happens at scale : PostgreSQL has hit its ceiling, CockroachDB has not even started climbing. For organizations building AI applications, durable execution is ultimately an architectural capability rather than a performance benchmark. As autonomous workflows https://www.cockroachlabs.com/blog/agentic-ai-architecture-memory-control/ become longer-running and more business-critical, simplifying workflow orchestration while ensuring resilient execution can reduce operational overhead and provide a more scalable foundation for production AI systems. Related CockroachDB Webinars — Distributed SQL & AI Infrastructure On-demand webinars covering distributed SQL fundamentals, CockroachDB architecture, and AI infrastructure patterns. This is a practical on-ramp for teams exploring the distributed database concepts introduced in this article. Learn how CockroachDB enables resilient AI workflows with embedded durable execution. Speak with an expert https://www.cockroachlabs.com/contact/ . Amine El Kouhen, Ph.D. , is Sr. Partner Solutions Architect at Cockroach Labs. Amine is a Senior ISV Partnerships leader at Cockroach Labs CRL , working on technical certification, innovation, GTM, and co-sell initiatives with CRL's strategic GSI and ISV partners. With more than 20 years of experience in implementing and supporting SQL, NoSQL and NewSQL databases, he held leadership and hands-on roles that span new market entry, GTM planning, and partner development. Amine holds a Ph.D. in Computer Sciences and was a former research associate in data and software architecture at Concordia University.