Transcript #
Jeremy Edberg: Imagine I come to you and I say, we want to process millions of documents. We want to do it in a way that an AI will be able to learn from them and query against them. Imagine what that architecture would look like to you. You probably all have built this before, or something like it. It probably looks something like this. You're going to set up a service that's going to download the documents. You're going to put in a queue item or something. Then you're going to process the document and put it in the AI. Maybe you're going to use something like RabbitMQ to keep track of the work that needs to be done. Or maybe you're going to be a little bit smarter and use something like Kafka and take advantage of its multiple streams and its shardability and all of the other things that Kafka can give you to try to make this a little bit more scalable.
What's going to happen when it breaks? All of this stuff can break. AIs break all the time. They give bad answers, no answers, wrong answers. Your user might just ask a question and then come back three weeks later and ask another one. Your coordinator might break. Down off of the internet is fraught with peril. That breaks all the time. How are you going to deal with all of that? Our proposal is to use your database, the database that you're already storing your data in. Put it in the middle and store all of your state and your workflows there too.
Who am I? I've worked at a few places. I've worked at Netflix, Reddit, Amazon. This is Qian. She was a grad student at Stanford where she created the DBOS project and logo. She also runs the South Bay Systems Meetup. She'll be giving you all the details about how DBOS is built, and the libraries and how to build your own.
Universal Challenge #
What is the challenge that we're trying to solve here? All modern applications need workflows. When you're writing your applications, you're probably already writing workflows. You're probably already writing it with steps and series of steps. That's usually how we write programs. Failures can get expensive. The more distributed they are, the more expensive they get. The more AI you use, the more expensive it gets, because using AI is expensive and you don't want to use it more than you have to. That can get really expensive. Angry customers. Nobody wants a website that's down or a service that's down. That's just more and more prevalent these days where customers have less and less patience for slowness, for bad answers, for anything like that. There weren't a lot of good solutions to this problem until just a few years ago. This is a real-world pain. AIs are now making decisions, real decisions that affect real people, and you really don't want them to get it wrong.
We have payment processing that has to complete. Exactly once execution is something that's becoming more and more important because we really need to get it right the first time because users are getting less and less patient for mistakes. When we use AI, we really have to get it right the first time. The current solutions really offer not a lot of visibility into what's going on. When you have a workflow solution, especially if you're using a queue, it's hard to peek in and see what exactly is in the queue, what failed, what had to retry. You either have to write a whole bunch of code to store all of that separately or you have to fly blind and hope that your system will recover from it properly. Today, the systems that we're building have orchestration scattered all about. There are multiple systems that are managing where data is going.
We have microservices. We have this coordinator and that coordinator, and multiple databases, and it's flying all over the place, data moving back and forth. The most expensive part of any distributed system is moving data around. We want to do it as little as possible, and right now we have to do it a lot. Reliable AI especially requires reliable workflows. The biggest problem we have is that a lot of our existing tools are just not designed for these AI workloads. For the last 30 years, we've been designing systems to support websites that often return an answer within a second or less, or batch jobs. We have the very quick responses and the very slow things that users aren't waiting for and AI falls somewhere in between. AI answers typically take on the order of seconds, and the user is waiting, and sometimes the user comes back hours, days later. This is a system that we really haven't built a lot of tools for, or an operating method.
Journey #
Today, what we want to tell you about is why you need durable execution which I've hopefully already convinced you of. What methods are available today which I've told you about already. We're going to introduce to you the Transact open-source library, and then we're going to tell you how to build it for yourself. We're going to show you how to build a durable execution library so that you can do that for yourself and then get all of the benefits of durable execution. Then at the end, we'll tell you about some of those benefits and ways you can exploit that.
Today's Solution #
Because today, people are throwing it together, do it yourself with whatever tools are available, and it's not great and there's lots of different ways to do it. We're going to give you a better solution than the one that's available today which is typically the external orchestrator. If you are using a durable computing solution today, you're most likely using one that's a coordinator where you have to build your software to be a worker or a producer of work. You have to take in work, you have to tell the orchestrator what to do. The orchestrator farms that work out to other places, and this just adds complexity to your system. It also adds new points of failure. The irony of all of this is that generally, we're using durability to increase reliability but all we're doing is increasing the attack surface by adding these external coordinators. We're actually decreasing reliability by making new places for things to fail.
We're also adding latency to the process, because you got to go to the coordinator and then come back. We're adding infrastructure to the process. We're making it more things that you have to run. You have to rewrite your application logic. There's the overhead of all the extra API calls. You're generally locking yourself into a vendor if you're using an external coordinator, because you're either paying them to run it on their cloud or paying a lot to run it in your own infrastructure. Our premise here is that we've been doing it wrong, that we've been doing durability wrong, that workflows are data. If you treat the workflows in your system, the state of your system as data, then this opens up a whole new way of thinking and a whole new way of building things. Databases turn out to be really good at handling data. We've been building databases for 50-plus years, and they're really good at manipulating, handling, and storing data. Why don't we use them to store the data of workflow execution state? What we hope you'll learn today is the database is all that you need.
Introducing Workflows as a (DB-backed) Library #
Qian Li: Now I'm going to talk about why you want to build workflows as the database-backed library. It sounds pretty trivial but I'm going to show you that it's not trivial. As Jeremy said, you already have a database every day. Everyone is running their application servers together with a database. Everyone is already using a database to store user data, to do transactions. Why don't we reuse that database to also do orchestration? We figure out that we don't need all sorts of moving parts. You don't need a separate orchestration. You don't need a separate worker pool. You don't need separate servers to orchestrate everything. What you can do, what you really need for durability, for durable execution is a way to checkpoint your workflow execution state into the database and have a way to resume from where it left off, and your databases can already do that.
What I'm going to show you is the idea of building a workflow library on top of a database. Specifically, when you build it as a library, the interface will look like this. Your workflows and steps are just ordinary functions in your code, and you only need to surface two core methods to an external world, to your users. Basically, you want to implement register_workflow to tell your library that this is going to be an orchestration workflow. Then another function is register_step. This is a register_workflow. Then, you also want to tell the library that you're going to run those functions as steps. You can configure retries, you can configure exponential backoff and stuff like that into the steps. Once you register your workflows and step functions, then the library can do the heavy lifting things.
The Transact Library #
I'm going to talk about the general idea of building a database-backed workflow library. If you want to see a reference implementation, we have four libraries available for Python, TypeScript, Go, and Java. They're MIT licensed open-source libraries, and they're just a library built on top of Postgres. However, today's talk is mostly about the technical details behind those libraries. In the rest of the talk, I'm going to talk about how you can build those core features on top of a database and how to solve some of the challenges you're going to face. Here's the basics. There are tons of durable execution engines out there, but the core is really just two things. First, you need a way to checkpoint your workflows and steps into some datastore, like a database. You want durability there. Second, you want to execute things exactly once. You don't want to repeat your work.
Whenever there's interruptions to your workflows, you need a way to recover those interrupted workflows from the checkpoint, from where it left off. You don't want to repeat from the very beginning. It's like playing video games. When you play video games, you want to autosave your progress so that if there's a power outage, if someone pulled the plug of your computer, you don't lose all your progress. When you reboot your computer, you can still resume from your last save point, so you can continue your play from there. Durable execution is like that. The real magic is really just a database. You can pick your favorite database. We build on top of Postgres, but this idea is universal to all databases. Under the hood, so once users register the workflows and steps, instead of directly running the bare functions, you're going to run the wrapped functions. What they're going to do is that for the workflow function, it will first checkpoint the workflow's inputs into the database.
Then after each step finishes, the step wrapper will checkpoint the step's output into the database. Say if during the execution, your workflow is interrupted, say like the server crashes, OOM, out of memory, stuff like that, your server crashes, the workflows got interrupted. No problem. You still get all the information you need in the database. When you recover from failures, you can just load those checkpoints from the database. Then instead of starting from the very beginning, you can just restart execution of the workflow from the last completed step, because you can safely skip those completed steps and use the checkpoints, use the recorded results. That way you can guarantee exactly once execution for your workflows. Then concretely, how will it store in the database? They're just database tables. Here's a reference implementation we have here. Essentially, you have a workflow_status table to record the overall metadata of the workflow execution, like workflow ID, name, the status of the workflow, and the inputs and outputs of it. Then for each individual step, you can store that information to another table called step_outputs table. There's a foreign key correlation where the workflow ID in the step_output table will refer to the workflow ID in the workflow_status table. Then you can store your steps output checkpoint in that step_output table. That's quite simple.
To show that it's actually really simple, I'm going to show you some code. I will walk you through the code. This is the function of the workflow wrapper that will run the actual workflow function. There are four steps. First, you want to generate a workflow ID. Usually think of it as an idempotency key for an app server. You want to uniquely identify the workflow instance, so that when you checkpoint a workflow instance into the database, you can later look it up and find it. Second, after that, after you get the workflow ID, you want to store the workflow ID and the workflow inputs into the database and mark the workflow as pending in the workflow_status table. This way, you successfully record the starting of the workflow. Even if something happens, you can still look at a database and find the pending workflows. After that, we can safely run the workflow function.
Because it's just a library, the workflow function will be just a normal function invocation in the language you choose. Finally, after the function executes, you can capture the result of it, or if there's any errors thrown from it, we'll capture the error and then checkpoint the output back into the database and mark the finished status of the workflow. That's a workflow. Within the workflow function, you have a series of steps. How do you run a step? It's also quite simple. Don't get scared by the code block here. I'm going to walk you through it. There are also four steps. The first step is to retrieve the workflow invocation ID and its own step ID from the context, so that you can uniquely identify a step execution as well. You can safely store the information in the database and retrieve it later. Then, this is a very important step. You want to check if there's a checkpoint in the database already for this step. If so, you want to return the checkpoints because you don't want to re-duplicate your work there. Then, you just execute a step again as a normal function. Finally, you can checkpoint the output of the step. It's quite simple.
Now you have all the checkpoints. What are you going to do with it? Now we're going to implement the second core functionality, which is resume from where it left off. Again, the code has four steps. After recovery, you can easily list all the pending workflows from the database. Then, for each workflow, you can re-execute them. In order to re-execute them, you're going to retrieve the workflow input from the database again. Then, because it's just a library, when you register the workflow, you can store the workflow function pointer into a map so that during recovery, you can look up the map and find a function definition. Now you get the input and the function definition. You can just rerun the workflow with its original ID and the recorded inputs using the run_workflow function we just talked about. During the re-execution, because we have the step checkpoints and the skip checkpoints functionality, it will basically safely skip all the checkpoints of already completed steps and resume from where it left off. That's basically the basics of durable execution as a library, but it's also the basics of durable execution.
The Good, The Bad, and The Ugly of a Library-Based Workflow Engine #
I'm going to talk about the good, the bad, and the ugly of this approach. I'm going to talk about the good, the advantages of this approach. Then I'm going to talk about the bad, the fundamental limitations of this approach. Finally, the ugly, the challenges you'll likely face when you are trying to implement a library-based workflow engine. Then besides the challenges, we're also going to talk about the solutions for those challenges. Let's first look at the good part, the advantages of this approach. The one biggest thing is this library-based approach gets along with others. This is really important because when you already have a complicated stack, a complicated codebase, when you want to add durability into it, you don't want to refactor an entire app just to fit into a durable execution framework. What you'll be able to do with a library is that you can just install the library, register your workflows and steps with the library, and then invoke the registered workflows and steps.
You don't have to change much to your existing codebase. The developer experience is really nice here. It works really well with all the other frameworks you're using. If you're using FastAPI to build an API server, it works well. If you use Spring Boot in Java, it works well. If you're using any AI frameworks like LangChain or others, it will nicely fit in because a library-based approach is not opinionated. You don't have to refactor your code to do that. Also, the operational cost is very low. Once you write a code, you need to deploy it to production. When you deploy it, because it's a library, you already have a database. You don't need to change the way you manage your application clusters. You can just reuse your existing Kubernetes cluster. You can reuse your existing serverless functions, or you can just use your existing application servers to run it.
Operational-wise, the library runs entirely in-process, and there's no separate workers or separate servers to manage, since it's all in your application. This simplicity also improves speed. If you think about this distributed approach, in order to run every single step, you're going to run a bunch of round trips to the orchestrator server, to the worker server, to your application database, to the checkpoint database. There'll be a lot of overhead. If you implement such functionalities into your database, then you will be able to reduce it to two database writes per workflow, and one database write per step. In production, what you're going to do is that you're going to co-locate your application server with your database. The latency to the database will typically be within a few milliseconds, instead of seconds to an external service. To just prove the point, we actually run a comparison of our implementation against AWS Step Functions.
On the x-axis here, we basically increase the number of steps in the workflow, and then we run those workflows 1,000 times, and then measure the average latencies of those workflows. As you can see, the y-axis here is the average latency of those workflows. You can see that as the number of steps increase, the latency for AWS Step Functions will increase a lot, because every step will have hundreds of milliseconds of latency, even though the step itself is only a few milliseconds. With a library-based approach, the overhead is minimum, because they are really just a database checkpointing overhead.
Then, because we store all the information in the database, we can unlock a bunch of powerful workflow management capability, just built on top of database. The first is that your workflow management can be done all through SQL queries. You can easily write queries to list all your workflows, or search your workflows, run at a specific time period, or search your workflow with certain criteria, all through SQL queries. Then you can also mark your workflow as canceled. You can also mark it as pending and resume it again. What's most interesting is that it unlocks forks. The fork is an idea where you can restart your workflow from a specific step. A nice analogy of it is, think of Git. When you write code, sometimes you commit bugs to it, but when you want to fix your code, you want to Git branch, branch out to a separate fork.
Then you will be able to fix your code there, and then fix the code path there. The idea applies to the workflow execution too. With workflow fork, what you're going to do is that you're going to maintain parts of the history, and you're going to branch out to the new history from a specific point. This is really useful, because in production, I guarantee you that your servers will fail, some APIs will definitely fail. It's going to be fine, because you get fork. How are we going to fix bugs with fork? I said, the idea is that you can restart a workflow from a specific step. Under the hood, what you're going to do with a library is that you're going to copy those old inputs and outputs up to a specific step, just those database rows, copy it to a new workflow ID, so that you can start a new workflow instance using those old checkpoints with the new code.
If that's too abstract, I have some illustrations for that. Say this is a workflow, ID:V1, it has three steps. Unfortunately, step two has a bug that caused step three to fail. What are we going to do? We're going to fix it with a fork. We're going to fork from step two. This library will create a new workflow ID, saying, workflow ID:V1-fixed. It's going to copy the workflow inputs and step one's output from the old workflow to the new workflow. After that, I can just safely run the wrong workflow function with a new code on this new V1-fixed instance. Because of the checkpoint, it will skip the step one and then resume the execution from step two with the new code. Now we're very happy because we recover from failures. Now we wrap up the good part. I'm going to talk about the bad things, the limitations of this approach. This is one fundamental issue for this library-based approach because we want to improve developer experience. We have to build a custom library for each individual language. In our case, we implement the same functionality four times for four different languages. Why do we want to do that? Because we want to make the library really familiar for developers. When developers use the Python library, they're going to run synchronized functions. There are also async Python functions. We want to support both. When you run the Java libraries, you want it to work with dependency injection with Spring Boot. We need to carefully design the developer experience for each individual language. Luckily, the database schema and the core SQL queries can be shared across all languages. Basically, the heavy lifting part is those SQL queries and the database tables.
Those can be shared so that what you can really focus on is the developer experience for an individual language. Then the second limitation is database-bound scalability. How do you scale this library-based approach? It looks simple. Basically, the same way you scale your application servers. You can just add more replicas pointing to the same database, add more replicas to scale the low-scale processing unit. Ultimately, every workflow has two database writes, and every step has one database write. Eventually, you're going to be bottlenecked by the database write throughput. However, don't get so scared about it. Databases scale really well. In our case, we chose Postgres. The anecdote is that a lot of large companies like Figma, or Notion, or even OpenAI run on a single write primary database for many years. Large Postgres bots can easily support tens of thousands of transactions per second. Beyond that, that'll be a really good problem for you to have.
You can use database sharding. You can use distributed Postgres offerings. In fact, I already see a bunch of database vendors sitting outside of the store. There are a lot of good people solving the hard problem of scaling the database. That's why we really like the idea of database-based approach, because you know that this is a solved problem and people are solving them.
Finally, let's dive into the fundamental challenges of running this library-based approach workflow engine, and then how we solve them. The overall theme is that there's no central orchestrator, there's only a database. Then you have a lot of workers pulling that database. The whole point is that you still need to implement those orchestration logic without an orchestrator. That's a real challenge. There are two sub-challenges I'm going to cover today. The first challenge is how to implement queues. Sure, we can run workflows already, but oftentimes, we don't want to directly run it. We don't want to run it synchronously. We want to queue up hundreds and thousands of workflows and then run them in batch, in the background, or want to run those workflows with certain rate limits. How are we going to do that? Instead of adding more complexity to the system, instead of introducing another queuing system, what we realized is that you can already use the workflow_status table.
What you can do is to extend that table, add more columns, to include the queue_name, the created_at, the started_at. You can also add a priority column to it. There's a nice synergy between workflows and queues. What are the queues, really? They're basically a way to group those workflows and a way to order those workflows for execution. Your database is already very good at that. If you use the workflow_status table also as the queues table, the benefit is that durability is already built in. Whenever you enqueue a workflow, a task, into the queue, it will be recorded in the database. If something happens, if your server crashes when you go back online, you still see all the queue tasks in your database. Then another nice feature is that flow control and rate limits are really easy to implement on this table. Say if I want to order those workflows based on a priority, then I just sort it by the priority column.
If you want to implement a FIFO queue, first in, first out queue, I just sort them by the created_at column. If I wanted to implement rate limits, I can just look at how many workflows that are started within the past minute, and then I can decide whether to start more workflows or not based on the aggregated queries of that table. That's all good. The real challenge is when you have a lot of workers, because there's no central orchestrator, so this queue will be a poll-based queue. Each worker will need to run a SQL query like this, like select from workflow_status where status is ENQUEUED, and then you order by the created_at and select the top-end oldest workflows to execute. This seems to be really straightforward when you have only a few workers in your system. It becomes a problem when you have hundreds of thousands of workers pulling from the same table. Suddenly, you are going to hit the lock contention issue into the database. Say the worker A will look at a table and see, I'm going to run the oldest workflow. It will select the first workflow. Then another worker B, because there's no communication between those workers, worker B will also see the same table and it's also going to run the first workflow.
They'll basically contend on the first row, and they're going to generate a lot of lock contention that can lead to really bad performance for a database, for a system. Luckily, because we are using a database, databases already have a way to solve the lock contention problem, which is the locking clauses. If you search online how to build a queuing system on top of a database, the top result will always be, use SKIP LOCKED. What is SKIP LOCKED? This is a special locking clause in Postgres, but in other databases you have other implementations. This FOR UPDATE SKIP LOCKED clause does two things. First, it will select the workflows that are going to execute, and then it will lock them so that other workers will not be able to lock the same workflow. Then, the second part is that it will also skip already locked rows. Instead of waiting on the locked rows, it will skip those and select next and available workflows to run.
Concretely, in this diagram, what we show here is that when worker A locks the first workflow, it will just say, I'm going to lock it, I'm going to work on it. Then when worker B comes in, it will just skip the already locked first row, and then it will lock on the second row. If you have many other workers, each of them will skip the already locked rows and then work on the next available task. Those workers will select rows that are not already being locked, and then that way you would basically eliminate the lock contention in the database. That's queue.
The second challenging thing is to do decentralized cron scheduling. It's a very common pattern where you want to execute a workflow on a cron schedule. Say, I want to run this test every 15 minutes, or I want to run this deployment every hour, or every 5 p.m. of the day. Because there's no central orchestrator, you don't have a way to dispatch those work. What we can do, if you're going to implement it in the library, is that you just ask each worker to run the same cron scheduler. Then the trick is that you use the scheduled time as the unique workflow ID or the idempotency key. Because we have a unique constraint, a primary key constraint on the workflow ID, the database can guarantee that only one will successfully write to the database, so you will not have duplication issues. The real issue is that when you have hundreds of thousands of workers in your cluster, everyone will try to enqueue the same workflow simultaneously.
Say, if you want to execute something at 5 p.m., then at 5 p.m. you have a vertical line of your database load. Everyone is trying to hit in the same database. These highly contending database writes will lead to really bad performance. It can even lead to outages. What can we do? The solution is to add a random jitter to the sleep. Instead of waking up exactly at 5 p.m., we add a small jitter so that those hundreds of workers will wake up at slightly different times by a few milliseconds to a few seconds. Then the first worker that wakes up will write into the database saying, I'm going to run this workflow. Then the following workers will wake up and first check the database and see if the workflow already exists. If it exists, then it will just skip the execution. Because database reads are much cheaper than database writes, this solves the main performance bottleneck around the cron scheduling. I hope that I convinced you that the database is all you need.
The Big Picture: Why This Way and Not the Old Way? #
Jeremy Edberg: Now that you've learned how to build these durable systems using a database, let's talk about some of the ways that you can use them that will enhance your experience. One is testing. Because this is in the database, because it's in a language library and not an external service, you can use all your existing language library tools. You can use your mocking and your testing that already exists in the language of your choice, which makes integration testing much easier, unit testing much easier, because it already exists. It's already there. You've already got your database there. This makes security and compliance a lot easier and better as well. All of your state is already in your database. Nothing is leaving your infrastructure. Your security people are going to be really happy about that. They're also going to be happy about how auditable everything is, because every function call, every input, every output is there, you can easily audit what happened.
You can go back and say what happened in the past really easily. You can also do anomaly detection with simple SQL queries. You can query your database to look for inputs and outputs that are out of band, out of range, have PII. Whatever it is that you need to audit for, whatever your regulatory infrastructure is, you can put it on top of this database because it's all there already. Really the idea that we want to do is get you to shift from this idea of external orchestration to internal orchestration. Orchestration as a library. Make things simpler. Don't add a bunch of extra stuff to make things more complex and more brittle. Just use what's already there. Use the database you're already using. By taking this library approach, it's a lot easier for a developer or an AI, because both of them can understand what is happening a lot easier.
You can build a completely valid, working, distributed system with a single file with this approach because you've already got your crons, your queues, your durability. It's all right there. You don't have to think about how are you going to do all this extra stuff. You don't have to have your AI do it either because AIs can be really good at reading a single file, getting that context, and adding in new stuff to it, especially if you give it a prompt as to how your durable execution library works. AIs are really bad at YAML and Terraform and creating a distributed system that actually works. You don't have to do that if you're using this library approach where everything is inside the process already. It's a lot easier for the developer because, as we've said a couple of times, you write your normal code. You don't re-architect.
You deploy it wherever you want. You don't have to worry about external access to the coordinators, things like that. The ecosystem is easy because you've already got your language ecosystem. The Transact library that we created is already integrated into a bunch of different places, Pydantic AI. The one that you build for yourself can use all of your existing integrations, all of your existing tools. That's another huge advantage of the library approach because you can use all the things that already exist. All of your connectors, all of that. If you're going to use our library, places where it really shines is if you're already using Postgres for your data, if latency really matters because as we said, you don't need that round trip, and you don't like writing YAML, because I know I don't and I imagine most people don't either.
Then, where do you go from here if you've built this library? You can always add more observability. There are just more SQL queries. You can do performance optimizations. There's tons of research on SQL performance optimizations, and you can start implementing them here. You can create built-in queries for everybody to use. You put built-in queries and everybody using that library can use those built-in queries. You can also build beautiful dashboards that are just based on SQL queries where you can show all of the pending workflows, all of the ones that are enqueued, all the ones that finished. You can show all the steps that are being involved in the processes because they're all just coming right out of your SQL database. You're making SQL queries. You're putting a UI around it. It's really easy to observe and visualize. You get that observability that we talked about was missing from a lot of solutions with simple SQL queries.
We have a contest going across our internal staff of the leads of each project. Please go and star your favorite language to make them feel good. We're seeing who can get the most stars by the end of the year. Also, this way you can find all the libraries that we built as our reference implementations. You can build it for yourself. You can take our MIT licensed code and use it. We highly encourage you. We would love to see everybody building everything durably. We think that in the future everyone will be building everything durably because it's just a better way to build software. It's a better way to operate software. If you're a developer and you build durably, your operators will love you. If you're an operator, you're going to love your developers.
Key Takeaways #
Hopefully at this point you agree with us that workflows are data. Your database is really powerful and good at managing that data. That simplicity is really better than complexity. At the beginning I told you to build this idea of ingesting millions of documents with tens of millions of AI queries. This is an actual company that's doing this. You should check them out because what they do is they provide you a way to query the documentation of all of your favorite open-source programs using simple language queries. That includes the Transact library, actually. It's all built on top of this idea of a library-based approach.
Questions and Answers #
Participant 1: What happens if you have a workflow for which the steps are spanning several different applications who may not share a database?
Qian Li: There are different ways to do that. If you have the idempotency key, you can install libraries, point to a different database on the other side, and then you communicate to each other with the idempotency key, so you know that once you start the workflow, it will finish exactly once.
Jeremy Edberg: Or another option would be to have a service that just runs the steps and calls to other services. Then those individual services could have their own workflows built in, but the ultimate controller of that is that main service that's calling to those other services to make sure all the steps actually happen.
Participant 1: An orchestrator?
Jeremy Edberg: Yes, you basically create your own internal orchestrator by yourself. That is a solution to that problem.
Participant 2: One thing we've been noticing building agentic workflows is maybe you're writing Go over here, maybe there's some really bespoke Python library over here that you want to use. It sounds like you need a separate orchestrator if you want one step to be completed by programming in one language and another system code in another.
Qian Li: What you can do is that you build some workflows in Go, some workflows in Python, and then you can have tiered workflows, like essentially one large overall workflow is dispatching tasks. Then to execute each task, you can also decompose the sub-tasks as also sub-workflows. You can, basically in the tiered approach.
Jeremy Edberg: Or you could attempt to build a custom serializer that lets them talk, which I wouldn't recommend.
Participant 2: If they're all sharing the same Transact database, though, do all those problems go out the window and you're just using steps and workflows at that point?
Qian Li: If you have your own way of defining the serialization and deserialization, that will work. The problem for now is that, for example, in Python, you can use pickle. In Go, you use gob. Basically, those serialization formats don't transfer across different languages.
Participant 2: Protobuf.
Jeremy Edberg: If someone wants to solve that problem, cross-language serialization, there's probably a lot of money in it for you.
Participant 3: How will you handle compensation in long-running workflows? Let's say you've done something with an external system, and then the step failed, you have to rerun it. How would you suggest handling it with Transact?
Qian Li: In the workflow code, you can say, catch this error, if this happens, then do some compensating steps. Workflows as code, everything is coded in your system.
Participant 3: Let's say I ran a few steps, then in step four, that failed, I need to undo some of it in step two. That's up to the developer to handle it?
Jeremy Edberg: Right. The nice thing is, in the database, you know how you got there. You know all the steps that were taken, the inputs, the outputs, and so you know how you got there. You can use that to go backwards. Actually, there's a paper coming out pretty soon about compensating steps.
Participant 4: Have you all considered serializing some workflow steps into stored procedures to avoid the round trip?
Qian Li: Yes, we do have a prototype to compile TypeScript code into V8 engine inside the Postgres. The environment is limited. You can do limited work there. Some Postgres extensions allows you to do that. It's another topic, basically how to compile everything into a database. It will be doable, essentially, if the database allows you to execute untrusted code.
Participant 5: You mentioned that you guarantee exactly once execution for the workflows. Is it dependent on the fact that the individual functions or the steps in the workflow, they're idempotent?
Qian Li: That's the basic requirement for durable workflows. The workflows need to be deterministic so you can deterministically find all the checkpoints. Then individual steps need to be idempotent so that if you re-execute it, it will give you the same result.
Jeremy Edberg: The first thing you're going to think is AI isn't idempotent. It's not deterministic. Things that have happened in the past are by definition determined, and so they're now deterministic. Once you've run your AI query once, it's stored in the database, it's now a deterministic step that can be replayed because it's already there. That will actually save you a lot of money to not do it.
See more presentations with transcripts