{"slug": "durable-workflows-are-great-and-the-wrong-choice-for-short-pipelines", "title": "Durable Workflows Are Great, and the Wrong Choice for Short Pipelines", "summary": "A developer argues that durable workflows, while powerful for long-running processes, are the wrong choice for short, cheap, idempotent pipelines. The infrastructure overhead, programming model constraints, and added latency outweigh the benefits when recovery can be achieved by simply rerunning the entire pipeline.", "body_md": "With the Microsoft Agent Framework and its Durable Task extension, durable execution is once again a major topic in the .NET world. The idea is tempting: define agents and workflows in ordinary code, then let the runtime handle persistence, crash recovery, state management, distributed scaling, and observability.\n\nThat is strong technology. At the time of writing, the Durable Task extension for Microsoft Agent Framework is still in preview, but the direction is already clear: durable execution is becoming easier to apply to agentic and workflow-based systems.\n\nPrecisely because it is powerful, it will also be used in places where it does not belong.\n\nMy thesis: for short, cheap, idempotent pipelines, durable workflows are usually the wrong abstraction. Not because the framework is bad, but because the nature of the problem is different.\n\nThere is an important distinction that is easy to blur.\n\nMicrosoft Agent Framework workflows let you define a graph of executors and agents. That can be useful even without durability. A workflow graph can model a sequence, a branch, a fan-out, or a handoff.\n\nDurable Task is a different decision: it adds persistence, recovery, replay semantics, and distributed coordination.\n\nThose two things are related, but they are not the same. A workflow does not automatically need to be durable. A three-step graph can still be just a three-step graph.\n\nThat distinction matters because the cost of a workflow model and the cost of durable execution are not the same.\n\nDurable execution solves a real and difficult problem: how do you keep a workflow alive when it runs for a long time, consists of several steps, waits for external events, and may fail along the way?\n\nThe runtime records progress in durable state. If the process crashes, the workflow can be reconstructed from history and resumed. Work that has already completed does not have to be blindly executed again. In serverless hosting models, a workflow can wait for a human approval or external event without consuming compute for the entire waiting period.\n\nThat is excellent engineering for workflows that actually have these properties:\n\nThis is where durable workflows shine. The extra machinery buys you something concrete: continuity, recovery, and visibility.\n\nBut durability is not magic. It does not remove the need to design side effects carefully. If a step sends an email, charges a card, writes to an external system, or publishes a message, you still need idempotency keys, deduplication, transactional boundaries, or an outbox pattern. Durable execution helps avoid repeating completed work, but it does not make the outside world exactly-once.\n\nA short pipeline is not just a workflow that finishes quickly.\n\nFor this discussion, a short pipeline has four properties:\n\nThat last point is the decisive one.\n\nA message comes in, gets transformed, validated, enriched, and forwarded. If the process dies halfway through, the simplest recovery strategy is often to run the whole pipeline again. If the pipeline is idempotent and the cost of repetition is tiny, replaying from durable history is solving a problem that the workload barely has.\n\nFor this kind of workload, durable execution can make you pay a price without giving you enough value in return.\n\n**Infrastructure overhead.**\n\nDurable execution depends on a persistent backend and a hosting environment that records workflow progress. For a process that finishes in milliseconds, that can be more machinery than the problem deserves.\n\n**Programming model constraints.**\n\nReplay-based orchestration requires discipline. You need clean step boundaries, deterministic orchestration logic, and careful handling of replay semantics. Those rules are worth their cost when you need recovery across time. If you do not, they become cognitive overhead.\n\n**Latency.**\n\nEvery durable boundary has a cost. In a workflow that waits for 24 hours, that cost disappears in the noise. In a low-latency pipeline, it lands exactly where you care most.\n\n**Debugging distance.**\n\nYou are no longer debugging simple control flow. You are debugging an orchestration with history, replay behavior, stored state, and runtime semantics. That is powerful for complex processes, but unnecessary distance from a three-step transformation.\n\n**Operational coupling.**\n\nThe pipeline now depends on the durability store, the worker model, the hosting configuration, and the runtime's interpretation of history. That can be a good trade when you need the guarantees. It is a bad trade when a normal retry would have solved the failure.\n\nThe question before choosing durable execution is not:\n\nWould reliability be nice?\n\nReliability is always nice.\n\nThe useful question is:\n\nWhat does it cost me to repeat the entire operation?\n\nIf the answer is \"almost nothing, because the pipeline is short, cheap, and idempotent,\" you probably do not need durable execution.\n\nIf the answer is \"a lot, because the workflow runs for hours, waits for external input, or contains expensive side effects,\" then the overhead may be worth every cent.\n\nI would ask five questions before introducing a durable workflow:\n\nIf most answers are no, durability is probably the wrong layer.\n\nFor a short pipeline, the honest architecture is usually boring:\n\nThat gives you reliability where it matters without turning ordinary control flow into orchestration.\n\nThe goal is not to avoid infrastructure at all costs. The goal is to put infrastructure at the correct boundary.\n\nDurable workflows are the right choice as soon as at least one of these points applies:\n\nIn those cases, the durable runtime earns its place. The workflow is no longer just a short transformation. It is a stateful business process.\n\nThe temptation with powerful frameworks is to use them everywhere because they are impressive. Durable workflows are impressive for good reasons.\n\nBut durability is not a free upgrade. It is a trade: persistence, replay semantics, storage dependency, latency, and a stricter programming model in exchange for recovery, continuity, and observability.\n\nFor short, cheap, idempotent pipelines, that trade is usually wrong. Repeating the whole operation is simpler than preserving every intermediate step.\n\nSometimes the right answer to \"Should we make this durable?\" is simply:\n\nNo. Make it repeatable.", "url": "https://wpnews.pro/news/durable-workflows-are-great-and-the-wrong-choice-for-short-pipelines", "canonical_source": "https://dev.to/ben-witt/durable-workflows-are-great-and-the-wrong-choice-for-short-pipelines-2n77", "published_at": "2026-07-29 08:00:00+00:00", "updated_at": "2026-07-29 08:04:03.408130+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-agents"], "entities": ["Microsoft Agent Framework", "Durable Task"], "alternates": {"html": "https://wpnews.pro/news/durable-workflows-are-great-and-the-wrong-choice-for-short-pipelines", "markdown": "https://wpnews.pro/news/durable-workflows-are-great-and-the-wrong-choice-for-short-pipelines.md", "text": "https://wpnews.pro/news/durable-workflows-are-great-and-the-wrong-choice-for-short-pipelines.txt", "jsonld": "https://wpnews.pro/news/durable-workflows-are-great-and-the-wrong-choice-for-short-pipelines.jsonld"}}