We built hash-chained workflow histories to make agent execution tamper-evident

Dapr has introduced cryptographic tamper detection for workflow execution histories by signing each history event with the sidecar's mTLS identity, creating an auditable hash chain. The feature requires careful root CA lifecycle planning, as expired or rotated roots will invalidate all signed workflows. This ensures that workflow events cannot be modified, reordered, or removed after being written.

Workflow history signing Dapr workflow history signing provides cryptographic tamper detection for workflow execution histories. Every history event produced during a workflow’s lifetime is signed using the sidecar’s mTLS identity X.509 SPIFFE Verifiable Identity Document SVID , creating an auditable chain of signatures that is verified each time the workflow state is loaded. Before you enable signing: plan your root CA lifecycle Workflow history signing trusts your Dapr root CA . The default Dapr-generated self-signed root is valid for one year . If that root expires, or if you rotate to a new root with a different private key, every signed workflow issued under the old root stops verifying and fails to load with error type SignatureVerificationFailed . There is no re-sign path. Before turning the feature on, decide which of the following you will commit to: Renew the leaf/issuer with the same root key recommended . Back up the Dapr-generated root private key now and reuse it for every renewal, or Bring your own CA with a root key you control and store securely HSM or secret store , and reuse it for all issuer renewals, or Drain before rotating to a new root. Only run workflows short enough to complete or be purged inside one root-CA validity window, and complete or purge all signed workflows before rotating the root. If you cannot guarantee one of these for the full lifetime of your longest workflow, do not enable signing yet . See long-running workflows and root CA expiry /developing-applications/building-blocks/workflow/workflow-history-signing/ long-running-workflows-and-root-ca-expiry for the full guidance. About SPIFFE Verifiable Identity Documents SVIDs An SVID is the workload’s digital passport. Each Dapr sidecar gets one from Sentry and uses it both for mTLS and for signing workflow history. SPIFFE ID : embedded in the X.509 certificate in the URI Subject Alternative Name as spiffe://<trust-domain /ns/<namespace /<app-id . It identifies the workload that produced the signature. Cryptographic proof : the sidecar holds the matching private key and uses it to sign each history batch. Trust roots : every SVID chains to a Sentry CA. Verifiers accept a signature only if its certificate chains to a CA in the trust bundle. For background on Sentry, mTLS, and trust domains, see setup & configure mTLS https://v1-18.docs.dapr.io/operations/security/mtls/ and security concepts https://v1-18.docs.dapr.io/concepts/security-concept/ . Overview Workflows in Dapr execute as a series of deterministic replay steps. Each step appends history events to the actor state store https://v1-18.docs.dapr.io/developing-applications/building-blocks/workflow/workflow-architecture/ . History signing ensures that those events have not been modified, reordered, or removed after they were written. When signing is active, Dapr: - Deterministically marshals each new history event. - Computes a SHA-256 digest over the batch of events. - Chains the new digest to the previous signature’s digest. - Signs the combined input using the sidecar’s SPIFFE https://spiffe.io/ X.509 private key SVID . - Persists the signature and the signing certificate alongside the history. On every subsequent load of that workflow’s state, Dapr walks the full signature chain and verifies every link before allowing execution to continue. Each signature covers a contiguous range of events and references the previous signature’s digest, forming a hash chain. A certificate table stores the DER-encoded X.509 certificate chains used for signing, indexed by position. When the sidecar’s SVID rotates for example, after a restart , a new certificate entry is appended and subsequent signatures reference the new index. Prerequisites History signing requires mTLS https://v1-18.docs.dapr.io/operations/security/mtls/ to be enabled. mTLS provides the SPIFFE X.509 identity that is used as the signing key. Without mTLS, there is no identity material available and signing is silently disabled. In a standard Dapr deployment with the Sentry service https://v1-18.docs.dapr.io/concepts/security-concept/ , mTLS is enabled by default. Configuration History signing is controlled by the WorkflowHistorySigning feature flag. It is disabled by default and must be explicitly enabled. Enabling signing To enable signing, set the feature flag to true in your Dapr configuration: apiVersion: dapr.io/v1alpha1 kind: Configuration metadata: name: my-config spec: features: - name: WorkflowHistorySigning enabled: true Conditions for signing to be active Both conditions must be true for signing to occur: | Condition | How to check | |---|---| | mTLS is enabled | Sentry service is running and the sidecar has a valid SVID | WorkflowHistorySigning is enabled | Feature flag is explicitly set to true | If mTLS is disabled no Sentry , the signer is nil regardless of the feature flag, and signing does not occur. Important Signing is a one-way commitment. Once a workflow is created with signing enabled, it must always run on signing-enabled hosts. Disabling signing on a host that loads a previously signed workflow will cause the workflow to fail. Similarly, enabling signing on a host that loads a previously unsigned workflow will cause the workflow to fail. See one-way commitment /developing-applications/building-blocks/workflow/workflow-history-signing/ one-way-commitment for details. Scope Signing is applied at the sidecar level. When the feature flag is enabled on a Dapr configuration, every workflow that runs on a sidecar using that configuration is signed - there is no per-workflow-type or per-instance opt-in. If you need to introduce signing only for a subset of workflows, run those workflows under a separate appID with a configuration that enables the feature. Activating signing on an existing deployment Because signing is a one-way commitment /developing-applications/building-blocks/workflow/workflow-history-signing/ one-way-commitment , turning the feature on against a cluster that already has running unsigned workflows will cause those in-flight workflows to fail to load. You do not need a brand new deployment, but you do need to drain unsigned work before flipping the flag. A typical rollout looks like: - Stop scheduling new workflow instances on the appIDs that will get signing. - Allow in-flight workflows to complete naturally, or purge https://v1-18.docs.dapr.io/developing-applications/building-blocks/workflow/howto-manage-workflow/ ones you don’t need to keep. - Confirm there is no unsigned workflow state remaining for those appIDs. - Update the Dapr configuration to set WorkflowHistorySigning to true . - Wait for the configuration change to propagate to the sidecars Dapr hot-reloads https://v1-18.docs.dapr.io/operations/components/component-updates/ configurations by default, so no restart is needed; just allow time for propagation across all sidecars . - Resume scheduling. New workflow instances are signed from the first event. If you cannot drain in-flight workflows, run a parallel appID with signing enabled for new work and let the original appID complete its workflows without signing. Once the original appID has no remaining instances, retire it. One-way commitment Signing is a permanent commitment per workflow. Once a workflow is created with signing enabled, all subsequent operations on that workflow must occur on signing-enabled hosts. There are two invariants: Signed workflow on non-signing host : If a workflow has signed history but the current host does not have a signer configured mTLS is off or the feature flag is disabled , loading the workflow fails. The workflow cannot execute and is effectively terminated. Unsigned workflow on signing host : If a workflow was created without signing the feature flag was off and is later loaded by a signing-enabled host, loading fails. The unsigned history has no integrity proof and cannot be retroactively signed. Migration guidance Before enabling signing cluster-wide, ensure all existing unsigned workflows have completed or been purged. Once signing is enabled, new workflows are signed and existing unsigned workflows fail to load. Certificate rotation Dapr handles certificate rotation transparently. When the sidecar’s SVID rotates for example, after a restart where Sentry issues a new short-lived certificate, or when the SVID naturally expires , the signing system: - Detects that the current certificate differs from the last entry in the certificate table. - Appends a new certificate entry to the table. - New signatures reference the new certificate index. Previous signatures remain valid because they reference their original certificate, which is still in the table and verifiable against the CA trust anchors. Both Cert A and Cert B chain to the same Sentry CA, so all signatures remain valid. In multi-replica deployments, each replica has its own private key and SVID certificate. When the workflow runtime migrates between replicas for example, due to scaling or rebalancing , the new replica’s certificate is appended to the table. All certificates are validated as belonging to the same app ID via SPIFFE identity binding. Important Certificate rotation new leaf SVID, same CA root works seamlessly. A full CA rotation completely different root CA will cause verification to fail for workflows signed under the old CA, because the old signing certificates will not chain to the new trust anchors. This is by design: if the trust root changes, previously signed data cannot be verified. Long-running workflows and root CA expiry This is the operational consideration that has the biggest practical impact on signed workflows, and is worth calling out separately. By default Dapr generates a self-signed root CA that is valid for one year . If that root expires or is replaced with a CA built from a different private key , any workflow whose signature chain trusts the old root stops being verifiable . Signed long-running workflows that outlive a CA rotation will fail to load and surface as FAILED with SignatureVerificationFailed . To keep long-running signed workflows healthy across CA renewals, do one of the following: Rotate the leaf/issuer cert, keep the same root key. This is the recommended path. The Dapr CLI supports it with dapr mtls renew-certificate -k --private-key <existing-root-key see renew certificates https://v1-18.docs.dapr.io/operations/security/mtls/ root-and-issuer-certificate-upgrade-using-cli-recommended . Existing signed workflows continue to verify against the same root. Bring your own CA. Provide a root certificate whose private key you control and store securely for example, in your own HSM or secret store . Renew the leaf/issuer with the same root key indefinitely. See bringing your own certificates https://v1-18.docs.dapr.io/operations/security/mtls/ bringing-your-own-certificates . Drain before rotating to a new root. If you must rotate to a brand new root key, complete or purge in-flight signed workflows first. Signing is a one-way commitment, so there is no “re-sign with the new root” operation. Plan your CA lifecycle before enabling signing If your workflows can run for weeks or months, back up your root private key and treat root CA rotation as a planned, drain-and-rotate event. Losing the root key or rotating to a freshly generated root will permanently break verification for any signed workflow that was issued under the old root. How signing works Signing new events After each workflow execution step, Dapr signs the newly appended history events. The signing process works as follows: Deterministic marshaling : Each new HistoryEvent is marshaled using protobuf’s deterministic mode, producing stable bytes for the same message. These exact bytes are both signed and persisted to the state store. Events digest : A SHA-256 hash is computed over the batch of marshaled events, with each event length-prefixed big-endian uint64 to prevent concatenation ambiguity. Chain linkage : The SHA-256 digest of the previous HistorySignature protobuf message is computed. The root signature first in the chain has no previous digest. Signature input : The final signing input is SHA-256 previousSignatureDigest || eventsDigest . Cryptographic signing : The input is signed using the sidecar’s SPIFFE X.509 private key. The signing key type is whatever Sentry issues to the sidecar; Dapr supports Ed25519, ECDSA P-256, and RSA. You don’t choose the key type per workflow: it follows from the mTLS https://v1-18.docs.dapr.io/operations/security/mtls/ setup Dapr-generated keys default to Ed25519 from 1.18 onwards, custom CAs sign with whatever algorithm the issuer key uses . Certificate resolution : If the current SVID certificate matches the last entry in the certificate table, the existing index is reused. Otherwise, a new entry is appended. This handles certificate rotation /developing-applications/building-blocks/workflow/workflow-history-signing/ certificate-rotation transparently. Persistence : The signature, any new certificate entry, and the history events are all persisted to the state store in a single transactional write, ensuring atomicity. Verification on load Every time workflow state is loaded whether for execution or a metadata query the full signature chain is verified. The verification steps for each signature in the chain are: | Step | Check | Detects | |---|---|---| | Chain linkage | previousSignatureDigest matches SHA-256 previous signature | Reordered or inserted signatures | | Contiguity | Event ranges are adjacent with no gaps | Missing signatures | | Events digest | Recompute SHA-256 from raw stored bytes | Tampered, inserted, or deleted events | | Cryptographic signature | Verify against public key from the signing certificate | Forged signatures | | Certificate validity | Certificate was valid at the time of the last signed event | Expired or backdated certificates | | Chain-of-trust | Certificate chains to a trusted Sentry CA root | Signing by untrusted identity | | App identity | SPIFFE ID in certificate matches the workflow’s owning app | Cross-app signature forgery | | Full coverage | Signatures cover every event from index 0 to the end | Partially unsigned history | Verification uses the raw bytes from the state store , not re-marshaled events. This ensures that any byte-level modification to persisted events is detected. Inbox event validation When signing is enabled, the Dapr workflow runtime validates inbox events before processing them. Result events TaskCompleted , TaskFailed , ChildWorkflowInstanceCompleted , ChildWorkflowInstanceFailed must reference an operation that was actually scheduled in the signed history. Events that reference non-existent operations, such as a TaskCompleted for a task ID that was never scheduled, are considered injected and are purged from the inbox. This prevents an attacker with state store access from injecting fake activity or child workflow results that would otherwise be signed into the history chain. Child workflow and activity attestation The same-workflow signature chain only protects history that the workflow itself produced. Cross-identity completion events a child workflow or activity running under a different SPIFFE identity reporting back to the parent need their own cryptographic proof. Dapr emits an attestation on every cross-identity completion and verifies it before the event enters the parent’s inbox. When a child workflow or activity completes, the executor attaches one of: ChildCompletionAttestation on ChildWorkflowInstanceCompleted / ChildWorkflowInstanceFailed . ActivityCompletionAttestation on TaskCompleted / TaskFailed . Each attestation commits to a deterministic, language-independent payload: | Field | Description | |---|---| parentInstanceId + parentTaskScheduledId | Bind the proof to a specific invocation. Prevents cross-instance and cross-task replay. | ioDigest | SHA-256 over canonicalized, NFC-normalized input and output bytes. Identical across SDKs because the encoding is wire-format-independent and spec-versioned. | signerCertDigest | SHA-256 of the signer’s DER-encoded X.509 chain. The chain itself travels alongside the attestation as a wire-only companion field. | terminalStatus | The workflow or activity outcome. For activities, the activity name is also committed. | On receipt, the parent’s Dapr workflow runtime runs verifyInboxAttestation before the event is appended to the inbox: - The attestation must be present signing on implies sender must attest . - The companion certificate’s digest must match the committed signerCertDigest . - The certificate chain must validate against a Sentry trust anchor. - The signature must verify over the exact attestation payload bytes no re-marshaling on the receiver side . parentInstanceId must equal the receiving workflow’s actor ID. parentTaskScheduledId must resolve to a TaskScheduled or ChildWorkflowInstanceCreated event already present in the signed history. ioDigest must equal the canonical digest of the parent’s scheduling input together with the reported output. terminalStatus must match the enclosing event type. If any check fails, the inbound completion is rejected and the workflow is tombstoned into failure. Once verified, the foreign certificate is absorbed into a content-addressed ext-sigcert-NNNNNN table on the receiver so the same signer is not re-validated on every subsequent completion. The chain-of-trust result is also cached per workflow instance for the lifetime of that instance, so a workflow that calls the same foreign signer repeatedly pays chain validation cost only once. Lineage propagation verification When a workflow’s history is forwarded to another workflow as IncomingHistory under PropagateLineage typically multi-hop child invocations across apps , the receiving workflow needs to verify the forwarded content. Because the propagated bytes do not become part of the receiver’s own signed History , an independent per-chunk signature is attached at dispatch time. Each PropagatedHistoryChunk carries: - The original events as produced by the chunk’s authoring app, byte-for-byte. - A fresh chunk-local signature over those events, produced by the chunk’s app using its current SPIFFE X.509 key. - The DER-encoded certificate chain for that signing key. On ingestion, every chunk is verified independently: | Check | Detects | |---|---| | Chain-of-trust to a Sentry trust anchor reusing the per-instance cache | Forged or self-signed lineage | Leaf SPIFFE ID’s app component matches the chunk’s declared appId | Lineage produced by the wrong app | | Per-chunk signature covers exactly the events in the chunk, contiguously | Splicing, gaps, or partial omission | | Empty-with-signatures, missing-signatures, missing-certificate variants | Malformed chunks that would otherwise short-circuit verification | Verified foreign certificates are absorbed into the same ext-sigcert-NNNNNN table used by completion attestations, so downstream attestation lookups can content-address them. What happens when verification fails When signature verification fails, Dapr’s response depends on whether the workflow is still in custody of the executor, and on whether the failure is a genuine tamper or a host configuration mismatch. In every case the original history, signatures, and inbox are never modified ; the untrusted data is preserved for forensic analysis. Tamper of an in-flight workflow When the Dapr workflow runtime loads a running workflow and detects tampering, Dapr stops the executor from acting on forged input by appending a single unsigned terminal ExecutionCompleted event marked with the well-known error type DAPR WORKFLOW HISTORY TAMPERED . This:- Halts further execution: the workflow is now in a terminal state and the Dapr workflow runtime will not replay or schedule new work on it. - Provides a stable, machine-matchable signal for clients DAPR WORKFLOW HISTORY TAMPERED . - Preserves the original untrusted history and signatures so operators can inspect what was tampered with. From an application perspective the workflow surfaces as FAILED , and any client that calls GetWorkflow or the equivalent SDK method on the instance receives the failure with the DAPR WORKFLOW HISTORY TAMPERED error type in the failure details. SDKs raise this as the same exception type that a normal workflow failure would raise, so existing error-handling paths can match on the error type to specifically detect tampering. The Dapr sidecar also emits an error-level log entry naming the workflow instance ID and the specific verification check that failed. LoadWorkflowState recognises the tamper marker and bypasses signature verification on subsequent loads, so the workflow can be read back and surfaced as FAILED rather than failing to load entirely. Reminders for the workflow and its activities are deleted to stop the engine from endlessly retrying a compromised workflow. The tamper-recovery path is scoped to workflows still in custody of the executor. Workflows that had already completed before the tamper happened are left untouched: there is nothing for the executor to do, and the verification error is surfaced to readers metadata queries who are responsible for detecting the tampering at read time. Configuration error A mismatch between signed history and host signing configuration signed workflow loaded by a host without a signer, or unsigned workflow loaded by a signing-enabled host is reported as a distinct ConfigurationError , not as tampering. The tamper-recovery path described above does not run against these intact-but-unloadable workflows: their history is byte-identical to what was written, so appending a tamper-terminal event would itself be a destructive edit. Fix the host configuration, or purge the workflow, to recover. The error is visible in two places: the Dapr sidecar logs an error-level entry on the failed load naming the workflow instance ID and whether the host was expected to have or not have a signer , and callers that issue a workflow metadata query GET /v1.0/workflows/<id or the SDK equivalent receive the ConfigurationError directly in the response, allowing programmatic detection. Metadata queries API path When a workflow metadata query such as GET /v1.0/workflows/<id or FetchWorkflowMetadata encounters a verification error, the error is returned directly to the caller. The error message contains the specific reason for failure for example, digest mismatch, attestation verification failure, or certificate trust failure . Common failure causes | Cause | What happened | Detection | |---|---|---| | Tampered history | A history event was modified directly in the state store | Events digest mismatch | | Deleted event | A history event was removed from the state store | Event count or coverage mismatch | | Inserted event | An event was added outside of normal workflow execution | Events digest mismatch | | Reordered events | Events were rearranged in the state store | Events digest mismatch | | Injected inbox event | A fake result was written to the inbox in the state store | Inbox validation: no matching scheduled operation | | CA change | Sentry CA was rotated to a completely new root | Certificate chain-of-trust failure | | Cross-app forgery | A certificate from a different app was used to sign | SPIFFE app identity mismatch | | Corrupted signature | A signature entry was modified in the state store | Cryptographic signature verification failure or chain linkage mismatch | | Forged child or activity completion | A cross-identity completion event was injected without a valid attestation | Inbox attestation verification fails missing attestation, bad signature, wrong parent ID or task scheduled ID, ioDigest mismatch | | Tampered propagated lineage | A PropagatedHistoryChunk was modified, swapped, or signed by the wrong app | Per-chunk signature, chain-of-trust, or SPIFFE app-ID check fails | | Signing disabled ConfigurationError | A signed workflow was loaded by a non-signing host | “signed history but no signer is configured” | | Signing enabled on unsigned ConfigurationError | An unsigned workflow was loaded by a signing host | “unsigned history events but signing is enabled” | State store layout Workflow signing data is stored alongside the workflow state using the following key prefixes. All keys are scoped to the workflow instance’s ID. | Key pattern | Content | Format | |---|---|---| history-NNNNNN | History events | Protobuf HistoryEvent | signature-NNNNNN | Signature entries | Protobuf HistorySignature | sigcert-NNNNNN | Signing certificates own SPIFFE identity | Protobuf SigningCertificate DER-encoded X.509 chain | ext-sigcert-NNNNNN | Foreign signing certificates absorbed from verified completion attestations and lineage chunks deduped by digest | Protobuf SigningCertificate DER-encoded X.509 chain | metadata | Counts and generation | Protobuf WorkflowStateMetadata | The NNNNNN suffix is a zero-padded 6-digit index for example, signature-000000 , signature-000001 . The metadata entry tracks the count of each entry type so the loader knows exactly how many keys to fetch. All writes history events, signatures, certificates, metadata are persisted in a single transactional state operation, ensuring atomicity. Security properties | Property | Guarantee | |---|---| Tamper detection | Any modification to persisted history events changes the events digest, breaking verification | Chain integrity | The previousSignatureDigest linkage prevents reordering, inserting, or removing signatures | Non-repudiation | Each signature is bound to a specific X.509 identity SPIFFE SVID | App identity binding | The SPIFFE ID in each signing certificate is validated against the workflow’s owning app ID, preventing cross-app forgery | Time binding | Certificate validity is checked against the event timestamp, preventing use of expired credentials | Trust anchoring | All signing certificates are verified against the Sentry CA trust bundle | Inbox validation | Activity and child workflow results are validated against scheduled operations in signed history, preventing injection of fake results | Cross-identity attestation | Child workflow and activity completions are cryptographically attested by the executing identity and verified by the parent before they enter the inbox | Lineage integrity | Forwarded PropagatedHistoryChunk s are individually signed and verified against the producing app’s SPIFFE identity | Immutable history | Dapr never modifies the existing workflow history, signatures, or inbox; the only write on tamper detection is an unsigned terminal ExecutionCompleted event with error type DAPR WORKFLOW HISTORY TAMPERED | One-way commitment | Signing cannot be disabled for signed workflows or enabled for unsigned workflows | Frequently asked questions Does signing add latency to workflow execution? The signing operation itself SHA-256 hashing and ECDSA/Ed25519/RSA signing is fast and adds negligible CPU latency. The measurable cost is on the state store side: each workflow step now persists additional entries a signature entry, and on certificate rotation a new certificate entry alongside the history events. These extra entries are written in the same transactional batch as the history events, so they cost one larger transaction rather than additional round-trips, but they do increase the payload size and the storage footprint per workflow. The exact impact depends on your state store’s pricing and write characteristics. What happens if I disable signing on a workflow that was previously signed? The workflow fails to load . Signing is a one-way commitment /developing-applications/building-blocks/workflow/workflow-history-signing/ one-way-commitment : once a workflow has signed history, it must always run on a signing-enabled host. This prevents an attacker from disabling signing to bypass verification. Can I enable signing on workflows that were created without it? No. Enabling signing on a host that loads unsigned workflow history causes a verification error. The unsigned history has no integrity proof and cannot be retroactively signed, because events written without signing could have been tampered with. Ensure all unsigned workflows complete or are purged before enabling signing cluster-wide. What happens during a Sentry CA rotation? Certificate rotation new leaf SVID, same CA root : works seamlessly. Multiple certificates are stored in the certificate table and each signature references its specific certificate. All certificates chain to the same CA. CA rotation completely new root CA : verification fails for workflows whose signing certificates were issued by the old CA. The workflow is reported as FAILED with SignatureVerificationFailed . This is intentional: the trust root has changed and previously signed data cannot be verified against the new trust anchors. See long-running workflows and root CA expiry /developing-applications/building-blocks/workflow/workflow-history-signing/ long-running-workflows-and-root-ca-expiry for the operational guidance to avoid this. What about multi-replica deployments? Each replica of the same app ID has its own private key and SVID certificate. When the workflow runtime migrates between replicas, each replica’s certificate is stored in the certificate table and the signature chain remains valid. All certificates are verified as belonging to the same app ID via SPIFFE identity binding. What state store backends are supported? History signing works with any state store that supports the actor state transactional API. The signing data is stored as additional key-value entries alongside the existing workflow state.