{"slug": "decision-firewall-what-building-an-ai-governance-framework-taught-me", "title": "Decision Firewall: what building an AI governance framework taught me", "summary": "A developer built Decision Firewall, an open-source reference framework for authorizing and auditing AI-driven actions, using a subscription refund-recovery example to demonstrate the pattern. The system separates model assessment from execution credentials, uses SQLite for durable authorization state, Ed25519 signatures and canonical JSON for verifiable audit records, and integer minor units to keep financial arithmetic out of model output. On a 4 GB GTX 1650, the recorded Laya run loaded in 28.42 seconds and produced a 47.45 ms median inference across 34 assessments, though the developer notes the timings do not establish model accuracy.", "body_md": "A refund-recovery example, measured comparisons and lessons from building Decision Firewall, an open-source reference for AI action authorization and audit.\n\nThis is an adapted edition of the original article. It preserves the section argument and evidence; use the source link for the published version and any interactive material.\n\n01 / The starting problem\n\nI picked subscription refunds because the facts are inspectable. A duplicate-charge claim needs two payment records. Cancellation eligibility depends on timing and usage. A prior refund reduces the amount still available.\n\nThe example policy allowed verified duplicates and unused subscriptions cancelled within seven days. Eligible amounts up to ₹1,000 could proceed automatically; larger amounts needed review. A ₹10,000 daily automated limit sent excess demand to review. These were demonstration rules, with simulated payments.\n\nHere is an illustrative ₹800 duplicate refund, using the recovery contract I built. The amount is an example, not an additional benchmark result.\n\nFollow one refund through a lost response\n\nA timeout tells me the response is missing. It does not tell me the refund failed. That distinction survives a process restart only if the key and execution state are durable. The customer’s refund request also needs a status separate from a failed or malformed execution attempt.\n\nThis is useful application engineering. A framework can package it; it does not own the idea. [Executor contract](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/framework/executors.md).\n\n02 / How the plan grew\n\nI wanted developers to bring their own decision models and business rules. Refunds became a domain pack, with access control and simulated deployment examples testing whether the core travelled beyond payments.\n\nEach next request sounded reasonable. Import old decisions. Keep reviewed labels separate from historical approvals. Compare policies offline. Retrieve examples without leaking evaluation data. I had LangChain’s abstractions and Langfuse’s observability experience in mind.\n\nThe plan grew into history ingestion, dataset snapshots and named Python rules, with optional context retrieval and telemetry. Preparation before inference could request missing evidence before spending a model call. All of it had a plausible use.\n\nI had not yet established that another developer benefited from adopting the runtime underneath it. A coherent feature list had become my substitute for that evidence.\n\nThe project changed direction when the comparator changed. Scroll the map on small screens. Follow a branch: start · expand · compare · observe · repair · narrow.\n03 / What I actually built with\n\nMy GTX 1650 had 4 GB VRAM, alongside roughly 8 GB system RAM. I chose Laya for typed assessments, left Jev out, and required a fixture path that worked without model downloads. That made the control lifecycle testable before inference was available.\n\nI used Python 3.12 and Pydantic for contracts, SQLite for durable authorization state, and a separate simulated payment ledger for effects. Integer minor units kept financial arithmetic out of model output. Ed25519 signatures and canonical JSON supported verifiable audit records. Typer exposed the CLI; FastAPI and Jinja powered an optional refund inspector.\n\nThe model adapter could return an assessment but held no execution credentials. Domain code resolved trusted evidence and evaluated policy. The executor owned the downstream effect. Optional OpenTelemetry export sat alongside the required local audit path.\n\nThe recorded Laya run used CUDA in FP32: 28.42 seconds to load and 47.45 ms median inference across 34 assessments, including the first. [The report pins the checkpoint and SDK](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/benchmarks/post-review-refund/summary.md). Small hardware shaped the implementation; those timings did not establish model accuracy.\n\n04 / The question I asked too late\n\n“Is the same foot just sitting at two different places, or genuinely there is some distinction?”\n\nMy question during the build conversation.\n\nI asked for numbers with and without the framework. The hard part was choosing “without.” A model directly triggering refunds is an easy opponent. A competent application already validates the ledger and handles uncertain execution.\n\nThe synthetic comparison used a deliberately minimal model-directed arm and two governed arms, sharing evidence, fault schedules and a simulator with its own protections. With fixture assessments, both application controls and Decision Firewall produced zero contract-violating effects and zero unresolved attempts after recovery. The minimal arm produced 95 violating effects across 170 request trials.\n\nThat established the value of controls. It did not distinguish the two ways of implementing them.\n\nOn the matched straightforward-success subset, application p95 latency was 26.52 ms versus 67.89 ms for the framework. The paired median added cost was 23.23 ms across 25 pairs. The full run repeated 32 frozen scenarios five times; those repetitions measured runtime variation, not independent business situations. [Refund comparison](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/benchmarks/post-review-refund/summary.md).\n\nMy next question should have been about integration effort: if both enforce the same contract, does the package make that contract easier to implement and maintain?\n\n05 / More rows, then a different question\n\nInstead, I first asked for more data. The classification comparison accounted for 39,955 BANKING77 and Bitext source rows using a CPU naive Bayes classifier. Wrapping the same classifier left its predictions unchanged. BANKING77 official-test accuracy was 78.60% in every arm; the prepared path added 13.527 ms paired median latency.\n\nThere was no consequential action to authorize. I had measured overhead on a task that could not demonstrate the central benefit. [Full counts and separate out-of-fold results](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/benchmarks/paired-public/summary.md).\n\nA later tool-use pilot used `openai/gpt-oss-20b` through OpenRouter because Laya could not fill the conversational roles. It scheduled 90 selected τ-bench and AgentDojo episodes; 78 completed and 12 remained incomplete.\n\nOn the attacked AgentDojo subset, the baseline had eight attacker successes in nine completed episodes. Both controlled arms had zero in ten, but completed the legitimate task in only four of those ten. A blocked attack and a completed user task needed separate columns. This selected development pilot was not a full-suite score. [Agent pilot](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/benchmarks/agent-pilot/summary.md).\n\n06 / Where there was supplied functionality\n\nI thought audit and observability might justify reuse even when decisions matched. The application baseline needed eight explicit event call sites and four operation decorators. The framework integration supplied an observer with no application event hooks.\n\nThat was supplied functionality. It also had gaps: the measured framework spans omitted the evaluation ID, policy version, disposition and reasons. The manually instrumented application exported the first three. Neither exported the sensitive canaries. Exporter failure allowed execution to continue; required local audit failure blocked it. [Measured observability report](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/benchmarks/post-review-observability/summary.md).\n\nFor the lost-response refund, telemetry helps locate the attempt. The durable record and downstream ledger establish what happened. I can tolerate a missing diagnostic span; I cannot safely forget the unresolved refund and release its reservation.\n\nSigned receipts add a check that recorded content has not changed under a trusted key. They do not prove that the original evidence was true. I kept that distinction, while recognising that telemetry alone did not justify adopting the whole runtime.\n\n07 / Letting another build challenge it\n\nI had no human participants, so I commissioned a separate Claude Code agent evaluation. One builder used the framework; another implemented equivalent application controls for the same purchase-order requirements. Both then handled a policy change and an investigation.\n\nThis was one paired pilot, independent of the framework builder, using the same model for both arms. It gave me a concrete integration comparison:\n\nThe framework required eight library-contract workarounds and seven core-change requests. The builder duplicated checks to obtain specific refusal reasons before calling it. Reuse was fighting the integration.\n\nBoth failed the same held-out approval-after-outage case because the spec was ambiguous; no downstream effect occurred. The baseline’s zero workarounds refers to library contracts, not its two test/path adjustments.\n\nThe 374-line spec supplied much of the lifecycle design. Investigators had downstream exports and a harness problem prevented operator-CLI use. A spend-limit interruption also affected the framework build. Tool calls are not developer hours, and these conditions limit generalisation. [Full comparison and deviations](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/evaluations/2026-09-29/COMPARISON.md).\n\n“Stop feature expansion and adoption/migration claims.”\n\nThe independent evaluator’s [recommendation](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/evaluations/2026-09-29/COMPARISON.md).\n\n08 / What passing tests had missed\n\nThe review found four confirmed defects among 21 findings. Approval could execute against evidence the reviewer had not seen. Deactivating a rejecting reviewer could revive an older approval. Oversized integer metadata could strand an already-executed action. The repository tests missed six of nine injected runtime faults.\n\nThe mutation result mattered: downstream protections had hidden missing guarantees in the layer I meant to test. A green end-to-end suite was insufficient evidence that each boundary enforced its contract.\n\nThe 0.3.1 fixes bound review to the inspected evaluation, preserved rejection and handled large integers losslessly. Direct invariant tests detected all nine targeted mutations; recorded Windows verification had 298 passing tests, including 22 corrective tests. [Corrective release](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/verification/0.3.1/README.md).\n\nI fixed the defects and kept the earlier comparison unchanged. Repairing correctness did not retroactively demonstrate easier integration.\n\n09 / Where I would spend less\n\nThe global SQLite write lock still spans adapter I/O. It explains the roughly ninefold suite-time difference in the pilot, not a universal per-request slowdown. Fixing it safely would require a different concurrency and recovery design.\n\nI chose to document the limit rather than start another architecture project to rescue the adoption argument. History ingestion and model-context budgets could wait too. I had built those surrounding features before proving the central abstraction helped an unfamiliar integrator.\n\nThe review exposed assumptions inside the supposedly generic interface: a message-shaped submission contract, one caller identity per runtime instance and a fixed approval lifetime. Moving code into a shared package had moved those assumptions into someone else’s application.\n\nThat is the distinction I now look for. The refund needs recovery after a lost response. Whether recovery belongs in this dependency is a separate engineering decision.\n\n10 / What I kept\n\nI kept Decision Firewall as an open-source reference implementation and conformance kit. The review verified action/evidence binding and recovery without replacement dispatch. A capacity-one resource admitted exactly one dispatch across eight operating-system processes. Historical replay called neither the evidence resolver nor the executor.\n\nThose are useful behaviours to inspect in runnable code. The [repository](https://github.com/asadani/decision-firewall) includes the reports and failure cases, so a builder can study the contract without accepting an adoption recommendation.\n\nStart with the model-free path using Python 3.12, the tested runtime. Activate the virtual environment for your shell before installing:\n\nLocal, simulated effects only\n\ngit clone [https://github.com/asadani/decision-firewall.git\\](https://github.com/asadani/decision-firewall.git%5C)\n\ncd decision-firewall\\\n\npython -m venv .venv\\\n\n# Activate .venv for your shell, then:\\\n\npython -m pip install -r requirements-core.lock\\\n\npython -m pip install --no-deps -e .\\\n\nfirewall framework --domain refunds demo\\\n\nfirewall conformance decision_firewall.conformance_fixtures:deployments --save conformance.json\n\nAfter running it, inspect which exact action the authorization binds and how the execution attempt connects to its receipt. Then read the [17 declared deployment conformance checks](https://github.com/asadani/decision-firewall/blob/a9b9a52/docs/framework/release-tools.md): which cases pass, which are unsupported, and what the adapter must guarantee.\n\nFor an application of my own, I would turn the lost-response example into a test: restart after downstream success, reconcile the original key, then assert one effect and no remaining reservation once success is resolved. Passing a happy-path refund would not answer that question. Production identity and protection against bypassing the controls still need deployment integration.\n\nAfter publishing this, someone asked whether the conformance checks could run against plain application controls without the runtime. They couldn’t. That exposed a gap in calling the project a conformance kit: running its checks still meant using my framework.\n\nI extracted a small independent package covering stale approval, duplicate effects and recovery after a lost response. In the [new development pilot](https://github.com/asadani/decision-firewall/blob/9ed7457/docs/benchmarks/independent-conformance/summary.md), both the existing application and Decision Firewall passed all three scenarios. Six deliberately injected behavioural faults, three in each implementation, were detected. These are separate results from the earlier paired study.\n\nThe package needs no Decision Firewall runtime or model. From a repository checkout, in a Python 3.12 virtual environment, this runs the plain application's simulated example:\n\nIndependent checks, simulated application\n\npython -m pip install ./conformance\\\n\npython -m control_conformance benchmarks.control_conformance.adapters:application --output report.json\n\nTesting another application still takes work: its test adapter must expose the controls and an independent record of downstream effects. The [adapter guide](https://github.com/asadani/decision-firewall/blob/9ed7457/conformance/README.md) defines that contract. The examples use one simulated export domain; recovery reconstructs state in a fresh interpreter, without testing an operating-system crash. Three scenarios do not establish complete conformance coverage or production safety.\n\nThis doesn’t establish that the framework saves work. It makes a smaller contribution available: someone can test these behaviours without adopting my runtime. That feels like a better fit for where the evidence led me.\n\n11 / What I would change next time\n\nNext time I would build one consequential workflow and a competent application comparator before adding the developer platform. I would write down the proposed benefit first, then choose a measurement capable of finding it.\n\nIf the claim is easier integration, classification accuracy cannot establish it. If the claim is faster investigation, giving both investigators an answer-bearing export makes a weak test. Several integrations and human participants would be needed to investigate human-productivity benefits.\n\nMy useful contribution was pressing those questions, although I asked them late. I started by separating a model’s assessment from permission to act. I ended by separating a useful control from an unproven reason to adopt a framework.\n\nThe refund still needs to survive a lost response. The library around that behaviour has to earn its place.", "url": "https://wpnews.pro/news/decision-firewall-what-building-an-ai-governance-framework-taught-me", "canonical_source": "https://dev.to/anuj-sadani/decision-firewall-what-building-an-ai-governance-framework-taught-me-k2n", "published_at": "2026-11-22 09:37:44+00:00", "updated_at": "2026-11-22 09:42:30.093176+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools"], "entities": ["Decision Firewall", "Laya", "LangChain", "Langfuse", "Pydantic", "SQLite", "FastAPI", "OpenTelemetry"], "also_reported_by": [], "takeaway": "Separate the model adapter, which returns assessments without execution credentials, from the executor that owns the downstream effect, and persist the idempotency key and execution state in SQLite so a lost response is distinguishable from a failed action across restarts.", "recommendation": {"advice": "Keep execution credentials out of the model adapter and store idempotency keys and execution state durably so a timeout is treated as a missing response, not a failed action.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/decision-firewall-what-building-an-ai-governance-framework-taught-me", "markdown": "https://wpnews.pro/news/decision-firewall-what-building-an-ai-governance-framework-taught-me.md", "text": "https://wpnews.pro/news/decision-firewall-what-building-an-ai-governance-framework-taught-me.txt", "jsonld": "https://wpnews.pro/news/decision-firewall-what-building-an-ai-governance-framework-taught-me.jsonld"}}