{"slug": "the-agentic-data-development-lifecycle-addlc", "title": "The Agentic Data Development Lifecycle (ADDLC)", "summary": "A proposed Agentic Data Development Lifecycle (ADDLC) framework — 3 actors, 3 loops, 7 phases and 4 gates built on the rule that \"agents propose, controls enforce, humans decide\" — was applied to the reference and streaming market data of one large platform, cutting human effort per integration by about 4-5x against a target of roughly 15x. The observational, non-controlled results also showed lead time from request to data in production about 5x shorter, about 8x more integrations delivered per quarter by the same team, and a roughly 17x lower share of incidents first reported by consumers rather than monitoring, with specs and packages accepted at first review nearly 2x higher. The framework, which builds on DAMA-DMBOK, Data Mesh and Data Management at Scale, argues agents made writing pipeline code cheap but not being wrong, moving the bottleneck to specifying meaning and verifying outcomes, and grants autonomy per class of change by reversibility and blast radius rather than per agent.", "body_md": "Agents made data pipelines cheap. Being wrong is still expensive. Here is who decides, what gets verified and how autonomy is earned.\n\nTL;DR\n\nAI agents made producing data pipelines almost free. They did not make being wrong any cheaper. The bottleneck has moved from writing code to specifying meaning and verifying outcomes.\n\nADDLC is a lifecycle designed around that shift: 3 actors, 3 loops, 7 phases, 4 gates.\n\nThe operating rule fits in one line: agents propose, controls enforce, humans decide.\n\nAutonomy is granted per class of change, by reversibility and blast radius ‐ not per agent, and not by enthusiasm. It is earned with evidence and lost with escaped defects.\n\nBuild and Run are segregated by design, for humans and agents alike. Master and reference data get a stewarded change channel of their own.\n\nThe model comes with gate criteria, an autonomy scale from Manual to Self-organizing, metrics, a maturity model and a self-assessment. It builds on DAMA-DMBOK, Data Mesh, Data Management at Scale rather than replacing them.\n\nNumbers from applying this model to the reference and streaming market data of one large platform, before and after. They are observational, not a controlled experiment, and part of the gain comes from organizational changes made at the same time.\n\nHuman effort per integration: about 4‐5x lower, with a target of roughly 15x.\n\nLead time from request to data in production: about 5x shorter.\n\nIntegrations delivered per quarter by the same team: about 8x more.\n\nShare of incidents first reported by consumers rather than by monitoring: about 17x lower.\n\nSpecs and packages accepted at the first review: nearly 2x higher, approaching all of them.\n\nIncident triage done by agents: from none to a target of about three quarters.\n\n1. The cost curve flipped\n\nIn the last year the economics of data engineering inverted. A coding agent with access to a warehouse, a dbt project and a catalog drafts an ingestion job, staging models, tests and documentation in minutes. What used to be the expensive part of a data integration ‐ writing the code ‐ is now the cheap part.\n\nWhat did not get cheaper is being wrong. A mis-specified join, a metric that silently changes meaning, a wrong identifier mapping in reference data: each costs as much as before, and often more, because agents produce change faster than people can review it.\n\nTeams that adopt agents without changing their process see the same symptoms:\n\nSprawl. Creating a new table is cheaper than understanding an existing one, so every request gets its own pipeline.\n\nSelf-certified quality. The agent writes the transformation and the tests that confirm it. Both pass. Both encode the same misunderstanding.\n\nDocumentation nobody trusts. Generated descriptions multiply, nobody reviewed them, and readers learn to ignore all of them.\n\nShadow autonomy. The agent runs with an engineer’s production credentials, so the real permission model is “whatever the person who started the session could do”.\n\nArtifact chaos. Agents create copies instead of versions: several near-identical files for one diagram, with the file name as the only version control.\n\nThese are lifecycle problems. DevOps spread through shared practices and, later, a shared yardstick ‐ the four key delivery metrics from the DevOps Research and Assessment program, popularized by the book Accelerate. Agentic data work needs the same, and it needs it before every vendor defines it in terms of its own product.\n\nDefinition. The Agentic Data Development Lifecycle (ADDLC) is the end-to-end process by which data products are framed, specified, built, verified, released, operated and retired when AI agents perform most of the execution and humans retain decision rights over intent, meaning and acceptance.\n\n2. What already exists, and what is missing\n\nCoalesce, “The Agentic Data Development Lifecycle in Practice” (Mikkel Dengsøe, October 2026). The closest prior art: a product workflow in four stages ‐ Define, Build, Govern, Observe ‐ with a clear split between what the agent does and what people decide, and spec-driven development in Git. It is anchored in one product and does not define gate criteria, accountable deciders, autonomy policy, metrics or maturity.\n\nThoughtworks, “Making Your Data Ready for Agentic AI” (Pramod Sadalage and Prem Chandrasekaran, martinfowler.com, August 2026). Five attributes of AI-ready data, a context layer (domain, semantic and capability models) and a staged autonomy ladder. It treats agents mainly as consumers and curators of data rather than as builders of pipelines.\n\n“The Self-Organizing Data Stack” (Ananth Packkildurai, Data Engineering Weekly, October 2026). An end state in which an engine maintains the dependency graph through five loops, with assets as contracts and a firm rule: never let an agent grade its own work. It describes a possible destination; the path to it is out of scope.\n\nHolistics, “Agentic Data Engineering” (Huy Nguyen, September 2026). What agents can do across the stack and why the semantic layer is the contract between engineering and analytics. No lifecycle.\n\n“From Data Quality to Quality of Agentic Data Use” (Ania Cravero, Jorge Díaz-Villegas, Zihao Xiao, Applied Sciences, September 2026). Six dimensions of agentic data use and a staged control lifecycle for an agent acting under delegated authority. Rigorous, but focused on executing a data task rather than on a team’s operating model.\n\nThe ADLC family (IBM, EPAM, AWS AI-DLC and others). Lifecycles for building agents, or for software built by agents. A comparative review in Towards AI found six definitions and zero consensus, and none of them targets data engineering.\n\nThe gap: there is no vendor-neutral model that (1) assigns decision rights between humans, agents and controls, (2) defines gates with evidence criteria, (3) segregates Build from Run, (4) governs autonomy explicitly, (5) defines metrics and maturity, and (6) treats master and reference data as the special case it is. ADDLC is an attempt to fill that gap.\n\n3. The bottleneck moved\n\nIn the classic SDLC the scarce resource was engineering time, and the lifecycle was organized to spend it well. In agentic data work, three other resources are scarce:\n\nJudgment about meaning. What exactly is an “active instrument”? Which source wins when two vendors disagree on a listing date? Is a negative trade size an error or a correction? These are decisions, not outputs, and an agent cannot make them on anyone’s behalf.\n\nVerification capacity. Agents produce changes faster than people can check them. Unverified change is not velocity ‐ it is deferred incidents.\n\nOperational attention. Every new asset is something somebody has to run. Generation without a path to operation moves cost from Build to Run, where it is more expensive.\n\nADDLC is organized to protect these three resources. Its operating rule:\n\nAgents propose. Controls enforce. Humans decide.\n\nDAMA-DMBOK defines data governance as “the exercise of authority and control (planning, monitoring, and enforcement) over the management of data assets” and keeps it separate from data management itself ‐ oversight apart from execution, which DMBOK compares to the separation of audit and accounting in finance. ADDLC applies that separation literally: governance is exercised by humans and enforced by deterministic controls, while management is increasingly executed by agents. The computational half of Zhamak Dehghani’s federated computational governance is exactly what “controls enforce” requires: policies as code, applied automatically, everywhere.\n\n4. Ten principles\n\nThe specification is the system of record; code is derived data. Martin Kleppmann distinguishes systems of record from derived data that can be recomputed from them. ADDLC applies the idea by analogy: the specification ‐ contract, semantics, quality expectations, acceptance examples ‐ is the system of record, and pipelines, DAGs and even tests are derived artifacts an agent can regenerate. When code and spec disagree, that is a defect: fix the code or formally revise the spec ‐ neither wins silently. Unlike derived data, generated code is not a deterministic function of its spec, which is why verification exists.\n\nMeaning is human-owned. Grain, keys, survivorship rules, metric definitions, glossary terms and quality thresholds are decisions with an accountable owner. Agents draft them, owners approve them, and nobody infers them at runtime.\n\nNo agent certifies its own work. The actor that built a change cannot be the only one that verifies it. Verification relies on independent evidence: tests derived from the spec, reconciliation against an independent source and ‐ where an agent verifies ‐ a separate agent that sees the spec and the output but not the builder’s reasoning.\n\nAutonomy is granted per change class, by reversibility and blast radius. “How autonomous is our agent?” is the wrong question. The right one is “which classes of change may be executed without a human, and why?”. A reversible documentation update and an irreversible change to golden-record matching rules deserve different answers, even from the same agent.\n\nContext is a product. Glossary, domain model, contracts, standards, agent instruction files and skills are what an agent knows about your organization. They need owners, versions, reviews and tests, like any other product. This is DMBOK’s metadata management with a new and very demanding consumer.\n\nEvery action leaves evidence. Every agent action is attributable (which agent identity, on whose behalf), explainable (which spec, which decision) and reconstructable (inputs, tool calls, outputs). Lineage extends from “where did this data come from” to “why did this change happen”.\n\nBuild and Run are segregated ‐ for humans and for agents. Build agents have no write access to production data. Run agents cannot change code or models. They use different identities, credentials and queues. It separates identities and permissions, not accountability: a team that builds and runs its own product still uses separate Build and Run identities, because non-human workers are otherwise happy to do everything.\n\nMaster-data changes are data operations, not deployments. Correcting a record, remapping an identifier or patching history goes through a stewarded change channel that is isolated, auditable and reversible. Nobody ‐ human or agent ‐ edits master data through a code path or an ad hoc SQL session.\n\nReuse before you generate. Before building, an agent must show that no existing asset already fulfills the need, or that extending one is worse than creating a new one. Otherwise near-free generation turns into permanent sprawl.\n\nMeasure flow and trust, not output. Lines of generated code and the number of pipelines created are vanity metrics. ADDLC measures lead time, human effort, failure rates, detection and verification yield ‐ per change class.\n\n5. The model at a glance: 3 actors, 3 loops, 7 phases, 4 gates\n\nThe figure at the top of the article shows the whole model on one page.\n\nThree actors\n\nHumans hold decision rights: intent, meaning, acceptance and exceptions. Typical roles are the data product owner, data steward, data engineer (increasingly an architect and supervisor of agents rather than an author of code), DataOps operator and consumer.\n\nAgents hold execution rights ‐ and only execution rights. They draft, build, test, document, triage and propose fixes. Build, verify and run agents are separate identities with separate permissions.\n\nControls hold enforcement: CI pipelines, contract validators, policy as code, access control, quality gates and promotion rules. Controls are deterministic. They do not use judgment, and they cannot be persuaded.\n\nThree loops\n\nIntent loop (business and governance): Frame and Specify. Humans lead, agents prepare.\n\nDelivery loop (the Build side): Build, Verify and Release. Agents lead, controls gate, humans accept.\n\nRun loop (the Run side): Operate, then Evolve and Retire. Operators decide, agents triage and prepare, controls watch.\n\nThe specification is the hinge between all three. Delivery cannot start without an approved spec, and Run reopens the spec when reality disagrees with it.\n\nFour gates: Intent, Meaning, Evidence, Operability.\n\nADDLC does not replace the data engineering lifecycle described by Joe Reis and Matt Housley ‐ generation, storage, ingestion, transformation, serving. It wraps it. Data flows through those stages inside Build and Operate, and the undercurrents ‐ security, data management, DataOps, architecture, orchestration, software engineering ‐ become context that agents consume and controls enforce.\n\n6. The seven phases\n\nFor each phase: what agents do, what humans decide, what controls enforce, and what the phase must leave behind.\n\nPhase 1 ‐ Frame\n\nPurpose: decide whether the work is worth doing, for whom, and how risky it is.\n\nAgents draft an intent brief from tickets, requests or call transcripts, find similar past requests and existing assets, and estimate effort and running cost.\n\nHumans confirm the consumer and the value hypothesis, name the accountable owner and assign a risk tier.\n\nControls enforce the intake template: no work item without an owner and a tier.\n\nThe risk tier combines data criticality, reversibility and blast radius. A one-off analytical extract is low. A new feed into a golden record that every product consumes is high. The tier sets the strictness of every later gate; the autonomy ceiling of the change class applies on top.\n\nPhase 2 ‐ Specify\n\nPurpose: agree on what the data means and what “good” looks like before any code exists.\n\nAgents profile candidate sources, search the catalog for reusable assets and write a reuse report, check licensing and entitlements, and draft the specification package: a data contract (for example in the Open Data Contract Standard), semantic definitions linked to glossary terms, the mapping to the canonical model, quality expectations with thresholds per dimension (completeness, validity, uniqueness, timeliness, consistency, accuracy), and acceptance examples ‐ concrete input-output cases that any correct implementation must reproduce.\n\nHumans ‐ the owner and the steward ‐ decide meaning: grain, keys, source precedence, survivorship, thresholds, what is out of scope. They resolve every ambiguity the agent escalates.\n\nControls validate the contract against its schema and check that every field is linked to a glossary term or explicitly marked local, and that every expectation has a threshold and a severity.\n\nExit artifact: approved specification package.\n\nGate: Meaning gate.\n\nThis is where most of the human time in ADDLC should go. The new code review is the spec review.\n\nPhase 3 ‐ Build\n\nPurpose: produce an implementation of the spec.\n\nAgents generate ingestion into Bronze, cleaning and conforming into Silver, and business and semantic models in Gold. Tests are derived from the spec in a separate context that never sees the implementation, acceptance examples run unchanged, and agents also generate documentation and lineage. They work in a sandbox on sampled, masked or synthetic data.\n\nHumans review architecture only where a deterministic template-conformance check flags a departure from a known pattern ‐ the agent does not get to decide that itself ‐ and answer escalations.\n\nControls encode standards as templates and agent skills, run linters and naming rules, apply policy as code (no personal data in Gold without classification, no new source without a contract) and keep production credentials out of the build environment.\n\nExit artifact: a change set with generated tests, documentation and lineage, linked to a spec version.\n\nCertify, don’t delete. Quality rules decide what is published, not what exists. Every conformed record stays one layer below Gold with its quality flags; Gold, and the agents that read it, get only what passed; failures go to the steward queue with evidence instead of disappearing.\n\nPhase 4 ‐ Verify\n\nPurpose: produce independent evidence that the implementation satisfies the spec.\n\nVerification combines lines of evidence that do not share the builder’s assumptions:\n\nDeterministic checks derived from the spec ‐ contract tests, schema checks, data tests such as dbt tests, acceptance examples.\n\nReconciliation against an independent source or a known invariant ‐ totals, counts, identities, temporal rules. Packkildurai calls this adversarial verification: schema, null and key tests can all pass while a join silently double-counts after historical entity merges.\n\nAn independent verifier agent with its own instructions and, ideally, a different model. It sees the spec and the output, never the builder’s reasoning, and tries to break the claim. This is the weakest line of evidence ‐ models share training data and blind spots ‐ so it supplements the other three and never replaces them.\n\nHuman acceptance on a sample by the consumer ‐ or, for master and reference data, the steward ‐ for medium and high risk tiers.\n\nExit artifact: evidence pack ‐ test results, coverage against the spec, reconciliation report, verifier findings, sample sign-off, cost of the run.\n\nGate: Evidence gate.\n\nPhase 5 ‐ Release\n\nPurpose: put the change into production safely and hand it over to the people who will run it.\n\nAgents prepare the promotion, backfill and rollback plans and the consumer notification, and assemble the handover package.\n\nHumans approve the release where the tier requires it. DataOps decides on operational acceptance.\n\nControls promote through environments via CI/CD, publish the versioned contract, perform an atomic cutover in Packkildurai’s sense (consumers are switched only if no contract changed during verification) and apply automatic rollback criteria.\n\nExit artifact: a released, versioned data product and an accepted handover package.\n\nGate: Operability gate.\n\nThe handover package is what makes Build and Run separable. It contains: ownership and escalation path; data flow; runbook; alerts mapped to each known failure mode and routed to the right queue; operational skills ‐ instructions that let run agents triage this product; operator access; quality rules and tests; contract, classification and license; expected capacity and cost.\n\nEarly-life support: the authors stay on call for 30 days or one full event cycle, whichever is longer. For market data, a cycle includes expirations, holidays, index rebalances and vendor maintenance windows ‐ many failure modes simply do not occur in a quiet month.\n\nPhase 6 ‐ Operate\n\nPurpose: keep the product within its commitments.\n\nAgents (read-mostly) watch the five observability pillars described by Barr Moses, Lior Gavish and Molly Vorwerck ‐ freshness, volume, distribution, schema, lineage. They deduplicate alerts, gather evidence, run root-cause analysis, classify incidents and draft fixes.\n\nHumans ‐ operators and stewards ‐ decide what the incident is and which channel fixes it.\n\nControls route alerts, enforce noise budgets and suppression rules with expiry dates, and guard the stewarded change channel.\n\nThe critical decision in Operate is classification, because each class has a different owner and a different fix path:\n\nDefect ‐ the implementation is wrong. It goes back to the Delivery loop as a change.\n\nData correction ‐ the code is right, the data is wrong (a vendor error, a late correction). It goes through the stewarded change channel.\n\nDecision ‐ a guard fired on a legitimate but unusual event, such as a mass change on a corporate-action day. It needs a human call, prepared by an agent with evidence.\n\nSpec gap ‐ reality produced a case the spec never described. It reopens Specify.\n\nIn feed operations the last two classes are easy to underestimate. Routed to developers as bugs, they get closed, not fixed.\n\nPhase 7 ‐ Evolve and Retire\n\nPurpose: keep the estate small, coherent and current.\n\nAgents detect overlapping assets and propose consolidation, propose contract versions for schema evolution, flag assets whose consumers and commitments have expired, and draft deprecation notices.\n\nHumans approve consolidation and retirement and set deprecation windows.\n\nControls enforce contract versioning, deprecation windows and consumer impact analysis from lineage.\n\nExit artifacts: consolidation and retirement decisions, new contract versions, context updates.\n\nAny change Evolve produces ‐ a consolidation, a new contract version ‐ re-enters the Delivery loop under Build identities. Run proposes; Build implements.\n\nPhase 7 also closes the learning loop. Every rejected gate, escaped defect and incident updates the context ‐ a skill, a standard, a glossary entry, a test template. If the same failure happens twice, the context failed, not the agent.\n\n7. The four gates\n\nA gate is passed with evidence, not with checkboxes. Each gate has exactly one accountable decider, a defined evidence pack, automated checks that run before any human looks, and explicit outcomes.\n\nGates need quality control of their own. A reviewer handed a polished, agent-assembled pack drifts toward rubber-stamping. Track approval rates and review depth per gate, and periodically re-review a sample of approved packs.\n\nIntent gate ‐ is this worth doing, who owns it, how risky is it?\n\nOutcomes: go; go with a smaller scope; reuse an existing asset; no.\n\nMeaning gate ‐ do we agree on what this data means and what “good” looks like?\n\nAccountable: data product owner. For master and reference data, the data steward’s approval is also required.\n\nEvidence: contract, semantic definitions linked to the glossary, canonical-model mapping, quality expectations with thresholds and severities, acceptance examples, licensing and entitlement check, reuse report.\n\nOutcomes: approved; approved with listed open questions that block release but not build; rejected.\n\nEvidence gate ‐ does the implementation demonstrably satisfy the spec?\n\nAccountable: for medium and high tiers, a release approver who did not author the change; for low tiers, the change-class owner ‐ the person accountable for that class of change. For classes at the Delegated level (section 8), controls enforce the gate automatically and the change-class owner stays accountable, auditing after the fact.\n\nEvidence: the evidence pack from Verify.\n\nOutcomes: pass; pass with dated exceptions; fail ‐ back to Build with findings.\n\nOperability gate ‐ can the Run side operate this without its authors?\n\nAccountable: DataOps lead.\n\nEvidence: the handover package plus an operational test ‐ replay historical incidents, fire a test alert end to end, walk through the access scenario, and let a run agent triage a seeded failure using only the provided skills.\n\nOutcomes: accepted; accepted with dated gaps; accepted for specific failure classes only; not accepted.\n\nOne rule makes the Operability gate work: if the Run side does not accept, the Build side keeps the pager.\n\nMinimum viable ADDLC. For low-risk change classes the lifecycle collapses to a spec template, spec-derived tests, an automated Evidence gate and a named change-class owner ‐ minutes, not meetings. If the gates cost more human time than the work they protect, the risk tiering is wrong. Ownership follows the same split as decision rights: whoever owns data governance ‐ usually the CDO ‐ owns the autonomy policy, change-class owners sit in the domains, and DataOps owns the Operability gate.\n\nFitting into what you already run. Gates ride on existing mechanics: the Meaning gate on a spec pull request with the steward as code owner, the Evidence gate on required CI checks, the Operability gate on the production-readiness review. Roles are hats, not headcount, and the platform team owns the controls and the agent identities.\n\n8. Autonomy levels\n\nADDLC borrows the shape of SAE J3016 (the levels of driving automation) and defines six autonomy levels. A level is assigned to a change class, never to an agent as a whole.\n\nManual. No agent involvement.\n\nAssist. The agent suggests; a human authors.\n\nDraft. The agent authors; a human reviews every change before it merges.\n\nSupervised. The agent executes the change class end to end; humans decide at gates and sample-review the internals.\n\nDelegated. The agent releases changes in the class without a human; controls enforce; humans audit after the fact; rollback is automatic.\n\nSelf-organizing. Agents propose and reorganize the asset graph within policy, in the spirit of Packkildurai’s engine. Humans own meaning and policy only.\n\nThe ceiling for a change class is set by three factors: reversibility (can we undo it completely, and how fast?), blast radius (how many consumers can it affect?) and data criticality. The Thoughtworks article reaches a similar conclusion through reversibility classes and staged autonomy.\n\nExample ceilings:\n\nDescriptions and documentation on Bronze and Silver: Delegated.\n\nDescriptions of Gold and glossary-linked fields: Supervised ‐ that is meaning.\n\nIncident triage and evidence gathering, read-only: Delegated.\n\nNew Bronze ingestion from a known source type and template: Supervised.\n\nNew or changed Silver transformation: Supervised.\n\nChange to a certified Gold metric definition: Draft.\n\nMatching, survivorship or identifier-mapping rules in master data: Draft, as a hard ceiling.\n\nCorrection of master records in production: Assist, through the stewarded change channel ‐ agents prepare, humans approve.\n\nChanges to the controls themselves ‐ CI rules, test thresholds, policy as code, agent permissions: Assist. An agent that can edit its own gates has no gates.\n\nAutonomy is earned with evidence and lost with escapes. A change class moves up one level only after a run of consecutive changes that passed every gate first time with no escaped defects. The bar scales with risk: 20 clean changes bound the failure rate only to about 15%, which is acceptable for documentation and far too weak for identity rules. Any escaped defect above an agreed severity demotes the class by one level automatically, and the count restarts.\n\n9. Master and reference data: where ADDLC gets strict\n\nAnalytics teams can often afford a looser version of this lifecycle. Teams that own master and reference data cannot, for three reasons:\n\nFan-out. A reference record is read by every product. A wrong instrument classification is wrong everywhere at once.\n\nSilent failure. Mapping errors pass every structural test. The row is valid; it is attached to the wrong entity.\n\nIdentity is semantic. Tickers are reused, instruments are renamed, corporate actions merge and split entities, and vendors disagree on listing dates. Identity resolution is a stream of judgments, not a join.\n\nADDLC adds four requirements for this domain:\n\nMatching and survivorship rules are meaning. They are decided at the Meaning gate and capped at Draft. Agents propose candidate matches with confidence scores; matches above the agreed threshold are applied only under rules a steward approved; everything below goes to the steward queue with evidence. Candidate matches stay in the layer below Gold until survivorship produces the golden record.\n\nA stewarded change channel. Corrections and patches are data operations. They are isolated from code deployment, validated by the same contracts as the pipeline, attributable (who, what, why, on whose request), reversible (versioned or bitemporal storage, so any correction can be undone and any past state reconstructed), and subject to four-eyes approval for high-criticality entities. Integrators and operators ‐ and their agents ‐ cannot unilaterally break master data.\n\nEntitlements travel with the data. License terms and entitlements are metadata that agents must read and propagate. Moving licensed data to a new consumer is a Meaning-gate decision.\n\nReconciliation is mandatory in Verify. Every release that touches identity is reconciled against at least one independent source before cutover.\n\nA hypothetical walkthrough\n\nOnboarding a new exchange’s instrument reference feed ‐ a composite scenario, not a case study:\n\nFrame. Consumers and value are clear. The risk tier is high: the feed populates the canonical instrument model.\n\nSpecify. The agent profiles the feed, maps vendor fields to the canonical instrument model, flags the attributes it cannot map with confidence, and drafts expectations (for example, every active instrument has exactly one primary listing) and acceptance examples from well-known instruments. The steward resolves the ambiguities and approves the matching rules.\n\nBuild. The agent generates Bronze ingestion from the standard vendor template, Silver conformance and matching, and Gold publication, with tests derived from the spec.\n\nVerify. Deterministic tests run; instrument counts and identities are reconciled against an independent source; a verifier agent hunts for instruments matched to two entities; the steward signs off a sample.\n\nRelease. A shadow run alongside the old path, then an atomic cutover. The handover package includes a run skill for the typical failure classes of this feed type.\n\nOperate. On the first corporate-action day a mass-change guard fires. The run agent classifies it as a decision and attaches evidence, and the steward approves the change through the stewarded channel ‐ no developer is paged.\n\n10. Risk, regulation and licensing\n\nFor regulated firms, most of ADDLC maps onto controls they already run:\n\nSegregating Build from Run is segregation of duties, extended to non-human identities.\n\nThe stewarded change channel and bitemporal history are what accuracy and integrity requirements for critical data, such as BCBS 239 for banks, expect.\n\nEvidence packs and decision records are the audit trail an examiner will ask for. In the EU, the AI Act’s logging requirements for high-risk systems point the same way.\n\nYour model provider is an ICT third party ‐ for EU financial entities, under the Digital Operational Resilience Act (DORA ‐ the EU regulation, not the DevOps metrics) ‐ and belongs in third-party risk management like any other critical vendor.\n\nMarket data needs one more warning. A model provider that processes licensed data is itself a consumer of that data. Exchange and vendor licenses often restrict redistribution and derived data, so sending licensed fields to a third-party model can breach them. Check licenses at the Meaning gate before any licensed field reaches a model outside your perimeter, and prefer deployments where the provider neither retains nor trains on your inputs.\n\n11. Metrics\n\nAccelerate gave software delivery four numbers everyone could compare. ADDLC proposes four families, always reported per change class and autonomy level, against a baseline taken before agents were introduced.\n\nFlow\n\nLead time for data: from the Intent gate to data in production, broken down by phase, so you can see where time goes.\n\nHuman effort per change: hours by role ‐ owner, steward, engineer, operator. This is where the economic case has to be proven: more integrations from the same team, at an equal or lower change failure rate.\n\nThroughput: changes or integrations released per period, per team.\n\nStability and trust\n\nChange failure rate: the share of releases that cause an incident, rollback or correction.\n\nData downtime, as defined in Data Quality Fundamentals: number of incidents × (time to detection + time to resolution).\n\nConsumer-detected share: the share of incidents first reported by consumers rather than by monitoring. If users find your incidents, Operate is not working, however green the dashboards look.\n\nEscaped defects per gate: defects found after a gate that the gate should have caught.\n\nAgent effectiveness\n\nFirst-pass yield per gate: the share of agent-produced packages that pass the Meaning, Evidence or Operability gate without rework.\n\nHuman review minutes per change ‐ read together with escaped defects: minutes trending to zero with a flat defect rate is rubber-stamping, not efficiency.\n\nEscalation rate and quality: did the agent ask the right questions, with evidence?\n\nCost per change: tokens plus compute plus human time.\n\nContext health\n\nCoverage: the share of assets with an owner, a contract and glossary-linked semantics.\n\nFreshness: the age of skills and standards relative to the last relevant incident.\n\nLearning rate: the share of incidents and rejected gates that produced a context update.\n\nGenerated lines of code, pipelines created and “share of code written by AI” are not success metrics. They reward sprawl.\n\n12. Maturity model\n\nAd hoc. Individuals use coding agents on their own initiative. No shared context, no agent identities, no evidence. Agent output is indistinguishable from human output ‐ including its mistakes.\n\nAssisted. Shared context exists: agent instruction files, standards, templates. Agents draft code and documentation; humans review everything. Gates are informal.\n\nSpec-driven. Specifications and contracts are the system of record. All four gates are defined, with evidence packs. Build and run agents have separate identities. Baseline metrics exist.\n\nGoverned autonomy. Autonomy is set per change class and moves with evidence. Run agents triage incidents. Master-data changes go only through the stewarded channel. Context is owned, versioned and tested. Metrics drive autonomy decisions.\n\nSelf-organizing. An engine maintains the asset graph ‐ synthesizing, consolidating, repairing ‐ within policy. Humans own meaning and policy.\n\nFor most organizations, and for every master-data domain, Governed autonomy is the right target. Self-organizing is optional, and some change classes should never go beyond Draft, whatever the maturity level.\n\n13. Anti-patterns\n\nPrompt-and-pray pipelines. Code generated from a chat message, with no spec to verify against. If nothing defines correct, nothing can be verified.\n\nGeneration sprawl. A new asset for every request, because generating is cheaper than understanding. Fix: reuse before you generate, with a reuse search at the Intent gate and a reuse report at the Meaning gate.\n\nSelf-certification. The same agent, in the same context, writes the code and the tests. Fix: tests derived from the spec, an independent verifier, reconciliation.\n\nDocumentation slop. Long, generated, unreviewed text. It costs readers more time than it saves writers and teaches people to ignore documentation altogether. Fix: generated documentation goes through the gate of its change class, and shorter wins.\n\nShadow autonomy. Agents running on a person’s production credentials. Fix: separate agent identities with scoped, short-lived credentials. Be especially strict when an agent reads untrusted content ‐ vendor emails, web pages, tickets ‐ while also having access to private data and the ability to communicate externally. Simon Willison calls this combination the lethal trifecta.\n\nDual authority. A generator regenerates an artifact that humans also edit by hand, and the next run silently erases their edits. Fix: exactly one authoritative source per artifact ‐ the trusted-source principle DMBOK applies to master and reference data, extended to every artifact an agent touches.\n\nContext rot. Instruction files written once and never maintained. Fix: context has owners and is updated by the learning loop.\n\nAutonomy by enthusiasm. Granting the Delegated level because a demo looked good. Fix: earned autonomy (section 8).\n\n14. How ADDLC maps to the canon\n\nMost of ADDLC is existing practice, reassigned to non-human executors:\n\nDAMA-DMBOK. Data governance becomes gates and decision rights. Metadata management becomes the context layer. Data quality runs through Specify, Verify and Operate. Reference and master data management gets the stewarded change channel. Data integration and interoperability lives in Specify and Build. Data security covers agent identities and entitlements. Data modeling and design provides the domain and semantic models. Data storage and operations is the Run loop.\n\nData Mesh (Zhamak Dehghani). Data as a product: the handover package is what makes a product operable by someone other than its authors. Domain ownership: owners decide at the Intent and Meaning gates. Self-serve platform: agents with standard skills become the platform’s interface. Federated computational governance: “controls enforce”.\n\nData Management at Scale (Piethein Strengholt). Metadata-driven automation, data contracts and domain interfaces ‐ the preconditions for letting agents execute safely at scale.\n\nFundamentals of Data Engineering (Joe Reis, Matt Housley). The data engineering lifecycle runs inside Build and Operate, and its undercurrents become agent context and controls.\n\nData Quality Fundamentals (Barr Moses, Lior Gavish, Molly Vorwerck). The observability pillars and data downtime anchor Operate and the stability metrics.\n\nDesigning Data-Intensive Applications (Martin Kleppmann). Systems of record versus derived data inspire the first principle; immutability and replayability make verification, rollback and reversible corrections possible.\n\nISO standards. ADDLC is not an ISO standard, but it is meant to sit alongside them: ISO/IEC 5338 defines life cycle processes for AI systems, ISO/IEC 42001 the management system around AI, the ISO/IEC 5259 series data quality for analytics and machine learning, and ISO 8000 data quality and master data. ADDLC covers what they leave open ‐ how data products are built and run when agents do most of the execution.\n\nCurrent building blocks that fit ADDLC well: the Open Data Contract Standard for contracts, OpenLineage for data lineage, OpenTelemetry-based tracing for agent actions, semantic layers such as dbt MetricFlow, the Model Context Protocol for governed agent access to tools, spec-driven development tooling such as GitHub Spec Kit, and AWS AI-DLC as an analogous lifecycle on the software side.\n\n15. Getting started: the first 90 days\n\nADDLC is meant to be adopted incrementally. A pragmatic sequence:\n\nWeeks 1‐2: baseline and inventory. Measure today’s lead time, human effort per integration and incident profile. Find out where agents are already in use, and with whose credentials.\n\nWeeks 3‐6: context and identities. Write agent instruction files and encode standards as skills, link the glossary, give agents their own identities, and introduce the spec template and the Meaning gate.\n\nWeeks 7‐10: one change class, full lifecycle. Pick a frequent, bounded class ‐ for example, new ingestion from a known source type ‐ and run it at Draft or Supervised level through all four gates. Shape the evidence pack on real changes.\n\nWeeks 11‐13: handover and decision. Run the Operability gate with an operational test and run-agent skills. Compare the metrics with the baseline. Decide, with evidence, whether to promote the class and which class comes next.\n\nKeep the Meaning gate strict and everything else light at first. Most of the value of ADDLC comes from agreeing on meaning before generating code.\n\n16. What ADDLC is not\n\nNot a tool or a platform. Any stack can implement it.\n\nNot a replacement for SDLC, DataOps or data governance. It is a profile of them for agentic execution.\n\nNot an argument for full autonomy. It is a way to decide, one change class at a time, how much autonomy is justified.\n\nNot only for analytics. It applies to ingestion, integration, master and reference data, and data operations. Low-latency streaming paths need adapted gates ‐ shadow runs instead of backfills, for example ‐ but the decision rights stay the same.\n\n17. Help shape it\n\nThe specification lives at github.com/savvakrasavasavva/addlc: 67 numbered requirements, three conformance levels, the MRD profile, templates and a self-assessment. Disagreement backed by practice is the most useful contribution: open an issue with what you ran, what broke and what you would change.\n\nADDLC grew out of practical work on market-data reference and master data. The early results above come from one platform; whether the model holds elsewhere is exactly what the metrics in section 11 are for. The model will be wrong in places, and the fastest way to improve it is disagreement backed by practice.\n\nI am particularly interested in:\n\nchange classes and autonomy ceilings from your domain;\n\ngate criteria that worked, or failed, in real teams;\n\nmetric baselines from before and after agents;\n\nwhere the model breaks for streaming, unstructured data or ML features.\n\nLeave a response here. Material improvements will be folded into the next revision.\n\nAcceptance examples cover normal, edge and known-bad cases\n\nLicense and entitlement check done\n\nOpen questions listed, each with an owner\n\nEvidence gate\n\nAll spec-derived tests pass; coverage of expectations reported\n\nReconciliation against an independent source or invariant\n\nIndependent verifier findings resolved or explicitly accepted\n\nConsumer (or steward) sample sign-off for medium and high tiers\n\nCost of build and verification recorded\n\nOperability gate\n\nOwner, escalation path and on-call defined\n\nRunbook covers each known failure class\n\nAlerts fire end to end into the correct queue; noise within budget\n\nRun-agent skills triage a seeded failure correctly\n\nOperator access verified\n\nRollback plan tested\n\nEarly-life support window agreed\n\nAppendix B. Self-assessment\n\nAnswer yes or no. Your level is the highest one for which every answer at that level and below is yes. If any Assisted answer is no, you are at Ad hoc.\n\nCravero, A., Díaz-Villegas, J., Xiao, Z. “From Data Quality to Quality of Agentic Data Use: A Conceptual Framework for Agentic Data Engineering”. Applied Sciences 16(17), 8887, 2026. https://doi.org/10.3390/app16178887\n\nISO/IEC 42001:2023. Information technology ‐ Artificial intelligence ‐ Management system.\n\nISO/IEC 5259 series. Artificial intelligence ‐ Data quality for analytics and machine learning (ML).\n\nISO 8000 series. Data quality.\n\nI lead data management ‐ data quality, ops, stewardship and integration. ADDLC is a personal proposal: examples are composite, results are rounded and relative, and the views are mine.", "url": "https://wpnews.pro/news/the-agentic-data-development-lifecycle-addlc", "canonical_source": "https://pub.towardsai.net/agentic-data-development-lifecycle-addlc-d40122a46e2c?source=rss----98111c9905da---4", "published_at": "2026-11-11 01:54:06+00:00", "updated_at": "2026-11-11 02:21:26.557030+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "mlops", "ai-tools"], "entities": ["Agentic Data Development Lifecycle (ADDLC)", "DAMA-DMBOK", "Data Mesh", "Data Management at Scale", "Mikkel Dengsøe", "Coalesce", "DevOps Research and Assessment (DORA)", "Accelerate"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-agentic-data-development-lifecycle-addlc", "markdown": "https://wpnews.pro/news/the-agentic-data-development-lifecycle-addlc.md", "text": "https://wpnews.pro/news/the-agentic-data-development-lifecycle-addlc.txt", "jsonld": "https://wpnews.pro/news/the-agentic-data-development-lifecycle-addlc.jsonld"}}