Your model upgrade is a change. Is it treated like one?

Part 3 of 3: The SDLC reimagined for the AI era

#

TL;DR

• Behaviour in AI systems changes with the model, the prompt, the retrieval data, the tools and how the model is served, some of it on the vendor's calendar.
• Treat each as a change with change control, and replace testing with evaluation.
• Decision for the next leadership agenda: list what your AI systems can do, rank it by how hard it is to undo, and decide where the model stops deciding. Enforcing that needs a gateway, a policy layer outside the model, and evaluation in the pipeline.

#

A change nobody ticketed

Part 1 argued that AI breaks the software lifecycle from two sides. Part 2 covered the first: AI writing the code. This part covers the second: software that contains AI, where behaviour can change without a single line of code changing. It was also the question readers raised most after Part 1: what happens when the model changes and the code doesn't?

"An endpoint that still responds doesn't mean your app still behaves correctly." That line comes from Microsoft's own guide to moving between models. On Azure, pay-as-you-go ("Standard") deployments are moved to a new model version automatically when the old one retires, unless you opt out. If you opt out, the deployment stops working at retirement, and retirement dates "aren't extendable".

How does a change like that reach production in your organisation: with a ticket, a test run and an approval?

#

Behaviour changes in more places than code

In a system built on a model, behaviour can change in at least five places besides the code:

the model , when a vendor retires a version or an alias moves; #

the prompt and context , where, as the FSB notes in itsOctober report , small changes in wording can significantly change the output; #

the retrieval data , which can shift behaviour when the index behind it is refreshed; #

the tools and permissions the model can use; #

the model as served , which can degrade throughinfrastructure bugs with no model change at all, as Anthropic found in 2025.

Microsoft spells out what a model swap can do: "a different model can quietly shift tone, formatting, JSON shape, latency, cost, or tool-calling behavior". In an integrated estate that drift doesn't stay local. A downstream service can receive messages that still pass schema checks but now mean something different, which contract tests built for deterministic systems may not catch. That is why the deterministic gates in Part 2 are necessary here but not enough. Germany's financial supervisor, BaFin, names "unannounced model changes" by third-party vendors as a testing challenge in its guidance on ICT risks in AI.

Few teams treat them that way yet. In a study of open-source repositories, only about one in five prompt changes was documented in the commit message. Every one of these changes needs the treatment a code change gets.

#

Testing becomes evaluation

A unit test checks that the same input gives the same output. With a model it often won't, so you evaluate behaviour across a set of cases instead.

Freeze the test set and the success criteria before the change. Microsoft's guide says: "Freeze the dataset. Keep the inputs, ground truths, and success criteria fixed for the whole migration." It also says that "Most teams don't have an evaluation suite". For high-risk systems, the EU AI Act asks for testing "against prior defined metrics and probabilistic thresholds".

Count confident wrong answers, not just accuracy. On one benchmark in OpenAI's own example, two of its models scored 22% and 24% accuracy. The first gave wrong answers on 26% of questions and declined on 52%. The second was wrong on 75% and declined on 1%. They are different models, but the table shows what accuracy hides. In OpenAI's words, "errors are worse than abstentions".

Keep the grader independent, for the same reason Part 2 gave for AI reviewers. I hold my own work to the same rule. In Signal Radar, the platform I designed and built with Claude Code, a blind-graded evaluation harness decides which model does which job. On TraceGov, the Quantamix Solutions platform where I am technical architect, we benchmarked candidate models on blind-labelled cases, so the grader did not know which model produced an answer.

Accept that evaluation won't catch everything. In April 2025 OpenAI rolled out a GPT-4o update that made ChatGPT noticeably more sycophantic; its offline evaluations "generally looked good", and it was later rolled back. Anthropic's evaluations missed its serving bugs. When you can't find the line of code to fix, root cause moves up a level: the missing evaluation, the unannounced change, the permission the system should never have had. OpenAI's lasting fix was a process change: behaviour issues became "blocking concerns" "even when metrics like A/B testing look good". That is why evaluation has to be paired with limits on what the system may do.

#

Decide by irreversibility

This is the part I would put in front of any CIO, and my answer to a question readers asked after the first two parts: how do you decide what still needs a person? Rank every action an AI system can take by how hard it is to undo. Reading data and drafting a recommendation are easy to undo. Changing something you can roll back is harder. Changing the system of record, or sending something outside the organisation, may not be undoable at all. Judge it by what a change reaches, not only what it touches. As a reader put it on X, a bad backfill looks reversible right up until someone screenshots the dashboard. A rollback restores the table, not the decisions people made from the dashboard in the meantime.

That ranking is the action ladder, and the model's authority should shrink as you go up it. It may read what its permissions allow and draft freely. A change that can be rolled back needs an approved draft first, under the kind of rules Part 2 describes. A write to the system of record is never the model's decision.

Three rules follow.

Agents never approve a write to the system of record, and whoever approves is never the author. (Part 2 allows a measured AI reviewer to approve routine code changes; this rule is about actions on the system of record.) #

Checks fail closed and can't be traded off against each other. One failed check stops the action, however good the others look. OpenAI shipped its sycophantic update on positive user signals although expert testers had said it felt off: a strong aggregate signal outweighed a qualitative warning. That is my reading of the case, not OpenAI's. Meta'sAgents Rule of Two is a hard limit of the same kind, though it is triggered by untrusted input rather than irreversibility.

• When the evidence is thin, the system asks instead of guessing.

OWASP and two vendors use the same trigger for human confirmation: whether an action can be undone. OWASP's Top 10 for Agentic Applications asks for "human confirmation for high-impact or destructive actions (delete, transfer, publish)". OpenAI's guidance for computer use says to "Keep users in control of purchases, data transmission, destructive changes, and other actions that are hard to reverse." Meta's Muse Spark Open model card, published on 15 October, recommends "human-in-the-loop confirmation for irreversible actions".

Approval follows irreversibility, not volume.

#

What the architecture has to provide

None of this holds if each team implements it differently. These are architecture decisions, not tooling choices left to each team.

One gateway for model access. Put every model call behind a gateway. In my design it handles version pins and aliases, fallback, canary routing, and budget and latency caps in one place, instead of in each service. Signal Radar has the same shape in miniature: a model router with fallback and a budget cap. #

Enforcement outside the model. The action ladder only holds if something other than the model enforces it. OWASP's agentic guidance asks for an external policy engine separating planning from execution, and for blast-radius limits such as quotas and circuit breakers. My translation for the ladder: a policy layer or proxy that blocks state-changing calls to the system of record unless an approval is attached. I have not built that layer for an agentic system; it is the design I would require. #

Pinned versions with known expiry. Pin models, prompts and tool definitions so you can roll back. Keep the pins together in the release artifact, so one rollback restores all of them. Pinning only covers the changes you choose: a serving bug like Anthropic's arrives with no version bump, which is why the next item keeps evaluating live traffic. A pinned model still retires on the vendor's date, so the rollback corridor Microsoft's guide asks you to preserve has an end. #

Evaluation in the pipeline and in release. Run the evaluation suite on every change to a model, prompt, retrieval index or tool. Release in stages, and where regulated flows cannot expose real customers, run the new model in shadow mode on production inputs first, as Microsoft's guide recommends. Then keep evaluating a sample of production traffic. OpenAI'sguidance is to "run evals on every change", and Microsoft's migration guide says to "wire up continuous evaluation to score a sampled percentage of production traffic". #

Versioned retrieval. Version the index and its chunking settings like code, and re-run the evaluation suite when either changes. #

A record of what is running. Formats such asCycloneDX's ML-BOM , which records "models, datasets, and their dependencies", give models the kind of inventory a software bill of materials gives code. Tie each evaluation result to what is actually deployed, not only to the commit. In Part 2's TraceGov incident, review and tests passed on a commit while the deployed package held an old library. A reader on X suggested binding the verdict to the exact build that ships. For an AI system, that build includes the model version, the prompt and the index. #

Tracing on every call. Trace prompts, token counts, latency and tool calls using theOpenTelemetry conventions for generative AI , still marked as in development, so AI calls land in the monitoring you already run. On TraceGov we audited every LLM call in production; afterwards a routine summariser moved to a cheaper model and a score the model had been producing moved into code. I wrote about why this matters inYou can't govern AI agents you can't see .

#

Who owns what

Accountability does not move to the model or the vendor. When Air Canada argued in effect that its chatbot was responsible for its own actions, a British Columbia small-claims tribunal rejected that in 2024: "It makes no difference whether the information comes from a static page or a chatbot."

I would split ownership like this:

Product owners own the evaluation criteria and the cases that matter. #

Risk and compliance own the action ladder: which actions need which checks. #

Engineers own versions, rollback and observability. #

Architects own the decisions in the previous section, so that a model swap at one supplier goes through the same controls as a code change.

Don't assume your model risk function already covers this. Extending it to models you didn't train is a decision each organisation has to make deliberately.

#

What regulators already say

In the US, the interagency guidance of April 2026 on model risk management is non-binding and says in a footnote that generative and agentic AI models "are not within the scope of this guidance", leaving the controls to each bank's own risk management. In Europe, ECB supervisor Pedro Machado said in February that "AI does not dilute responsibility. If anything, it raises the bar."

In financial services, the EU Digital Operational Resilience Act requires change-management procedures "to ensure that all changes to ICT systems are recorded, tested, assessed, approved, implemented and verified in a controlled manner". BaFin reads that onto AI in its non-binding guidance of January 2026: "Any changes to AI systems (whether to the software or hardware) are generally subject to strict change management". That a prompt edit counts is my reading; no text says so explicitly.

The EU AI Act adds duties for high-risk systems, including AI used "to evaluate the creditworthiness of natural persons or establish their credit score" (fraud detection excluded). After this year's amendments, the high-risk regime for these Annex III systems applies from 2 December 2027. Providers need "procedures for the management of modifications", and a substantial modification means a new conformity assessment. A bank that builds its own scoring model is a provider, and the shortcut for financial institutions does not cover its risk management system, post-market monitoring or serious-incident reporting.

The limit: in an ECB workshop series with 13 banks, self-reported and not a supervisory assessment, "No banks reported using GenAI in the context of credit scoring". For most generative AI in banks today, the nearest binding text is the change management required by the EU Digital Operational Resilience Act, which BaFin reads onto AI, and the behavioural testing comes from the bank's own model risk management.

#

Where I would start

1. Put model, prompt, retrieval and tool changes through change management, using the same risk tiers as code changes in Part 2, with versions pinned and their expiry dates tracked.
2. Build a frozen evaluation set per use case, with success criteria written before the run, and count confident wrong answers.
3. Shadow-run every model change on production inputs before exposure, and keep the old version available while you can.
4. Put model access behind one gateway and the action ladder in a policy layer outside the model, with checks that fail closed.
5. Trace every call, review outcomes, and feed what you learn back into the ladder.

#

The decision for the agenda

If I were a CIO, I would put one decision on the next leadership agenda: list every action your AI systems can take, rank them by how hard they are to undo, and decide where the model stops deciding. Then put model and prompt changes through change management before a vendor's calendar does it for you. That is where the series ends up. In Part 2 the answer was to place each check in the right layer. Here it is to treat every change to behaviour as a change, and let a model's authority shrink as its actions get harder to undo. Both come back to the point of Part 1: the bottleneck has moved from building software to deciding what may change.

Where does the model stop deciding in your organisation today? I'd like to hear in the comments.

Thanks to Harish Kumar from Quantamix Solutions for the conversations that shaped the principles on irreversibility and independent checks.

Researched and drafted with the help of AI tools. The views and experience are mine, and every source was checked.

First published on LinkedIn.

#

Sources

1. Anthropic. "A postmortem of three recent issues." 17 September 2025. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues
2. BaFin. Guidance on ICT risks in the use of AI (English version), January 2026. https://www.bafin.de/SharedDocs/Downloads/EN/Anlage/dl_Anlage_orientierungshilfe_IKT_Risiken_bei_KI_en.pdf?__blob=publicationFile&v=1
3. Civil Resolution Tribunal (British Columbia). Moffatt v. Air Canada, 2024 BCCRT 149, 14 February 2024. https://decisions.civilresolutionbc.ca/crt/crtd/en/item/525448/index.do
4. CycloneDX. "Machine Learning Bill of Materials (ML-BOM)." https://cyclonedx.org/capabilities/mlbom/
5. ECB Banking Supervision. Annex to the supervisory newsletter of 20 November 2025 on AI workshops. https://www.bankingsupervision.europa.eu/ecb/pub/pdf/annex/ssm.nl251120_1_annex.en.pdf
6. ECB Banking Supervision. Speech by Pedro Machado, KPMG RiskTech Conference, 24 February 2026. https://www.bankingsupervision.europa.eu/press/speeches/date/2026/html/ssm.sp260224~6c5b64a77a.en.html
7. Federal Reserve. SR 26-2, "Revised Guidance on Model Risk Management", attachment, 17 April 2026. https://www.federalreserve.gov/supervisionreg/srletters/SR2602a1.pdf
8. Financial Stability Board. "Sound Practices for Responsible Adoption of Artificial Intelligence (AI): Final Report." 15 October 2026. https://www.fsb.org/2026/10/sound-practices-for-responsible-adoption-of-artificial-intelligence-ai-final-report/
9. Meta AI. "Agents Rule of Two: A Practical Approach to AI Agent Security." 31 October 2025. https://ai.meta.com/blog/practical-ai-agent-security/
10. Meta. Muse Spark Open model card. Hugging Face, 15 October 2026. https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B
11. Microsoft Learn. "Foundry Models lifecycle and support policy." July 2026. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirements
12. Microsoft Learn. "Model migration: upgrade or switch models in Microsoft Foundry." August 2026. https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/model-migration

13. OpenAI. "Computer use" guide. [https://developers.openai.com/api/docs/guides/tools-computer-use](https://developers.openai.com/api/docs/guides/tools-computer-use)
14. OpenAI. "Evaluation best practices." [https://developers.openai.com/api/docs/guides/evaluation-best-practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)

15. OpenAI. "Expanding on what we missed with sycophancy." Update rolled out 25 April 2025; post date not shown on the page. https://openai.com/index/expanding-on-sycophancy/
16. OpenAI. "Why language models hallucinate." September 2025. https://openai.com/index/why-language-models-hallucinate/
17. OpenTelemetry. Semantic conventions for generative AI (status: Development). https://github.com/open-telemetry/semantic-conventions-genai
18. OWASP GenAI Security Project. "OWASP Top 10 for Agentic Applications for 2026." 9 December 2025. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
19. Regulation (EU) 2022/2554 (EU Digital Operational Resilience Act), Article 9(4)(e). https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32022R2554
20. Regulation (EU) 2024/1689 (EU AI Act), as amended by Regulation (EU) 2026/1744. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
21. Shetty, J. "You can't govern AI agents you can't see." LinkedIn, 12 October 2026. https://www.linkedin.com/pulse/you-cant-govern-ai-agents-see-jay-shetty-907qe/
22. Tafreshipour et al. "Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories." MSR 2025. https://arxiv.org/abs/2412.17298

source & further reading

dev.to — original article Building a Private AI Control Plane for PS5 Porting: What We Learned After SQLite Locked Us Out at 2 AM How to Ship a PoC Fast Without Getting Burned Later How to Compare Voice AI Agents: a 16-Scenario Acceptance Test