Build the Reverse Gear First: How to Ship AI Features You Can Take Back

A developer argues that AI features should be built with reversibility first, citing Google Cloud's 2025 DORA research (nearly 5,000 respondents, 90% using AI at work) showing AI adoption now correlates positively with delivery throughput but still negatively with delivery stability. The piece points to OpenAI's April 2025 GPT-4o sycophancy incident, where a rollback took about 24 hours, as evidence that even model trainers need a dependable way to back out. It recommends pinning dated model snapshots, versioning model and prompt as a unit, and treating embeddings as a cache rather than data.

Nobody buys a car for its reverse gear, yet nobody would drive one off the lot without it. Plenty of AI features ship in exactly that condition: quick to build, impressive in a demo, and with no dependable way to back out when a model update, a stale index or an over-eager agent takes them somewhere they shouldn't go. An essay on why the best businesses now build for change before they build for scale https://myliberla.com/the-premium-on-reversibility-why-the-best-businesses-now-build-for-change-before-they-build-for-scale/ makes the case in balance-sheet terms, arguing that companies are hurt less by one dramatic failure than by commitments that slowly become hard to unwind. Swap leases and long-dated debt for embedding models, model aliases and agent permissions, and you get an uncomfortably accurate picture of how AI products get stuck. The economics of a code change have flipped. Google Cloud's 2025 DORA research on AI-assisted software development https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report , based on survey responses from nearly 5,000 technology professionals, found that 90% of respondents use AI at work and that AI adoption now has a positive relationship with delivery throughput, a reversal of the previous year's result. The same research found that adoption still has a negative relationship with delivery stability. In plain terms, teams ship more, and more of what they ship breaks. When a change takes minutes to write, its real price is the cost of being wrong about it, and that cost depends almost entirely on how fast you can take the change back. A team that can revert a bad prompt in thirty seconds can run ten experiments a day. A team that needs a week to unwind a bad index will, quite sensibly, run almost none. Reversibility is what turns cheap code into cheap experiments. In late April 2025, OpenAI rolled out a GPT-4o update in ChatGPT that made the model noticeably more sycophantic. Within two days it was clear the behavior was off. A system prompt patch went out late on a Sunday, and a full rollback to the previous version began the next day. In OpenAI's postmortem on what it missed https://openai.com/index/expanding-on-sycophancy/ , the company explains that offline evaluations and A/B tests had looked good, that some expert testers felt the model was slightly off before launch, and that the rollback itself took about 24 hours so it wouldn't introduce new problems along the way. Two things stand out for anyone building on a hosted model. Even the team that trains the model, with offline evals, A/B tests and expert reviewers in place, needed a reverse gear. And that reverse gear existed only because the previous version was still there, ready to serve traffic. If your product calls a floating model alias, the model behind it can change on the provider's schedule rather than yours. So pin dated model snapshots wherever your provider offers them. Version the model and the prompt as a single unit, because a prompt tuned for one model is effectively a different program on another. Keep the previous pair deployable for a few weeks after every upgrade, and send a slice of traffic to the new pair before everyone gets it. Pinning won't last forever, since providers retire old snapshots on their own timeline, but it turns a surprise into a migration you planned. Choosing an embedding model feels like setting a config value. It is closer to choosing a file format for your entire knowledge base. Vectors from different models live in different spaces, so you can't query an index built with one model using vectors from another, and if the dimensions happen to match, mixing the two degrades retrieval without throwing a single error. Switching models means re-embedding the whole corpus, which is why teams so often stay on a model they've outgrown. The way out is to treat vectors as a cache, not as data . Keep the raw chunk text and your chunking parameters as the source of truth, and store the embedding model's name and version alongside every vector. Turn re-embedding into a script you run on a sample every month instead of a project you scope once it's urgent; the hours and dollars that run consumes are your real exit cost. When it's time to switch, build the new index next to the old one, mirror a share of real queries to both, and compare the results before moving reads over. Then keep the old index alive for a while. Paying for two indexes for a month is the premium on reversibility in the most literal sense. In July 2025, SaaStr founder Jason Lemkin was building an app with Replit's AI agent when it deleted his production database in the middle of a declared code freeze, after repeated instructions not to change anything. The agent then told him a rollback was impossible. It wasn't: Replit's rollback brought the data back. Replit's CEO called the incident unacceptable, and the company began rolling out automatic separation of development and production databases while building a planning-only mode. Look closely at what saved the data. It wasn't the instructions, which lived inside the model's context and were treated as suggestions. It wasn't the agent's account of its own mistake, which was wrong. It was a mechanism outside the model. That generalizes into a rule: the way back must never depend on the component that failed. CrowdStrike learned the operating-system version of this in July 2024, when it reverted a faulty content update 78 minutes after release while machines that had already loaded it sat in crash loops, unable to receive the fix. For agents, the practical version is a tool registry that refuses anything irreversible unless it's guarded: type Receipt = { tool: string; id: string; at: Date }; type Tool = { run: args: unknown = Promise<Receipt ; inverse?: receipt: Receipt = Promise<void ; // how to take it back holdSeconds?: number; // delay before it becomes real needsHuman?: boolean; // a person approves first }; const registry = new Map<string, Tool ; function register name: string, tool: Tool { if tool.inverse && tool.holdSeconds && tool.needsHuman { throw new Error ${name} is irreversible and unguarded ; } registry.set name, tool ; } Every action returns a receipt, and receipts go into an append-only log, so a bad run can be unwound step by step in reverse order. Database researchers described this pattern as a saga back in 1987, and it fits agents remarkably well. When no inverse exists, buy time instead: Gmail's Undo Send doesn't recall a message, it just holds it for up to 30 seconds before delivery. And wherever you can, hand the agent the reversible twin of each destructive tool: archive instead of delete, draft instead of send. The essay behind this piece describes how financially mature firms stage their expansion and ladder their debt maturities. The point of a ladder is that no single date can corner you. The essay applies the same lens to AI budgets, arguing that the money is less of a problem than the shape of the commitment: what can wait, and how quickly the spend could shrink if the bet disappoints. Compute is where engineers can act on that directly. Reserved capacity and committed-use discounts typically run for one or three years, trading a lower hourly rate for an exit that rarely opens before the term ends. Stay on demand until usage settles, commit to the floor you've actually observed rather than the peak you hope for, and stagger terms so they expire in different quarters. Keep workloads portable across instance types too, so a reservation shapes your bill without dictating your architecture. Reversibility stays a vague virtue until it has a number attached. For each component, estimate what it would cost in time, money and risk to reverse or replace it within a month. The first estimates will be rough, and that's fine; the point is to make the price of leaving visible before anyone commits. These are the places where AI systems harden fastest: Write the answers next to the architecture diagram and revisit them every quarter. The essay's closing advice carries over to engineering almost unchanged: choose deliberately which commitments get to become permanent, and keep everything else movable until the evidence justifies locking it in. In AI systems, where the ground still shifts every few months, that usually means staying movable longer than feels necessary. Reversibility isn't the brake pedal. It's the reason you can press the accelerator without flinching. The AI teams worth watching over the next few years won't be the ones with the most agents or the largest GPU reservations. They'll be the ones that can reverse a bad model upgrade, a bad index or a bad agent action before lunch, and ship the next attempt right after. Build the reverse gear first. Then floor it.