Ship Gate: The Pre-Deploy Checklist Most AI Features Skip

AI architect Wasim Sheikh published a pre-deployment checklist called "Ship Gate" that he argues most AI features skip before going live. The five-gate framework covers threat modeling, data boundaries, eval sets of at least 20 cases, human override for irreversible actions, and observability that lets an engineer reconstruct a bad turn in under five minutes. Sheikh cites a case where an AI email summarizer shipped into customer-facing replies and invented a discount a customer had never been offered, with no eval set, human gate, or kill switch in place.

Most AI features don't fail in the model. They fail at the gate. Someone demos a shiny prompt. Leadership loves it. It ships. Then a customer pastes something weird, an agent calls the wrong tool, or a "helpful" answer invents an offer that never existed. The postmortem is always the same: we optimized for the demo, not for production. I call the missing step Ship Gate — a short, boring checklist you run before any AI feature leaves the sandbox. Not a 40-page risk memo. A gate you can clear in an afternoon. Situation. A product team shipped an AI email summarizer into customer-facing replies. Staging looked clean. Leadership wanted it live before the quarter closed. What broke. Day one, the model invented a discount a customer had never been offered. Support scrambled. There was no eval set of "known trap" emails, no human gate on outbound AI text, and no kill switch short of a full deploy rollback. The fix Ship Gate . Then they added the ops layer every Ship Gate implies: canary traffic, a one-click kill switch, and a documented rollback owner before the flag flipped on. Ship Gate isn't bureaucracy. It's the minimum checklist so an AI feature earns production — instead of learning in front of customers. Demos are curated. You pick the happy path. You soft-prompt around the edge cases. You never paste a raw customer email with secrets still in it. Production is the opposite. Users paste junk. Models invent confident nonsense. Tools fire with the wrong args. Logs capture more than you think. If your only test was "it looked good in the meeting," you didn't ship AI — you shipped a vibe. Ship Gate exists to make that gap visible before customers feel it. Run these in order. Fail any one and you don't ship. Pass all five and you can sleep. Before code, write three sentences: If you can't name the top three failure modes, you're not ready to build — you're ready to be surprised. Answer out loud: Then enforce it in code, not in a slide. Strip secrets before the model. Scope tools to the minimum. Prefer retrieval over stuffing the whole corpus into context. If the model doesn't need a field, don't give it the field. Pick 20–50 real-ish cases: happy path, hostile prompts, empty input, long paste, "ignore previous instructions," the ticket that always breaks things. Score each on three axes: Ship Gate rule: you don't tune the UI until the eval set is green enough that you'd trust a teammate to use the feature unsupervised. Every autonomous or semi-autonomous path needs an escape hatch: If the only recovery path is "hope the model was right," you don't have a product. You have a liability with a chat box. When it breaks at 2 a.m., can you answer: If the answer is "we'd have to dig through JSON for an hour," fix the logs before you grow the feature. Ship Gate isn't complete until a tired engineer can reconstruct a single bad turn. | Gate | Pass criteria | |---|---| | Threat model | Top 3 failure modes named + owner | | Data boundary | See / write / never-leave documented + enforced | | Eval set | ≥20 cases; safe + useful bar met | | Human override | Irreversible actions confirmed; stop path clear | | Observability | One bad turn reconstructable in <5 minutes | No green across the board → no ship. Ask one question before any AI feature goes live: Which of the five gates is still red — and who owns fixing it? If nobody owns the red gate, you don't have a launch plan. You have a calendar date. I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production. Canonical: https://sheikhwasim.com/insights/ship-gate-pre-deploy-checklist/ https://sheikhwasim.com/insights/ship-gate-pre-deploy-checklist/