{"slug": "feature-flags-for-a-b-testing-ai-models-and-prompts", "title": "Feature Flags for A/B Testing AI Models and Prompts", "summary": "A developer outlines how to use feature flags for A/B testing AI models and prompts, arguing that model selection must be evaluated server-side rather than in client code so callers cannot choose which model gets paid for. The post details deterministic variant assignment via salted SHA-256 hashing, pinning assignments to conversations rather than users, and separating experiment flags from paid entitlements, while recommending cost and latency be tracked on every variant.", "body_md": "*This post was created with AI assistance and reviewed for accuracy before publishing.*\n\nChanging the model behind a feature is a deploy-sized decision dressed up as a config change. The output shape shifts, the latency profile shifts, the cost per request shifts, and none of it shows up in your test suite because the test suite does not call the model.\n\nFeature flags are the right tool, but AI rollouts stress them in ways ordinary flags do not. The main differences are that the thing you are testing is non-deterministic, the quality signal is delayed, and a bad variant can be expensive rather than just broken.\n\nModel selection must never be decided by client code. A flag evaluated in the browser can be read and changed by anyone, which means the caller chooses which model you pay for.\n\n``` js\n// app/api/chat/route.ts\nexport async function POST(req: Request) {\n  const session = await getSession(req);\n  const variant = await flags.evaluate('chat-model', {\n    userId: session.userId,\n    plan: session.plan,\n  });\n\n  const model = MODELS[variant] ?? MODELS.control;   // fall back on unknown\n  // ...\n}\n```\n\nThe fallback on an unknown variant matters. Flag services fail, configs get edited, and a variant name can disappear while sessions still reference it. Defaulting to the control means the worst case is no experiment, rather than an exception in your main request path.\n\nThese get conflated because both are \"flags\", and the consequences differ enormously.\n\nAn experiment decides which of two equivalent implementations a user gets. It can be reassigned, ramped, and rolled back freely. An entitlement decides what someone has paid for. It is authorisation, and it belongs with your billing logic, checked server-side on every request.\n\nThe failure is putting a paid capability behind an experiment flag. Ramp percentages then control who gets a feature they bought, and a rollback removes it from paying customers. Different systems, or at minimum different code paths with different review requirements.\n\nA user reassigned between variants mid-session gets a conversation where the model changes underneath them. That produces incoherent behaviour and contaminates your results.\n\nHash a stable identifier so assignment is deterministic:\n\n``` js\nfunction variantFor(userId: string, salt: string, split: number) {\n  const h = createHash('sha256').update(`${salt}:${userId}`).digest();\n  return (h.readUInt32BE(0) % 100) < split ? 'treatment' : 'control';\n}\n```\n\nInclude the experiment name in the salt. Without it, the same users land in the treatment group of every experiment you run, and effects compound invisibly.\n\nFor conversational products, consider pinning to the conversation rather than the user, so a single thread always uses one model even if the user's assignment later changes.\n\nThis is where AI experiments differ most from ordinary ones. There is no click-through rate for \"gave a good answer\", and the honest signals are indirect.\n\n| Signal | Reads as | Caveat | \n|---|---|---|\n| Explicit thumbs up or down | Quality | Very low response rate, skewed to extremes | \n| Regeneration rate | Dissatisfaction | Also rises when responses are slow | \n| Conversation length | Engagement or struggle | Ambiguous on its own | \n| Task completion | Real success | Only measurable if the task has an endpoint | \n| Cost and latency per request | Operational fit | Unambiguous, measure always | \n\nPick the primary metric before launching. Choosing afterwards from whatever moved is how a worse model gets promoted on the strength of a metric that happened to rise.\n\nCost and latency are worth tracking on every variant regardless, because they are the two that are never ambiguous and they frequently decide the question on their own.\n\nEvery request should record which variant served it, alongside the tokens used, the latency, and any feedback that arrives later.\n\nWithout that join, you cannot attribute anything. A week into the experiment someone asks whether the treatment group's costs went up, and if the variant is not on the usage record, the answer is unavailable and the experiment was wasted.\n\nTreat prompt text in these logs carefully. It is user content, and an experiment log is an easy place for it to end up with looser retention than the rest of your data.\n\nRamp deliberately: internal users, then a small percentage, then wider. But the ramp matters less than the ability to stop.\n\nA kill switch has to be a config change that takes effect immediately, with no deploy and no cache to wait out. Test that it works before you need it, and make sure whoever is on call can operate it without a code review.\n\nThe scenarios it exists for are specific and worth naming: a variant is producing unsafe output, a provider incident makes one path unusable, or the cost per request turns out to be several times the control. All three are discovered in production, and in all three the time between noticing and stopping is the entire cost of the mistake.", "url": "https://wpnews.pro/news/feature-flags-for-a-b-testing-ai-models-and-prompts", "canonical_source": "https://dev.to/ganeshjoshi/feature-flags-for-ab-testing-ai-models-and-prompts-2k8g", "published_at": "2026-09-27 14:32:13+00:00", "updated_at": "2026-09-27 15:01:19.233819+00:00", "lang": "en", "topics": ["ai-tools", "mlops", "developer-tools", "large-language-models"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/feature-flags-for-a-b-testing-ai-models-and-prompts", "markdown": "https://wpnews.pro/news/feature-flags-for-a-b-testing-ai-models-and-prompts.md", "text": "https://wpnews.pro/news/feature-flags-for-a-b-testing-ai-models-and-prompts.txt", "jsonld": "https://wpnews.pro/news/feature-flags-for-a-b-testing-ai-models-and-prompts.jsonld"}}