Multi-Tenant AI Chat: From Hardcoded Config to BYOK in 4 Steps

A developer describes evolving a multi-tenant AI chat system from hardcoded configuration to a bring-your-own-key (BYOK) model in four stages, driven by real customer requests. The progression moves from a static config object to a database-backed settings table, then to event-sourced configuration for auditability, and finally to tenant-provided API keys. The developer emphasizes that each stage was prompted by a genuine user need, such as self-service prompt editing and the ability to trace configuration changes.

Two tenants, two AI providers, two prompts. Sounds simple, and on day one it is. That's the trap. It stays simple right up until customer number two sends their first "quick question," and eighteen months later you're running a small distributed system to answer it. Here's the honest version of that slide, four stages, each one caused by a real human typing a real request into Slack. Tenant A wants OpenAI. Tenant B wants Claude. Both want their own system prompt. The obvious first version: one config object, one row per tenant. What could possibly go wrong. Everything. Everything could go wrong. But not yet. ┌─────────────┐ ┌──────────────────┐ ┌─────────────┐ │ Request │ → │ TENANT CONFIG │ → │ Provider │ │ tenantId │ │ hardcoded obj │ │ SDK call │ └─────────────┘ └──────────────────┘ └─────────────┘ js const TENANT AI CONFIG = { tenantA: { provider: 'openai', model: 'gpt-5', prompt: 'You are terse and technical.' }, tenantB: { provider: 'anthropic', model: 'claude-sonnet-5', prompt: 'You are friendly. Antworte auf Deutsch.' }, } as const async function handleChat tenantId: string, userMessage: string { const config = TENANT AI CONFIG tenantId const client = config.provider === 'openai' ? openai : anthropic return client.chat config.model, config.prompt, userMessage } Ships in an afternoon. Two tenants, two rows, demo goes great, everyone claps 👏. Put this moment in a frame, it's the calmest the codebase will ever be. A week in a week, we didn't even get a full sprint , tenant B messages: "can we change the prompt ourselves, without waiting for a deploy?" Fair ask, they know their users, we don't, and also nobody wants to be the on-call engineer who gets paged to edit a string literal. A hardcoded object can't answer that, it needs a rebuild to change a comma. ┌─────────────┐ ┌──────────────────┐ ┌─────────────┐ │ Request │ → │ tenant settings │ → │ Provider │ │ tenantId │ │ DB row, admin │ │ SDK call │ │ │ │ editable │ │ │ └─────────────┘ └──────────────────┘ └─────────────┘ Config moves from a code constant to a table the tenant's own admin UI can write to. js async function getAiConfig tenantId: string { const row = await ctx.db.tenantSettings.findOne { tenantId } return row.aiConfig // { provider, model, prompt } } Same shape as before, different source. handleChat doesn't change at all, it has no idea any of this happened, which is exactly the point of putting the lookup behind one function. Self-service is great until it isn't: tenant B's prompt quietly changed last Tuesday, their bot started answering in pirate-speak for reasons nobody can reconstruct, support gets a ticket, and the honest answer is "we have no idea, the database doesn't remember either." A plain DB row just gets overwritten, the past has no representation, it's Ctrl+Z with no undo history. ┌──────────────┐ ┌────────────────┐ ┌──────────────────┐ │ Admin edits │ → │ ConfigChanged │ → │ current config │ │ the prompt │ │ event who, │ │ = fold events │ │ │ │ when, diff │ │ │ └──────────────┘ └────────────────┘ └──────────────────┘ The config becomes event-sourced instead of a mutable row: every change is an event, the current value is a projection over them. async function updateAiConfig tenantId: string, patch: Partial<AiConfig , actor: string { await ctx.emit 'AiConfigChanged', { tenantId, patch, actor, at: ctx.now } } async function getAiConfig tenantId: string : Promise<AiConfig { const events = await ctx.db.events.find { tenantId, type: 'AiConfigChanged' } return events.reduce cfg, e = { ...cfg, ...e.patch } , DEFAULT AI CONFIG } Now "who changed it and when" is a query, not a seance 🔮. handleChat still hasn't changed, it just calls getAiConfig , blissfully unaware it's now talking to an event log instead of a table. A bigger tenant shows up, the kind that gets its own Slack channel, with two demands: they want to use their own OpenAI key cost control, their own rate limits, their own finance team breathing down their neck , and they want a hard cap on monthly spend so an over-caffeinated intern's script can't turn into a five-figure invoice. ┌──────────────┐ ┌───────────────────────┐ ┌──────────────┐ │ Request │ → │ config.apiKey? │ → │ usage < cap?│ │ │ │ BYOK, encrypted │ │ → call │ │ │ │ else our shared key │ │ → else 429 │ └──────────────┘ └───────────────────────┘ └──────────────┘ js async function callProvider tenantId: string, userMessage: string { const config = await getAiConfig tenantId const usage = await getMonthlyUsage tenantId if config.usageCap && usage = config.usageCap { throw new UsageCapExceeded tenantId } const apiKey = config.byokApiKey ?? process.env.SHARED API KEY // BYOK overrides shared key const client = getClient config.provider, apiKey const reply = await client.chat config.model, config.prompt, userMessage await recordUsage tenantId, reply.usage.totalTokens return reply } Two additive fields on the same config, byokApiKey and usageCap , and one counter check before the call. No new architecture, no rewrite of the first three stages, no "sorry, we need a full quarter to redesign this." Every stage kept getAiConfig tenantId → { provider, model, prompt, ... } as the seam. Storage changed underneath it four times constant, DB row, event-sourced projection, projection with encrypted secrets and the call site never noticed, never cared, never even asked. That's the actual lesson: don't design the multi-tenant AI system upfront, design one seam that can absorb whatever the next Slack message throws at it. Skipped on purpose: provider fallback, streaming, per-model cost tables. Add those when a tenant actually asks, same as everything above. If a tenant asks for a fifth provider before you've read this sentence, that's not a counterexample, that's Tuesday. 🙃