{"slug": "what-to-restore-when-litellm-holds-team-keys-budgets-and-spend-data", "title": "What to restore when LiteLLM holds team keys, budgets and spend data", "summary": "LiteLLM's gateway stores state in PostgreSQL, including virtual-key records, teams, budgets, and spend data. A developer explains that restoring the gateway requires the database, configuration, and decryption key together, and recommends rehearsing restores on throwaway instances. The team emphasizes that virtual keys are the state being protected, and provider credentials may be stored externally.", "body_md": "LiteLLM forwards requests, but a team deployment is not stateless. PostgreSQL holds virtual-key records, teams, budgets and spend data. Stored credentials also depend on the key used to encrypt them.\n\nThat makes the database, configuration and decryption key part of one recovery plan. Restoring only the container or only the database is not enough. This article explains what must be restored, what can be rebuilt, and how the salt key changes master-key rotation.\n\nBehind the OpenAI-compatible endpoint, the gateway keeps its state in a PostgreSQL database. The database stores hashed virtual-key records, their scopes, users and teams, budgets, rate limits and spend data. Provider credentials are a separate case: they may be stored in the database encrypted, or supplied from an external secret store, in which case the database never holds them. The gateway also depends on the configuration that defines the model map, and on the key that decrypts any credentials the database does hold.\n\nIf you lose the database, you lose the virtual-key records and the spend history. Provider credentials you keep in an external secret store can be re-injected. So the operational unit is the database plus the configuration plus the decryption key, not the container that serves traffic.\n\nA consistent restore is three things brought back together: the PostgreSQL database, the configuration that points at it, and the key that decrypts stored credentials — the salt key if you set one, or the master key if you did not. The database on its own will not start a working gateway. Version the database and the configuration together as one restore set. The decryption key has to be available to that restore too, but keep it in a separate secret store rather than inside the backup, so the encrypted database and the key that opens it are not sitting in the same place.\n\nThen rehearse the restore on a throwaway instance. Bring the config and database back, start the gateway on the same pinned version, log into the admin UI, confirm a virtual key still resolves and routes a request, and confirm a model whose credentials live in the database still works.\n\nA backup you have never restored has not been shown to work. Set backup frequency and retention from how much spend history and how many live keys you could accept losing between backups, and treat that restore rehearsal as the launch gate, not an afterthought.\n\nLiteLLM documents PostgreSQL through `DATABASE_URL`\n\nas a requirement for virtual keys, budgets and spend tracking. That is the documented position, and it is the reason the database is the system of record here.\n\nThe rest of this section is general database operations practice rather than a documented requirement for running the proxy. Once the database holds your keys, users, teams and usage, treat it as production data: give it persistent storage, size the connection pool against the total connection budget your instances and workers share rather than a per-worker guess, and apply the backup-and-restore discipline above.\n\nSpend logging writes on the request path, so watch that write volume as traffic grows — the production docs give concrete settings for batching those writes.\n\nThe reason to run the gateway instead of sharing one provider key is virtual keys, and they are the state you are protecting.\n\nInstead of handing an app your real provider key, you issue a gateway key through `/key/generate`\n\n, scoped to an app or team with a `team_id`\n\nor `user_id`\n\nand the models it may call. The real provider keys stay in the gateway; the app only holds a key you can revoke or rotate without touching the provider account.\n\nA key request is small:\n\n```\n{\n  \"team_id\": \"team-web\",\n  \"models\": [\"gpt-4o\", \"claude-3-5-sonnet\"],\n  \"max_budget\": 200,\n  \"budget_duration\": \"30d\",\n  \"rpm_limit\": 60\n}\n```\n\nThe `team_id`\n\nhas to reference a team you have already created, and you get back an `sk-`\n\nvirtual key the app uses as its API key.\n\n`max_budget`\n\nand `budget_duration`\n\nset a spend cap and its reset window; `rpm_limit`\n\n, `tpm_limit`\n\nand `max_parallel_requests`\n\ncap request rate, token rate and concurrency, so one runaway job cannot spend the month in an afternoon.\n\nThese apply only where you configure them and enforce against the gateway's own accounting, not the provider invoice. How a budget resets and what happens at the limit are version-specific — confirm the field names and behaviour against the docs for the version you run, because the key API changes between releases.\n\nAll of this is stored in PostgreSQL, which is why the restore above matters.\n\nTwo secrets protect that state, and they behave differently.\n\n`LITELLM_MASTER_KEY`\n\nis the proxy's admin credential — it must start with `sk-`\n\n, and it creates and manages every virtual key.\n\n`LITELLM_SALT_KEY`\n\n, when you set it, is the key that encrypts stored credentials — models, the config `environment_variables`\n\n, and the credentials table — in the database.\n\nSet the salt key from the start, because the master-key rotation procedure splits on exactly whether it is present, and the official rotation guide is worth reading before you rotate anything.\n\nIf the master key leaks, an attacker has proxy-admin control: they can create, revoke and re-scope virtual keys, change the configuration, and use the model access the gateway is configured with. Where no salt key is set, the master key is also the credential-encryption key, so they can decrypt the stored credentials too.\n\nBack up PostgreSQL before either rotation, whichever case you are in.\n\nWith a salt key configured, the master key is only an authentication credential. To rotate it you generate a new value, update whichever master-key source your deployment uses — the `LITELLM_MASTER_KEY`\n\nenvironment variable or `general_settings.master_key`\n\nin the config — so every instance gets the same new value, then restart them all.\n\nYou do not call `POST /key/regenerate`\n\nwith `new_master_key`\n\nhere — the docs warn that doing so would re-encrypt your stored credentials under a key the proxy never uses to decrypt, which strands them.\n\nWithout a salt key, the master key also encrypts those stored credentials, so rotating it has to re-encrypt them. That is why the database backup matters especially here: the re-encryption deletes and recreates rows and skips any row it cannot re-encrypt, so a partial failure can strand data.\n\nCall `POST /key/regenerate`\n\nwith the current master key as `key`\n\nand the new one as `new_master_key`\n\n. It re-encrypts the stored models, the config environment variables, and the credentials, MCP-server and user tables.\n\nAfter it returns, update the same master-key source — the `LITELLM_MASTER_KEY`\n\nenvironment variable or `general_settings.master_key`\n\n— so every instance gets the new value, restart them all, log into the admin UI with the new key, and test a request against a DB-stored model.\n\nNeither of those is the same as rotating a single virtual key, which is a different operation on a different endpoint, `POST /key/{key}/regenerate`\n\n.\n\nVirtual keys are stored hashed rather than encrypted, so they keep working through a master-key rotation. You regenerate one only when that specific key is compromised.\n\nThe documented rotation flows assume that you still hold the current master key. Keep it in a real secret store, keep the salt key separate from it, and rehearse a rotation before you need one.\n\nYou do not need Redis for a single instance, and when you do use it, it is not part of what you restore.\n\nIt becomes relevant once you run more than one proxy, where it shares rate-limit counters and router state across instances so limits apply across the fleet rather than per process.\n\nIt backs the response cache when you have enabled caching and pointed it at Redis, and it can cache virtual-key lookups when you set `enable_redis_auth_cache`\n\n, which keeps the database off the path of every request.\n\nPostgreSQL is the durable system-of-record state. Redis holds rebuildable cache, counter and router state, so losing it costs cache warmth and cross-instance counters, not your virtual-key records or spend history.\n\nRestore PostgreSQL and allow the Redis cache and distributed counters to rebuild.\n\nThe schema of that PostgreSQL database moves with LiteLLM, which matters because it is the thing you back up and restore.\n\nThe project ships Prisma migrations gated by `USE_PRISMA_MIGRATE`\n\nand `DISABLE_SCHEMA_UPDATE`\n\n, versioned so older releases keep using their own migrations.\n\nTreat an upgrade as a change to production data: pin the image to a specific tag instead of tracking `latest`\n\n, take a database backup first, and read the release notes.\n\nIn a multi-pod deployment, run migrations from one place and set `DISABLE_SCHEMA_UPDATE`\n\non the rest so pods do not race the same migration.\n\nA restore is only clean if you bring the database back on a version whose schema matches it, which is another reason the pinned version is part of the restore rehearsal above.\n\nThe admin UI at `/ui`\n\nis an access path to all of that state — it creates keys, adds models and shows spend.\n\nIt has its own login through `UI_USERNAME`\n\nand `UI_PASSWORD`\n\nor through SSO, and you can turn it off entirely with `DISABLE_ADMIN_UI=\"True\"`\n\n.\n\nThe docs do not say it must never be public; that is a call I make anyway.\n\nKeep the admin surface behind its login and, where you can, off the public internet, with the gateway behind a reverse proxy over TLS rather than on a raw published port. It serves traffic fine without the UI being publicly reachable.\n\nThe failures worth designing around are all about that state.\n\nA leaked master key exposes proxy-admin control, and the stored credentials too if no salt key is set.\n\nA database you cannot restore costs you the keys and the spend history, which is why the restore rehearsal is the gate.\n\nOne failure shows up in the spend the database records rather than in the storage itself: fallback between providers can raise spend sharply if a cheap model fails over to an expensive one, depending on how retries and billing are configured. Cap it with the per-key budgets you already keep in that database and watch the spend when it triggers.\n\nA public, unauthenticated admin UI is a direct path to the key namespace.\n\nIf one script uses one provider key, none of this applies — you would be adding a stateful service, a database and an admin surface to operate for no gain over calling the provider directly.\n\nThe restore question only becomes real when several apps, teams or people share model access through per-team keys, budgets and spend attribution.\n\nReach for LiteLLM when that shared state is what you need, and then treat the database and its restore as the load-bearing part.\n\nYNVAR can operate the container, TLS boundary, PostgreSQL, backups and upgrade window on European infrastructure. The customer keeps the provider accounts, provider keys, virtual-key policy and model-access decisions.\n\nThe [full write-up, including the backup scope and the operate-versus-own split](https://www.ynvar.com/blog/litellm-gateway-for-teams/?utm_source=devto&utm_medium=referral&utm_campaign=ynvar_earned_media_2026q3&utm_content=devto-litellm) goes further than this post.\n\nEuropean hosting documents the location of that platform layer. It does not determine where an external model provider processes requests, and it is not a GDPR-compliance claim.\n\nYNVAR is not an official partner of the LiteLLM project or of any model provider.\n\n*— Adrians Petrovs, Founder and Infrastructure Engineer at YNVAR*", "url": "https://wpnews.pro/news/what-to-restore-when-litellm-holds-team-keys-budgets-and-spend-data", "canonical_source": "https://dev.to/ynvar/what-to-restore-when-litellm-holds-team-keys-budgets-and-spend-data-4fac", "published_at": "2026-07-22 14:35:22+00:00", "updated_at": "2026-07-22 15:02:11.310267+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["LiteLLM", "PostgreSQL"], "alternates": {"html": "https://wpnews.pro/news/what-to-restore-when-litellm-holds-team-keys-budgets-and-spend-data", "markdown": "https://wpnews.pro/news/what-to-restore-when-litellm-holds-team-keys-budgets-and-spend-data.md", "text": "https://wpnews.pro/news/what-to-restore-when-litellm-holds-team-keys-budgets-and-spend-data.txt", "jsonld": "https://wpnews.pro/news/what-to-restore-when-litellm-holds-team-keys-budgets-and-spend-data.jsonld"}}