{"slug": "rotate-a-leaked-api-key-without-stopping-your-services-dotvault-hands-out-the", "title": "Rotate a leaked API key without stopping your services: dotvault hands out the new value first", "summary": "A developer building the dotvault secrets-management tool demonstrated a rotation workflow that hands out a new key value to callers before revoking the old one, avoiding the ordering problem that normally forces services to be stopped during key rotation. In the demo, a caller polling at 15-second intervals had its value accepted in all 111 calls that received a response during rotation, then moved to the new value automatically while the leaked value was rejected. The tool also supports automatic rotation at a user-chosen interval, and is slated for free release at the end of the 12-part series.", "body_md": "This is part 11 of the series \"dotvault, Start to Finish: One Command from Your Laptop to CI, Your Team, and Your AI Agents.\" dotvault will be released free of charge when this series ends.\n\nIn the demo, a script kept calling with the key at 15-second intervals during the whole rotation, and in all 111 calls that got a response, the caller's value was accepted. The caller moved to the new value on its own, and the leaked value was rejected from then on.\n\nThat matters because rotating a key between two of your services has an ordering problem. Switch the service that checks the key first, and the caller still holding the old value gets rejected. Switch the caller first, and the checker rejects a value it has never seen. Either way, calls fail until the other side catches up, and the usual way around it is to stop both and switch them together. This part shows how `dotvault rotate --reason leak` avoids that.\n\nThis part also covers a second feature: rotating a key automatically at an interval you choose.\n\nRotation is not only for leaks. It is part of routine key management. Set an interval when you register a key, and dotvault keeps rotating it on its own, even when nothing has leaked. Nobody has to rotate keys by hand any more, so you can keep each key's lifetime short.\n\nThis is part 11 of 12.\n\n`dotvault rotate <KEY> --reason leak`.` interval_seconds`) when you register the key, and as long as the holders and verifiers read from dotvault as part of their normal work, dotvault keeps rotating it at that interval, so you can keep the key's lifetime short.`--revoke-now` once every holder has the new value.\nThis follows part 4 (overwriting and rolling back values) and part 7 (creating service tokens). You can follow along without them.\n\nThe demo: `order-worker` processes orders and calls `payment-api`, the payment API. It sends `PAYMENT_API_KEY`, and `payment-api` checks that the value is right. Assume `PAYMENT_API_KEY` has leaked.\n\nThe terms used here:\n\n| Term | Meaning | \n|---|---|\n| Rotation management | dotvault generates the key's values and decides who receives them. Only registered keys are covered | \n| Holder | The side that holds the value and calls the other service. In the demo, the `order-worker` token | \n| Verifier | The side that checks the value it receives. In the demo, the `payment-api` token | \n| Rotate | Hand out a new value, then revoke the old one | \n| Retire | Revoke the value at once, without handing out a new one | \n\nRotation works for keys that both sides receive from dotvault, which means keys between your own services. dotvault generates the new value. An API key issued by another company's service has to be replaced in that service's dashboard.\n\nIn the demo project `shop-backend`, the first value of `PAYMENT_API_KEY` went into production with `dotvault set`. `order-worker` and `payment-api` each got a Read only service token. A service token is the string, starting with `est_`, that CI or a service uses instead of logging in. Part 7 shows how to create one.\n\nRegistering takes a token of the Credential rotation type. In Service Tokens, click Create token and pick \"Credential rotation\" as the Role.\n\nAs the dialog says, this token can issue tokens and register keys only inside one project and the environments you pick. It is not needed once the key is registered, so it expires in 1 day.\n\nWith that token, send one HTTP PUT, the request type that stores data at a URL. The body is JSON naming the holder and verifier tokens by Token ID.\n\n`HTTP 201` (created) and `\"created\":true` mean the key is registered. The fields:\n\n| Field | What it holds | \n|---|---|\n| `bindings` | Who receives the key. `tokenId` is the service token's Token ID,`role` is`holder` or`verifier` , and`label` is a name you choose | \n| `residual_seconds` | How long to wait, after every holder has moved to the new value, before revoking the old one. 60 seconds in the demo | \n| `generator` (in the response) | How new values are made. `random-base64url-32` is 32 random bytes written as letters, digits and symbols | \n\n`key=` in the URL is the key name, and `project=`, `env=` and `file=` say where it lives. `$DOTVAULT_TOKEN` in the `Authorization` header holds the Credential rotation token.\n\nA rotation cannot start in the first 5 minutes after registering. Register service-to-service keys when you store them, not after a leak.\n\nAfter registering, pull the same project as a person, with the holder token and with the verifier token.\n\nOnly the holder, `order-worker`, gets the value. This screen was captured after the rotation in section 4; a registered key is handed out the same way before and after a rotation. The person (an org admin) and the verifier see the key name in a \"withheld 1 rotation-managed key(s)\" warning, with no value. If you need a key with that name for local development, store a development value under a different key name.\n\nThe verifier does not pull. It reads the set of values it should accept from `GET /v1/env/key-set`. Each entry has the value, an ID for the value (` keyId`) and a state (` state`). The verifier accepts a value that is in this set.\n\nTo see whether calls keep working during the rotation, a small script, `service.mjs`, ran at 15-second intervals. Each round, it:\n\n`PAYMENT_API_KEY` with the `order-worker` token`payment-api` token, and prints `accepted` if the value is in it, `REJECTED` if not`phase`)\nIt prints only the first 8 characters of each `keyId`, never the value. With the script running, an org admin signed in to the CLI runs:\n\n```\ndotvault rotate PAYMENT_API_KEY -E production --reason leak\n```\n\n`started rotation` means it is running. `generation 2` means this is the key's second rotation; the first was the routine rotation in section 7, which was aborted.\n\nA leak rotation cannot be stopped. `--abort` returns `ABORT_REFUSED_LEAK` with exit code 7, so there is no path back to the leaked value.\n\n`--status` shows how far the rotation has got. In the audit log, the rotation reached `STAGED` in the same second it started.\n\nThe rotation goes in this order.\n\n| `phase` | What happens | \n|---|---|\n| `STAGED` | dotvault generates a new value and adds it to the verifier's set. The holder still gets the old value | \n| `PROMOTED` | About 15 minutes after the verifier is seen reading the new value. The holder's pull starts returning the new value. The leaked value stays in the set | \n| `HOLDERS_OK` | About 15 minutes after the holder is seen reading the new value | \n| `DONE` | `residual_seconds` (60 seconds) after`HOLDERS_OK` , the leaked value is revoked and leaves the set | \n\nThe table leaves out `LOCKED`, the phase at the start (the audit log shows `STAGED` in the same second as the start), and `VERIFIERS_OK`, the mark that the verifier has read, which is recorded in the same second as `PROMOTED`.\n\nThe 15-minute waits exist because holders and verifiers may keep a value or a set they read for up to 5 minutes. Several instances can share one token and they do not all read at the same moment, so 10 more minutes are added. `--reason leak` does not skip these waits on its own. In a leak rotation, if one side never reads, the rotation moves on anyway after 30 minutes (plus the 5 minutes a read may be kept).\n\nHere is the script's output around each change of `phase`. The first two `ABORTED` lines are the state left by the routine rotation aborted in section 7.\n\nThree things show up:\n\n`STAGED`, the new value (` k_iVota-`, state `staged`) joins the set, while the holder keeps using the old value (`k_Z0Njns`) and is still accepted.` PROMOTED`, the holder's value is `retiring`.` DONE`, the set holds only the new value, and the leaked value is Over the whole run, the script made 124 rounds. In all 111 that got a response, the holder's value was `accepted`. The other 13, `error fetch failed`, were rounds where the script's network request failed, so no response from dotvault came back.\n\nThe console shows the rotation too. In the project's Keys, select `PAYMENT_API_KEY` and a Rotation section appears on the right.\n\nBesides Phase, the Rotation panel shows when the rotation is next due to move on (`Next due`) and, in a leak rotation, when it moves on even if a side has not read the new value (`Deadline`). The figure leaves those two lines out.\n\nAfter `DONE`, check the state and try writing the leaked value back.\n\nIn `dotvault versions`, the new v3 is `state=current` and the leaked v1 is `state=revoked`. v3 and v2 were made by `system:rotation-driver`, which is dotvault itself. v2 belongs to the routine rotation aborted in section 7.\n\nWriting the leaked value with `dotvault set`, or rolling back to v1 with `dotvault rollback --to 1`, both fail with `ROTATION_MANAGED`: a key under rotation management changes only through rotation. As part 4 showed, an ordinary key can be rolled back to an old value after an overwrite. A key under rotation management cannot go back to the leaked value, even if someone tries with good intentions.\n\nIn the console's Audit Log, filter with `shop-backend:production:env:PAYMENT_API_KEY` as the Target ID: project, environment, file and key name joined by `:`.\n\nThe leak rotation's rows, from the bottom up. `14582408-…` is the user ID of dev-e, the demo user.\n\n| Action | Actor | Meaning | \n|---|---|---|\n| `secret.rotation.started` | `14582408-…` , cli | dev-e started the rotation from the CLI | \n| `secret.rotation.staged` | `system:rotation-driver` | A new value was generated and added to the verifier's set | \n| `secret.rotation.verifiers_ok` | `system:rotation-driver` | The 15-minute wait after the verifier read the new value is over | \n| `secret.rotation.promoted` | `system:rotation-driver` | The holder started receiving the new value | \n| `secret.rotation.holders_ok` | `system:rotation-driver` | The 15-minute wait after the holder read the new value is over | \n| `secret.rotation.revoked` | `system:rotation-driver` | The leaked value was revoked | \n| `secret.rotation.completed` | `system:rotation-driver` | The rotation finished | \n\nA person started the rotation and dotvault did the rest. That the leaked value was revoked is one row you can point to later.\n\nWhen nothing has leaked, a routine rotation runs without `--reason` (the same as `--reason manual`).\n\nTwo differences from a leak rotation:\n\n`--abort` stopped it in a few seconds: the new value became `revoked` and the old value stayed `current`.\n`waiting on verifier payment-api` means `payment-api` has not read the new value yet. `last ok` is when it last read the set. The `phase deadline` of a routine rotation (24 hours after the start) is how long it waits for a side that has not read the new value. Past it, the rotation does not move on; it stops (STALLED).\n\nEven when nothing has leaked, you may not want to keep one key for a long time. Add `interval_seconds` (the rotation interval, in seconds) when you register the key, and dotvault starts a routine rotation on its own at that interval. Rotation stops being a person's job.\n\nIn the demo, a new key, `WEBHOOK_SECRET`, was registered with the same holder and verifier tokens as in section 1, with `interval_seconds` set to 600 (10 minutes). Ten minutes is only for the recording; in real use you would pick something like 30 days (2592000).\n\nRight after registering, `--status` says `managed but has never run`: the key is registered but has never been rotated. Nobody typed a rotate command after that (the only thing running was the script that reads values). Once the 10-minute interval had passed, `--status` showed `reason scheduled` (dotvault started this rotation itself because the interval came round), and the rotation had already added the new value to the verifier's accepted set (STAGED).\n\nThe audit log agrees. After the registration (`rotation_provisioner.config_created`), both `secret.rotation.started` and `secret.rotation.staged` have the Actor `system:rotation-driver`. In the leak rotation in section 6, a person was the Actor of `started`; here dotvault started the rotation too.\n\nA scheduled rotation is the same as the routine rotation in section 7: the old and new values both stay valid for at least 24 hours, and the new value goes to the verifier first, then to the holder. The next rotation starts once the interval has passed after the previous one finished. Right after you register a key, or shorten its interval, nothing starts for 5 minutes.\n\nA rotation only reaches the end when the holders and verifiers read from dotvault as part of their normal work. If a service never reads again, the rotation waits on it with the `waiting on …` line from section 7, and past the deadline it stops (STALLED).\n\nA rotation accepts the leaked value until the new one is everywhere. Even after the holder has read the new value, there is a 15-minute wait and the 60-second residual. To make the leaked value stop working sooner, skip the remaining waits with `--revoke-now`.\n\n```\ndotvault rotate REFUND_API_KEY --revoke-now --confirm REFUND_API_KEY -E production\n```\n\nThis was recorded after PAYMENT_API_KEY had been retired (later in this section), on a second key, `REFUND_API_KEY`, registered the same way. One of the script's calls already carried the new value (PROMOTED). Then `--revoke-now`: in the audit log, `revoke_now`, `revoked` and `completed` share the same second, and from the next call that got a response, the leaked value was REJECTED. The holder's value stayed accepted. The script ran 74 times during this recording, and in all 57 calls that got a response, the holder's value was accepted (the other 17 failed on the script's side before reaching dotvault).\n\nRun `--revoke-now` once every holder has the new value, that is, after PROMOTED. It skips every wait from that point on, so if you run it earlier, dotvault switches values and revokes the old one without waiting for the verifier or the holder to read the new one, and a holder still on the old value is rejected.\n\nThe `waiting on holder …` line in `--status` is per token. When several services share one token, the line disappears as soon as one of them reads. Any other service still holding the old value is rejected after `--revoke-now`; that is what 10 of the 15 minutes of waiting are for. Run it only when you know every service has pulled again, for example after restarting them. In the demo the holder's value stayed accepted because there was one holder and it pulled on every call.\n\nOne more thing: revoking takes the value out of dotvault's accepted set. While a verifier keeps its copy of the set (up to 5 minutes), it may still accept the leaked value. The demo's verifier read the set on every call, so the leaked value was REJECTED from the next call.\n\n`--confirm` repeats the key name; one wrong character stops it with `VALIDATION_ERROR` and nothing is sent. Only a signed-in person can run it; a service token cannot.\n\nIf you cannot even wait for the holder to move to the new value, retire the key.\n\n```\ndotvault rotate PAYMENT_API_KEY --retire --confirm PAYMENT_API_KEY -E production\n```\n\nThe retire ran on the same key after the PAYMENT_API_KEY rotation was recorded. `revoked version(s): 3` means it revoked v3, the current value from the rotation; v1 and v2 were already revoked. The verifier can no longer read the set (`HTTP 403`, `NOT_BOUND`), and writing the leaked value back with `set` fails with `VALUE_REVOKED`: a revoked value can never be written again.\n\nRetire hands out no new value, so the holder and the verifier both stop at once. `--confirm` repeats the key name, and one wrong character means nothing happens. Only a signed-in person can retire; a service token cannot. Use it only when the leaked value has to stop working right now, even at the cost of stopping your services.\n\nWhat to use depends on what leaked.\n\n| What leaked | What to use | What happens | Covered in | \n|---|---|---|---|\n| A service token ( `est_…` ) | Revoke in Service Tokens | The token stops working with dotvault. The values do not change | Parts 7, 8 and 10 | \n| The value of a key that is not registered | Overwrite it with `dotvault set` | The new value goes out. The old one stays in the history and `rollback` can restore it | Part 4 | \n| The value of a registered key | `dotvault rotate --reason leak` | The new value goes out first, then the leaked value is revoked. Services keep running as long as the holders read the new value | This part | \n| The same, once every holder has the new value and you want to skip the remaining waits | `dotvault rotate --revoke-now` | The leaked value leaves dotvault's accepted set within seconds. Any service still holding the old value is rejected | This part | \n| The same, when it must stop right now | `dotvault rotate --retire` | Revoked at once. The holder and the verifier both stop | This part | \n\nFor a routine rotation when nothing has leaked, use `dotvault rotate` (`--reason manual`); the old value stays usable for at least 24 hours. Add `interval_seconds` to the registration, and dotvault starts that routine rotation on its own at the interval you set.\n\nRotating a key between two services fails calls whichever side you switch first, until the other side catches up. Register service-to-service keys for rotation management when you store them, and when one leaks, run `dotvault rotate <KEY> --reason leak`. dotvault gives the new value to the verifier, then to the holder, and revokes the leaked value only after both have read the new one, so the leaked value stops working without stopping your services. Set an interval when you register the key, and routine rotation before any leak stops being a person's job too.\n\ndotvault stores the secrets in your .env and config files server-side and hands them to any authorised developer, CI job or AI agent with a single command ([https://dotvault.io](https://dotvault.io)). It will be released free of charge when this series ends, and the steps in these articles are the steps you will use on day one.\n\nIf you find this useful, please like and follow.\n\nOur products are listed at [https://www.eln.ne.jp/products](https://www.eln.ne.jp/products)\n\n**EarthLink Network** builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:\n\n→ [The 18 products EarthLink Network builds in-house](https://dev.to/uehara/the-18-products-earthlink-network-builds-in-house-hb1)\n\nFor more about the company and each product, visit [www.eln.ne.jp](https://www.eln.ne.jp).", "url": "https://wpnews.pro/news/rotate-a-leaked-api-key-without-stopping-your-services-dotvault-hands-out-the", "canonical_source": "https://dev.to/uehara/rotate-a-leaked-api-key-without-stopping-your-services-dotvault-hands-out-the-new-value-first-408", "published_at": "2026-11-09 10:42:53+00:00", "updated_at": "2026-11-09 10:52:45.647330+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools"], "entities": ["dotvault", "order-worker", "payment-api"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rotate-a-leaked-api-key-without-stopping-your-services-dotvault-hands-out-the", "markdown": "https://wpnews.pro/news/rotate-a-leaked-api-key-without-stopping-your-services-dotvault-hands-out-the.md", "text": "https://wpnews.pro/news/rotate-a-leaked-api-key-without-stopping-your-services-dotvault-hands-out-the.txt", "jsonld": "https://wpnews.pro/news/rotate-a-leaked-api-key-without-stopping-your-services-dotvault-hands-out-the.jsonld"}}