Rotate a leaked API key without stopping your services: dotvault hands out the new value first

A developer building the dotvault secrets-management tool demonstrated a rotation workflow that hands out a new key value to callers before revoking the old one, avoiding the ordering problem that normally forces services to be stopped during key rotation. In the demo, a caller polling at 15-second intervals had its value accepted in all 111 calls that received a response during rotation, then moved to the new value automatically while the leaked value was rejected. The tool also supports automatic rotation at a user-chosen interval, and is slated for free release at the end of the 12-part series.

This is part 11 of the series "dotvault, Start to Finish: One Command from Your Laptop to CI, Your Team, and Your AI Agents." dotvault will be released free of charge when this series ends. In the demo, a script kept calling with the key at 15-second intervals during the whole rotation, and in all 111 calls that got a response, the caller's value was accepted. The caller moved to the new value on its own, and the leaked value was rejected from then on. That matters because rotating a key between two of your services has an ordering problem. Switch the service that checks the key first, and the caller still holding the old value gets rejected. Switch the caller first, and the checker rejects a value it has never seen. Either way, calls fail until the other side catches up, and the usual way around it is to stop both and switch them together. This part shows how dotvault rotate --reason leak avoids that. This part also covers a second feature: rotating a key automatically at an interval you choose. Rotation is not only for leaks. It is part of routine key management. Set an interval when you register a key, and dotvault keeps rotating it on its own, even when nothing has leaked. Nobody has to rotate keys by hand any more, so you can keep each key's lifetime short. This is part 11 of 12. dotvault rotate <KEY --reason leak . interval seconds when you register the key, and as long as the holders and verifiers read from dotvault as part of their normal work, dotvault keeps rotating it at that interval, so you can keep the key's lifetime short. --revoke-now once every holder has the new value. This follows part 4 overwriting and rolling back values and part 7 creating service tokens . You can follow along without them. The demo: order-worker processes orders and calls payment-api , the payment API. It sends PAYMENT API KEY , and payment-api checks that the value is right. Assume PAYMENT API KEY has leaked. The terms used here: | Term | Meaning | |---|---| | Rotation management | dotvault generates the key's values and decides who receives them. Only registered keys are covered | | Holder | The side that holds the value and calls the other service. In the demo, the order-worker token | | Verifier | The side that checks the value it receives. In the demo, the payment-api token | | Rotate | Hand out a new value, then revoke the old one | | Retire | Revoke the value at once, without handing out a new one | Rotation works for keys that both sides receive from dotvault, which means keys between your own services. dotvault generates the new value. An API key issued by another company's service has to be replaced in that service's dashboard. In the demo project shop-backend , the first value of PAYMENT API KEY went into production with dotvault set . order-worker and payment-api each got a Read only service token. A service token is the string, starting with est , that CI or a service uses instead of logging in. Part 7 shows how to create one. Registering takes a token of the Credential rotation type. In Service Tokens, click Create token and pick "Credential rotation" as the Role. As the dialog says, this token can issue tokens and register keys only inside one project and the environments you pick. It is not needed once the key is registered, so it expires in 1 day. With that token, send one HTTP PUT, the request type that stores data at a URL. The body is JSON naming the holder and verifier tokens by Token ID. HTTP 201 created and "created":true mean the key is registered. The fields: | Field | What it holds | |---|---| | bindings | Who receives the key. tokenId is the service token's Token ID, role is holder or verifier , and label is a name you choose | | residual seconds | How long to wait, after every holder has moved to the new value, before revoking the old one. 60 seconds in the demo | | generator in the response | How new values are made. random-base64url-32 is 32 random bytes written as letters, digits and symbols | key= in the URL is the key name, and project= , env= and file= say where it lives. $DOTVAULT TOKEN in the Authorization header holds the Credential rotation token. A rotation cannot start in the first 5 minutes after registering. Register service-to-service keys when you store them, not after a leak. After registering, pull the same project as a person, with the holder token and with the verifier token. Only the holder, order-worker , gets the value. This screen was captured after the rotation in section 4; a registered key is handed out the same way before and after a rotation. The person an org admin and the verifier see the key name in a "withheld 1 rotation-managed key s " warning, with no value. If you need a key with that name for local development, store a development value under a different key name. The verifier does not pull. It reads the set of values it should accept from GET /v1/env/key-set . Each entry has the value, an ID for the value keyId and a state state . The verifier accepts a value that is in this set. To see whether calls keep working during the rotation, a small script, service.mjs , ran at 15-second intervals. Each round, it: PAYMENT API KEY with the order-worker token payment-api token, and prints accepted if the value is in it, REJECTED if not phase It prints only the first 8 characters of each keyId , never the value. With the script running, an org admin signed in to the CLI runs: dotvault rotate PAYMENT API KEY -E production --reason leak started rotation means it is running. generation 2 means this is the key's second rotation; the first was the routine rotation in section 7, which was aborted. A leak rotation cannot be stopped. --abort returns ABORT REFUSED LEAK with exit code 7, so there is no path back to the leaked value. --status shows how far the rotation has got. In the audit log, the rotation reached STAGED in the same second it started. The rotation goes in this order. | phase | What happens | |---|---| | STAGED | dotvault generates a new value and adds it to the verifier's set. The holder still gets the old value | | PROMOTED | About 15 minutes after the verifier is seen reading the new value. The holder's pull starts returning the new value. The leaked value stays in the set | | HOLDERS OK | About 15 minutes after the holder is seen reading the new value | | DONE | residual seconds 60 seconds after HOLDERS OK , the leaked value is revoked and leaves the set | The table leaves out LOCKED , the phase at the start the audit log shows STAGED in the same second as the start , and VERIFIERS OK , the mark that the verifier has read, which is recorded in the same second as PROMOTED . The 15-minute waits exist because holders and verifiers may keep a value or a set they read for up to 5 minutes. Several instances can share one token and they do not all read at the same moment, so 10 more minutes are added. --reason leak does not skip these waits on its own. In a leak rotation, if one side never reads, the rotation moves on anyway after 30 minutes plus the 5 minutes a read may be kept . Here is the script's output around each change of phase . The first two ABORTED lines are the state left by the routine rotation aborted in section 7. Three things show up: STAGED , the new value k iVota- , state staged joins the set, while the holder keeps using the old value k Z0Njns and is still accepted. PROMOTED , the holder's value is retiring . DONE , the set holds only the new value, and the leaked value is Over the whole run, the script made 124 rounds. In all 111 that got a response, the holder's value was accepted . The other 13, error fetch failed , were rounds where the script's network request failed, so no response from dotvault came back. The console shows the rotation too. In the project's Keys, select PAYMENT API KEY and a Rotation section appears on the right. Besides Phase, the Rotation panel shows when the rotation is next due to move on Next due and, in a leak rotation, when it moves on even if a side has not read the new value Deadline . The figure leaves those two lines out. After DONE , check the state and try writing the leaked value back. In dotvault versions , the new v3 is state=current and the leaked v1 is state=revoked . v3 and v2 were made by system:rotation-driver , which is dotvault itself. v2 belongs to the routine rotation aborted in section 7. Writing the leaked value with dotvault set , or rolling back to v1 with dotvault rollback --to 1 , both fail with ROTATION MANAGED : a key under rotation management changes only through rotation. As part 4 showed, an ordinary key can be rolled back to an old value after an overwrite. A key under rotation management cannot go back to the leaked value, even if someone tries with good intentions. In the console's Audit Log, filter with shop-backend:production:env:PAYMENT API KEY as the Target ID: project, environment, file and key name joined by : . The leak rotation's rows, from the bottom up. 14582408-… is the user ID of dev-e, the demo user. | Action | Actor | Meaning | |---|---|---| | secret.rotation.started | 14582408-… , cli | dev-e started the rotation from the CLI | | secret.rotation.staged | system:rotation-driver | A new value was generated and added to the verifier's set | | secret.rotation.verifiers ok | system:rotation-driver | The 15-minute wait after the verifier read the new value is over | | secret.rotation.promoted | system:rotation-driver | The holder started receiving the new value | | secret.rotation.holders ok | system:rotation-driver | The 15-minute wait after the holder read the new value is over | | secret.rotation.revoked | system:rotation-driver | The leaked value was revoked | | secret.rotation.completed | system:rotation-driver | The rotation finished | A person started the rotation and dotvault did the rest. That the leaked value was revoked is one row you can point to later. When nothing has leaked, a routine rotation runs without --reason the same as --reason manual . Two differences from a leak rotation: --abort stopped it in a few seconds: the new value became revoked and the old value stayed current . waiting on verifier payment-api means payment-api has not read the new value yet. last ok is when it last read the set. The phase deadline of a routine rotation 24 hours after the start is how long it waits for a side that has not read the new value. Past it, the rotation does not move on; it stops STALLED . Even when nothing has leaked, you may not want to keep one key for a long time. Add interval seconds the rotation interval, in seconds when you register the key, and dotvault starts a routine rotation on its own at that interval. Rotation stops being a person's job. In the demo, a new key, WEBHOOK SECRET , was registered with the same holder and verifier tokens as in section 1, with interval seconds set to 600 10 minutes . Ten minutes is only for the recording; in real use you would pick something like 30 days 2592000 . Right after registering, --status says managed but has never run : the key is registered but has never been rotated. Nobody typed a rotate command after that the only thing running was the script that reads values . Once the 10-minute interval had passed, --status showed reason scheduled dotvault started this rotation itself because the interval came round , and the rotation had already added the new value to the verifier's accepted set STAGED . The audit log agrees. After the registration rotation provisioner.config created , both secret.rotation.started and secret.rotation.staged have the Actor system:rotation-driver . In the leak rotation in section 6, a person was the Actor of started ; here dotvault started the rotation too. A scheduled rotation is the same as the routine rotation in section 7: the old and new values both stay valid for at least 24 hours, and the new value goes to the verifier first, then to the holder. The next rotation starts once the interval has passed after the previous one finished. Right after you register a key, or shorten its interval, nothing starts for 5 minutes. A rotation only reaches the end when the holders and verifiers read from dotvault as part of their normal work. If a service never reads again, the rotation waits on it with the waiting on … line from section 7, and past the deadline it stops STALLED . A rotation accepts the leaked value until the new one is everywhere. Even after the holder has read the new value, there is a 15-minute wait and the 60-second residual. To make the leaked value stop working sooner, skip the remaining waits with --revoke-now . dotvault rotate REFUND API KEY --revoke-now --confirm REFUND API KEY -E production This was recorded after PAYMENT API KEY had been retired later in this section , on a second key, REFUND API KEY , registered the same way. One of the script's calls already carried the new value PROMOTED . Then --revoke-now : in the audit log, revoke now , revoked and completed share the same second, and from the next call that got a response, the leaked value was REJECTED. The holder's value stayed accepted. The script ran 74 times during this recording, and in all 57 calls that got a response, the holder's value was accepted the other 17 failed on the script's side before reaching dotvault . Run --revoke-now once every holder has the new value, that is, after PROMOTED. It skips every wait from that point on, so if you run it earlier, dotvault switches values and revokes the old one without waiting for the verifier or the holder to read the new one, and a holder still on the old value is rejected. The waiting on holder … line in --status is per token. When several services share one token, the line disappears as soon as one of them reads. Any other service still holding the old value is rejected after --revoke-now ; that is what 10 of the 15 minutes of waiting are for. Run it only when you know every service has pulled again, for example after restarting them. In the demo the holder's value stayed accepted because there was one holder and it pulled on every call. One more thing: revoking takes the value out of dotvault's accepted set. While a verifier keeps its copy of the set up to 5 minutes , it may still accept the leaked value. The demo's verifier read the set on every call, so the leaked value was REJECTED from the next call. --confirm repeats the key name; one wrong character stops it with VALIDATION ERROR and nothing is sent. Only a signed-in person can run it; a service token cannot. If you cannot even wait for the holder to move to the new value, retire the key. dotvault rotate PAYMENT API KEY --retire --confirm PAYMENT API KEY -E production The retire ran on the same key after the PAYMENT API KEY rotation was recorded. revoked version s : 3 means it revoked v3, the current value from the rotation; v1 and v2 were already revoked. The verifier can no longer read the set HTTP 403 , NOT BOUND , and writing the leaked value back with set fails with VALUE REVOKED : a revoked value can never be written again. Retire hands out no new value, so the holder and the verifier both stop at once. --confirm repeats the key name, and one wrong character means nothing happens. Only a signed-in person can retire; a service token cannot. Use it only when the leaked value has to stop working right now, even at the cost of stopping your services. What to use depends on what leaked. | What leaked | What to use | What happens | Covered in | |---|---|---|---| | A service token est … | Revoke in Service Tokens | The token stops working with dotvault. The values do not change | Parts 7, 8 and 10 | | The value of a key that is not registered | Overwrite it with dotvault set | The new value goes out. The old one stays in the history and rollback can restore it | Part 4 | | The value of a registered key | dotvault rotate --reason leak | The new value goes out first, then the leaked value is revoked. Services keep running as long as the holders read the new value | This part | | The same, once every holder has the new value and you want to skip the remaining waits | dotvault rotate --revoke-now | The leaked value leaves dotvault's accepted set within seconds. Any service still holding the old value is rejected | This part | | The same, when it must stop right now | dotvault rotate --retire | Revoked at once. The holder and the verifier both stop | This part | For a routine rotation when nothing has leaked, use dotvault rotate --reason manual ; the old value stays usable for at least 24 hours. Add interval seconds to the registration, and dotvault starts that routine rotation on its own at the interval you set. Rotating a key between two services fails calls whichever side you switch first, until the other side catches up. Register service-to-service keys for rotation management when you store them, and when one leaks, run dotvault rotate <KEY --reason leak . dotvault gives the new value to the verifier, then to the holder, and revokes the leaked value only after both have read the new one, so the leaked value stops working without stopping your services. Set an interval when you register the key, and routine rotation before any leak stops being a person's job too. dotvault stores the secrets in your .env and config files server-side and hands them to any authorised developer, CI job or AI agent with a single command https://dotvault.io https://dotvault.io . It will be released free of charge when this series ends, and the steps in these articles are the steps you will use on day one. If you find this useful, please like and follow. Our products are listed at https://www.eln.ne.jp/products https://www.eln.ne.jp/products EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build: → The 18 products EarthLink Network builds in-house https://dev.to/uehara/the-18-products-earthlink-network-builds-in-house-hb1 For more about the company and each product, visit www.eln.ne.jp https://www.eln.ne.jp .