Rate limits and budgets constrain resource use. They can reduce the impact of some abuse and software failures, but they cannot establish whether a request is authorized. Choose each limit according to the resource it protects and the legitimate workload it must support.
Match the control to the resource #
| Control | What it bounds | What it can miss |
|---|---|---|
| Requests per interval | Request arrival rate | Large or expensive individual requests |
| Tokens per interval | Token throughput under the counter's semantics | Model price differences and accounting delay |
| Concurrent requests | Simultaneous in-flight work | Sequential misuse over a long period |
| Spend budget | Cost according to the configured accounting model | Delayed usage updates and in-flight work |
| Request/output size and time bounds | Some per-request resource exposure | Many requests below each limit |
OWASP's Unrestricted Resource Consumption guidance recommends limiting resource use and interaction frequency. For an inference service, the operational design still needs to account for model selection, variable token use, and the scope of each counter.
Choose the identity and aggregation scope #
A per-key limit protects one credential's allowance, while a shared organization budget can constrain aggregate use across many keys. Consider whether new credentials reset the relevant entitlement. Use network-level controls carefully because several legitimate customers may share infrastructure.
Document whether limits apply per process, deployment, region, or globally. A locally enforced counter should not be described as a global cap without testing the coordination mechanism. Define what happens when the counter store is unavailable.
Test the boundary, not just the happy path #
- Run a legitimate representative workload near the proposed limit.
- Test bursts, streaming cancellation, retries, and simultaneous requests across replicas.
- Check when usage is reserved and reconciled, including missing or delayed terminal events.
- Observe customer errors and retry behavior after throttling; a badly behaved client can amplify attempts.
- Verify the budget resets and administrative override process.
A budget is not always an exact cutoff #
Synthetic example: recorded usage is $9.90 against a $10 budget when several requests are already running. Their final usage can arrive after the budget check. The resulting overshoot depends on the implementation and reservation strategy. Measure it in staging rather than promising that the bill cannot exceed $10.
State whether the budget is an alert, an admission decision, or another enforcement mechanism. Those are different behaviors. Also state whether amounts use estimated cost or provider-confirmed charges.
Combine constraints with investigation #
A compromised key can remain under every consumption limit. Keep ownership records, credential response procedures, and metadata-based review alongside controls. When increasing a legitimate customer's limit, preserve who approved the change and why.
Track rejected legitimate work, control failures, overshoot, and the incidents the limits contained. Continue with account farming, usage-spike triage, or spend baselines.