AI Systems Need Runtime Budgets A developer argues that AI systems need runtime budgets that span entire workflows, not just per-call timeouts. The post details how individual dependencies can each stay within their limits while the overall request exceeds acceptable time, tokens, and cost, and proposes a unified budget with immutable limits and thread-safe accounting. A five-second model timeout does not give you a five-second AI feature. Retrieval may take one second. The first model call takes four. A tool takes another two, and the next model call gets its own five-second timeout. Every dependency stayed within its local limit, yet the request took 12 seconds and paid for two model calls. The complete use case needs one runtime budget. Its remaining allowance follows every model call, tool invocation, and retry. Timeouts still matter. They are just one line in the budget. Consider a support-answer workflow: php load the ticket - retrieve policy documents - ask the model - call a customer-history tool - ask the model again - validate the answer Each dependency can have a sensible timeout while the workflow still runs too long. The same applies to tokens and cost. Three calls with a 1,000-token output cap can generate 3,000 tokens. A retry repeats input, and tool results enlarge the next prompt. A runtime budget gives the workflow one shared envelope: | Dimension | What it bounds | What happens when it is exhausted | |---|---|---| | Wall-clock time | Total useful lifetime of the execution | Request cancellation and return a deadline result | | Input tokens | Context sent across all model calls | Reduce context before the call or stop | | Output tokens | Generated tokens across all model calls | Lower the next call's cap or stop | | Model calls | Initial calls, follow-ups, repairs, and retries | Do not start another call | | Tool calls | Automatic and application-directed invocations | Stop the loop before another tool runs | | Retries | Repeated attempts after transient failures | Return the chosen failure or recovery outcome | | Estimated cost | Accumulated cost of metered work | Stop or use a fallback whose cost already fits | These limits belong together because the dimensions interact. A retry can spend more time, tokens, and money. A tool call spends time and may make the next prompt larger. A longer output consumes more of the deadline. A budget will stop some work that might have succeeded. Good. A late or overpriced result is not a successful execution for that product. Keep the configured limits immutable. The live ledger belongs to one execution, which may or may not be an HTTP request. public sealed record RuntimeBudget TimeSpan MaxDuration, TimeSpan MaxModelAttemptDuration, TimeSpan MinUsefulModelAttemptDuration, TimeSpan CompletionHeadroom, int MaxModelCalls, int MaxToolCalls, int MaxRetries, long MaxInputTokens, long MaxOutputTokens, decimal MaxEstimatedCostUsd ; Validate the policy when configuration is loaded. The three attempt and execution durations must be positive. Headroom, counters, token limits, and cost limits cannot be negative. Require a positive output cap. The time limits must satisfy both invariants: MinUsefulModelAttemptDuration <= MaxModelAttemptDuration MinUsefulModelAttemptDuration + CompletionHeadroom <= MaxDuration Apply the same checks to reservations. Negative estimates must never create allowance. Use checked arithmetic or subtraction helpers to avoid overflow. Keep counters private and protect every compound read or mutation with the same lock. Return immutable snapshots created under that lock. Making one reservation method atomic does not make unrelated public properties thread-safe. I use four separate cancellation sources: The execution scope owns its deadline. Both elapsed-time accounting and cancellation use the same TimeProvider and start when the scope is created: js long startedAt = timeProvider.GetTimestamp ; using var deadlineCts = new CancellationTokenSource limits.MaxDuration, timeProvider ; TimeSpan RemainingTime { TimeSpan remaining = limits.MaxDuration - timeProvider.GetElapsedTime startedAt ; return remaining TimeSpan.Zero ? remaining : TimeSpan.Zero; } Using one clock keeps tests honest when they use FakeTimeProvider . A system timer should not expire independently from the fake clock used by the ledger. The per-attempt timeout must fit inside the time still available. If an execution has 1.4 seconds left, starting a model call with its normal five-second timeout is dishonest. Reserve completion headroom first, then calculate: attempt timeout = min configured attempt timeout, remaining execution time - completion headroom Reject the call when the result is below MinUsefulModelAttemptDuration . A one-millisecond attempt is traffic, not a serious chance of success. For each dependency call, create a time-provider-aware attempt source and link the sources only for delivery to that dependency: js using var attemptCts = new CancellationTokenSource reservation.AttemptTimeout, timeProvider ; using var callCts = CancellationTokenSource.CreateLinkedTokenSource callerToken, hostStoppingToken, deadlineCts.Token, attemptCts.Token ; Keep the original tokens. The linked token asks observing work to stop, but it does not record which source won. In this example, classification precedence is host shutdown, caller cancellation, execution deadline, then attempt timeout. Pick an order deliberately and use it everywhere. Cancellation is cooperative. It does not prove that local work stopped immediately, or that a remote provider stopped processing and billing. Post-call accounting is too late. The workflow has already spent the money and time. A model reservation needs a stable ID, estimated input tokens, an output cap, estimated cost, and an attempt timeout. Create it under the ledger lock only when every required dimension has enough allowance. The reservation lifecycle matters: php Reserved - Started - Reconciled Reserved work only when the application never handed it to the inner client or tool. Started work when required usage fields are available.At this abstraction level, Started means control was handed to the next layer. Local validation, serialization, or option conversion may still fail before a network request. Precise dispatch tracking requires instrumentation inside the provider or tool adapter. Without it, charge pessimistically after handoff. A caller timeout does not prove that the provider consumed nothing. The reservation operation should validate its arguments and calculate the attempt timeout inside the same lock that reserves calls, tokens, and cost. The core calculation is: available attempt time = remaining execution time - completion headroom attempt timeout = min configured attempt timeout, available attempt time output allowance = limit - committed output - reserved output Reject the reservation when the attempt timeout is below the minimum or the output allowance is zero. Subtract validated non-negative values and return zero once committed or reserved usage reaches the limit. Use the same approach for input and cost. The reservation supplies the per-call ChatOptions.MaxOutputTokens . That option limits one model request when the underlying client honors it. It is not a cumulative workflow limit. FunctionInvokingChatClient.GetResponseAsync ... can make several requests to its inner client. It sends tool results back to that inner client and continues until the tool loop ends. The final response aggregates available usage from those turns. One reservation around the outer call is wrong. It counts one model call even when the wrapper makes three, and its MaxOutputTokens value can be applied to each inner request. It also sees none of the locally invoked tools. Place model-call enforcement inside the function-invocation wrapper. Put any retry middleware you control outside that budgeting boundary so each attempt must reserve again: php application - FunctionInvokingChatClient - application retry layer - BudgetingChatClient - provider SDK There is a lifetime trap here. AddChatClient registers its pipeline as a singleton by default. A budgeting adapter that captures one execution's ledger or cancellation sources must not live in that singleton pipeline. Create the budgeting layer and its FunctionInvokingChatClient for each execution, or keep the adapter stateless and pass the current ledger through explicit per-call context. The following fragment is illustrative pseudocode. Types such as PromptEstimate , ModelReservation , and the ledger are application contracts, not a companion library. callCts.Token is the linked per-attempt token from the previous section. List