A five-second model timeout does not give you a five-second AI feature.
Retrieval may take one second. The first model call takes four. A tool takes another two, and the next model call gets its own five-second timeout. Every dependency stayed within its local limit, yet the request took 12 seconds and paid for two model calls.
The complete use case needs one runtime budget. Its remaining allowance follows every model call, tool invocation, and retry.
Timeouts still matter. They are just one line in the budget.
Consider a support-answer workflow:
load the ticket
-> retrieve policy documents
-> ask the model
-> call a customer-history tool
-> ask the model again
-> validate the answer
Each dependency can have a sensible timeout while the workflow still runs too long. The same applies to tokens and cost. Three calls with a 1,000-token output cap can generate 3,000 tokens. A retry repeats input, and tool results enlarge the next prompt.
A runtime budget gives the workflow one shared envelope:
| Dimension | What it bounds | What happens when it is exhausted |
|---|---|---|
| Wall-clock time | Total useful lifetime of the execution | Request cancellation and return a deadline result |
| Input tokens | Context sent across all model calls | Reduce context before the call or stop |
| Output tokens | Generated tokens across all model calls | Lower the next call's cap or stop |
| Model calls | Initial calls, follow-ups, repairs, and retries | Do not start another call |
| Tool calls | Automatic and application-directed invocations | Stop the loop before another tool runs |
| Retries | Repeated attempts after transient failures | Return the chosen failure or recovery outcome |
| Estimated cost | Accumulated cost of metered work | Stop or use a fallback whose cost already fits |
These limits belong together because the dimensions interact. A retry can spend more time, tokens, and money. A tool call spends time and may make the next prompt larger. A longer output consumes more of the deadline.
A budget will stop some work that might have succeeded. Good. A late or overpriced result is not a successful execution for that product.
Keep the configured limits immutable. The live ledger belongs to one execution, which may or may not be an HTTP request.
public sealed record RuntimeBudget(
TimeSpan MaxDuration,
TimeSpan MaxModelAttemptDuration,
TimeSpan MinUsefulModelAttemptDuration,
TimeSpan CompletionHeadroom,
int MaxModelCalls,
int MaxToolCalls,
int MaxRetries,
long MaxInputTokens,
long MaxOutputTokens,
decimal MaxEstimatedCostUsd);
Validate the policy when configuration is loaded. The three attempt and execution durations must be positive. Headroom, counters, token limits, and cost limits cannot be negative. Require a positive output cap. The time limits must satisfy both invariants:
MinUsefulModelAttemptDuration <= MaxModelAttemptDuration
MinUsefulModelAttemptDuration + CompletionHeadroom <= MaxDuration
Apply the same checks to reservations. Negative estimates must never create allowance. Use checked arithmetic or subtraction helpers to avoid overflow.
Keep counters private and protect every compound read or mutation with the same lock. Return immutable snapshots created under that lock. Making one reservation method atomic does not make unrelated public properties thread-safe.
I use four separate cancellation sources:
The execution scope owns its deadline. Both elapsed-time accounting and cancellation use the same TimeProvider
and start when the scope is created:
long startedAt = timeProvider.GetTimestamp();
using var deadlineCts = new CancellationTokenSource(
limits.MaxDuration,
timeProvider);
TimeSpan RemainingTime()
{
TimeSpan remaining = limits.MaxDuration
- timeProvider.GetElapsedTime(startedAt);
return remaining > TimeSpan.Zero ? remaining : TimeSpan.Zero;
}
Using one clock keeps tests honest when they use FakeTimeProvider
. A system timer should not expire independently from the fake clock used by the ledger.
The per-attempt timeout must fit inside the time still available. If an execution has 1.4 seconds left, starting a model call with its normal five-second timeout is dishonest. Reserve completion headroom first, then calculate:
attempt timeout = min(
configured attempt timeout,
remaining execution time - completion headroom)
Reject the call when the result is below MinUsefulModelAttemptDuration
. A one-millisecond attempt is traffic, not a serious chance of success.
For each dependency call, create a time-provider-aware attempt source and link the sources only for delivery to that dependency:
using var attemptCts = new CancellationTokenSource(
reservation.AttemptTimeout,
timeProvider);
using var callCts = CancellationTokenSource.CreateLinkedTokenSource(
callerToken,
hostStoppingToken,
deadlineCts.Token,
attemptCts.Token);
Keep the original tokens. The linked token asks observing work to stop, but it does not record which source won. In this example, classification precedence is host shutdown, caller cancellation, execution deadline, then attempt timeout. Pick an order deliberately and use it everywhere.
Cancellation is cooperative. It does not prove that local work stopped immediately, or that a remote provider stopped processing and billing.
Post-call accounting is too late. The workflow has already spent the money and time.
A model reservation needs a stable ID, estimated input tokens, an output cap, estimated cost, and an attempt timeout. Create it under the ledger lock only when every required dimension has enough allowance.
The reservation lifecycle matters:
Reserved -> Started -> Reconciled
Reserved
work only when the application never handed it to the inner client or tool.Started
work when required usage fields are available.At this abstraction level, Started
means control was handed to the next layer. Local validation, serialization, or option conversion may still fail before a network request. Precise dispatch tracking requires instrumentation inside the provider or tool adapter. Without it, charge pessimistically after handoff. A caller timeout does not prove that the provider consumed nothing.
The reservation operation should validate its arguments and calculate the attempt timeout inside the same lock that reserves calls, tokens, and cost. The core calculation is:
available attempt time = remaining execution time - completion headroom
attempt timeout = min(configured attempt timeout, available attempt time)
output allowance = limit - committed output - reserved output
Reject the reservation when the attempt timeout is below the minimum or the output allowance is zero. Subtract validated non-negative values and return zero once committed or reserved usage reaches the limit. Use the same approach for input and cost.
The reservation supplies the per-call ChatOptions.MaxOutputTokens
. That option limits one model request when the underlying client honors it. It is not a cumulative workflow limit.
FunctionInvokingChatClient.GetResponseAsync(...)
can make several requests to its inner client. It sends tool results back to that inner client and continues until the tool loop ends. The final response aggregates available usage from those turns.
One reservation around the outer call is wrong. It counts one model call even when the wrapper makes three, and its MaxOutputTokens
value can be applied to each inner request. It also sees none of the locally invoked tools.
Place model-call enforcement inside the function-invocation wrapper. Put any retry middleware you control outside that budgeting boundary so each attempt must reserve again:
application
-> FunctionInvokingChatClient
-> application retry layer
-> BudgetingChatClient
-> provider SDK
There is a lifetime trap here. AddChatClient
registers its pipeline as a singleton by default. A budgeting adapter that captures one execution's ledger or cancellation sources must not live in that singleton pipeline. Create the budgeting layer and its FunctionInvokingChatClient
for each execution, or keep the adapter stateless and pass the current ledger through explicit per-call context.
The following fragment is illustrative pseudocode. Types such as PromptEstimate
, ModelReservation
, and the ledger are application contracts, not a companion library. callCts.Token
is the linked per-attempt token from the previous section.
List<ChatMessage> messageList = [.. messages];
ChatOptions callOptions = options?.Clone() ?? new ChatOptions();
PromptEstimate estimate = promptEstimator.Estimate(
messageList,
callOptions,
modelProfile);
ModelReservation reservation = budget.ReserveModelCall(
estimate,
callOptions.MaxOutputTokens);
callOptions.MaxOutputTokens = reservation.OutputTokenCap;
budget.MarkStarted(reservation); // Controlled handoff, not confirmed dispatch.
ChatResponse response = await innerClient.GetResponseAsync(
messageList,
callOptions,
callCts.Token);
budget.ReconcileModel(reservation, response.Usage);
Materializing messages
once avoids enumerating an arbitrary IEnumerable<ChatMessage>
twice. The estimator needs more than message text. Depending on the provider, the effective prompt can include ChatOptions.Instructions
, tool declarations and JSON schemas, structured-output schemas, multimodal content, and provider-specific framing. Estimate
is the honest name because local preflight cannot always produce an exact count.
Stateful clients make that limit clearer. If ConversationId
refers to history stored by the provider, the application may not possess the complete context. Reserve conservatively or disable local claims of exact input accounting for that path.
After the controlled handoff, charge the reservation on an exception unless reliable usage data allows reconciliation. The fragment omits that branch and the streaming override. A production adapter needs both.
Guard locally executed tools through FunctionInvokingChatClient.FunctionInvoker
. Validate known preconditions before reserving, then mark the reservation Started
immediately before context.Function.InvokeAsync(context.Arguments, cancellationToken)
.
On success or a known failure, reconcile from the request, response metadata, usage record, and failure details available to that tool adapter. If those sources cannot resolve the cost after handoff, charge the reservation. This is deliberately pessimistic. Parallel tool calls need atomic reservations before any task starts.
Hard tool-call limits apply only to execution you control. Provider-managed or server-side tools can be opaque. In that case, enforce the controls the provider exposes, account for reported usage, and avoid claiming that the application counted every tool call.
An application-owned tool loop is the other valid design. It is more code, but it makes every model and tool boundary visible without relying on wrapper placement.
ChatResponse.Usage
being non-null does not mean every count is present. InputTokenCount
, OutputTokenCount
, cached input, and reasoning counts are nullable. Reconcile each field required by your policy. For a missing required field, charge its reservation or stop with unknown usage.
Per-call interception avoids another ambiguity. FunctionInvokingChatClient
adds the non-null usage objects returned by its inner calls. If one turn omits usage and another reports it, the outer response can contain a plausible but incomplete total.
Do not add cached input tokens to InputTokenCount
again. Do not add reasoning tokens to OutputTokenCount
again. Those detail counts classify pricing; the documented totals already include them.
Cost estimation needs a reviewed pricing key, not only a model ID. Depending on the provider, it may include the provider, deployment or model snapshot, region, service tier, modality, batch mode, and cached-token rules.
An application-estimated cost budget is not a provider billing ceiling. Rounding, missing usage, and price changes can make the invoice differ. Use provider-side quotas and billing controls separately, and know whether they alert or block spend.
The total cost dimension must include every paid boundary the article claims to budget. Retrieval services, rerankers, paid tools, and external APIs need their own reservations. If the ledger only accounts for model calls, name the limit MaxEstimatedModelCostUsd
instead.
A retry does not receive a fresh deadline or token allowance. Do not start one that cannot finish inside the remaining time or would exceed another dimension.
Put application retry middleware outside BudgetingChatClient
, as shown above. Each retry attempt then crosses the budgeting boundary and needs a new reservation.
Before repeating an attempt, the retry layer must atomically reserve one retry slot. The inner budgeting client separately reserves the repeated model or dependency call.
Retries inside the provider SDK are different. Configure or disable them when possible. If the SDK retries internally without an attempt callback, the application cannot claim an exact retry or model-call count. Treat that layer as opaque and reserve conservatively.
A cost-exhausted execution may use a fallback only when that fallback is free, has separately reserved capacity, or still fits inside the remaining allowance.
OperationCanceledException
alone does not explain what happened. It might mean the user disconnected, the host stopped, a dependency attempt timed out, or the complete runtime deadline expired.
Select a stop reason from the original source tokens, not from the linked token. Apply the same precedence at every boundary. This is a policy decision based on the tokens observed at inspection time; it does not prove which token caused the cancellation first.
For example, a SelectStopReason
helper can check host shutdown first, then caller cancellation, the execution deadline, and finally the attempt timer. Map those to unavailable, cancelled, time-budget exhaustion, and dependency timeout results. Ledger failures should carry the exhausted dimension directly rather than masquerading as cancellation.
The user-facing message need not mention tokens or internal prices. Keep the reason in the application result for telemetry, support, and fallback decisions.
A partial answer needs its own policy. Some features can label and return one. Others should return nothing because an incomplete answer would be misleading.
Record both sides of the ledger. A trace that says model.duration = 3.2s
does not say whether 3.2 seconds was healthy for this feature.
For one execution, I would record:
Do not copy prompts, retrieved documents, or tool results into telemetry just to explain the numbers. Operational metadata usually answers the budget question without duplicating sensitive content.
Per-execution budgets stop one workflow from running away. They do nothing about ten thousand executions that each stay inside budget. Rate limits, concurrency limits, tenant quotas, and dependency-level retry budgets protect that shared capacity.
There is no universal runtime budget. A chat interaction and an overnight evaluation run have different definitions of "too late." Start with how long the caller will wait, what one result may cost, which tools it needs, and whether a fallback still has value.
Test the complete path under normal load and injected failure. Measure tail latency as well as averages. Include prompt growth, tool output, throttling, and retry delays. Test the effects of opaque SDK retries even when the SDK does not expose an exact attempt count.
Leave headroom between the internal deadline and the external timeout. The application still needs to map the result, write telemetry, and respond. If both expire together, the caller may receive a connection failure instead of the intended outcome.
Treat the first values as a hypothesis. Production traces and cost data will show where the budget is too loose, too strict, or spent on the wrong step.
Use an execution-scoped budget when a workflow can make more than one metered or remote call, invoke tools, retry, or fan out. It gives the orchestration layer one answer to the question: may this execution start more work?
For interactive work, that execution normally observes caller cancellation. Deliberate background work needs an independent budget instead of silently outliving the request. Durable workflows also need persisted reservations and idempotent reconciliation so a restart cannot reset or double-charge the ledger.
A local spike or one low-risk model call may only need cancellation, a timeout, and an output cap. Add a shared ledger when the workflow can multiply time or spend, not when the object would add ceremony without changing a decision.
Pick one AI use case and write down the maximum wall-clock time, cumulative tokens, model calls, tool calls, retries, and estimated cost for one execution.
Put those limits in one policy. Create one ledger when the execution begins. Before every new attempt, reserve what it can spend. After each metered operation, reconcile its reservation with the available usage or cost data. When a required dimension lacks enough allowance, return a named application outcome instead of starting one more call.
The next model call now starts only when the complete execution can still afford it.
ChatOptions.MaxOutputTokens
ChatOptions.Clone()
ChatResponse.Usage
AddChatClient
service lifetimeFunctionInvokingChatClient
FunctionInvokingChatClient.cs
FunctionInvokingChatClient.FunctionInvoker
UsageDetails
CachedInputTokenCount
ReasoningTokenCount