The Developer Guide for Setting Up the AI Gateway

Traffic is now reaching OpenAI through the gateway, which is the point at which most teams want a limit on it. Rate limits are normally counted in requests, but a request is a weak unit for model traffic: one can use fifty tokens and the next fifty thousand. The [AI Rate Limiting Advanced](https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/)AI Rate Limiting Advanced policy counts the tokens the provider reports in each response, which is the number that tracks spend.

Open your gateway, go to the Policies tab, and select New policy. Find AI Rate Limiting Advanced in the policy list. The form has three numbered sections.

1. General information

• • This policy is enabled : leave it on, you can turn it off when it’s not needed.
• • Display name : a name you will recognise, for example`chat-default-token-limit` . 2. Scope

Choose Scoped, which applies the policy to specific models, MCP servers and agents.. Global would apply it to every model on the gateway, which is not what you want while one model is under test.

3. Policy configuration

Identifier sets what the counter is keyed by. Accepted values are `ip`, `credential`, `consumer`, `service,` `header`, `path` and `consumer-group.` It defaults to `consumer,` which gives each authenticated caller its own budget. You have not configured consumers yet, so for now the limit applies across all traffic to this model. It starts behaving per caller once you add authentication . Leave it at `consumer`.

Policies hold the rules. Select Add policies to create one. A rule has four parts.

• • ID : an optional reference to a rate limiting policy stored elsewhere. Leave it empty and define the rule inline.
• • Match : optional. Leave it empty and the rule applies to all traffic reaching this model. Covered below if you need to narrow it.
• • Limits : required. This is where the number goes. Covered below.
• • Window type :`sliding` ,`fixed` or`calendar. sliding ` is the default and the right choice here, because it measures the trailing window continuously rather than resetting on a clock boundary.

Inside the rule, select Add limits:

• • Limit : the number of tokens allowed in the window. Enter`60` , which is low enough to trigger while you are watching.
• • Window size : the window in seconds, used by` sliding ` and`fixed` . Enter`60` .
• • Period :`week` or`month` , used only whenWindow type is`calendar` , as an alternative toWindow size . Leave it empty.
• • Week start day :`monday` through`sunday` , used only with a week period. Leave it empty.
• • Month day : required whenPeriod is`month` , ignored otherwise. Leave it empty.
• • Tokens count strategy : which figure from the provider's`usage` object is counted.`total_tokens` counts prompt plus completion, and is the default. The alternatives are`prompt_tokens, completion_tokens` and`cost` . Leave it at`total_tokens` .

Limit and Window size are the only two values this form actually needs. Everything else above has a working default.

If you later want one rule per team rather than one for the whole model, select Add match inside the rule:

• • Type : what to match on. Accepted values are`consumer` ,`consumer_group` ,`credential` ,`ip` ,`header` ,`path` ,`model` ,`provider` and`service` .
• • Values : the values that satisfy the match, one per entry, added withAdd values .
• • Key : the name of the thing being read, for header and path matches. For a`header ` match on`X-Team` , the key is`X-Team` and the values are the team names.
• • Partition by : off by default, which counts all matching traffic against one shared budget. Turn it on to give each matched value its own counter.

Below the rules:

• • Strategy : where the counters live.`local` keeps them in the node's memory and is correct for the single node you are running.`redis` is required once you run more than one node, because`local` counts per node and the effective limit becomes your number multiplied by the node count.`cluster ` is the third option.
• • Dictionary name : the shared memory zone the`local ` strategy writes counters to. Leave it at`kong_rate_limiting_counters` .

Leave Show additional settings closed. Create the policy, then attach it to `chat-default `when the wizard asks which entities it applies to.

When you set a real limit later, account for agent traffic. A tool loop can fire five or six calls to satisfy one user request, so a number that looks generous per user can cut a conversation off partway through.

To verify: