# The Developer Guide for Setting Up the AI Gateway

> Source: <https://konghq.com/blog/engineering/setting-up-the-ai-gateway>
> Published: 2026-10-28 08:42:00+00:00

Traffic is now reaching OpenAI through the gateway, which is the point at which most teams want a limit on it. Rate limits are normally counted in requests, but a request is a weak unit for model traffic: one can use fifty tokens and the next fifty thousand. The [\[___AI Rate Limiting Advanced___\](https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/)___AI Rate Limiting Advanced___](https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/) policy counts the tokens the provider reports in each response, which is the number that tracks spend.

Open your gateway, go to the ****Policies**** tab, and select ****New policy****. Find ****AI Rate Limiting Advanced**** in the policy list. The form has three numbered sections.

****1. General information****

- - ****This policy is enabled**** : leave it on, you can turn it off when it’s not needed.
- - ****Display name**** : a name you will recognise, for example`` `chat-default-token-limit` `` .

****2. Scope****

Choose ****Scoped****, which applies the policy to specific models, MCP servers and agents.. ****Global**** would apply it to every model on the gateway, which is not what you want while one model is under test.

****3. Policy configuration****

****Identifier**** sets what the counter is keyed by. Accepted values are `` `ip` ``, `` `credential` ``, `` `consumer` ``, `` `service,` `` `` `header` ``, `` `path` `` and `` `consumer-group.` `` It defaults to `` `consumer,` `` which gives each authenticated caller its own budget. You have not configured consumers yet, so for now the limit applies across all traffic to this model. It starts behaving per caller once you add authentication . Leave it at `` `consumer` ``.

****Policies**** hold the rules. Select ****Add policies**** to create one. A rule has four parts.

- - ****ID**** : an optional reference to a rate limiting policy stored elsewhere. Leave it empty and define the rule inline.
- - ****Match**** : optional. Leave it empty and the rule applies to all traffic reaching this model. Covered below if you need to narrow it.
- - ****Limits**** : required. This is where the number goes. Covered below.
- - ****Window type**** :`` `sliding` `` ,`` `fixed` `` or`` `calendar. sliding ` `` is the default and the right choice here, because it measures the trailing window continuously rather than resetting on a clock boundary.

Inside the rule, select ****Add limits****:

- - ****Limit**** : the number of tokens allowed in the window. Enter`` `60` `` , which is low enough to trigger while you are watching.
- - ****Window size**** : the window in seconds, used by`` ` sliding ` `` and`` `fixed` `` . Enter`` `60` `` .
- - ****Period**** :`` `week` `` or`` `month` `` , used only when****Window type**** is`` `calendar` `` , as an alternative to****Window size**** . Leave it empty.
- - ****Week start day**** :`` `monday` `` through`` `sunday` `` , used only with a week period. Leave it empty.
- - ****Month day**** : required when****Period**** is`` `month` `` , ignored otherwise. Leave it empty.
- - ****Tokens count strategy**** : which figure from the provider's`` `usage` `` object is counted.`` `total_tokens` `` counts prompt plus completion, and is the default. The alternatives are`` `prompt_tokens, completion_tokens` `` and`` `cost` `` . Leave it at`` `total_tokens` `` .

****Limit**** and ****Window size**** are the only two values this form actually needs. Everything else above has a working default.

If you later want one rule per team rather than one for the whole model, select ****Add match**** inside the rule:

- - ****Type**** : what to match on. Accepted values are`` `consumer` `` ,`` `consumer_group` `` ,`` `credential` `` ,`` `ip` `` ,`` `header` `` ,`` `path` `` ,`` `model` `` ,`` `provider` `` and`` `service` `` .
- - ****Values**** : the values that satisfy the match, one per entry, added with****Add values**** .
- - ****Key**** : the name of the thing being read, for header and path matches. For a`` `header ` `` match on`` `X-Team` `` , the key is`` `X-Team` `` and the values are the team names.
- - ****Partition by**** : off by default, which counts all matching traffic against one shared budget. Turn it on to give each matched value its own counter.

Below the rules:

- - ****Strategy**** : where the counters live.`` `local` `` keeps them in the node's memory and is correct for the single node you are running.`` `redis` `` is required once you run more than one node, because`` `local` `` counts per node and the effective limit becomes your number multiplied by the node count.`` `cluster ` `` is the third option.
- - ****Dictionary name**** : the shared memory zone the`` `local ` `` strategy writes counters to. Leave it at`` `kong_rate_limiting_counters` `` .

Leave ****Show additional settings**** closed. Create the policy, then attach it to `` `chat-default ` ``when the wizard asks which entities it applies to.

When you set a real limit later, account for agent traffic. A tool loop can fire five or six calls to satisfy one user request, so a number that looks generous per user can cut a conversation off partway through.

****To verify:****