The Developer Guide for Setting Up the AI Gateway

Kong published a developer guide for configuring its AI Rate Limiting Advanced policy on the AI Gateway, which counts the tokens a provider reports in each response rather than counting requests, since a single request can consume 50 or 50,000 tokens. The guide walks through the policy's three form sections — general information, scope, and policy configuration — and instructs users to choose Scoped rather than Global, keep the identifier at its default of `consumer`, and set a limit of 60 tokens over a 60-second sliding window with the `total_tokens` count strategy. The guide states that limit and window size are the only two values the form requires, with every other field carrying a working default.

Traffic is now reaching OpenAI through the gateway, which is the point at which most teams want a limit on it. Rate limits are normally counted in requests, but a request is a weak unit for model traffic: one can use fifty tokens and the next fifty thousand. The \ AI Rate Limiting Advanced \ https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/ AI Rate Limiting Advanced https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/ policy counts the tokens the provider reports in each response, which is the number that tracks spend. Open your gateway, go to the Policies tab, and select New policy . Find AI Rate Limiting Advanced in the policy list. The form has three numbered sections. 1. General information - - This policy is enabled : leave it on, you can turn it off when it’s not needed. - - Display name : a name you will recognise, for example chat-default-token-limit . 2. Scope Choose Scoped , which applies the policy to specific models, MCP servers and agents.. Global would apply it to every model on the gateway, which is not what you want while one model is under test. 3. Policy configuration Identifier sets what the counter is keyed by. Accepted values are ip , credential , consumer , service, header , path and consumer-group. It defaults to consumer, which gives each authenticated caller its own budget. You have not configured consumers yet, so for now the limit applies across all traffic to this model. It starts behaving per caller once you add authentication . Leave it at consumer . Policies hold the rules. Select Add policies to create one. A rule has four parts. - - ID : an optional reference to a rate limiting policy stored elsewhere. Leave it empty and define the rule inline. - - Match : optional. Leave it empty and the rule applies to all traffic reaching this model. Covered below if you need to narrow it. - - Limits : required. This is where the number goes. Covered below. - - Window type : sliding , fixed or calendar. sliding is the default and the right choice here, because it measures the trailing window continuously rather than resetting on a clock boundary. Inside the rule, select Add limits : - - Limit : the number of tokens allowed in the window. Enter 60 , which is low enough to trigger while you are watching. - - Window size : the window in seconds, used by sliding and fixed . Enter 60 . - - Period : week or month , used only when Window type is calendar , as an alternative to Window size . Leave it empty. - - Week start day : monday through sunday , used only with a week period. Leave it empty. - - Month day : required when Period is month , ignored otherwise. Leave it empty. - - Tokens count strategy : which figure from the provider's usage object is counted. total tokens counts prompt plus completion, and is the default. The alternatives are prompt tokens, completion tokens and cost . Leave it at total tokens . Limit and Window size are the only two values this form actually needs. Everything else above has a working default. If you later want one rule per team rather than one for the whole model, select Add match inside the rule: - - Type : what to match on. Accepted values are consumer , consumer group , credential , ip , header , path , model , provider and service . - - Values : the values that satisfy the match, one per entry, added with Add values . - - Key : the name of the thing being read, for header and path matches. For a header match on X-Team , the key is X-Team and the values are the team names. - - Partition by : off by default, which counts all matching traffic against one shared budget. Turn it on to give each matched value its own counter. Below the rules: - - Strategy : where the counters live. local keeps them in the node's memory and is correct for the single node you are running. redis is required once you run more than one node, because local counts per node and the effective limit becomes your number multiplied by the node count. cluster is the third option. - - Dictionary name : the shared memory zone the local strategy writes counters to. Leave it at kong rate limiting counters . Leave Show additional settings closed. Create the policy, then attach it to chat-default when the wizard asks which entities it applies to. When you set a real limit later, account for agent traffic. A tool loop can fire five or six calls to satisfy one user request, so a number that looks generous per user can cut a conversation off partway through. To verify: