{"slug": "allow-google-agent-traffic-cdn-waf-and-web-bot-auth-for-agency-sites", "title": "Allow Google-Agent Traffic: CDN, WAF, and Web Bot Auth for Agency Sites", "summary": "Google published documentation explaining how site operators can allowlist Google-Agent, the user-triggered fetcher that navigates the web on users' behalf, through CDNs, WAFs and bot-detection services. The guidance identifies three signals — the Google-Agent User-Agent string, IP ranges published in user-triggered-agents.json, and optional Web Bot Auth signatures under RFC 9421 using the identity agent.bot.goog — and warns that signature-only rules will reject much legitimate traffic while Web Bot Auth remains partial.", "body_md": "Google’s crawling infrastructure docs now include a dedicated page on how to [allow Google-Agent traffic](https://developers.google.com/crawling/docs/crawlers-fetchers/allow-google-agent) through a content delivery network, web application firewall, or bot-detection service. The page exists for a practical reason: agencies and hosts have spent years tightening “block AI agents” packs, so user-triggered agents that run on Google’s infrastructure often never reach search, product detail, or checkout. Allowlisting is edge operations. It does not replace crawl readiness for indexing bots, and it does not replace lab and field Core Web Vitals on the URLs that still convert when a human loads the page. What follows is a working translation of Google’s rules for multi-tenant portfolios, plus the monitoring boundary that keeps the retainer honest.\n\n`Google-Agent` is the name Google uses for agents hosted on Google infrastructure that navigate the web and perform actions when a user asks. In Google’s [user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) table, the User-Agent strings include `compatible; Google-Agent` with a documentation URL, for both mobile and desktop shapes. Traffic for this fetcher uses IP ranges published in [`user-triggered-agents.json`](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json).\n\nThat is a different job from **Googlebot**, which crawls for Search. It is also a different product conversation from Google Cloud or Gemini “agent builder” marketing pages that dominate results for the bare query `google-agent`. When a client asks you to “allow the Google agent,” it helps to confirm they mean this crawling User-Agent and IP list, not a Vertex or Workspace product toggle.\n\nIt is also distinct from classic AI crawlers such as GPTBot. Those systems are usually discussed through `robots.txt`, fetch budgets, and whether the HTML becomes parseable quickly. Our earlier guide on [why AI crawlers need fast, crawlable pages](https://apogeewatcher.com/blog/why-ai-crawlers-need-fast-crawlable-pages-and-how-to-stay-ready) covers that lane. Google-Agent allowlisting sits one layer earlier on many stacks: if the WAF challenges or drops the request, crawlability and performance never get a chance.\n\nLighthouse’s experimental Agentic Browsing audits ask whether a page is structured for machine interaction; they do not configure your CDN. [Lighthouse agentic browsing scoring](https://apogeewatcher.com/blog/lighthouse-agentic-browsing-scoring) belongs in the page-construction conversation. The CDN and WAF rules below belong in the edge-allow conversation.\n\nGoogle documents three signals, and the split matters in production: the first two (User-Agent and IP) work as the baseline today, while Web Bot Auth stays optional for as long as the experiment remains partial.\n\n| Signal | What to check | Role | \n|---|---|---|\n| User-Agent | Header contains `Google-Agent` | Cheap first filter; spoofable alone | \n| Client IP | Address is listed in [`user-triggered-agents.json`](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json) | Makes an allow rule safe | \n| Web Bot Auth (optional) | `Signature-Agent` is`g=\"https://agent.bot.goog\"` ; verify`Signature` /`Signature-Input` labelled`g` per[RFC 9421](https://www.rfc-editor.org/rfc/rfc9421.html) | Cryptographic identity when present | \n\nParticipating signed requests use the identity `https://agent.bot.goog`. The public key set lives at [` https://agent.bot.goog/.well-known/http-message-signatures-directory`](https://agent.bot.goog/.well-known/http-message-signatures-directory). Google’s [Web Bot Auth guide](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) is explicit: not every agent request is signed yet, and verification still falls back to User-Agent plus IP. A rule that only accepts signatures will fail a large share of legitimate traffic for as long as the experiment stays partial.\n\nIn provider language, the safe shape is usually: confirm User-Agent contains `Google-Agent`, confirm the IP is on the published list, verify the signature when headers are present, and for verified requests skip bot challenges. Matching the User-Agent string alone is not enough, because anyone can send that string. The IP check (and signatures when present) is what keeps the allow rule from becoming an open door for spoofed bots.\n\nProvider verified-bot or managed allowlist features are the least fragile starting point. Cloudflare, AWS WAF, Fastly, and several bot-detection vendors already document Web Bot Auth or Google crawler categories. Their supported path usually ages better than a hand-rolled regex, but the security event log still matters: a checkbox is not the same as an end-to-end journey that clears search and cart APIs.\n\nWhen the rule has to be expressed by hand, Google’s four checks map cleanly into most host syntax:\n\n`Google-Agent`.` user-triggered-agents.json` ranges.`Signature-Agent` / signature headers are present, verify them against `https://agent.bot.goog` (Web Bot Auth); if they are absent, steps 1–2 remain the authority.\nThe useful scope is the full journey, not only the homepage. Google’s troubleshooting list calls out on-site search, filtering, detail pages, add-to-cart, booking, and the underlying XHR, GraphQL, or `api.*` endpoints. An allow rule that only clears `/` still fails the moment an assistant filters a catalogue or posts to a cart API.\n\nLoad balancers and reverse proxies often rewrite the picture. When the edge sees a substituted client IP, a stripped User-Agent, or missing `Signature` / `Signature-Input` / `Signature-Agent` headers, verification fails even if Google sent a valid request. IP-based rules also drift after a one-time paste of ranges. Teams that already refresh other verified-bot CIDR feeds can point the same job at [`user-triggered-agents.json`](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json), or use a provider feed that pulls Google’s lists.\n\nWhen support says “we allowed it” and journeys still die, Google’s failure modes tend to appear in a predictable order.\n\n**Rule priority.** Allow rules need to evaluate before broad cloud or data-centre ASN blocks, managed bot packs, and category-level “block AI agents” policies. On agency accounts, the last pack someone enabled for “AI scraping” often sits above a carefully written Google-Agent exception. The rule order is usually the bug, not the rule text.\n\n**Interactive challenges.** Skipping a 403 is not enough. Challenges still break journeys when CAPTCHA or JavaScript widgets fire on search or form submit, even after the HTML document itself loads. Agents that can fetch the first page and then stall on a widget never finish the path.\n\n**In-journey actions and APIs.** Coverage has to include the paths and hosts the agent calls after the first document, because search autocomplete, filter endpoints, GraphQL, and `api.` subdomains are common blind spots when the homepage rule looks green in the UI and the journey still dies one click later.\n\n**Shared-IP rate limits.** Multiple users’ agents share Google’s egress pool. Per-IP rate limits that treat those addresses like a single abusive scraper cut legitimate concurrency. Exempting or raising limits for verified Google-Agent ranges is the usual fix.\n\n**Stale IP lists.** CDN logs that show Google-Agent User-Agents from new ranges while your rule misses them almost always point at sync, so it is worth confirming that automation still pulls [`user-triggered-agents.json`](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json) rather than a months-old paste.\n\n**Proxy header loss.** Original client IP and signature headers need to survive every hop. Stacked Cloudflare → origin WAF → application firewall setups are a frequent place where each layer “helpfully” normalises traffic and breaks verification before the application ever sees a clean request.\n\nBoth the CDN security events and the origin access log help locate the cut: requests that never appear at origin were blocked upstream, while requests that reach origin with a challenge cookie requirement still need the skip-challenge step completed.\n\nMost portfolios are not one Cloudflare zone, and a short, repeatable checklist per client keeps tickets from restarting discovery every quarter:\n\nThat last step is where PageSpeed work returns. Allowlisting stops verified agents from being rejected at the edge. It does not tell you whether LCP on the product detail page is still eight seconds after a theme release. For the difference between a one-off paste into PageSpeed Insights and a schedule you can defend in a retainer, see [PageSpeed Insights vs automated monitoring](https://apogeewatcher.com/blog/pagespeed-insights-vs-automated-monitoring-when-manual-checks-arent-enough).\n\nAgencies sometimes treat “agent readiness” as a single green badge, but edge allow rules, `robots.txt` for crawlers, Lighthouse agentic audits, and Core Web Vitals answer different questions. Mixing them into one score produces false confidence on both the security ticket and the performance report:\n\nSynthetic schedules still matter after an edge change because WAF packs and theme deploys move on different calendars. Field data still matters because CrUX reflects real browsers, not the egress pool of an agent. For when synthetic measurement fits versus real-user monitoring, see [synthetic vs real user monitoring](https://apogeewatcher.com/blog/when-to-use-synthetic-vs-real-user-monitoring-performance). The tools layer; none of them replaces the others.\n\nIf agent journeys start working in staging logs but checkout still fails Core Web Vitals budgets in production, the firewall moved and the performance debt stayed. That is a clearer problem than a silent block. It is still worth scheduling and alerting on before the next client review, because the board will ask about speed long after the WAF ticket closes.\n\n**Is Google-Agent the same as Googlebot?**\n\nNo. Googlebot crawls for Search. Google-Agent identifies user-triggered agents on Google infrastructure that navigate and act on request. Allow rules and IP lists differ; the [allow Google-Agent](https://developers.google.com/crawling/docs/crawlers-fetchers/allow-google-agent) and [user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) docs are the source of truth.\n\n**Do we need Web Bot Auth to allow Google-Agent?**\n\nNot as the only control. Google still expects User-Agent plus IP verification, and signatures are experimental and incomplete. Web Bot Auth helps when headers are present and the provider supports verification; the IP fallback stays required.\n\n**Will allowing Google-Agent open the site to spoofed bots?**\n\nA User-Agent-only allow would. Pairing the User-Agent with Google’s published IP ranges (and signatures when present) is the control Google documents. Broad bot packs can stay in place for unverified traffic.\n\n**Does** `robots.txt` **control Google-Agent the way it controls GPTBot?**\n\nCrawl directives and WAF allowlists are separate layers. A `robots.txt` change does not clear a Cloudflare challenge on an API path. Edge logs are the proof.\n\n**What should we tell the client after we change the WAF?**\n\nThat verified Google-Agent traffic should now reach the journey paths you listed, that bot defence remains for unverified traffic, and that lab and field performance on those checkout, search, and product URLs stay on the monitoring programme because allowlisting does not measure LCP or INP.\n\nWhen a client’s “block AI” pack and a new Google allow doc collide, the durable pattern is UA + IP (plus Web Bot Auth when present), an allow rule above category AI blocks, challenge skips on the full journey, and a sync job for `user-triggered-agents.json`. After the edge change, checkout and search still need proof for humans. A lab check now, those URLs on a schedule with budgets, and an alert path when a theme or WAF pack regresses them again keep the retainer honest.\n\nA multi-page [domain PageSpeed check](https://apogeewatcher.com/check) is enough to spot-check after a WAF edit. [Apogee Watcher sign-up](https://apogeewatcher.com/sign-up) fits when the same checks need to run across many client organisations without a separate cron and key for each site. Allowlisting gets the agent through the door. Scheduled lab and field data tell you what happens after it walks in.", "url": "https://wpnews.pro/news/allow-google-agent-traffic-cdn-waf-and-web-bot-auth-for-agency-sites", "canonical_source": "https://dev.to/apogeewatcher/allow-google-agent-traffic-cdn-waf-and-web-bot-auth-for-agency-sites-243a", "published_at": "2026-11-20 08:01:51+00:00", "updated_at": "2026-11-20 08:12:28.439372+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "ai-agents", "developer-tools"], "entities": ["Google", "Google-Agent", "Googlebot", "GPTBot", "Web Bot Auth", "RFC 9421", "Lighthouse"], "also_reported_by": [], "takeaway": "Allowlist Google-Agent at the edge by matching the User-Agent header plus the IP ranges in user-triggered-agents.json, and treat Web Bot Auth signatures (Signature-Agent g=\"https://agent.bot.goog\", RFC 9421) as optional until Google signs all agent requests.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/allow-google-agent-traffic-cdn-waf-and-web-bot-auth-for-agency-sites", "markdown": "https://wpnews.pro/news/allow-google-agent-traffic-cdn-waf-and-web-bot-auth-for-agency-sites.md", "text": "https://wpnews.pro/news/allow-google-agent-traffic-cdn-waf-and-web-bot-auth-for-agency-sites.txt", "jsonld": "https://wpnews.pro/news/allow-google-agent-traffic-cdn-waf-and-web-bot-auth-for-agency-sites.jsonld"}}