{"slug": "google-trends-ties-its-data-tokens-to-your-ip-and-it-broke-my-scraper-in-a-way-i", "title": "Google Trends ties its data tokens to your IP and it broke my scraper in a way I didn't expect", "summary": "A developer building a Google Trends scraper on Apify discovered that Google Trends ties its data tokens to the IP address that made the initial request, causing silent failures when proxies rotate IPs. The fix requires pinning a proxy session to reuse the same IP for all requests in a run. The scraper, now available as an Apify Actor, charges only for terms that return validated data.", "body_md": "I just shipped a Google Trends scraper as an Apify Actor ([https://apify.com/swiftscrape/google-trends-fast-scraper](https://apify.com/swiftscrape/google-trends-fast-scraper)), and the most interesting part wasn't building it — it was a blocking bug that only showed up **in production**, on the platform, that I couldn't reproduce locally no matter what I tried.\n\nIf you've ever tried to scrape Google Trends and had it work on your laptop but mysteriously fail behind proxies, this one's for you. The root cause is non-obvious and I couldn't find it documented anywhere, so here's the whole thing.\n\nGoogle Trends has no official API. There are scrapers out there, but the most-used one on the market is **slow** (runs take minutes), it **charges you even when it returns nothing**, and its multi-term comparison breaks when called from AI agents. Its reviews are full of \"ran for 20 minutes, returned nothing, cost me money.\"\n\nThat's a wedge. Same data, but: fast, and **you only pay for terms that actually return data** — failed, blocked or empty terms are free. If a scraper is unreliable, that's the developer's problem, not the user's bill.\n\nGoogle Trends' UI is backed by internal JSON endpoints. You don't need a headless browser — you can talk to them directly:\n\n`GET /trends/api/explore`\n\nwith your terms → returns `GET /trends/api/widgetdata/multiline?token=…`\n\n→ interest over time.`GET /trends/api/widgetdata/relatedsearches?token=…`\n\n→ related queries.No Chrome, no Puppeteer. Direct HTTP is why it runs in ~1.5s locally instead of minutes.\n\nLocally (running from my home IP, no proxy): flawless. Timeline, related queries, everything.\n\nOn Apify (behind a datacenter proxy): `explore`\n\nsucceeded, `timeline`\n\nsucceeded... but **related queries silently failed** for every term. Then when I switched to residential proxies to \"fix\" it, it got *worse* — now even the timeline failed with `blocked`\n\n.\n\nThat combination made no sense to me at first:\n\n| explore | timeline | related | |\n|---|---|---|---|\n| Home IP (no proxy) | ✅ | ✅ | ✅ |\n| Datacenter proxy | ✅ | ✅ | ❌ |\n| Residential proxy | ✅ | ❌ | ❌ |\n\nIf residential IPs are \"better,\" why did they break *more*?\n\nHere's the thing nobody tells you: the widget tokens Google hands you in step 1 are **tied to the IP address that made the explore call**. When you then call the\n\n`multiline`\n\n/ `relatedsearches`\n\nendpoints with those tokens, Google checks that the request comes from the `blocked`\n\n.Now the table makes sense:\n\n`explore`\n\nand `timeline`\n\nsometimes landed on the same IP by luck — but the parallel related calls hit fresh IPs and got rejected.`explore`\n\ncame from a different IP. Even the timeline broke.The \"better\" proxy broke things precisely *because* it rotated IPs more aggressively.\n\nThe fix is to force every request in a single run through the **same IP** by pinning a proxy session:\n\n``` js\nconst proxy = await Actor.createProxyConfiguration({ groups: ['RESIDENTIAL'] });\nconst sessionId = `gt${Date.now()}`;\n\n// Every request in this run reuses the same IP:\nconst proxyUrl = await proxy.newUrl(sessionId);\n```\n\nOne sticky session → `explore`\n\n, `timeline`\n\n, and `related`\n\nall come from the same IP → tokens stay valid. Result after the fix, on the platform:\n\n`chatgpt`\n\n: 53 weekly data points, 16 top related queries, 3 rising`gemini`\n\n: 53 data points, 25 top, 18 risingThe lesson generalizes beyond Google Trends: **any two-step \"get a token, then use the token\" API can be IP-bound**. If step 2 fails behind a proxy but step 1 succeeds, suspect IP-pinned tokens before you suspect rate limiting.\n\nBecause I only charge for terms that return validated data, I had to make the billing code honest: a term with no search volume comes back from Google as all-zeros with a `hasData: false`\n\nflag, not an empty array. If you don't check `hasData`\n\n, you'll \"successfully\" return a series of zeros and charge for a term that had no data. That check is the difference between \"no result = no charge\" being a slogan and being true.\n\n`hasData`\n\nflags or your \"free on failure\" promise is a lie.The scraper is live here if you want to try it (pay-per-result, failed terms are free): ** Fast Google Trends Scraper on Apify**.\n\nHappy to answer questions about the internal endpoints or the token behavior in the comments.", "url": "https://wpnews.pro/news/google-trends-ties-its-data-tokens-to-your-ip-and-it-broke-my-scraper-in-a-way-i", "canonical_source": "https://dev.to/swiftscrape/google-trends-ties-its-data-tokens-to-your-ip-and-it-broke-my-scraper-in-a-way-i-didnt-expect-dgp", "published_at": "2026-07-21 16:56:46+00:00", "updated_at": "2026-07-21 17:23:36.646938+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Google Trends", "Apify", "Google"], "alternates": {"html": "https://wpnews.pro/news/google-trends-ties-its-data-tokens-to-your-ip-and-it-broke-my-scraper-in-a-way-i", "markdown": "https://wpnews.pro/news/google-trends-ties-its-data-tokens-to-your-ip-and-it-broke-my-scraper-in-a-way-i.md", "text": "https://wpnews.pro/news/google-trends-ties-its-data-tokens-to-your-ip-and-it-broke-my-scraper-in-a-way-i.txt", "jsonld": "https://wpnews.pro/news/google-trends-ties-its-data-tokens-to-your-ip-and-it-broke-my-scraper-in-a-way-i.jsonld"}}