{"slug": "how-to-scrape-google-ai-mode-citation-pills-place-cards", "title": "How to scrape Google AI Mode: citation pills + place cards", "summary": "Google AI Mode and AI Overview cite the same URLs only 13.7% of the time on semantically similar queries, according to ALM Corp's analysis of 1.3 million AI Mode citations, and google.com self-cites in 17.42% of AI Mode answers — more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. A technical guide details that AI Mode hides source URLs in HTML comments keyed by UUID rather than anchor tags and streams answers via a separate call to /async/folwr, so scrapers must intercept that request and rejoin citation buttons to their metadata by UUID. The guide also cites Seer Interactive's 2026 trigger data putting AI surfaces on 95.4% of comparison queries and 85.9% of question-format queries, and Alphabet's Q3 2025 earnings call reporting AI Mode passed 75 million daily active users by late 2025.", "body_md": "# How to scrape Google AI Mode: citation pills + place cards\n\nKey takeaways AI Mode and AI Overview cite the same URLs only 13.7% of the time, so scraping one tells you nothing reliable about the other. Source URLs live in HTML comments keyed by UUID, not in anchor tags, so a scraper reading <a href> returns an empty citation list rather than an error. Strip #\n\n:~:text scroll-to-text fragments before counting, or one article reads as several distinct URLs and inflates your citation numbers. Google AI Mode does not put its sources in anchor tags. It renders each one as a button, and hides the actual URL in an HTML comment keyed by UUID. A scraper that reads <a href> finds nothing.\n\nThat one design choice is most of the work in this post. The rest is that nothing is in the initial HTML at all. The shell loads, then the answer streams in over a separate call to /async/folwr. Parse on page load and you get an empty container.\n\nBuild the URL with udm=50 and aep=11 to force the interface. Register a network interceptor on https://www.google.com/async/folwr before navigating. Navigate, and treat a non-200 as a CAPTCHA rather than a dead page. Poll until the intercepted response lands, with a hard timeout.\n\nRead the answer text off data-session-thread-id, not off a snapshot. Collect the citation buttons, then rejoin them to their HTML-comment metadata by UUID. Rewrite the pills into anchors before converting anything to markdown. Steps 6 and 7 are where the work is.\n\nThe rest is plumbing. AI Mode is not AI Overview with a different skin. Per ALM Corp's analysis of 1.3 million AI Mode citations, the two cite the same URLs only 13.7% of the time on semantically similar queries, and google.com self-cites in 17.42% of AI Mode answers, more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. Scraping one tells you nothing reliable about the other.\n\nIt is also where the comparison queries go. Seer Interactive's 2026 trigger data puts AI surfaces on 95.4% of comparison queries and 85.9% of question-format ones, and AI Mode passed 75 million daily active users by late 2025 per Alphabet's Q3 2025 earnings call. The query fan-out behind it, where Google splits your question into subtopics and runs many searches at once, is why one answer stitches together so many cited pages. Surface Trigger Citation style Difficulty AI Mode udm=50 Pills, metadata in HTML comments High: async streaming, drifting selectors AI Overview Organic query, varies by class Inline source list above the SERP Medium: single fetch Gemini gemini.google.com Grounded sources panel High: separate auth domain ChatGPT Search chatgpt.com with search on Numbered citations Highest: SSE plus Cloudflare udm=50 is the switch.\n\naep=11 goes with it. search_url = build_url_with_params( \"https://www.google.com/search\", { \"udm\": 50, # AI Mode \"aep\": 11, \"q\": prompt, \"hl\": google_params[\"hl\"], \"gl\": google_params[\"gl\"], }, ) We drive Playwright and capture the async call, then poll until it lands. Sixty seconds, then fail loudly. A hard timeout beats a silent partial parse, because a half-streamed answer looks like a real answer with fewer citations.\n\npage_interceptor = PlaywrightInterceptor(do_not_block_resources=True) page_interceptor.add_capture_urls([\"https://www.google.com/async/folwr\"]) await page_interceptor.setup_page_interceptor(page) for _ in range(120): # 500ms intervals, 60s cap if len(page_interceptor.captured_responses): break await sleep(500) else: raise Exception(\"Never received AI Mode response after 60 seconds\") The answer text hangs off data-session-thread-id. Target that and read its parent, because AI Mode rewrites its own container several times while streaming and a locator survives that where a snapshot does not. thread_element = page.locator(\"[data-session-thread-id]\") text = await thread_element.locator(\"..\").text_content() or \"\" Here is the shape you are working against. A UUID ties the visible button to a comment block carrying the real URLs.\n\n<!--Sv6Kpe[[\"uuid-12345\",[\"label\",\"description\"],[\"https://example.com\",\"source2\"]]]--> <button data-icl-uuid=\"uuid-12345\" data-amic=\"true\">[1]</button> So: find the buttons, read their UUIDs, regex the comments back out of the page HTML, and filter Google's own domains out of the URL list. One pill can carry several sources. async def extract_aimode_citation_pills(page: Page) -> Dict[str, List[LinkData]]: citation_pills: Dict[str, List[LinkData]] = {} pill_locators = page.locator('button[data-icl-uuid][data-amic=\"true\"]') page_html = html.unescape(await page.content()) for i in range(await pill_locators.count()): pill_button = pill_locators.nth(i) if not await pill_button.is_visible(): continue uuid = await pill_button.get_attribute(\"data-icl-uuid\") if not uuid: continue pattern = rf'<!--Sv6Kpe\\[\\[\"{re.escape(uuid)}\".*?]]-->' current_pill: List[LinkData] = [] for content in re.findall(pattern, page_html, re.DOTALL): desc_match = re.search( rf'\"{re.escape(uuid)}\"\\s*,\\s*\\[\\s*\"[^\"]+\"\\s*,\\s*\"([^\"]+)\"', content ) url = next( (u for u in re.findall(r'\"(https://[^\"]+)\"', content) if not any(skip in u for skip in [\"google.com\", \"gstatic.com\", \"encrypted-tbn\"])), None, ) if url: url = url.split(\"#:~:text\")[0] url = url.replace(\"\\\\u003d\", \"=\").replace(\"\\\\u0026\", \"&\") current_pill.append(LinkData( position=len(current_pill) + 1, label=f\"Source {len(current_pill) + 1}\", url=url, description=desc_match.group(1) if desc_match else None, )) if current_pill: citation_pills[uuid] = current_pill return citation_pills The #:~:text strip matters more than it looks. Google appends scroll-to-text fragments, so the same article comes back as several distinct URLs and your citation counts inflate quietly.\n\nThe sources panel needs its own selector per layout. Web-results pages expose it one way, the dialog another. Playwright's actionability checks handle the waiting, so log and continue on timeout rather than killing the run. sources_selector = ( '[data-container-id=\"rhs-col\"] [role=\"dialog\"] a' if not is_web_results_page else \"a.ZbQNgf\" ) Hand the raw HTML to html2text and every pill becomes a dead button.\n\nRewrite the pills into real anchors first, using the UUID map you already built, then convert. soup = BeautifulSoup(html_content, \"html.parser\") for button in soup.find_all(\"button\", attrs={\"data-icl-uuid\": True, \"data-amic\": \"true\"}): uuid = button.get(\"data-icl-uuid\") for link_data in reversed(citation_pills.get(uuid, [])): anchor = soup.new_tag(\"a\", href=link_data.get(\"url\")) anchor.string = link_data.get(\"label\") button.insert_after(anchor) button.decompose() h = html2text.HTML2Text() h.ignore_links = False h.body_width = 0 markdown = h.handle(str(soup)).strip() A non-200 is usually a CAPTCHA, not a dead page, so solve before you give up. One blocked request should not tank a batch. response = await page.goto(search_url, timeout=20_000) if not is_http_success(response.status): if not await solve_captcha(page, page_interceptor): raise Exception(f\"HTTP error: {response.status} (probably captcha)\") The real running cost is selector drift.\n\nAI Mode's DOM changes without notice, and a scraper that worked last month returns empty rather than throwing. Alert on success rate, not on exceptions. A sudden drop is a layout change, and you will find it weeks late if you are only watching for errors. You will also want a proxy pool and fingerprint rotation before this runs at any volume.\n\ncloro maintains this pipeline as a managed endpoint, which is the disclosure to weigh this section against. One POST, parsed text and sources back, P50 under 8 seconds. response = requests.post( \"https://api.cloro.dev/v1/monitor/aimode\", headers={\"Authorization\": \"Bearer sk_live_your_api_key\"}, json={\"prompt\": \"What do you know about Tesla's latest updates?\", \"country\": \"US\", \"include\": {\"markdown\": True}}, ) result = response.json()[\"result\"] print(f\"{len(result['sources'])} citations, fan-out: {result['searchQueries']}\") Built in-house, this runs $5,000 to $10,000 a month once you count engineering time, browser instances and proxies. Below a few thousand queries a month that maths does not favour us, and you should build it.\n\nAbove that it does, and cloro's AI Mode endpoint absorbs the selector drift.\n\n## Key Takeaways\n\n- •Google AI Mode does not put its sources in anchor tags.\n- •This story was reported by **Dev.to** , covering developments in the**dev** space.\n- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.\n\n📖 Continue reading the full article:\n\n[Read Full Article on Dev.to →](https://dev.to/cloro/how-to-scrape-google-ai-mode-citation-pills-place-cards-1igj)", "url": "https://wpnews.pro/news/how-to-scrape-google-ai-mode-citation-pills-place-cards", "canonical_source": "https://ainexusdaily.vercel.app/article/2026-11-15-how-to-scrape-google-ai-mode-citation-pills-place-cards", "published_at": "2026-11-15 13:31:23+00:00", "updated_at": "2026-11-15 13:52:09.533594+00:00", "lang": "en", "topics": ["ai-search", "generative-engine-optimization", "ai-crawlers"], "entities": ["Google", "Google AI Mode", "Google AI Overview", "ALM Corp", "Seer Interactive", "Alphabet", "Playwright"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-scrape-google-ai-mode-citation-pills-place-cards", "markdown": "https://wpnews.pro/news/how-to-scrape-google-ai-mode-citation-pills-place-cards.md", "text": "https://wpnews.pro/news/how-to-scrape-google-ai-mode-citation-pills-place-cards.txt", "jsonld": "https://wpnews.pro/news/how-to-scrape-google-ai-mode-citation-pills-place-cards.jsonld"}}