How to scrape Google AI Mode: citation pills + place cards

Google AI Mode and AI Overview cite the same URLs only 13.7% of the time on semantically similar queries, according to ALM Corp's analysis of 1.3 million AI Mode citations, and google.com self-cites in 17.42% of AI Mode answers — more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. A technical guide details that AI Mode hides source URLs in HTML comments keyed by UUID rather than anchor tags and streams answers via a separate call to /async/folwr, so scrapers must intercept that request and rejoin citation buttons to their metadata by UUID. The guide also cites Seer Interactive's 2026 trigger data putting AI surfaces on 95.4% of comparison queries and 85.9% of question-format queries, and Alphabet's Q3 2025 earnings call reporting AI Mode passed 75 million daily active users by late 2025.

How to scrape Google AI Mode: citation pills + place cards Key takeaways AI Mode and AI Overview cite the same URLs only 13.7% of the time, so scraping one tells you nothing reliable about the other. Source URLs live in HTML comments keyed by UUID, not in anchor tags, so a scraper reading <a href returns an empty citation list rather than an error. Strip :~:text scroll-to-text fragments before counting, or one article reads as several distinct URLs and inflates your citation numbers. Google AI Mode does not put its sources in anchor tags. It renders each one as a button, and hides the actual URL in an HTML comment keyed by UUID. A scraper that reads <a href finds nothing. That one design choice is most of the work in this post. The rest is that nothing is in the initial HTML at all. The shell loads, then the answer streams in over a separate call to /async/folwr. Parse on page load and you get an empty container. Build the URL with udm=50 and aep=11 to force the interface. Register a network interceptor on https://www.google.com/async/folwr before navigating. Navigate, and treat a non-200 as a CAPTCHA rather than a dead page. Poll until the intercepted response lands, with a hard timeout. Read the answer text off data-session-thread-id, not off a snapshot. Collect the citation buttons, then rejoin them to their HTML-comment metadata by UUID. Rewrite the pills into anchors before converting anything to markdown. Steps 6 and 7 are where the work is. The rest is plumbing. AI Mode is not AI Overview with a different skin. Per ALM Corp's analysis of 1.3 million AI Mode citations, the two cite the same URLs only 13.7% of the time on semantically similar queries, and google.com self-cites in 17.42% of AI Mode answers, more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. Scraping one tells you nothing reliable about the other. It is also where the comparison queries go. Seer Interactive's 2026 trigger data puts AI surfaces on 95.4% of comparison queries and 85.9% of question-format ones, and AI Mode passed 75 million daily active users by late 2025 per Alphabet's Q3 2025 earnings call. The query fan-out behind it, where Google splits your question into subtopics and runs many searches at once, is why one answer stitches together so many cited pages. Surface Trigger Citation style Difficulty AI Mode udm=50 Pills, metadata in HTML comments High: async streaming, drifting selectors AI Overview Organic query, varies by class Inline source list above the SERP Medium: single fetch Gemini gemini.google.com Grounded sources panel High: separate auth domain ChatGPT Search chatgpt.com with search on Numbered citations Highest: SSE plus Cloudflare udm=50 is the switch. aep=11 goes with it. search url = build url with params "https://www.google.com/search", { "udm": 50, AI Mode "aep": 11, "q": prompt, "hl": google params "hl" , "gl": google params "gl" , }, We drive Playwright and capture the async call, then poll until it lands. Sixty seconds, then fail loudly. A hard timeout beats a silent partial parse, because a half-streamed answer looks like a real answer with fewer citations. page interceptor = PlaywrightInterceptor do not block resources=True page interceptor.add capture urls "https://www.google.com/async/folwr" await page interceptor.setup page interceptor page for in range 120 : 500ms intervals, 60s cap if len page interceptor.captured responses : break await sleep 500 else: raise Exception "Never received AI Mode response after 60 seconds" The answer text hangs off data-session-thread-id. Target that and read its parent, because AI Mode rewrites its own container several times while streaming and a locator survives that where a snapshot does not. thread element = page.locator " data-session-thread-id " text = await thread element.locator ".." .text content or "" Here is the shape you are working against. A UUID ties the visible button to a comment block carrying the real URLs. < --Sv6Kpe "uuid-12345", "label","description" , "https://example.com","source2" -- <button data-icl-uuid="uuid-12345" data-amic="true" 1 </button So: find the buttons, read their UUIDs, regex the comments back out of the page HTML, and filter Google's own domains out of the URL list. One pill can carry several sources. async def extract aimode citation pills page: Page - Dict str, List LinkData : citation pills: Dict str, List LinkData = {} pill locators = page.locator 'button data-icl-uuid data-amic="true" ' page html = html.unescape await page.content for i in range await pill locators.count : pill button = pill locators.nth i if not await pill button.is visible : continue uuid = await pill button.get attribute "data-icl-uuid" if not uuid: continue pattern = rf'< --Sv6Kpe\ \ "{re.escape uuid }". ? -- ' current pill: List LinkData = for content in re.findall pattern, page html, re.DOTALL : desc match = re.search rf'"{re.escape uuid }"\s ,\s \ \s " ^" +"\s ,\s " ^" + "', content url = next u for u in re.findall r'" https:// ^" + "', content if not any skip in u for skip in "google.com", "gstatic.com", "encrypted-tbn" , None, if url: url = url.split " :~:text" 0 url = url.replace "\\u003d", "=" .replace "\\u0026", "&" current pill.append LinkData position=len current pill + 1, label=f"Source {len current pill + 1}", url=url, description=desc match.group 1 if desc match else None, if current pill: citation pills uuid = current pill return citation pills The :~:text strip matters more than it looks. Google appends scroll-to-text fragments, so the same article comes back as several distinct URLs and your citation counts inflate quietly. The sources panel needs its own selector per layout. Web-results pages expose it one way, the dialog another. Playwright's actionability checks handle the waiting, so log and continue on timeout rather than killing the run. sources selector = ' data-container-id="rhs-col" role="dialog" a' if not is web results page else "a.ZbQNgf" Hand the raw HTML to html2text and every pill becomes a dead button. Rewrite the pills into real anchors first, using the UUID map you already built, then convert. soup = BeautifulSoup html content, "html.parser" for button in soup.find all "button", attrs={"data-icl-uuid": True, "data-amic": "true"} : uuid = button.get "data-icl-uuid" for link data in reversed citation pills.get uuid, : anchor = soup.new tag "a", href=link data.get "url" anchor.string = link data.get "label" button.insert after anchor button.decompose h = html2text.HTML2Text h.ignore links = False h.body width = 0 markdown = h.handle str soup .strip A non-200 is usually a CAPTCHA, not a dead page, so solve before you give up. One blocked request should not tank a batch. response = await page.goto search url, timeout=20 000 if not is http success response.status : if not await solve captcha page, page interceptor : raise Exception f"HTTP error: {response.status} probably captcha " The real running cost is selector drift. AI Mode's DOM changes without notice, and a scraper that worked last month returns empty rather than throwing. Alert on success rate, not on exceptions. A sudden drop is a layout change, and you will find it weeks late if you are only watching for errors. You will also want a proxy pool and fingerprint rotation before this runs at any volume. cloro maintains this pipeline as a managed endpoint, which is the disclosure to weigh this section against. One POST, parsed text and sources back, P50 under 8 seconds. response = requests.post "https://api.cloro.dev/v1/monitor/aimode", headers={"Authorization": "Bearer sk live your api key"}, json={"prompt": "What do you know about Tesla's latest updates?", "country": "US", "include": {"markdown": True}}, result = response.json "result" print f"{len result 'sources' } citations, fan-out: {result 'searchQueries' }" Built in-house, this runs $5,000 to $10,000 a month once you count engineering time, browser instances and proxies. Below a few thousand queries a month that maths does not favour us, and you should build it. Above that it does, and cloro's AI Mode endpoint absorbs the selector drift. Key Takeaways - •Google AI Mode does not put its sources in anchor tags. - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/cloro/how-to-scrape-google-ai-mode-citation-pills-place-cards-1igj