{"slug": "i-analyzed-1000-keywords-with-python-to-see-how-ai-overviews-relate-to-google", "title": "I Analyzed 1,000 Keywords with Python to See How AI Overviews Relate to Google Rankings", "summary": "A developer analyzed approximately 1,000 keywords using Python, LangChain, and LLM APIs to investigate the relationship between Google AI Overviews and organic search rankings. The research, published on heysho.com, found that traditional SEO rankings do not strongly predict visibility in AI-generated answers, suggesting that brands should aim for top positions but also focus on other factors. The study used Semrush for keyword collection and SearchApi for SERP retrieval, narrowing down from 100,000 to 1,000 queries across five product categories.", "body_md": "Recently, I conducted a research project on the relationship between Google AI Overviews and organic search rankings, and published the findings on my website:\n\n[https://heysho.com/en/geo/research-aio-organic-ranking.html](https://heysho.com/en/geo/research-aio-organic-ranking.html)\n\nRather than going into detail about the findings themselves, in this article I’d like to share some of the behind-the-scenes aspects of the research that I couldn’t fully cover in the original report, including:\n\nI’ll also share some of the challenges, discoveries, and interesting lessons I encountered throughout the research process.\n\nOver the past six months or so, I’ve been spending a lot of time researching GEO and AI search. I’ve created my own reports and looked closely at how brands appear in actual AI-generated answers.\n\nOne question kept coming up:\n\n**“How much does a brand’s Google ranking affect its chances of appearing in AI-generated answers?”**\n\nAs I looked at different brands, I started noticing some interesting patterns.\n\nSome brands don’t seem particularly focused on SEO, yet they are frequently recommended by ChatGPT and other AI search platforms.\n\nOn the other hand, some companies perform very well in Google Search, but their brand names don’t appear nearly as often in AI-generated answers.\n\nThat made me wonder:\n\n**How much do traditional SEO rankings really influence visibility in AI search?**\n\nThere was also a more practical question I wanted to answer: **What ranking position should we actually aim for?**\n\nWhen working on SEO for a large website, you might track KPIs such as:\n\nBut once AI search enters the picture, it becomes less clear which ranking thresholds actually matter.\n\nThe answer could affect how we prioritize SEO initiatives.\n\nThat’s what motivated me to collect the data myself and compare Google organic rankings with visibility in AI Overviews.\n\nIn the end, I analyzed around 1,000 keywords, but I didn’t start by simply putting together a list of 1,000 queries.\n\nFirst, I used Semrush to collect a large set of product-category keywords. The initial dataset contained nearly 100,000 keywords.\n\nAfter removing obviously irrelevant terms, I narrowed the list down to around 50,000.\n\nClassifying that many keywords manually would have taken a huge amount of time, so I used Python, LangChain, and LLM APIs to help automate the process.\n\nFor this study, I then narrowed the scope further to queries in the format **“product category + recommended.”**\n\nFor example:\n\nThe reason I chose this query format was simple: **it makes it easier to analyze whether specific brands appear in AI Overview responses.**\n\nIf someone searches only for “T-shirt,” the intent is relatively broad and ambiguous.\n\nWith a query like “recommended T-shirts,” however, the response is much more likely to include specific product or brand recommendations.\n\nI created keyword pools across five categories—beauty, consumer electronics, apparel, furniture, and outdoor—and eventually narrowed the dataset down to around 15,000 candidate keywords.\n\nFrom there, I selected the top 200 keywords in each category by search volume, giving me roughly 1,000 queries in total.\n\nI didn’t jump straight to 1,000 queries, though.\n\nI started small, testing with 20, then 50, and finally 100 queries.\n\nBy the time I had analyzed around 100 queries, the overall pattern was already becoming fairly clear. That suggested I probably didn’t need to analyze tens of thousands of keywords to see the broader trend.\n\nSo for the final study, I scaled the analysis up to around 1,000 queries.\n\nI used SearchApi to retrieve Google search results.\n\nEach request returns a JSON response containing a wide range of SERP data, including:\n\nTechnically, it’s possible to build your own Google Search scraping pipeline, and I’ve written similar scripts myself in the past.\n\nBut once you need to collect hundreds or thousands of SERPs reliably, managing the entire process yourself can become quite a lot of work.\n\nBeing able to retrieve all of this data quickly through an API made the research process much easier.\n\nI think this kind of service is particularly useful for SEO agencies that need to collect SERP data at scale across multiple clients, as well as for teams working on large websites or global SEO projects.\n\nFor individual use, however, there is still a trade-off between convenience and monthly cost. That was my impression after using it for this project.\n\nFor this research, I mainly extracted **Google organic rankings** along with the **AI Overview response text and citations**, then linked the two datasets together for analysis.\n\nOne thing that turned out to be surprisingly difficult during the implementation was identifying brand names accurately.\n\nBrand names can appear in many different forms in AI-generated answers.\n\nFor example, the same brand might appear as:\n\nIf you rely only on exact string matching, the same brand can easily be treated as multiple different brands.\n\nTo address this, I created a brand-name dictionary and added a normalization process so that different variations could be mapped back to the same brand.\n\nEven then, it was difficult to automate the process completely, so I also manually reviewed some of the results.\n\nThis made me realize that **brand-name variation is an important issue when using GEO / AI search tracking tools for Japanese queries.**\n\nIf you’re using a GEO tool developed outside Japan, it may be worth checking how well it recognizes an English brand name, its katakana version, and the Japanese legal entity name as the same brand.\n\nA dashboard may show clean-looking numbers, but if the underlying brand matching is inaccurate, the reported mention rate can change significantly.\n\nIt’s not the most glamorous part of the process, but it ended up being one of the most important lessons from this research.\n\nOf the roughly 1,000 queries I analyzed, **AI Overviews appeared for around 70% of them.**\n\nPersonally, I found that surprisingly high.\n\nOf course, this was probably influenced by the fact that the study focused heavily on **“product category + recommended”** queries.\n\nFor example, Google may show very different AI Overview results for a search like “bicycle” or “T-shirt” compared with “recommended bicycles” or “recommended T-shirts.”\n\nSo if your goal is to measure **how often your brand is recommended in AI search**, it may be more useful to track queries that are likely to trigger brand recommendations rather than relying only on generic category terms.\n\nFor example:\n\nDesigning queries around recommendation intent makes it much easier to measure how often your brand appears in AI-generated answers.\n\nThis was the question I was most interested in answering.\n\n**If a brand ranks #1 on Google, is it also more likely to appear in AI Overviews?**\n\nThe results were quite interesting.\n\nAmong companies and brands ranking #1 in Google Search:\n\nThe citation rate was higher than I expected. In other words, pages ranking #1 in organic search were also cited as a source in AI Overviews in roughly half of the cases.\n\nThe brand mention rate, however, was much lower at around 20%.\n\nBefore running the study, I had assumed that if a brand ranked #1 on Google, its name would also appear in AI Overviews fairly often.\n\nIn reality, that happened only about **one in five times**.\n\nThat gap between citation rate and brand mention rate was one of the findings that stood out to me the most.\n\nWhen I grouped the rankings into buckets, another interesting pattern emerged.\n\nThe **brand mention rates** were:\n\nThe **citation rates** were:\n\nWhat stood out to me most was the **gap between the Top 5 and lower-ranking positions**.\n\nThe difference was especially noticeable for citations. The citation rate was around 38% for positions 1–5, but dropped to roughly 11% for positions 6–10.\n\nOf course, based on this dataset alone, I can’t say that **ranking in the Top 5 will cause a brand to appear in AI Overviews**.\n\nStill, the results suggest that, when AI Overview visibility is taken into account, reaching the Top 5 may be significantly more valuable than simply thinking, **“As long as we’re in the Top 10, that’s good enough for SEO.”**\n\nOne of the questions that originally motivated this research was: **Should our SEO KPI be Top 3, Top 5, or Top 10?**\n\nSo for me, this was one of the most useful findings from the study.\n\nLooking at the citation rates, I found a fairly strong relationship between Google organic rankings and AI Overview citations, at least within the dataset I analyzed.\n\nAI search and SEO are sometimes discussed as if they are completely separate disciplines. But when it comes to Google AI Overviews, I’m starting to think we may not need to treat them as entirely separate.\n\nAI Overviews draw on information from across the web when generating answers.\n\nIn that sense, it seems reasonable that pages already performing well in Google Search may also be more likely to be used as sources in AI-generated answers.\n\nThere is an important caveat, though: **what I observed here is correlation, not causation.**\n\nThis study does not prove that improving a page’s organic ranking will directly increase its likelihood of being cited in an AI Overview.\n\nIt is also possible that both organic rankings and AI Overview citations are influenced by some of the same underlying factors, such as:\n\nSo the takeaway is not:\n\n**“Improve your rankings, and your citation rate will automatically increase.”**\n\nRather, the results suggest that organic search visibility and AI Overview citations are closely related and may, at least in part, be influenced by the same underlying signals.\n\nThis is where it becomes important to separate **citations** from **brand mentions**.\n\nWhen companies invest in GEO, I think what they ultimately want to increase is not just citations, but brand mentions.\n\nFor example, imagine a user asks an AI:\n\n**“What are some recommended T-shirt brands?”**\n\nFrom a company’s perspective, what matters more is probably not whether the AI says, “I used this page as a source,” but whether **the brand itself is recommended in the answer.**\n\nAnd in this study, even brands ranking #1 in Google Search had a brand mention rate of only around 20%.\n\nSo ranking highly in organic search alone does not appear to be enough.\n\nWhat else might matter?\n\nFrom this point on, I’m going beyond what this dataset can prove, so the following is my own hypothesis.\n\nWhen AI systems answer questions about brands, they are likely drawing on information from across the web, not just from a brand’s own website.\n\nIf that is the case, the important question may not simply be:\n\n**“Does our own website rank highly?”**\n\nbut also:\n\n**“Is our brand being recommended across many different places on the web?”**\n\nFor an e-commerce brand, that could include:\n\nFor example, if a brand is consistently recommended on comparison or review sites that also rank well in Google Search, such as Kakaku.com or mybest, that could potentially help AI systems recognize and recommend the brand as well.\n\nMy current hypothesis is that the brands that perform well in AI search will not simply be the ones with strong SEO.\n\nInstead, they may be the brands with strong visibility across the web as a whole: **ranking well in search, being discussed on social media, appearing in YouTube videos, being featured on comparison sites, and accumulating a large number of reviews.**\n\nIn other words, broader digital visibility may ultimately put brands in a stronger position in AI search.\n\nThis is one of the areas I’d like to investigate next.\n\nFor this research, I combined a range of tools, including Semrush, Python, LangChain, LLM APIs, and SearchApi.\n\nI also used Google Antigravity and Codex to support the coding process.\n\nFor tasks such as classifying large numbers of keywords, where I didn’t necessarily need the most powerful models, I used lightweight LLM APIs to keep processing costs down.\n\nOne thing this project really made me realize is that **it has become possible for an individual to conduct research at a surprisingly large scale.**\n\nA few years ago, doing all of the following would probably have required a huge amount of time and effort:\n\nToday, by combining Python, APIs, and AI coding tools, one person can handle a surprisingly large portion of that workflow.\n\nThere is still a lot we don’t know about GEO and AI search.\n\nSo rather than only reading other people’s research, I’d like to keep taking the following approach:\n\n**“If something makes me curious, collect the data and test it myself.”**\n\nThat’s the mindset I want to keep as this field continues to evolve.\n\nIf I had to summarize the three most important takeaways from this research, they would be:\n\n**1. Google rankings and AI Overview visibility appear to be closely related**\n\nThe relationship was particularly clear for citations: the higher a page ranked in Google Search, the more likely it was to be cited in an AI Overview.\n\n**2. The Top 5 seems especially important**\n\nThere was a clear gap in both brand mention rates and citation rates between positions 1–5 and positions 6–10.\n\nSo when setting SEO KPIs with AI search in mind, aiming for the Top 5 may be more meaningful than simply reaching the first page of Google.\n\n**3. But even at #1, the brand mention rate was only around 20%**\n\nThis suggests that GEO probably cannot be solved through SEO alone.\n\nIf you want AI systems to actually recommend your brand, it may be necessary to build a presence beyond your own website and create an environment where your brand is recognized, discussed, and recommended across different parts of the web.\n\nGoing forward, I’d like to conduct similar research not only on AI Overviews, but also on **Google AI Mode, ChatGPT, Gemini, and other AI search platforms.**\n\nI’m also interested in exploring how visibility on YouTube, social media, review sites, and other platforms relates to brand mentions in AI-generated answers.\n\nIf you’re interested in the full methodology, charts, and data behind this study, I’ve published a more detailed version of the research on my website, heysho.com:\n\n[https://heysho.com/en/geo/research-aio-organic-ranking.html](https://heysho.com/en/geo/research-aio-organic-ranking.html)\n\nI currently work in SEO and GEO / AI search.\n\nI’ve been using Python for around three to four years, and I also spent about five years working as a frontend engineer. I enjoy using programming for marketing analysis, research, and automation.\n\nThere is still a lot to learn about GEO and AI search, so I plan to keep exploring the field by collecting real data and testing things for myself.\n\nIf I come across any interesting findings, I’ll share them again.", "url": "https://wpnews.pro/news/i-analyzed-1000-keywords-with-python-to-see-how-ai-overviews-relate-to-google", "canonical_source": "https://dev.to/heysho/i-analyzed-1000-keywords-with-python-to-see-how-ai-overviews-relate-to-google-rankings-4p80", "published_at": "2026-08-29 01:55:52+00:00", "updated_at": "2026-08-29 02:18:33.856556+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-research", "developer-tools"], "entities": ["Google", "Semrush", "SearchApi", "LangChain", "heysho.com"], "alternates": {"html": "https://wpnews.pro/news/i-analyzed-1000-keywords-with-python-to-see-how-ai-overviews-relate-to-google", "markdown": "https://wpnews.pro/news/i-analyzed-1000-keywords-with-python-to-see-how-ai-overviews-relate-to-google.md", "text": "https://wpnews.pro/news/i-analyzed-1000-keywords-with-python-to-see-how-ai-overviews-relate-to-google.txt", "jsonld": "https://wpnews.pro/news/i-analyzed-1000-keywords-with-python-to-see-how-ai-overviews-relate-to-google.jsonld"}}