Building a Keyword Rank Tracker That Also Flags AI Overviews (Python)

SteadyScrape built a Python-based keyword rank tracker that records organic and absolute positions, ranking URLs, and SERP features including AI Overviews, flagging cannibalization when a different page of the same site starts ranking. The design diffs successive runs to report position changes, new SERP features, and marks runs not comparable when result depth changes, and is packaged as the steady-scrape/google-rank-tracker Apify Actor built on a licensed SERP data API. The company disclosed it earns revenue when the hosted tool is used.

Rank tracking sounds simple: search a keyword, find your site, write down the position. The useful part is everything after that. Did it move? Did a different page of yours start ranking cannibalization ? Did Google add an AI Overview that pushed your "position 2" below the fold? This post walks through the design of a small rank tracker: what to store per keyword, how to diff runs, and which SERP features to record. The code works with any source of SERP data. At the end there's a hosted option. Don't scrape Google results yourself. It's against Google's terms, gets blocked quickly, and results vary by IP location. In practice everyone uses a SERP API : you send keyword + location + language + device and get back parsed results. Whatever provider you choose, make sure it returns: Store one record per keyword, keyed by everything that changes the SERP: python def key keyword, location, language, device : return f"{keyword.lower }|{location}|{language}|{device}" def snapshot row keyword, serp, target domain : mine = r for r in serp "organic" if r "domain" == target domain or r "domain" .endswith "." + target domain best = mine 0 if mine else None return { "keyword": keyword, "position": best "position" if best else None, "url": best "url" if best else None, "otherUrls": r "url" for r in mine 1: , "features": sorted serp.get "features", , "top3": r "domain" for r in serp "organic" :3 , } Count subdomains as you make it a setting , and keep the URL, not just the position : most rank-tracking insights come from URL changes. python def compare prev, cur : if prev is None: return {"changeStatus": "first check", "positionChange": None} p, c = prev "position" , cur "position" if p is None and c is None: status, change = "not ranked", None elif p is None: status, change = "new", None elif c is None: status, change = "lost", None else: change = p - c positive = moved up status = "up" if change 0 else "down" if change < 0 else "same" return { "changeStatus": status, "positionChange": change, "rankingUrlChanged": bool p and c and prev "url" = cur "url" , "newFeatures": sorted set cur "features" - set prev "features" , } rankingUrlChanged is your cannibalization alert : two of your pages are competing for one keyword, and Google is swapping between them. newFeatures tells you when, say, ai overview or local pack appears on a SERP where it wasn't before. Also, mark a run not comparable if the depth changed top 10 vs top 100 instead of reporting false "lost" rankings. An organic position doesn't tell you how visible the result is. Position 2 under an AI Overview, a Local Pack and four ads sits far lower on the page than position 2 on a plain SERP. Record the absolute position too, and flag the features that matter to you: AI Overview, People Also Ask, featured snippet, Local Pack, shopping, video and forums. Disclosure: we SteadyScrape built the tool below and get paid when it's used. Google Rank Tracker API: Daily Keyword Positions https://apify.com/steady-scrape/google-rank-tracker steady-scrape/google-rank-tracker is an Apify Actor that implements this design on top of a licensed SERP data API. You give it a target a domain, or an exact URL if only that page should count , up to 2,000 keywords, and a location, language and device. Each keyword comes back with its organic and absolute position, the ranking URL and any other URLs of yours on the page, the SERP features and whether your site appears inside them , the top 3 competing domains, positions for up to 10 competitors you name, and, if you reuse the same trackerId , the change since the last run. python import os from apify client import ApifyClient client = ApifyClient os.environ "APIFY TOKEN" run = client.actor "steady-scrape/google-rank-tracker" .call run input={ "target": "apify.com", "keywords": "scrape amazon prices", "instagram scraper", "web scraping api" , "location": "United States", "language": "English", "device": "desktop", "depth": "10", "queueMode": "standard", "trackerId": "apify-us-desktop", } for r in client.dataset run "defaultDatasetId" .iterate items : print r "keyword" , r "position" , r "changeStatus" , r "serpFeatures" , r "aiOverviewCited" That's the Actor's example input. Here's what our test run of it returned on October 31, 2026: Run it again with the same trackerId , or put it on an Apify schedule, and every row gets positionChange , changeStatus up , down , same , new , lost and so on , previousRankingUrl and rankingUrlChanged . Set loadAiOverview: true to also load AI Overviews that Google shows asynchronously and check whether they cite you. A few practical notes: You pay per keyword that was checked successfully. Errors, invalid keywords and timeouts are free. | Event | Price | Apify Bronze / Silver / Gold plans | |---|---|---| | Keyword checked, top 10 | $0.003 $3 per 1,000 | $0.0027 / $0.0024 / $0.00225 | | Keyword checked, top 30 | $0.007 | $0.0063 / $0.0056 / $0.00525 | | Keyword checked, top 100 | $0.025 | $0.0225 / $0.02 / $0.01875 | | Priority queue surcharge | $0.0015 per 10 results checked | same | | Live mode surcharge | $0.003 per 10 results checked | same | | AI Overview citations only when one was loaded | $0.002 per keyword | same | | Run start only after the first successful check | $0.015 | same | Apify also adds its own $0.00005 start fee per run. Examples at list price: You can set a maximum cost per run. The Actor works out how many keywords fit before it starts and marks the rest as skipped, free of charge. Whatever data source you use, the design is what makes a tracker useful: key by keyword + location + language + device, store the URL as well as the position, diff run over run, and record SERP features. AI Overviews are changing what "ranking 2" is worth, and a tracker that doesn't record them will report rankings that look better than they are.