Building a Keyword Rank Tracker That Also Flags AI Overviews (Python)

Rank tracking sounds simple: search a keyword, find your site, write down the position. The useful part is everything after that. Did it move? Did a different page of yours start ranking (cannibalization)? Did Google add an AI Overview that pushed your "position 2" below the fold?

This post walks through the design of a small rank tracker: what to store per keyword, how to diff runs, and which SERP features to record. The code works with any source of SERP data. At the end there's a hosted option.

Don't scrape Google results yourself. It's against Google's terms, gets blocked quickly, and results vary by IP location. In practice everyone uses a SERP API: you send keyword + location + language + device and get back parsed results. Whatever provider you choose, make sure it returns:

Store one record per keyword, keyed by everything that changes the SERP:

def key(keyword, location, language, device):
    return f"{keyword.lower()}|{location}|{language}|{device}"

def snapshot_row(keyword, serp, target_domain):
    mine = [r for r in serp["organic"] if r["domain"] == target_domain
            or r["domain"].endswith("." + target_domain)]
    best = mine[0] if mine else None
    return {
        "keyword": keyword,
        "position": best["position"] if best else None,
        "url": best["url"] if best else None,
        "otherUrls": [r["url"] for r in mine[1:]],
        "features": sorted(serp.get("features", [])),
        "top3": [r["domain"] for r in serp["organic"][:3]],
    }

Count subdomains as you (make it a setting), and keep the URL, not just the position: most rank-tracking insights come from URL changes.

def compare(prev, cur):
    if prev is None:
        return {"changeStatus": "first_check", "positionChange": None}
    p, c = prev["position"], cur["position"]
    if p is None and c is None:
        status, change = "not_ranked", None
    elif p is None:
        status, change = "new", None
    elif c is None:
        status, change = "lost", None
    else:
        change = p - c                       # positive = moved up
        status = "up" if change > 0 else "down" if change < 0 else "same"
    return {
        "changeStatus": status,
        "positionChange": change,
        "rankingUrlChanged": bool(p and c and prev["url"] != cur["url"]),
        "newFeatures": sorted(set(cur["features"]) - set(prev["features"])),
    }

rankingUrlChanged is your cannibalization alert: two of your pages are competing for one keyword, and Google is swapping between them. newFeatures tells you when, say, ai_overview or local_pack appears on a SERP where it wasn't before. Also, mark a run not_comparable if the depth changed (top 10 vs top 100) instead of reporting false "lost" rankings.

An organic position doesn't tell you how visible the result is. Position 2 under an AI Overview, a Local Pack and four ads sits far lower on the page than position 2 on a plain SERP. Record the absolute position too, and flag the features that matter to you: AI Overview, People Also Ask, featured snippet, Local Pack, shopping, video and forums.

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Google Rank Tracker API: Daily Keyword Positions (steady-scrape/google-rank-tracker) is an Apify Actor that implements this design on top of a licensed SERP data API. You give it a target (a domain, or an exact URL if only that page should count), up to 2,000 keywords, and a location, language and device. Each keyword comes back with its organic and absolute position, the ranking URL and any other URLs of yours on the page, the SERP features (and whether your site appears inside them), the top 3 competing domains, positions for up to 10 competitors you name, and, if you reuse the same trackerId, the change since the last run.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/google-rank-tracker").call(run_input={
    "target": "apify.com",
    "keywords": ["scrape amazon prices", "instagram scraper", "web scraping api"],
    "location": "United States",
    "language": "English",
    "device": "desktop",
    "depth": "10",
    "queueMode": "standard",
    "trackerId": "apify-us-desktop",
})
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(r["keyword"], r["position"], r["changeStatus"], r["serpFeatures"], r["aiOverviewCited"])

That's the Actor's example input. Here's what our test run of it returned on October 31, 2026:

Run it again with the same trackerId, or put it on an Apify schedule, and every row gets positionChange, changeStatus (up, down, same, new, lost and so on), previousRankingUrl and rankingUrlChanged. Set loadAiOverview: true to also load AI Overviews that Google shows asynchronously and check whether they cite you.

A few practical notes:

You pay per keyword that was checked successfully. Errors, invalid keywords and timeouts are free.

Apify also adds its own $0.00005 start fee per run.

Examples at list price:

You can set a maximum cost per run. The Actor works out how many keywords fit before it starts and marks the rest as skipped, free of charge.

Whatever data source you use, the design is what makes a tracker useful: key by keyword + location + language + device, store the URL as well as the position, diff run over run, and record SERP features. AI Overviews are changing what "ranking #2" is worth, and a tracker that doesn't record them will report rankings that look better than they are.