{"slug": "i-built-a-simple-ai-visibility-tracker-in-python-heres-what-breaks-when-you-it", "title": "I Built a Simple AI Visibility Tracker in Python. Here’s What Breaks When You Scale It", "summary": "A developer at CrawlSpider described building an AI visibility tracker in Python that queries LLM APIs to check whether brands appear in model answers, and detailed how the naive approach breaks at scale. The simple loop of prompting a model and string-matching the brand name fails under load — 100 brands across 50 prompts and 3 models means 15,000 daily requests — requiring a scheduler, job queue, worker pool, and response store, while basic substring matching also fails for ambiguous names like Apple and must be replaced with entity extraction and competitor classification. The developer argued that tracking change over time, not a single response, is where the real value lies.", "body_md": "I Built a Simple [AI Visibility Tracker](https://www.crawlspider.com/) in Python. Here’s What Breaks When You Scale It\n\nAI visibility tracking sounds like a fairly simple programming problem.\n\nAsk ChatGPT a question.\n\nCheck whether a brand appears in the answer.\n\nSave the result.\n\nRepeat tomorrow.\n\nAnd honestly, at first, it **is** that simple.\n\nYou can build a primitive AI visibility tracker in a few lines of Python.\n\n``` python\nfrom openai import OpenAI\nfrom datetime import datetime\n\nclient = OpenAI()\n\nbrand = \"Acme\"\n\nprompts = [\n    \"What are the best project management tools?\",\n    \"What are good alternatives to Trello?\",\n    \"What project management software is best for small businesses?\",\n    \"Which project management tools have AI features?\",\n    \"What tools can remote teams use to organize projects?\"\n]\n\nresults = []\n\nfor prompt in prompts:\n\n    response = client.responses.create(\n        model=\"gpt-5.4-mini\",\n        input=prompt\n    )\n\n    answer = response.output_text\n\n    results.append({\n        \"prompt\": prompt,\n        \"mentioned\": brand.lower() in answer.lower(),\n        \"response\": answer,\n        \"checked_at\": datetime.utcnow().isoformat()\n    })\n\nvisibility = (\n    sum(r[\"mentioned\"] for r in results)\n    / len(results)\n) * 100\n\nprint(f\"{brand} visibility: {visibility:.1f}%\")\n```\n\nIf Acme appears in two of five responses, we could call that 40% visibility.\n\nDone.\n\nWell... not quite.\n\nI recently worked through this problem while [building the AI visibility tracking system behind CrawlSpider](https://www.crawlspider.com/how-to-build-an-ai-visibility-tracker-from-scratch/), and the interesting part wasn't making the LLM API call.\n\nIt was everything that happened after that.\n\nConceptually, an AI visibility tracker looks something like this:\n\n```\nfor each brand:\n    for each prompt:\n        for each model:\n            ask the model\n            save the response\n            find the brand\n            find competitors\n            calculate metrics\n```\n\nThat looks harmless.\n\nBut consider:\n\n```\n100 brands\n× 50 prompts\n× 3 models\n× daily scans\n```\n\nThat's **15,000 requests every day**.\n\nOr 450,000 model responses every month.\n\nMove to 1,000 brands and you're dealing with millions.\n\nAnd suddenly this:\n\n```\nfor prompt in prompts:\n    call_llm(prompt)\n```\n\nisn't really the architecture anymore.\n\nThe first thing that breaks is the simple loop.\n\nWhat happens if request #8,742 fails?\n\nWhat happens when an API starts returning rate-limit errors?\n\nWhat if one provider slows down?\n\nWhat if your worker crashes halfway through a batch?\n\nYou don't want to restart everything.\n\nSo the architecture starts becoming something like:\n\n```\nScheduler\n    ↓\nScan Generator\n    ↓\nJob Queue\n    ↓\nWorker Pool\n    ↓\nLLM Provider\n    ↓\nResponse Store\n```\n\nNow you need:\n\nWe've moved surprisingly far away from our original Python script.\n\n`brand in response` breaks\nOur prototype has another wonderfully naive line:\n\n```\nbrand.lower() in answer.lower()\n```\n\nTry that with:\n\n```\nbrand = \"Apple\"\n```\n\nDid the model mention Apple Inc.?\n\nOr an apple?\n\nWhat about abbreviations?\n\nProduct names?\n\nParent companies?\n\nAnd simply knowing that a brand appeared isn't particularly interesting.\n\nSuppose the response says:\n\n```\nFor enterprise teams I'd consider Acme or Monday.com,\nwhile smaller teams might prefer Trello.\n```\n\nNow I probably want something closer to:\n\n```\n{\n  \"target_brand\": {\n    \"mentioned\": true,\n    \"position\": 1\n  },\n  \"competitors\": [\n    {\"name\": \"Monday.com\", \"position\": 2},\n    {\"name\": \"Trello\", \"position\": 3}\n  ]\n}\n```\n\nThe tracker has quietly turned into an entity extraction and classification system too.\n\nHere's another realization I had while working on this.\n\nA single AI response isn't particularly valuable.\n\n**Change is valuable.**\n\nImagine seeing this:\n\n```\n\"Best project management software for small businesses?\"\n\nWeek 1    Acme not mentioned\nWeek 2    Acme #5\nWeek 3    Acme #3\nWeek 4    Acme #2\n```\n\nThat's interesting.\n\nBut now every observation potentially needs:\n\n```\nbrand_id\nprompt_id\nmodel_id\nmodel_version\ntimestamp\nraw_response\nbrand_mentioned\nbrand_position\ncompetitors\nsentiment\ncitations\ntoken_usage\nlatency\nstatus\n```\n\nMultiply that by millions of responses.\n\nYou're not storing API results anymore.\n\nYou're building a historical analytics dataset.\n\nThen you decide that monitoring one AI model isn't enough.\n\nMaybe you want:\n\n```\nproviders = [\n    \"openai\",\n    \"anthropic\",\n    \"google\"\n]\n```\n\nEach has different APIs, response structures, rate limits, model identifiers, errors, citations and pricing.\n\nEventually you want an abstraction like:\n\n```\n                 ┌── OpenAI Adapter\nPrompt Engine ───┼── Anthropic Adapter\n                 └── Google Adapter\n                         ↓\n                 Normalized Response\n```\n\nOtherwise provider-specific logic ends up everywhere.\n\nThen users ask for:\n\n```\nPrompt A → Daily\nPrompt B → Weekly\nPrompt C → Daily\nPrompt D → Manual\n```\n\nNow something has to determine what is due.\n\nAnd prevent duplicate runs.\n\nAnd recover failed jobs.\n\nAnd calculate the next run.\n\nAnd make sure one huge account doesn't consume the entire worker pool.\n\nAt this point the \"AI visibility tracker\" is really a distributed job-processing and analytics application that happens to call LLMs.\n\nThere's another subtle problem.\n\nRun:\n\n```\nWhat are the best tools for X?\n```\n\ntoday and your brand might appear.\n\nRun the exact same prompt tomorrow and it might not.\n\nThat doesn't necessarily mean the brand suddenly became less visible.\n\nLLM responses vary.\n\nSo when a dashboard says:\n\n```\nVisibility\n\nLast week: 42%\nThis week: 38%\n```\n\nwhat does that actually mean?\n\nIs something changing?\n\nOr are we observing normal model variation?\n\nThis makes prompt consistency, sample size, model versions and historical comparison surprisingly important.\n\nIt's natural to focus on token costs.\n\nThose certainly matter when you're running hundreds of thousands or millions of requests.\n\nBut I found the more interesting cost to be **engineering complexity**.\n\nAt scale you're paying for much more than inference:\n\n```\nLLM inference\n+ queues\n+ workers\n+ databases\n+ storage\n+ scheduling\n+ retries\n+ observability\n+ analytics\n+ provider maintenance\n+ engineering time\n```\n\nAnd every new dimension multiplies the workload:\n\n```\nbrands\n× prompts\n× models\n× scan frequency\n× time\n```\n\nThat's the equation I'd pay attention to when designing one of these systems.\n\nOne reason we were able to build this into CrawlSpider is that we weren't starting completely from zero.\n\nI'd previously built pieces of this kind of infrastructure for other projects.\n\nInfoCaptor had given us experience with analytics, visualization and AI-driven workflows.\n\nCrawlSpider's existing internal-linking system already dealt with crawling, page analysis, background processing and large collections of URLs.\n\nOther projects had already forced us to solve problems around scheduled jobs, APIs, queues and asynchronous processing.\n\nThe AI visibility tracker became less about inventing every component and more about assembling those existing patterns around a new workflow:\n\n```\nBrand\n  ↓\nPrompts\n  ↓\nModels\n  ↓\nScheduled scans\n  ↓\nResponses\n  ↓\nMentions + competitors\n  ↓\nHistorical metrics\n  ↓\nDashboard\n```\n\nThat reuse turned out to be extremely valuable.\n\nCould you build an AI visibility tracker yourself?\n\nAbsolutely.\n\nIn fact, I think building the five-prompt Python version is a great weekend project.\n\nThe core algorithm can fit on one screen.\n\nBut that's also what makes this problem interesting.\n\nThere is a huge gap between:\n\n```\n\"Call an LLM and see if my brand appears.\"\n```\n\nand:\n\n```\n\"Reliably monitor thousands of brands across\nmultiple models every day and explain how their\nvisibility is changing.\"\n```\n\nThe first is an API call.\n\nThe second is a platform.\n\nI wrote a much deeper breakdown of the architecture, scaling math, infrastructure and costs while documenting how we approached this at CrawlSpider:\n\n[How to Build an AI Visibility Tracker From Scratch](https://www.crawlspider.com/how-to-build-an-ai-visibility-tracker-from-scratch/)\n\nIf you're building something similar, I'd be interested in hearing how you're approaching the scheduling, normalization and non-determinism problems.\n\nPS:\n\nI also built a [AI Adoption visualization Dashboard](https://www.crawlspider.com/pages/ai-adoption-rate/) , check out!\n\nI also maintain LLM cutoff dates for major providers\n\n[https://www.crawlspider.com/llm-knowledge-cutoff-dates/](https://www.crawlspider.com/llm-knowledge-cutoff-dates/)\n\nLastly there are 50+ [brands monitored for their AI Visibility](https://www.crawlspider.com/geo/)", "url": "https://wpnews.pro/news/i-built-a-simple-ai-visibility-tracker-in-python-heres-what-breaks-when-you-it", "canonical_source": "https://dev.to/nilesh_jethwa_d90f22baf69/i-built-a-simple-ai-visibility-tracker-in-python-heres-what-breaks-when-you-scale-it-24ee", "published_at": "2026-09-22 20:20:14+00:00", "updated_at": "2026-09-22 20:22:44.629839+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-search", "large-language-models", "ai-tools", "natural-language-processing"], "entities": ["CrawlSpider", "OpenAI", "ChatGPT", "Python", "Monday.com", "Trello", "Acme"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-simple-ai-visibility-tracker-in-python-heres-what-breaks-when-you-it", "markdown": "https://wpnews.pro/news/i-built-a-simple-ai-visibility-tracker-in-python-heres-what-breaks-when-you-it.md", "text": "https://wpnews.pro/news/i-built-a-simple-ai-visibility-tracker-in-python-heres-what-breaks-when-you-it.txt", "jsonld": "https://wpnews.pro/news/i-built-a-simple-ai-visibility-tracker-in-python-heres-what-breaks-when-you-it.jsonld"}}