{"slug": "how-to-build-an-ai-visibility-tracker-from-scratch", "title": "How to Build an AI Visibility Tracker from Scratch", "summary": "A technical guide from CrawlSpider details how to build an AI visibility tracker that monitors whether models like ChatGPT mention a brand, scaling from a five-prompt Python script to a platform handling 100 brands × 50 prompts × 3 AI models × 1 scan per day, or 15,000 AI requests daily and 450,000 model responses per 30-day month. At 1,000 brands the same configuration produces 4.5 million responses per month, making the core unit of an AI visibility platform brands × prompts × models × executions × time. The guide notes that API token cost is not the biggest problem, since inexpensive models such as OpenAI's GPT-5.4-mini keep raw inference relatively affordable.", "body_md": "## [Tracking whether ChatGPT mentions your company](https://www.crawlspider.com/app/llm/check) sounds deceptively simple.\n\nGive an AI model a question such as:\n\n“What are the best expense-splitting apps?”\n\nSave the response. Search it for your brand name. Repeat tomorrow.\n\nAt the smallest scale, that really is almost all you need.\n\nThe interesting engineering problem starts when you turn that experiment into a product.\n\nSuppose you want to monitor 20 prompts for a brand, check them every day, compare several AI models, identify competitors, preserve historical responses, measure changes in visibility and eventually do the same thing for hundreds or thousands of brands.\n\nSuddenly you are no longer building a script that calls an API.\n\nYou are building a ***data collection and analytics platform***.\n\nCheckout \n[Where businesses are using artificial intelligence](https://www.crawlspider.com/pages/ai-adoption-rate/?tab=survey)\n\n## A Tiny AI Visibility Tracker in Python\n\nConsider a company called Acme.\n\nPerhaps Acme wants to monitor five questions:\n\n- What are the best tools for project management?\n- What are good alternatives to Trello?\n- What project management software is best for small businesses?\n- Which project management tools have AI features?\n- What tools can remote teams use to organize projects?\n\nAt the simplest level, the algorithm looks something like this:\n\n``` python\nfrom openai import OpenAI\nfrom datetime import datetime\n\nclient = OpenAI()\n\nbrand = \"Acme\"\n\nprompts = [\n    \"What are the best tools for project management?\",\n    \"What are good alternatives to Trello?\",\n    \"What project management software is best for small businesses?\",\n    \"Which project management tools have AI features?\",\n    \"What tools can remote teams use to organize projects?\"\n]\n\nresults = []\n\nfor prompt in prompts:\n\n    response = client.responses.create(\n        model=\"gpt-5.4-mini\",\n        input=prompt\n    )\n\n    answer = response.output_text\n\n    mentioned = brand.lower() in answer.lower()\n\n    results.append({\n        \"brand\": brand,\n        \"prompt\": prompt,\n        \"mentioned\": mentioned,\n        \"response\": answer,\n        \"checked_at\": datetime.utcnow().isoformat()\n    })\n\nvisibility = (\n    sum(1 for result in results if result[\"mentioned\"])\n    / len(results)\n) * 100\n\nprint(f\"{brand} AI visibility: {visibility:.1f}%\")\n```\n\nConceptually, the algorithm is straightforward:\n\n```\nFOR each brand\n    FOR each prompt\n        FOR each AI model\n            submit prompt\n            capture response\n            detect brand mention\n            detect competitor mentions\n            store raw response\n            calculate metrics\n        END\n    END\nEND\n```\n\nFor one company and five prompts, this is easy.\n\nIf Acme appears in two of the five answers, its simple visibility score is 40%.\n\nYou could put the results in a CSV file and have a basic AI visibility experiment running before lunch.\n\nBut that is very different from building an AI visibility platform.\n\n## The Multiplication Problem\n\nThe first challenge is simply scale.\n\nImagine moving from the experiment above to:\n\n**100 brands × 50 prompts × 3 AI models × 1 scan per day**\n\nThat becomes:\n\n**15,000 AI requests every day.**\n\nOver a 30-day month:\n\n**450,000 model responses.**\n\nNow increase the platform to 1,000 brands:\n\n**1,000 × 50 × 3 × 30 = [4.5 million responses per month](https://www.crawlspider.com/geo/).**\n\nAnd this is only one daily observation.\n\nIf some customers want more prompts, additional models or more frequent monitoring, the request count grows very quickly.\n\nThe core unit of an **AI visibility platform** therefore isn't really a \"brand.\"\n\nIt is something closer to:\n\n**brands × prompts × models × executions × time**\n\nEvery new dimension multiplies the workload.\n\n## API Cost Is Surprisingly Not the Biggest Problem\n\nToken consumption is an obvious expense, but inexpensive models can make raw inference relatively affordable.\n\nFor example, as of September 2026, OpenAI lists GPT-5.4 mini at **$0.75 per million input tokens and $4.50 per million output tokens**. Actual costs depend heavily on model choice, prompt length, response length, tools and whether web search is involved.\n\nAssume, purely for illustration, that an average monitoring request consumes:\n\n- 100 input tokens\n- 500 output tokens\n\nThe approximate model cost would be:\n\n```\nInput:\n100 × $0.75 / 1,000,000\n= $0.000075\n\nOutput:\n500 × $4.50 / 1,000,000\n= $0.00225\n\nTotal:\n≈ $0.002325 per scan\n```\n\nAt that hypothetical usage:\n\n| Monthly scans | Approx. model cost | \n|---|---|\n| 10,000 | $23 | \n| 100,000 | $233 | \n| 450,000 | $1,046 | \n| 1,000,000 | $2,325 | \n| 4,500,000 | $10,463 | \n\nThese are deliberately ballpark numbers rather than a CrawlSpider cost sheet. Different models and response lengths can move the numbers substantially, and search-enabled queries may introduce additional charges.\n\nOpenAI also offers asynchronous Batch API processing at a 50% discount to standard synchronous pricing, which can be attractive for monitoring workloads that do not require an immediate answer.\n\nSo inference cost matters.\n\nBut once a visibility tracker reaches meaningful scale, **orchestration becomes at least as important as token price.**\n\n## You Can't Just Run 15,000 Requests in a Loop\n\nOur Python example works because there are five requests.\n\nNow imagine a scheduler launching 15,000.\n\nSome requests succeed.\n\nSome time out.\n\nSome receive rate-limit responses.\n\nSome need retrying.\n\nOne provider may be experiencing elevated latency while another is operating normally.\n\nA process could crash after request 8,742.\n\nNow the system needs to know exactly which requests completed and which ones should be restarted without duplicating successful work.\n\nOpenAI itself recommends techniques such as exponential backoff for handling rate limits, and API limits vary according to usage tier.\n\nA production architecture starts looking more like:\n\n```\nScheduler\n   ↓\nScan Generator\n   ↓\nJob Queue\n   ↓\n┌─────────┬─────────┬─────────┐\n│ Worker  │ Worker  │ Worker  │\n│ Pool A  │ Pool B  │ Pool C  │\n└─────────┴─────────┴─────────┘\n   ↓\nLLM Providers\n   ↓\nRaw Response Store\n   ↓\nResponse Analyzer\n   ↓\nMetrics Engine\n   ↓\nHistorical Database\n   ↓\nDashboard / API\n```\n\nNow we need queues, workers, concurrency controls, retries, dead-letter handling, provider-specific rate limits, job states, idempotency and monitoring.\n\nThat is a considerably different project from our original Python loop.\n\n## Detecting a Mention Isn't Always `brand in response`\n\nOur toy implementation contains this line:\n\n```\nmentioned = brand.lower() in answer.lower()\n```\n\nProduction systems need considerably more nuance.\n\nImagine monitoring Apple.\n\nDoes the word \"apple\" refer to Apple Inc. or the fruit?\n\nWhat if the model says \"Apple's iPhone division\"?\n\nWhat about a company with an acronym?\n\nWhat if the AI gives the company's product name without mentioning its parent brand?\n\nThen there are competitor mentions.\n\nAn answer might say:\n\n\"For enterprise teams consider Acme or Monday.com, while smaller teams may prefer Trello.\"\n\nA useful visibility system might want to extract:\n\n```\n{\n  \"target_brand\": {\n    \"name\": \"Acme\",\n    \"mentioned\": true,\n    \"position\": 1\n  },\n  \"competitors\": [\n    {\n      \"name\": \"Monday.com\",\n      \"position\": 2\n    },\n    {\n      \"name\": \"Trello\",\n      \"position\": 3\n    }\n  ]\n}\n```\n\nNow the system is doing entity extraction and classification in addition to generating responses.\n\nAnd businesses eventually want more than mention/no mention.\n\nThey may want to know:\n\n- How frequently am I mentioned?\n- Which competitors appear instead?\n- Where am I positioned in recommendations?\n- Is the description positive, neutral or negative?\n- Which prompts produce visibility?\n- Which prompts have lost visibility?\n- Which model mentions me most?\n- Is visibility increasing over time?\n- What sources appear to influence the answer?\n\nEach question adds another processing and data-modeling problem.\n\n## History Changes Everything\n\nA single AI response isn't particularly interesting.\n\nChange is.\n\nSuppose today's scan produces:\n\n```\nPrompt: Best project management software for small businesses?\n\nSeptember 1:\nAcme not mentioned\n\nSeptember 8:\nAcme position #5\n\nSeptember 15:\nAcme position #3\n\nSeptember 22:\nAcme position #2\n```\n\nNow we have useful information.\n\nBut providing it means every observation needs dimensions such as:\n\n```\nbrand_id\nprompt_id\nmodel_id\nmodel_version\nscan_id\ntimestamp\nraw_response\nbrand_mentioned\nbrand_position\ncompetitors\nsentiment\ncitations\ntoken_usage\ncost\nlatency\nstatus\n```\n\nRun millions of scans and this becomes a genuine historical analytics dataset.\n\nThe product needs both transactional infrastructure for running scans and analytical infrastructure for answering questions about the resulting history.\n\n## AI Answers Are Also Non-Deterministic\n\nThere is another complication.\n\nAsk an AI system the same question twice and you aren't guaranteed the identical response.\n\nThat means:\n\n```\nBrand mentioned yesterday\n```\n\nversus:\n\n```\nBrand not mentioned today\n```\n\ndoesn't automatically prove that something fundamental changed.\n\nThe platform has to distinguish signal from normal model variation.\n\nThis becomes particularly important when showing charts.\n\nA user may see visibility move from 42% to 38% and assume something happened to their brand. With a small sample of prompts, the movement could simply represent normal response variability.\n\nGood AI visibility analytics therefore requires careful treatment of sampling, prompt sets, model versions and historical comparisons.\n\n## Then Add Multiple AI Providers\n\nUsers don't search through only one AI system.\n\nA broader visibility platform may monitor several models or providers.\n\nConceptually:\n\n```\nproviders = [\n    \"openai\",\n    \"anthropic\",\n    \"google\"\n]\n```\n\nBut each provider has its own:\n\n- API\n- authentication\n- model names\n- pricing\n- rate limits\n- response structure\n- citation behavior\n- tool/search behavior\n- errors\n- model updates\n\nSo a useful architecture needs a normalized abstraction:\n\n```\n                 ┌── OpenAI Adapter\nPrompt Engine ───┼── Anthropic Adapter\n                 └── Google Adapter\n                         ↓\n                Normalized Response\n```\n\nWithout that layer, provider-specific logic eventually spreads throughout the application and becomes difficult to maintain.\n\n## Scheduling Becomes Its Own Product\n\nCustomers don't just want to click \"scan.\"\n\nThey want:\n\n```\nPrompt A → Daily\nPrompt B → Weekly\nPrompt C → Daily\nPrompt D → Manual\n```\n\nThat creates another subsystem.\n\nThe scheduler has to determine what is due, prevent duplicate execution, distribute work, recover failed jobs and calculate the next execution time.\n\nIt also needs to prevent one large customer from consuming all available workers.\n\nAt sufficient scale, scheduling AI scans becomes a resource-allocation problem.\n\n## What Would the Infrastructure Cost?\n\nCloud infrastructure can actually start modestly.\n\nA small system might use:\n\n```\nWeb/API servers                  $20–$100/month\nManaged database                 $30–$150\nQueue / job infrastructure       $10–$100\nObject storage                   $5–$50\nLogging / monitoring             $0–$100+\nBackups / bandwidth / misc.      $20–$100\n```\n\nA small commercial implementation could therefore plausibly operate with **roughly $100–$500 per month of core cloud infrastructure**, before significant AI usage.\n\nAt larger scale, perhaps with millions of observations, larger databases, multiple workers, extensive logs and high availability, infrastructure could move into **hundreds or thousands of dollars per month**.\n\nBut model inference can quickly become the larger variable expense.\n\nUsing our illustrative GPT-5.4-mini assumptions, 4.5 million monthly scans would be around $10,000 in model tokens alone before search/tool costs or additional providers.\n\nThis is why optimization becomes important.\n\nYou start asking questions such as:\n\nCan jobs use batch processing?\n\nCan static instructions benefit from prompt caching?\n\nShould different workloads use different models?\n\nCan failed scans be retried without repeating successful ones?\n\nShould raw responses live in cheaper object storage while derived metrics remain in the primary database?\n\nCan analysis be separated from expensive generation?\n\nThese decisions barely matter when you have five prompts.\n\nThey matter enormously at five million.\n\n## CrawlSpider Had an Unusual Head Start\n\nThis is also where the development path behind CrawlSpider becomes interesting.\n\nCrawlSpider's AI visibility tracker wasn't built in isolation.\n\nSeveral engineering pieces already existed from earlier products and internal projects.\n\n**[InfoCaptor](https://www.infocaptor.com)** provided the project management, data visualization, analytics and lot of other things.\n\nThe **[CrawlSpider internal linking system](https://www.crawlspider.com/product/internal-link-building-wordpress-ai-seo/)** already dealt with crawling, page analysis, large collections of URLs, content relationships, background processing and structured recommendations.\n\nOther projects had required scheduled jobs, API integrations, databases, queues and asynchronous processing.\n\nSo when CrawlSpider moved into [AI visibility tracking](https://www.crawlspider.com/), the project didn't begin with:\n\n```\nCreate API account\n↓\nLearn queues\n↓\nLearn scheduling\n↓\nDesign database\n↓\nBuild workers\n↓\nBuild dashboards\n```\n\nMany of the architectural patterns already existed.\n\nThe new problem was how to assemble those pieces around a different unit of work:\n\n```\nBrand\n   ↓\nPrompts\n   ↓\nModels\n   ↓\nScheduled Scans\n   ↓\nResponses\n   ↓\nMentions + Competitors\n   ↓\nHistorical Visibility\n   ↓\nDashboard\n```\n\nThat is a substantial advantage when building this kind of product.\n\n## The Hidden Cost Is Engineering\n\nSomeone evaluating an AI visibility product might look at an API price and think:\n\n*\"Couldn't I just build this myself?\"*\n\nFor one brand and five prompts?\n\nAbsolutely.\n\nThe Python script is evidence of that.\n\nFor 500 brands, 50 prompts per brand, multiple models, daily schedules, historical comparisons, retries, rate limits, competitor extraction, dashboards and millions of stored observations?\n\nThat is a different question.\n\nThe cost is no longer simply:\n\n```\nAPI tokens\n```\n\nIt becomes:\n\n```\nAI inference\n+ engineering\n+ orchestration\n+ storage\n+ scheduling\n+ observability\n+ analytics\n+ provider maintenance\n+ reliability\n```\n\nAnd engineering is often the expensive part.\n\n## From Five Prompts to a Platform\n\nThat distinction is useful well beyond AI visibility tracking.\n\nGenerative AI has made prototypes remarkably inexpensive.\n\nA few dozen lines of Python can demonstrate an idea that might previously have required weeks of development.\n\nBut production software still has to solve the same old problems:\n\nreliability, scale, concurrency, data modeling, failure recovery, security, monitoring and cost control.\n\nAI visibility tracking is a good example.\n\nThe fundamental algorithm can fit on a screen.\n\nThe platform required to run that algorithm reliably millions of times, preserve the results and transform them into useful business intelligence cannot.\n\nThat infrastructure is what turns an API call into a product.\n\n----------------------------------------------------------------------------", "url": "https://wpnews.pro/news/how-to-build-an-ai-visibility-tracker-from-scratch", "canonical_source": "https://www.crawlspider.com/how-to-build-an-ai-visibility-tracker-from-scratch/", "published_at": "2026-09-23 19:58:07+00:00", "updated_at": "2026-09-23 20:31:55.296108+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-search", "large-language-models", "ai-tools"], "entities": ["CrawlSpider", "OpenAI", "ChatGPT", "Acme", "Trello", "GPT-5.4-mini", "GPT-5"], "alternates": {"html": "https://wpnews.pro/news/how-to-build-an-ai-visibility-tracker-from-scratch", "markdown": "https://wpnews.pro/news/how-to-build-an-ai-visibility-tracker-from-scratch.md", "text": "https://wpnews.pro/news/how-to-build-an-ai-visibility-tracker-from-scratch.txt", "jsonld": "https://wpnews.pro/news/how-to-build-an-ai-visibility-tracker-from-scratch.jsonld"}}