How to Build an AI Visibility Tracker from Scratch

A technical guide from CrawlSpider details how to build an AI visibility tracker that monitors whether models like ChatGPT mention a brand, scaling from a five-prompt Python script to a platform handling 100 brands × 50 prompts × 3 AI models × 1 scan per day, or 15,000 AI requests daily and 450,000 model responses per 30-day month. At 1,000 brands the same configuration produces 4.5 million responses per month, making the core unit of an AI visibility platform brands × prompts × models × executions × time. The guide notes that API token cost is not the biggest problem, since inexpensive models such as OpenAI's GPT-5.4-mini keep raw inference relatively affordable.

Tracking whether ChatGPT mentions your company https://www.crawlspider.com/app/llm/check sounds deceptively simple. Give an AI model a question such as: “What are the best expense-splitting apps?” Save the response. Search it for your brand name. Repeat tomorrow. At the smallest scale, that really is almost all you need. The interesting engineering problem starts when you turn that experiment into a product. Suppose you want to monitor 20 prompts for a brand, check them every day, compare several AI models, identify competitors, preserve historical responses, measure changes in visibility and eventually do the same thing for hundreds or thousands of brands. Suddenly you are no longer building a script that calls an API. You are building a data collection and analytics platform . Checkout Where businesses are using artificial intelligence https://www.crawlspider.com/pages/ai-adoption-rate/?tab=survey A Tiny AI Visibility Tracker in Python Consider a company called Acme. Perhaps Acme wants to monitor five questions: - What are the best tools for project management? - What are good alternatives to Trello? - What project management software is best for small businesses? - Which project management tools have AI features? - What tools can remote teams use to organize projects? At the simplest level, the algorithm looks something like this: python from openai import OpenAI from datetime import datetime client = OpenAI brand = "Acme" prompts = "What are the best tools for project management?", "What are good alternatives to Trello?", "What project management software is best for small businesses?", "Which project management tools have AI features?", "What tools can remote teams use to organize projects?" results = for prompt in prompts: response = client.responses.create model="gpt-5.4-mini", input=prompt answer = response.output text mentioned = brand.lower in answer.lower results.append { "brand": brand, "prompt": prompt, "mentioned": mentioned, "response": answer, "checked at": datetime.utcnow .isoformat } visibility = sum 1 for result in results if result "mentioned" / len results 100 print f"{brand} AI visibility: {visibility:.1f}%" Conceptually, the algorithm is straightforward: FOR each brand FOR each prompt FOR each AI model submit prompt capture response detect brand mention detect competitor mentions store raw response calculate metrics END END END For one company and five prompts, this is easy. If Acme appears in two of the five answers, its simple visibility score is 40%. You could put the results in a CSV file and have a basic AI visibility experiment running before lunch. But that is very different from building an AI visibility platform. The Multiplication Problem The first challenge is simply scale. Imagine moving from the experiment above to: 100 brands × 50 prompts × 3 AI models × 1 scan per day That becomes: 15,000 AI requests every day. Over a 30-day month: 450,000 model responses. Now increase the platform to 1,000 brands: 1,000 × 50 × 3 × 30 = 4.5 million responses per month https://www.crawlspider.com/geo/ . And this is only one daily observation. If some customers want more prompts, additional models or more frequent monitoring, the request count grows very quickly. The core unit of an AI visibility platform therefore isn't really a "brand." It is something closer to: brands × prompts × models × executions × time Every new dimension multiplies the workload. API Cost Is Surprisingly Not the Biggest Problem Token consumption is an obvious expense, but inexpensive models can make raw inference relatively affordable. For example, as of September 2026, OpenAI lists GPT-5.4 mini at $0.75 per million input tokens and $4.50 per million output tokens . Actual costs depend heavily on model choice, prompt length, response length, tools and whether web search is involved. Assume, purely for illustration, that an average monitoring request consumes: - 100 input tokens - 500 output tokens The approximate model cost would be: Input: 100 × $0.75 / 1,000,000 = $0.000075 Output: 500 × $4.50 / 1,000,000 = $0.00225 Total: ≈ $0.002325 per scan At that hypothetical usage: | Monthly scans | Approx. model cost | |---|---| | 10,000 | $23 | | 100,000 | $233 | | 450,000 | $1,046 | | 1,000,000 | $2,325 | | 4,500,000 | $10,463 | These are deliberately ballpark numbers rather than a CrawlSpider cost sheet. Different models and response lengths can move the numbers substantially, and search-enabled queries may introduce additional charges. OpenAI also offers asynchronous Batch API processing at a 50% discount to standard synchronous pricing, which can be attractive for monitoring workloads that do not require an immediate answer. So inference cost matters. But once a visibility tracker reaches meaningful scale, orchestration becomes at least as important as token price. You Can't Just Run 15,000 Requests in a Loop Our Python example works because there are five requests. Now imagine a scheduler launching 15,000. Some requests succeed. Some time out. Some receive rate-limit responses. Some need retrying. One provider may be experiencing elevated latency while another is operating normally. A process could crash after request 8,742. Now the system needs to know exactly which requests completed and which ones should be restarted without duplicating successful work. OpenAI itself recommends techniques such as exponential backoff for handling rate limits, and API limits vary according to usage tier. A production architecture starts looking more like: Scheduler ↓ Scan Generator ↓ Job Queue ↓ ┌─────────┬─────────┬─────────┐ │ Worker │ Worker │ Worker │ │ Pool A │ Pool B │ Pool C │ └─────────┴─────────┴─────────┘ ↓ LLM Providers ↓ Raw Response Store ↓ Response Analyzer ↓ Metrics Engine ↓ Historical Database ↓ Dashboard / API Now we need queues, workers, concurrency controls, retries, dead-letter handling, provider-specific rate limits, job states, idempotency and monitoring. That is a considerably different project from our original Python loop. Detecting a Mention Isn't Always brand in response Our toy implementation contains this line: mentioned = brand.lower in answer.lower Production systems need considerably more nuance. Imagine monitoring Apple. Does the word "apple" refer to Apple Inc. or the fruit? What if the model says "Apple's iPhone division"? What about a company with an acronym? What if the AI gives the company's product name without mentioning its parent brand? Then there are competitor mentions. An answer might say: "For enterprise teams consider Acme or Monday.com, while smaller teams may prefer Trello." A useful visibility system might want to extract: { "target brand": { "name": "Acme", "mentioned": true, "position": 1 }, "competitors": { "name": "Monday.com", "position": 2 }, { "name": "Trello", "position": 3 } } Now the system is doing entity extraction and classification in addition to generating responses. And businesses eventually want more than mention/no mention. They may want to know: - How frequently am I mentioned? - Which competitors appear instead? - Where am I positioned in recommendations? - Is the description positive, neutral or negative? - Which prompts produce visibility? - Which prompts have lost visibility? - Which model mentions me most? - Is visibility increasing over time? - What sources appear to influence the answer? Each question adds another processing and data-modeling problem. History Changes Everything A single AI response isn't particularly interesting. Change is. Suppose today's scan produces: Prompt: Best project management software for small businesses? September 1: Acme not mentioned September 8: Acme position 5 September 15: Acme position 3 September 22: Acme position 2 Now we have useful information. But providing it means every observation needs dimensions such as: brand id prompt id model id model version scan id timestamp raw response brand mentioned brand position competitors sentiment citations token usage cost latency status Run millions of scans and this becomes a genuine historical analytics dataset. The product needs both transactional infrastructure for running scans and analytical infrastructure for answering questions about the resulting history. AI Answers Are Also Non-Deterministic There is another complication. Ask an AI system the same question twice and you aren't guaranteed the identical response. That means: Brand mentioned yesterday versus: Brand not mentioned today doesn't automatically prove that something fundamental changed. The platform has to distinguish signal from normal model variation. This becomes particularly important when showing charts. A user may see visibility move from 42% to 38% and assume something happened to their brand. With a small sample of prompts, the movement could simply represent normal response variability. Good AI visibility analytics therefore requires careful treatment of sampling, prompt sets, model versions and historical comparisons. Then Add Multiple AI Providers Users don't search through only one AI system. A broader visibility platform may monitor several models or providers. Conceptually: providers = "openai", "anthropic", "google" But each provider has its own: - API - authentication - model names - pricing - rate limits - response structure - citation behavior - tool/search behavior - errors - model updates So a useful architecture needs a normalized abstraction: ┌── OpenAI Adapter Prompt Engine ───┼── Anthropic Adapter └── Google Adapter ↓ Normalized Response Without that layer, provider-specific logic eventually spreads throughout the application and becomes difficult to maintain. Scheduling Becomes Its Own Product Customers don't just want to click "scan." They want: Prompt A → Daily Prompt B → Weekly Prompt C → Daily Prompt D → Manual That creates another subsystem. The scheduler has to determine what is due, prevent duplicate execution, distribute work, recover failed jobs and calculate the next execution time. It also needs to prevent one large customer from consuming all available workers. At sufficient scale, scheduling AI scans becomes a resource-allocation problem. What Would the Infrastructure Cost? Cloud infrastructure can actually start modestly. A small system might use: Web/API servers $20–$100/month Managed database $30–$150 Queue / job infrastructure $10–$100 Object storage $5–$50 Logging / monitoring $0–$100+ Backups / bandwidth / misc. $20–$100 A small commercial implementation could therefore plausibly operate with roughly $100–$500 per month of core cloud infrastructure , before significant AI usage. At larger scale, perhaps with millions of observations, larger databases, multiple workers, extensive logs and high availability, infrastructure could move into hundreds or thousands of dollars per month . But model inference can quickly become the larger variable expense. Using our illustrative GPT-5.4-mini assumptions, 4.5 million monthly scans would be around $10,000 in model tokens alone before search/tool costs or additional providers. This is why optimization becomes important. You start asking questions such as: Can jobs use batch processing? Can static instructions benefit from prompt caching? Should different workloads use different models? Can failed scans be retried without repeating successful ones? Should raw responses live in cheaper object storage while derived metrics remain in the primary database? Can analysis be separated from expensive generation? These decisions barely matter when you have five prompts. They matter enormously at five million. CrawlSpider Had an Unusual Head Start This is also where the development path behind CrawlSpider becomes interesting. CrawlSpider's AI visibility tracker wasn't built in isolation. Several engineering pieces already existed from earlier products and internal projects. InfoCaptor https://www.infocaptor.com provided the project management, data visualization, analytics and lot of other things. The CrawlSpider internal linking system https://www.crawlspider.com/product/internal-link-building-wordpress-ai-seo/ already dealt with crawling, page analysis, large collections of URLs, content relationships, background processing and structured recommendations. Other projects had required scheduled jobs, API integrations, databases, queues and asynchronous processing. So when CrawlSpider moved into AI visibility tracking https://www.crawlspider.com/ , the project didn't begin with: Create API account ↓ Learn queues ↓ Learn scheduling ↓ Design database ↓ Build workers ↓ Build dashboards Many of the architectural patterns already existed. The new problem was how to assemble those pieces around a different unit of work: Brand ↓ Prompts ↓ Models ↓ Scheduled Scans ↓ Responses ↓ Mentions + Competitors ↓ Historical Visibility ↓ Dashboard That is a substantial advantage when building this kind of product. The Hidden Cost Is Engineering Someone evaluating an AI visibility product might look at an API price and think: "Couldn't I just build this myself?" For one brand and five prompts? Absolutely. The Python script is evidence of that. For 500 brands, 50 prompts per brand, multiple models, daily schedules, historical comparisons, retries, rate limits, competitor extraction, dashboards and millions of stored observations? That is a different question. The cost is no longer simply: API tokens It becomes: AI inference + engineering + orchestration + storage + scheduling + observability + analytics + provider maintenance + reliability And engineering is often the expensive part. From Five Prompts to a Platform That distinction is useful well beyond AI visibility tracking. Generative AI has made prototypes remarkably inexpensive. A few dozen lines of Python can demonstrate an idea that might previously have required weeks of development. But production software still has to solve the same old problems: reliability, scale, concurrency, data modeling, failure recovery, security, monitoring and cost control. AI visibility tracking is a good example. The fundamental algorithm can fit on a screen. The platform required to run that algorithm reliably millions of times, preserve the results and transform them into useful business intelligence cannot. That infrastructure is what turns an API call into a product. ----------------------------------------------------------------------------