{"slug": "setting-up-ai-search-rank-tracking-easily", "title": "SEtting up AI search rank tracking easily", "summary": "A new open-source repository offers a self-hosted, scheduled tool for tracking whether AI search engines such as ChatGPT, Claude, Perplexity, and Gemini mention or cite a website in response to buying questions, providing a DIY alternative to commercial AI visibility tools that cost $90 or more per month. The tool, which requires only a fork of the repo and API keys, scores answers for mentions and citations separately and trends results over time, with engine modules for OpenAI, Anthropic, Perplexity, and Gemini.", "body_md": "Track whether AI search engines (ChatGPT, Claude, Perplexity, Gemini) mention or cite **your** site when users ask buying questions. Self-hosted, scheduled, no servers - a fork of this repo and a couple of API keys is the whole setup.\n\nThis is the DIY version of what commercial \"AI visibility\" tools sell for $90+/month: scheduled prompt sampling against web-grounded AI APIs, scored for mentions and citations, trended over time.\n\nYou cannot see real user conversations with AI assistants. Nobody can - not you, not the $90/month tools. What you CAN do is **sample**: define the buying questions your customers would plausibly ask, put the same questions to the AI engines on a schedule, and measure how often your site appears in the answers.\n\n**It is a poll, not a census.** Run it weekly and you get a trend line for \"do I rank on AI?\" - which is the thing you actually need to manage.\n\nEvery answer is scored for two distinct outcomes, never conflated:\n\n**Mention**- the answer's text names your brand (\"...Acme Coffee Gear is a solid choice for beginners...\"). The user*saw*you.**Citation**- your domain appears in the answer's cited sources. Your content*fed*the answer, even if the text never named you.\n\nBoth matter. A mention without a citation means the model knows you from training data. A citation without a mention means your content is doing work for someone else's answer. Absent means neither - and that is the number you are trying to move.\n\n- Buyers increasingly ask AI assistants where to buy, what to buy, and whom to trust. Those answers name specific stores and cite specific sites.\n- You cannot manage what you cannot measure. If AI answers are a channel, you need a number for it, the same way you have one for Google rankings.\n- Commercial AI-visibility tools exist and are fine - but the core loop (ask, score, trend) is simple enough to self-host for the price of the API calls. This repo is that loop, readable in an afternoon.\n\nFour steps, four small files.\n\n[ prompts.json](/razz1000/ai-search-rank-tracking-example-repo/blob/main/prompts.json) holds the buying questions. The shipped ones are for a fictional specialty coffee equipment store -\n\n**replace them with your own**(see\n\n[Writing good prompts](#writing-good-prompts)).\n\n```\n{\n  \"id\": \"where-buy-grinder\",\n  \"prompt\": \"Where should I buy a good specialty coffee grinder online? I want a shop with real expertise, not just a marketplace.\",\n  \"tags\": [\"store-intent\", \"broad\"]\n}\n```\n\nEach engine module takes a prompt and returns the same shape: the answer text plus its cited sources. Web grounding is the point - without it the models answer from training data and cite nothing.\n\n``` js\n// src/engines/perplexity.ts (the simplest of the three)\nconst res = await fetch(\"https://api.perplexity.ai/chat/completions\", {\n  method: \"POST\",\n  headers: { authorization: `Bearer ${process.env.PERPLEXITY_API_KEY}`, ... },\n  body: JSON.stringify({ model: \"sonar\", messages: [{ role: \"user\", content: prompt }] }),\n});\n// -> { text: string, sources: string[] }\n```\n\n**OpenAI**- Responses API with the built-in`web_search`\n\ntool ()`src/engines/openai.ts`\n\n**Anthropic**- Claude Messages API with the server-side web search tool; cited sources come back as citations on the answer text ()`src/engines/anthropic.ts`\n\n**Perplexity**-`sonar`\n\n, search-grounded by default ()`src/engines/perplexity.ts`\n\n**Gemini**- Google Search grounding tool ()`src/engines/gemini.ts`\n\nEvery engine is optional: if its API key is not set, it is skipped with a console note. Any subset works.\n\nModel names rot. Each engine file has a single\n\n`MODEL`\n\nconstant at the top with a comment - that is the only thing to update when a provider rotates model versions. The names in this repo were current when it was built.\n\n[ src/score.ts](/razz1000/ai-search-rank-tracking-example-repo/blob/main/src/score.ts) checks each answer against the brands in\n\n[: a case-insensitive alias match in the answer](/razz1000/ai-search-rank-tracking-example-repo/blob/main/config.json)\n\n`config.json`\n\n*text*is a mention (the matching snippet is stored so you can read how you were mentioned); a domain match in the\n\n*sources*is a citation.\n\n```\n{\n  \"runs\": 3,\n  \"brands\": [\n    { \"name\": \"Acme Coffee Gear\", \"aliases\": [\"Acme Coffee Gear\", \"Acme Coffee\"], \"domains\": [\"acmecoffeegear.com\"] },\n    { \"name\": \"Prima Coffee (competitor example)\", \"aliases\": [\"Prima Coffee\"], \"domains\": [\"prima-coffee.com\"] }\n  ]\n}\n```\n\nBecause answers vary between identical asks, each prompt is asked `runs`\n\ntimes (default 3) per engine, and results are always reported as X out of N runs - never as a binary \"you rank / you don't\".\n\nThe example config tracks a fictional brand plus one real retailer as a competitor example, so your very first run produces non-zero data. Replace both with your brand and your actual competitors.\n\nEach run appends one JSONL file to [ data/](/razz1000/ai-search-rank-tracking-example-repo/blob/main/data) - one line per (prompt, engine, run) with the full answer text, sources, and scores - and regenerates\n\n[. The workflow commits both, so](/razz1000/ai-search-rank-tracking-example-repo/blob/main/REPORT.md)\n\n`REPORT.md`\n\n**the git history is the time series**. No database.\n\n```\nnpm install\ncp .env.example .env   # add at least one API key\nnpm run track          # sample all engines, write data/ + REPORT.md\nnpm run report         # regenerate REPORT.md from existing data (no API calls)\n```\n\nThe repo ships a GitHub Actions workflow ([ .github/workflows/track.yml](/razz1000/ai-search-rank-tracking-example-repo/blob/main/.github/workflows/track.yml)) that runs weekly and commits the results back. Setup:\n\n**Fork this repo****Add secrets**: repo Settings → Secrets and variables → Actions → add`OPENAI_API_KEY`\n\n/`ANTHROPIC_API_KEY`\n\n/`PERPLEXITY_API_KEY`\n\n/`GEMINI_API_KEY`\n\n(any subset works)**Edit** for your siteand`prompts.json`\n\n`config.json`\n\nEvery Monday the Action runs the tracker and commits a fresh `REPORT.md`\n\nplus the raw data. No servers. You can also trigger it manually from the Actions tab (workflow_dispatch) - do that once after setup to check everything works.\n\nThe tracker is only as good as the questions you feed it. Rules of thumb:\n\n**Real buying intent.**\"Where should I buy X\" and \"what is the best X for Y\" - the questions that end in a purchase, not \"what is a coffee grinder\".**Phrased like a customer.** Write the way a person talks to a chatbot: first person, context, constraints (\"for a beginner\", \"under $800\"). Not keyword strings.**Mix broad and long-tail.** Broad prompts (\"best home espresso machine\") tell you about the big race you are probably losing; long-tail prompts (\"which stores sell the Comandante C40 in the US\") are where a specialty site realistically appears first.**5 to 15 prompts is plenty.** More prompts means more cost and more report to read. Start small; add prompts when you have a hypothesis to test.**Keep prompt** The trend line is per prompt id - renaming an id starts its history over.`id`\n\ns stable.\n\n[ REPORT.md](/razz1000/ai-search-rank-tracking-example-repo/blob/main/REPORT.md) has three parts:\n\n**Per-brand summary table** (prompt x engine) for the latest session, with a trend arrow against the previous session:\n\n| Prompt | openai | perplexity | gemini |\n|---|---|---|---|\n| where-buy-grinder | 0/3 M - 1/3 C ↑ | 1/3 M - 2/3 C = | 0/3 M - 0/3 C ↓ |\n\n*(Illustrative numbers, not real output.)* `1/3 M`\n\n= mentioned in 1 of 3 runs; `2/3 C`\n\n= cited in 2 of 3. Expect variance - that is why runs exist.\n\n**Trend over time** - the same rates per session, so you can see whether a content push or a new competitor moved anything.\n\n**Sources that fed the answers** - every domain the engines cited this session, most-cited first. This is the competitive intel: these sites are being read to answer *your* buying questions. If your domain is not on the list, the list tells you exactly who is there instead of you - and what kind of content (reviews, comparisons, guides) is winning the citations.\n\nMention snippets are stored too, so you can read *how* you were mentioned - \"great for beginners\" and \"had shipping problems\" are both mentions.\n\nRough math so nobody is surprised by an API bill. Per tracking session with the default config (5 prompts x 3 runs = 15 calls per engine):\n\n**OpenAI**(`gpt-5-mini`\n\n+ web search tool): the search tool call dominates, roughly $0.01 to $0.02 per call → about**$0.15 to $0.30****Anthropic**(`claude-haiku-4-5`\n\n+ web search tool): a per-search fee plus tokens - and note that search results are billed as*input tokens*, which is where the money goes on this engine. On Haiku, roughly $0.02 to $0.05 per call → about**$0.30 to $0.75**. A measured warning from running this in anger: on`claude-opus-5`\n\nthe exact same session cost ~20x more (search-result input tokens at Opus prices), so if you swap the`MODEL`\n\nconstant into a bigger model, check your first bill.`src/engines/anthropic.ts`\n\n**Perplexity**(`sonar`\n\n): request fees plus tokens, roughly**$0.10 to $0.15****Gemini**(`gemini-3.6-flash`\n\n+ Search grounding): grounded prompts have a free daily allowance at the time of writing; on paid billing roughly**$0.50**\n\nSo a full four-engine weekly run lands **around $1 to $2**, or a few dollars per month - and under $1 without the Anthropic engine or with a cheaper Claude model. Costs scale linearly: prompts x engines x runs x price per grounded call. Double the prompts, double the bill. Prices change - check each provider's current pricing page before scaling up.\n\nSame rules as the rest of this series - no pretending:\n\n**This samples the questions YOU define. It cannot see what real users ask or what they were told.** It is a controlled poll that approximates a channel you cannot observe directly.**API answers are a proxy for the consumer apps, not a screenshot of them.** The APIs use the same underlying engines and web grounding as ChatGPT, Perplexity and Gemini, but the consumer apps add their own layers (memory, location, A/B tests). Treat the trend as the signal, not any single answer.**Mentions and citations are different things** and the report never merges them.**Answers are nondeterministic.** The same prompt asked twice gives different answers - that is why every number is X out of N runs, and why the weekly trend matters more than any single session.\n\n**Why do results differ between runs?**\nThe models are nondeterministic and the live web results behind them shift constantly. That is not a bug in the tracker; it is the nature of the channel. It is exactly why the tracker asks each prompt several times and reports X/N instead of a checkmark.\n\n**Why not scrape the ChatGPT website instead?**\nScraping the consumer apps breaks their terms of service and breaks *technically* every few weeks. The APIs are the legitimate, stable path, and they use the same engines and grounding. A proxy you can run every week beats a perfect measurement you can run never.\n\n**Can I track competitors?**\nYes - that is why `config.json`\n\ntakes an array of brands. Add each competitor with their aliases and domains and the report shows all brands side by side. Watching a competitor's citation rate climb on your key prompts is usually the earliest signal you will get.\n\n**How is this different from AI crawler logs?**\nCrawler logs (GPTBot, PerplexityBot etc. in your server logs) tell you your site is being *read*. This repo tells you whether your site is being *surfaced* in answers. Being read is necessary but not sufficient - the two measurements are complementary halves of the same funnel.\n\n📺 **YouTube walkthrough:** [Track How You Rank on ChatGPT & Claude - for free](https://youtu.be/dk2LoGmvARU) - how the tracker works, full setup with \"Use this template\", and real results (Prometora vs Sharetribe on AI search).\n\nThis repo is the \"answer side\" of AI visibility: are you surfaced when people ask? The other half is the \"crawl side\": are AI engines reading your pages at all?\n\n[ Prometora](https://www.prometora.com) - a marketplace builder with AI Visibility built in - tracks the crawl side server-side, per listing: which AI crawlers fetched which pages, trended over time. See\n\n[their AI crawler research](https://www.prometora.com/learn/ai-crawler-data)for what that data looks like at scale. The crawl side is built into Prometora today; this repo is the answer side, self-hosted.\n\nMIT - see [LICENSE](/razz1000/ai-search-rank-tracking-example-repo/blob/main/LICENSE). Questions, ideas, corrections: [rasmus@prometora.com](mailto:rasmus@prometora.com)", "url": "https://wpnews.pro/news/setting-up-ai-search-rank-tracking-easily", "canonical_source": "https://github.com/razz1000/ai-search-rank-tracking-example-repo", "published_at": "2026-08-14 12:35:44+00:00", "updated_at": "2026-08-14 12:42:33.326916+00:00", "lang": "en", "topics": ["ai-tools", "ai-products"], "entities": ["OpenAI", "Anthropic", "Perplexity", "Gemini", "ChatGPT", "Claude"], "alternates": {"html": "https://wpnews.pro/news/setting-up-ai-search-rank-tracking-easily", "markdown": "https://wpnews.pro/news/setting-up-ai-search-rank-tracking-easily.md", "text": "https://wpnews.pro/news/setting-up-ai-search-rank-tracking-easily.txt", "jsonld": "https://wpnews.pro/news/setting-up-ai-search-rank-tracking-easily.jsonld"}}