Track whether AI search engines (ChatGPT, Claude, Perplexity, Gemini) mention or cite your site when users ask buying questions. Self-hosted, scheduled, no servers - a fork of this repo and a couple of API keys is the whole setup.
This is the DIY version of what commercial "AI visibility" tools sell for $90+/month: scheduled prompt sampling against web-grounded AI APIs, scored for mentions and citations, trended over time.
You cannot see real user conversations with AI assistants. Nobody can - not you, not the $90/month tools. What you CAN do is sample: define the buying questions your customers would plausibly ask, put the same questions to the AI engines on a schedule, and measure how often your site appears in the answers.
It is a poll, not a census. Run it weekly and you get a trend line for "do I rank on AI?" - which is the thing you actually need to manage.
Every answer is scored for two distinct outcomes, never conflated:
Mention- the answer's text names your brand ("...Acme Coffee Gear is a solid choice for beginners..."). The usersawyou.Citation- your domain appears in the answer's cited sources. Your contentfedthe answer, even if the text never named you.
Both matter. A mention without a citation means the model knows you from training data. A citation without a mention means your content is doing work for someone else's answer. Absent means neither - and that is the number you are trying to move.
- Buyers increasingly ask AI assistants where to buy, what to buy, and whom to trust. Those answers name specific stores and cite specific sites.
- You cannot manage what you cannot measure. If AI answers are a channel, you need a number for it, the same way you have one for Google rankings.
- Commercial AI-visibility tools exist and are fine - but the core loop (ask, score, trend) is simple enough to self-host for the price of the API calls. This repo is that loop, readable in an afternoon.
Four steps, four small files.
prompts.json holds the buying questions. The shipped ones are for a fictional specialty coffee equipment store -
replace them with your own(see
{
"id": "where-buy-grinder",
"prompt": "Where should I buy a good specialty coffee grinder online? I want a shop with real expertise, not just a marketplace.",
"tags": ["store-intent", "broad"]
}
Each engine module takes a prompt and returns the same shape: the answer text plus its cited sources. Web grounding is the point - without it the models answer from training data and cite nothing.
// src/engines/perplexity.ts (the simplest of the three)
const res = await fetch("https://api.perplexity.ai/chat/completions", {
method: "POST",
headers: { authorization: `Bearer ${process.env.PERPLEXITY_API_KEY}`, ... },
body: JSON.stringify({ model: "sonar", messages: [{ role: "user", content: prompt }] }),
});
// -> { text: string, sources: string[] }
OpenAI- Responses API with the built-inweb_search
tool ()src/engines/openai.ts
Anthropic- Claude Messages API with the server-side web search tool; cited sources come back as citations on the answer text ()src/engines/anthropic.ts
Perplexity-sonar
, search-grounded by default ()src/engines/perplexity.ts
Gemini- Google Search grounding tool ()src/engines/gemini.ts
Every engine is optional: if its API key is not set, it is skipped with a console note. Any subset works.
Model names rot. Each engine file has a single
MODEL
constant at the top with a comment - that is the only thing to update when a provider rotates model versions. The names in this repo were current when it was built.
src/score.ts checks each answer against the brands in
: a case-insensitive alias match in the answer
config.json
textis a mention (the matching snippet is stored so you can read how you were mentioned); a domain match in the
sourcesis a citation.
{
"runs": 3,
"brands": [
{ "name": "Acme Coffee Gear", "aliases": ["Acme Coffee Gear", "Acme Coffee"], "domains": ["acmecoffeegear.com"] },
{ "name": "Prima Coffee (competitor example)", "aliases": ["Prima Coffee"], "domains": ["prima-coffee.com"] }
]
}
Because answers vary between identical asks, each prompt is asked runs
times (default 3) per engine, and results are always reported as X out of N runs - never as a binary "you rank / you don't".
The example config tracks a fictional brand plus one real retailer as a competitor example, so your very first run produces non-zero data. Replace both with your brand and your actual competitors.
Each run appends one JSONL file to data/ - one line per (prompt, engine, run) with the full answer text, sources, and scores - and regenerates
. The workflow commits both, so
REPORT.md
the git history is the time series. No database.
npm install
cp .env.example .env # add at least one API key
npm run track # sample all engines, write data/ + REPORT.md
npm run report # regenerate REPORT.md from existing data (no API calls)
The repo ships a GitHub Actions workflow ( .github/workflows/track.yml) that runs weekly and commits the results back. Setup:
Fork this repo****Add secrets: repo Settings → Secrets and variables → Actions → addOPENAI_API_KEY
/ANTHROPIC_API_KEY
/PERPLEXITY_API_KEY
/GEMINI_API_KEY
(any subset works)Edit for your siteandprompts.json
config.json
Every Monday the Action runs the tracker and commits a fresh REPORT.md
plus the raw data. No servers. You can also trigger it manually from the Actions tab (workflow_dispatch) - do that once after setup to check everything works.
The tracker is only as good as the questions you feed it. Rules of thumb:
Real buying intent."Where should I buy X" and "what is the best X for Y" - the questions that end in a purchase, not "what is a coffee grinder".Phrased like a customer. Write the way a person talks to a chatbot: first person, context, constraints ("for a beginner", "under $800"). Not keyword strings.Mix broad and long-tail. Broad prompts ("best home espresso machine") tell you about the big race you are probably losing; long-tail prompts ("which stores sell the Comandante C40 in the US") are where a specialty site realistically appears first.5 to 15 prompts is plenty. More prompts means more cost and more report to read. Start small; add prompts when you have a hypothesis to test.Keep prompt The trend line is per prompt id - renaming an id starts its history over.id
s stable.
REPORT.md has three parts:
Per-brand summary table (prompt x engine) for the latest session, with a trend arrow against the previous session:
| Prompt | openai | perplexity | gemini |
|---|---|---|---|
| where-buy-grinder | 0/3 M - 1/3 C ↑ | 1/3 M - 2/3 C = | 0/3 M - 0/3 C ↓ |
(Illustrative numbers, not real output.) 1/3 M
= mentioned in 1 of 3 runs; 2/3 C
= cited in 2 of 3. Expect variance - that is why runs exist.
Trend over time - the same rates per session, so you can see whether a content push or a new competitor moved anything.
Sources that fed the answers - every domain the engines cited this session, most-cited first. This is the competitive intel: these sites are being read to answer your buying questions. If your domain is not on the list, the list tells you exactly who is there instead of you - and what kind of content (reviews, comparisons, guides) is winning the citations.
Mention snippets are stored too, so you can read how you were mentioned - "great for beginners" and "had shipping problems" are both mentions.
Rough math so nobody is surprised by an API bill. Per tracking session with the default config (5 prompts x 3 runs = 15 calls per engine):
OpenAI(gpt-5-mini
-
web search tool): the search tool call dominates, roughly $0.01 to $0.02 per call → about**$0.15 to $0.30****Anthropic**(
claude-haiku-4-5 -
web search tool): a per-search fee plus tokens - and note that search results are billed asinput tokens, which is where the money goes on this engine. On Haiku, roughly $0.02 to $0.05 per call → about**$0.30 to $0.75**. A measured warning from running this in anger: on
claude-opus-5
the exact same session cost ~20x more (search-result input tokens at Opus prices), so if you swap theMODEL
constant into a bigger model, check your first bill.src/engines/anthropic.ts
Perplexity(sonar
): request fees plus tokens, roughly**$0.10 to $0.15****Gemini**(gemini-3.6-flash
- Search grounding): grounded prompts have a free daily allowance at the time of writing; on paid billing roughly**$0.50**
So a full four-engine weekly run lands around $1 to $2, or a few dollars per month - and under $1 without the Anthropic engine or with a cheaper Claude model. Costs scale linearly: prompts x engines x runs x price per grounded call. Double the prompts, double the bill. Prices change - check each provider's current pricing page before scaling up.
Same rules as the rest of this series - no pretending:
This samples the questions YOU define. It cannot see what real users ask or what they were told. It is a controlled poll that approximates a channel you cannot observe directly.API answers are a proxy for the consumer apps, not a screenshot of them. The APIs use the same underlying engines and web grounding as ChatGPT, Perplexity and Gemini, but the consumer apps add their own layers (memory, location, A/B tests). Treat the trend as the signal, not any single answer.Mentions and citations are different things and the report never merges them.Answers are nondeterministic. The same prompt asked twice gives different answers - that is why every number is X out of N runs, and why the weekly trend matters more than any single session.
Why do results differ between runs? The models are nondeterministic and the live web results behind them shift constantly. That is not a bug in the tracker; it is the nature of the channel. It is exactly why the tracker asks each prompt several times and reports X/N instead of a checkmark.
Why not scrape the ChatGPT website instead? Scraping the consumer apps breaks their terms of service and breaks technically every few weeks. The APIs are the legitimate, stable path, and they use the same engines and grounding. A proxy you can run every week beats a perfect measurement you can run never.
Can I track competitors?
Yes - that is why config.json
takes an array of brands. Add each competitor with their aliases and domains and the report shows all brands side by side. Watching a competitor's citation rate climb on your key prompts is usually the earliest signal you will get.
How is this different from AI crawler logs? Crawler logs (GPTBot, PerplexityBot etc. in your server logs) tell you your site is being read. This repo tells you whether your site is being surfaced in answers. Being read is necessary but not sufficient - the two measurements are complementary halves of the same funnel.
📺 YouTube walkthrough: Track How You Rank on ChatGPT & Claude - for free - how the tracker works, full setup with "Use this template", and real results (Prometora vs Sharetribe on AI search).
This repo is the "answer side" of AI visibility: are you surfaced when people ask? The other half is the "crawl side": are AI engines reading your pages at all?
Prometora - a marketplace builder with AI Visibility built in - tracks the crawl side server-side, per listing: which AI crawlers fetched which pages, trended over time. See
their AI crawler researchfor what that data looks like at scale. The crawl side is built into Prometora today; this repo is the answer side, self-hosted.
MIT - see LICENSE. Questions, ideas, corrections: rasmus@prometora.com