My AI visibility score was 29.2. Here is what the one number was hiding

A developer built an AI citation tracker that scores how well web-grounded models from Gemini and OpenAI identify and describe a tracked entity, and found that a blended visibility score of 29.2 on 21 October concealed wide variation across question types. Direct name-based prompts scored 88.9% while genre recommendation, comparison and research buckets scored between 0% and 16.7%, meaning the headline number mostly reflected the question mix rather than model behavior. The developer released the tracker's code under MIT and an open dataset on Hugging Face, and cautions that with only 16 prompts per model, differences of one or two answers fall inside the noise.

The daily run of my AI citation tracker finished on 21 October with a blended score of 29.2. On its own that reads like a mediocre week. Split by question type it is something else entirely, and the split is the only part of it I would trust. Every prompt goes to two web-grounded models, Gemini and OpenAI's search model. Each answer is scored on a 0 to 3 scale for how well it identifies and describes the entity I track. The blended score is points earned over points possible. On 21 October that was 28 out of 96: 16 prompts, two models, 32 answers, three points each. Same run, broken out by question bucket. n is the number of answers in the bucket. | Bucket | Points | Possible | Percent | n | |---|---|---|---|---| | Direct asked about the author by name | 16 | 18 | 88.9 | 6 | | Long tail | 6 | 12 | 50.0 | 4 | | Genre recommendation | 6 | 36 | 16.7 | 12 | | Genre | 0 | 12 | 0 | 4 | | Comparison cluster | 0 | 12 | 0 | 4 | | Research | 0 | 6 | 0 | 2 | So the 29.2 is mostly a statement about the mix. The bucket where the entity does well is 6 of 32 answers. The genre recommendation bucket alone is 12 of 32, more than a third of the denominator. Bucket weights decide the headline. If I added six more direct questions tomorrow and they scored like today's, the blended score would go from 28/96 to 44/114, which is 38.6. Nothing about how the models treat the entity would have changed. Only my question list would have. Small buckets are coarse. With n=4 and three points per answer, a single answer is worth 25 percentage points of its bucket. A bucket that goes from 0 to 25 has moved by one answer. With n=2 one answer is 50 points. Any chart that shows these as smooth lines is drawing noise. The model gap is smaller than it looks. Gemini scored 35.4 17 of 48 , OpenAI search 22.9 11 of 48 . Twelve and a half points sounds like a ranking. In absolute terms it is six points, two fully correct answers, on one day. I would not publish a model comparison on that. Points, possible points and n for every bucket, and never a percentage without the n next to it. I also keep two questions apart that the blended score mixes together. The direct bucket answers "does the model know who this is when asked". The genre buckets answer "does the model think of him without being asked". Those are different problems with different fixes, and an average of the two answers neither. The honest reading of 21 October is short: asked by name, both models mostly get it right. Asked an open genre question, they almost never bring him up. That second line is the one worth tracking week to week, and it needs more prompts per bucket before a change in it means anything. One day, 16 prompts per model, two models. Anything that differs by one or two answers is inside the noise. The tracker runs daily, so the next step is pooling a week per bucket before comparing anything. Code MIT : https://github.com/marintkael/marin-research-tools?utm source=devto https://github.com/marintkael/marin-research-tools?utm source=devto Open dataset: https://huggingface.co/datasets/marintkael/ai-citation-fidelity?utm source=devto https://huggingface.co/datasets/marintkael/ai-citation-fidelity?utm source=devto Research overview: https://marin-t-kael.de/en/research?utm source=devto https://marin-t-kael.de/en/research?utm source=devto