I kept running into the same thing while researching AI visibility tools. Agencies have basically no idea whether ChatGPT even knows they exist, let alone whether it would recommend them over a competitor. So I built a small tool that asks ChatGPT the way a real prospect actually would, shows you the raw answer, and tells you who got mentioned ahead of you.
Stack is Next.js with the App Router, Supabase for storage, the OpenAI API for the actual audit calls, Tailwind for styling, deployed on Vercel.
While testing it I ran it against Mother, a real, well known independent ad agency founded in 1996. I asked two versions of basically the same question.
Direct: "what do you know about Mother?"
ChatGPT nailed it. Correct founding year, correct offices, correct reputation for their work.
Open ended, the way an actual client would ask it: "who's a good advertising agency?"
Mother didn't show up anywhere. Got Ogilvy, BBDO, DDB, and Leo Burnett instead, all bigger network agencies.
So the model clearly knows exactly who Mother is. It just doesn't reach for them when giving an open recommendation. My guess is it comes down to frequency in the training data rather than accuracy. Bigger, older agencies just have a lot more written about them across the web, and that seems to matter more than whether the answer is actually the best fit.
An early version of my scoring logic had the ranking direction backwards. If a competitor's name showed up after the brand in ChatGPT's answer, I was marking it as "ahead of you," when really it should have been the opposite. Appearing earlier in a list means ranked higher, so that should count as ahead, not the other way around.
I caught it by testing on Pentagram, a well known design studio, and getting a result that said "you're ranked number one" and "this competitor appeared ahead of you" in the same sentence, which obviously can't both be true. Good reminder to run a test on something you're confident should rank well before trusting your own scoring logic.
It's free, and you don't need to sign up to see your first result. algorithmofai.app
Curious if anyone else has run into something similar testing LLMs for anything close to search or recommendations.