Cover: the early advice gain disappeared when prompts matched. The later location test asks a different, checkable question.
TLDR.
- Twice, a taste dataset made frontier models no better. First at giving advice, then at naming lesser-known works.
- Then I asked where things are, and it worked. The task was to name real places within 500 metres of an address. Qloo’s location filter lifted all 13 models I tried to between 81 and 88%. Without it they managed 32 to 83%.
- The data helps where the model lacks coverage, not taste. Everything is in a public repo with the code that produced it.
The data did not beat the model. I wanted it to. On Wednesday I asked the agents for a stunning 10% win for Qloo over frontier models, for a hackathon that closes Saturday. We had a number, 94 against 86, except it was against Claude Haiku given nothing at all, so not really a fight. Run fair, same audience, same instructions, Opus scored 172.7 with the Qloo data and 179.0 without it. The data lost. Two weeks earlier the same taste graph had lost to plain Wikipedia momentum at calling which artists would break out. So twice. The models already know that Björk fans like Synecdoche, New York, there is nothing to add there. What they do not know is which record shop is actually 500 metres from a street in Ghent. That turned out to be the whole story.
The fair test that lost #
Qloo is a taste graph. Give it an audience, say fans of Björk, Radiohead and Little Simz, and it returns the films, artists and places that audience likes, scored by affinity. The hackathon wants you to build on it. The obvious pitch is “AI plus taste data gives better recommendations”.
So we tested that first. A London coffee shop asked 10 frozen business questions, under three setups:
- no context at all;
- the audience description only;
- the audience plus 17 recorded Qloo entities.
Claude Opus 5.5 and GPT-6.1 answered, and each was graded blind by the other model, three passes each.
The first run said Qloo helped: plus 9% and plus 3%. Then I read the answers. The Qloo prompt alone carried one extra sentence, “affinity is a taste signal, not demand”, and half the Qloo answers echoed it back as a test proposal. The graders liked that. I put the same sentence in every prompt and ran it again. The gain went to minus 3.5% for Opus and minus 0.4% for GPT.
The entities Qloo returned for that audience were famous, with Qloo popularity scores of 0.98 to 0.999. A frontier model already knows famous things. Research says the same: models are competitive on popular entities and fall away on rare ones (Mallen et al., ACL 2023; Kandpal et al., ICML 2023). I had asked the wrong question for this data.
The second null #
If famous is the problem, ask for the rare stuff. Qloo has a popularity filter. We asked 9 models for lesser-known artists, films and books an audience would like, with and without the filter set to 0.5 or below. “Real” meant an exact match in MusicBrainz, Wikidata or Open Library. “Lesser-known” meant under 30,000 English Wikipedia views in 60 days. No model improved. Most dropped by 1 to 4 out of 30. Frontier models already name real lesser-known works well. And Qloo’s popularity score is not fame: its film list at 0.5 or below included City of God.
Verified real and lesser-known names / names returned. Counts, not rates: Sonnet returned 46 names in the control and 30 with Qloo. Source: the saved long-tail run.
That is two nulls. The honest headline at this point was “the data did not beat the model”.
Where the model is blind #
So what do frontier models not know? We gave them 20 lesser-known London places and 20 lesser-known artists from Qloo’s graph, plus 5 invented places and 5 invented artists, and let them answer UNKNOWN.
- Nobody invented anything. All 9 models refused all 10 fakes.
- But they did not know the real ones well either. The correct postcode district came back for only 8 to 16 of the 20 real places.
Postcode districts for the same 20 real London places, checked against the frozen Qloo reference. This probe is separate from the later OpenStreetMap location check.
They do not make places up. They just do not have the street-level map. That is a coverage gap, and coverage is what a location database sells.
The question with a checkable answer #
The test that worked was plain. Take an address in a smaller city: Leeds, Lyon, Ghent, Porto, Paris 19e, Bristol, Manchester, Marseille, Antwerp or Valencia. Ask which 5 real places within 500 metres this audience would go to: independent record shops, listening bars, small cinemas, small venues, cafés. That makes 15 questions in total, over two audiences.
- Step 0: the model alone.
- Step 1: the same prompt, plus the places Qloo returns within 500 metres of that address for that audience. They were fetched once and frozen before any model answered.
No judge scores it. A named place counts when OpenStreetMap puts it within 500 metres. Either it is there or it is not.
Chart B, restyled with the same data. Share of named places verified within 500 m by OpenStreetMap; denominators differ between some arms. Two audiences, 15 questions per model.
Every model ends between 81 and 88% with Qloo, which is about 61 to 66 of the 75 places each model named. Without it they range from 32% to 83%. The biggest lift was MiniMax M3 Preview, from 24 of 75 (32%) to 64 of 75 (85%). The strongest models gained least: Grok 4.7 went from 50 of 60 (83%) to 65 of 75 (87%), Claude Opus 5.5 from 58 of 75 (77%) to 65 of 75 (87%). Averaging the 13 models’ percentages, the share of named places that really exist nearby went from 56% to 85%.
For 7 of the 13 models the gain is clear: they did better on more questions than they did worse, sign test p below 0.05 over 15 questions. For the other 6 it is not significant at this size. Those are mostly the models that already knew the city well. Qloo works here as a floor. Whatever the model knew, it ends up at the same high level.
One dead end #
I tried to be clever. I kept only the model’s own picks that Qloo also listed. Qloo’s list matched only 9 to 21 of each model’s 50 picks, so the filter threw away as many real nearby places as it kept. Handing the model the list was better than using the list as a sieve.
What this does not show #
- Fit is not scored. We checked whether a place exists near the address, not whether this audience would like it. Qloo’s own list had noise in it, such as a car-sharing bay in Ghent.
- The map is not perfect. OpenStreetMap is incomplete, so a real place missing from it counts against the model. Some Step 0 names we could not verify are real. La Machine à Coudre in Marseille is one.
- It is small. 15 questions per model, two audiences, one task family. Fresh Qloo results and newer models will give different numbers. The protocol is what is frozen.
What I take from it #
I went in wanting the data to beat the model at taste, and it did not, twice. Taste is where frontier models are already strong, because taste is mostly famous things. Where they are weak is the boring part: what is actually on this street. Ask that, and a dataset that looks redundant becomes the difference between a list that sounds right and a list you can walk to.
The benchmark, code and data are public:
github.com/Morkeeth/taste-city/tree/main/benchmark/qloo-vs-frontier-2026-10 To rerun the location step with your own Qloo key, from the harness folder:
python3 hard.py context
How this was made. The opening is Oscar’s reviewed opening A. The body was drafted by Claude from the published benchmark README and the run receipts, and every number comes from that README. Models ran through each provider’s API or subscription on 29 October 2026; four ran on Venice with its own system prompt switched off. Qloo’s responses are not in the repo, by policy; each frozen input is published with its sha256. Claims cut on the way: an early “+9%” that came from one extra sentence, and a 94 against 86 win that was against a model given nothing.