Running LLM jobs over hundreds of items, the obvious shortcut is to collapse execution and verification into one model pass: one call, one output, ship it. It fails at scale, and a scoring system for 146 countries across 11 categories shows exactly why.
Each score runs 0 to 100 on a single canonical dataset, the overall rating is the arithmetic mean of those 11, and there are no per-country exceptions. One yardstick, applied identically everywhere.
Ask a cheap model to generate all 146 in one pass and you get speed with a hidden cost: drift. One country's "friendliness" score reads high because the model read it as social warmth rather than visa bureaucracy. Another's culture score inflates after the prompt happened to emphasize food over history. None of these are bugs, they're quiet inconsistencies, and at 146 items a 5% drift rate means seven countries silently failing the canonicity requirement while every individual score still looks reasonable.
Step one: a cheap executor runs the mechanical pass. Fixed ruleset, all 146 countries in parallel batches, structured JSON out. No judgment calls, just apply rule X to field Y.
Step two: a stronger gate verifies before anything ships. Same scale everywhere? Any statistical outlier? Did a category get reweighted mid-run? This is judgment work, holding many items in view at once, and it's what a single combined pass can't do reliably.
A model doing both jobs at once optimizes for the wrong thing: it second-guesses the ruleset mid-run, adds nuance where the spec demanded consistency, and marks cases "exceptional" that shouldn't be. Splitting the two roles is faster and cheaper than one model trying to hold both contexts simultaneously.
The Country Comparison Tool's best-travel-months field works the same way: a month qualifies if it scores 70 or higher on a fixed weather index built from Open-Meteo data, no editorial override, no "tourists usually go in December anyway." One yardstick, all 146 countries, computed once and refreshed when the climate dataset updates. Add "let the model decide if a category doesn't apply here" and you've introduced exceptions. Add "one call to save tokens" and the 10th and 100th items get scored by two different reasoning paths without anyone noticing.
At five items you can eyeball drift and call it done. At 146 it's a data integrity problem, at 1,000 a regression nobody catches until a user does. The same split applies to any bulk LLM job against a fixed dataset: translation against a glossary, classification against a taxonomy. Collapse the layers and you get quiet inconsistency. Split them and drift gets caught before it ships.
Check out the Country Comparison Tool to see one canonical dataset and one scoring model produce consistent results across 146 countries.