Repeating a YouTube search query three times and keeping only results that appear in two or more of those fetches is more reliable than treating any single fetch as ground truth. The Jaccard similarity between two fetches of the same query — within a single minute — sits around 0.43–0.88. Today's ru
Repeating a YouTube search query three times and keeping only results that appear in two or more of those fetches is more reliable than treating any single fetch as ground truth. The Jaccard similarity between two fetches of the same query — within a single minute — sits around 0.43–0.88. Today's run measured 0.447 for the "more steam reviews than" query. That means roughly half the video IDs flipped between fetches. A single fetch isn't a signal; it's a snapshot of one shuffled result. I built this pattern into a daily market-signal collector for three directory sites (AI tools, indie games, open-source alternatives). The collection layer fetches five YouTube queries every morning, and the full implementation is in scripts/market-listening/collect.mjs. This article is about how I made that collection reliable despite YouTube search being inherently non-deterministic. YouTube's search ranking is legitimately non-deterministic. Freshness boosts, A/B test bucketing, CDN-level caching, and real-time personalization all interact to produce different rankings on the same query across consecutive requests. This isn't an API bug or detection avoidance — it's how the system works. I confirmed this by measuring. The script stores the mean pairwise Jaccard across all rep-pairs per query in the output file. Today's run: Query Mean pairwise Jaccard Items kept Items dropped as noise "more steam reviews than" 0.447 measured across 3 reps 28 The code stores the mean only — not individual pairwise values — so 0.447 is what I can actually cite. At that value, under half the video IDs appear in both halves of any given fetch-pair. Treating one fetch as "the ranking" means you're looking at roughly a coin flip on which videos actually show up. You can't tell signal from noise from a single observation. The problem is worse for market-signal use cases, where you're not trying to understand any one video's rank — you're trying to identify which videos appear consistently over time. A video that appears in one fetch but not the next isn't evidence of anything. Consistent appearance across independent fetches is. The fix is three fetches per query, with a jitter delay (5–15 seconds, randomized) between each. A video that appears in fewer than two of the three fetches is filtered out and counted in dropped_noise. Only items that appear in at least two of three fetches are considered signal. Today's run dropped 28 items as noise for the first query. Those videos were surfaced by YouTube's shuffler at least once, but didn't clear the consistency threshold. Some of them might be genuinely relevant; most are one-time shuffler artifacts. I'm trading recall for precision: I'd rather miss a few real signals than include noise that drives incorrect market conclusions. A few implementation notes that aren't obvious: The MIN_APPEARANCES=2 threshold is a parameter. At REPS=3, requiring 2/3 appearances means a video must appear consistently across the majority of fetches. Raising it to 3/3 would be stricter — better for high-confidence signals, worse for coverage. I've run with 2/3 since the collector was built and haven't found a reason to change it. The jitter delay is calibrated for request politeness, not cache-busting. The sleep intervals I use across different APIs are different for different services — YouTube gets 5–15s between requests. This introduces some time variance between reps, which helps with cache freshness but isn't guaranteed to force a cache miss. I don't use the official YouTube Data API for this. The search HTML surface is sufficient for market-listening purposes, and it avoids a key rotation dependency. I prefer polling public endpoints without API key registration where the data surface is adequate. Once I determine which videos cleared the consistency threshold, I face a second problem: what rank to assign them. A video might rank 2nd in one fetch, 3rd in another, and 15th in a third. None of those individual positions is trustworthy. The solution is median rank. For each kept video, I collect its rank from every rep where it appeared, sort those ranks, and take the median: // voteReps() in scripts/market-listening/collect.mjs const sorted = [...v.ranks].sort((a, b) => a - b); kept.push({ ...v, median_rank: sorted[Math.floor(sorted.length / 2)], }); A video ranked 2, 3, 40 across three reps gets median rank 3. The outlier position (40th) doesn't drag it down. A video ranked 1, 2, 1 gets median 1. For a two-element list — a video that only appeared in 2/3 reps — median is the second element: ranks [2, 8] returns 8, since floor(2/2) = 1. I use median specifically because search rank distributions have long tails. A video can appear in position 30+ on a bad shuffle while legitimately being a top-10 result. Mean would be dragged by those outlier positions; min would reward lucky placements. Median is stable against both. I've applied the same outlier-resistance property when using Jaccard for duplicate detection — the logic generalizes across signal-extraction problems. After computing the vote for each query, the script computes the mean Jaccard similarity across all pairs of reps for that query and stores it in the output: const pairs = []; for (let i = 0; i < reps.length; i++) { for (let j = i + 1; j < reps.length; j++) { pairs.push(jaccard( reps[i].map((v) => v.videoId), reps[j].map((v) => v.videoId) )); } } queries[query].jaccard_mean = pairs.length ? Number((pairs.reduce((a, b) => a + b, 0) / pairs.length).toFixed(3)) : null; This means every daily file in data/market-listening/ carries a per-query Jaccard mean. It's not just an implementation detail — it's the primary diagnostic for whether the repeat-and-vote pattern is still justified. If jaccard_mean climbs toward 1.0 for a query, the shuffler has become deterministic for that query. Either caching locked in, or the query narrowed to so few results there's no variation left. That's the cue to lower REPS (three fetches of the same cached result isn't three independent samples) or rethink the query entirely. The pipeline health monitor reads these files daily. I could add Jaccard-based alerts — flag if any query exceeds 0.9 (suspiciously deterministic) or drops below 0.25 (unusually chaotic). I haven't added those yet because 0.447 has been stable enough that I don't have a calibrated threshold. This is an honest gap in the current implementation. The Jaccard function itself: export function jaccard(a, b) { const A = new Set(a); const B = new Set(b); if (A.size === 0 && B.size === 0) return 1; let inter = 0; for (const x of A) if (B.has(x)) inter++; return inter / (A.size + B.size - inter); } Two empty sets return 1 (vacuously equal). That edge case matters: if two reps both returned empty results, this would report perfect similarity — which is wrong from a health standpoint. I handle this upstream via the sources_ok logic, which fails a source if any query produced zero accepted results. But the Jaccard function itself doesn't surface that failure; it needs external context. The repeat-and-vote pattern assumes independent fetches. Several failure modes break that assumption. CDN-level caching. If all three requests hit the same YouTube edge cache, they receive the same response — effectively one sample, not three. The jitter delay and randomized inter-rep timing help, but can't guarantee cache-misses. Watching jaccard_mean over time is the only way to catch this from the outside; a sustained climb toward 1.0 across all queries simultaneously is the clearest signal. Egress IP stability. GitHub Actions runners can change egress IPs between runs but typically don't change between steps within a single run. If all three reps within a run use the same egress IP, they might receive the same regional ranking. This is less likely than CDN determinism, but it's a real risk for queries with strong geo-sensitivity. Temporal autocorrelation. The 5–15s jitter between reps is short enough that trending content could genuinely rank differently between rep 1 and rep 3. For market-signal purposes, where I want stable-over-time signals, this is actually fine — a video that's trending at second-level resolution isn't what I'm tracking. But if you were trying to detect short-lived viral content, the jitter window would need to collapse. What I'd do differently now: log per-query Jaccard variance alongside the mean. The mean tells me how similar the fetches were on average, but variance would tell me whether they were clustered (three fetches all near 0.45) or spread out (one at 0.2, one at 0.8). That's diagnostic information I don't currently have. I'd also add reps_attempted vs. reps_succeeded at the per-rep level, so I can identify which specific fetch failed without reading runner logs. The broader pattern — collect, vote, report the vote metadata — applies to any non-deterministic API. The implementation details change (different HTML parsing, different jitter), but the logic is the same. And having Jaccard built into the output gives you a continuous health signal rather than having to re-instrument later when something starts behaving differently. This is related to what I cover in catching silent ETL failures and applies equally to survivorship-bias concerns in any analytics pipeline — the data you collect determines what you can conclude, and knowing its collection conditions is part of using it responsibly. Why not use the YouTube Data API's search endpoint instead? The search.list endpoint has its own caching and returns different results than the search HTML surface in ways that are harder to inspect. More importantly, I prefer polling public endpoints without registering for a key where sufficient — every API key is a credential to rotate and a quota to manage. For market-listening at this scale, the HTML surface is adequate. What happens if one of the three reps fails to return any results? A failed rep reduces the pool for voting. If only one rep succeeds, nothing can reach the MIN_APPEARANCES=2 threshold and the query produces zero accepted items. This flips sources_ok.youtube to false in the output artifact, and the daily health check surfaces it. It's not silent. Would REPS=5 produce better results? Probably yes, at linear cost. Each additional rep adds several minutes to the run (5–15s jitter per request × 5 queries). For a daily market-signal that only needs directional accuracy, REPS=3 filters most single-fetch noise without being prohibitively slow. I'd increase it if I needed sub-day granularity or was tracking a more volatile query set. Does this approach work for other search APIs? Yes. The repeat-and-vote pattern and the Jaccard diagnostic are search-API-agnostic. You'd change the HTML extraction and the appropriate jitter timing, but the voting logic and the sources_ok/Jaccard output structure are reusable. The only input the vote function needs is a list of per-rep result sets where each item has a stable ID and a rank. Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Key Takeaways #
- •Repeating a YouTube search query three times and keeping only results that appear in two or more of those fetches is more reliable than treating any single fetch as ground truth
- •This story was reported by Dev.to, covering developments in the** dev**space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article: