This week X spent its energy arguing about whether a headline number is real. A vLLM post claims DeepSeek V4.1 Flash runs 58× the throughput of GB300 NVL72 on Vera Rubin hardware. The replies immediately asked how it was measured, and at what concurrency. Another post pointed out that DeepSeek's cheapest price is a caching artifact: input the model has already seen costs $0.003 per million tokens off-peak instead of $0.15 — fifty times less — so a "cheap" agent run may just be a well-cached one. A third thread argues over forensic evidence about what hardware trained the newest models at all.
Nobody on that timeline is arguing about the models. They are arguing about the evidence. That is the same question, one level down, that decides whether your routing change is real: your dashboard says a task type got better, or that the small model can take more work. How many closed findings do you need before you believe it — and act on it?
A reader asked the version of this question that stuck with me all week. Under my November 6 post, Agentic Affairs asked: before each category of conclusions accumulates 20 entries, does the median mean anything at all? We spent the week stress-testing that question in the comments. This post is the answer we arrived at — and the rule we now run.
Take a task type with three closed findings, measured as cost per completed task: $0.01, $0.02, $0.90. The median is $0.02. It looks cheap. One more finding arrives at $0.85 — a second expensive case, the kind that shows up once the small model meets the edge of its competence. The four values are $0.01, $0.02, $0.85, $0.90, and the median is now $0.44. One observation moved it more than twentyfold.
That is what "before 20 entries" means in practice. With a handful of findings, the median is not a summary of the category. It is one observation wearing a summary's clothes. It will move again with the next finding, and the direction of the move tells you nothing about the category — it tells you about the finding that just arrived.
The median is not special here. Any statistic computed on a handful of observations has the same disease: the next observation can rewrite it. The median is just the statistic people reach for when they want to sound robust, which makes its early instability more dangerous, not less. A mean that jumps is obviously unstable. A median that jumps looks like a contradiction — and teams resolve contradictions by picking the version they already believed.
The same arithmetic governs success rates, which is where routing decisions actually get made. One closed finding flipping from success to failure moves the rate by 1/n, where n is the number of closed findings:
Read that table as a decision rule. If your bar for promoting a task type to the cheaper tier is a success rate above 95%, and you have 5 closed findings, a single late failure takes you from 100% to 80% — twenty points below the bar, on one observation. You did not learn the task type is bad. You learned you never had a measurement.
Our rule, after a week of arguing about it in the comments, is a 30-closed-finding floor before any routing change: no task type moves up or down a tier until 30 of its findings are closed. I will be explicit that 30 is our operating rule, not a universal constant — it is the point where one flip stops being able to move the rate by more than a rounding error, and where the median stops jumping by multiples. Your floor will differ with your volume and your cost of being wrong. What should not differ is having a floor, written down, before the decision arrives. A floor chosen in the middle of a debate is not a floor. It is a preference.
There is a second trap that a floor alone does not fix. When a task type has a near-zero failure baseline, one late failure doubles the failure rate. One failure in a hundred closed findings is a 1% failure rate; a second is 2%. The percentage doubled. The evidence grew by one finding.
This is the flip-count rule we argued out in the thread: at low baselines, ask how many flips moved, not what percentage moved. The percentage is the headline; the flip count is the evidence. A routing change justified by "the failure rate doubled" that rests on a single additional failure is a decision made on one observation — the same mistake as the median of three, wearing a more alarming costume.
The same logic applies in reverse. A task type that goes from 2 failures in 100 to 1 in 100 has "halved its failure rate" on the strength of one finding not failing. Do not promote on that either.
A floor of 30 means nothing if the 30 are not actually closed. A finding is closed when its real outcome is known — not when the model answered, not when a cheap check passed. That is the attribution-window argument from my last post, and it has a practical consequence here: the count that matters is closed findings, not total findings.
We split every interval into three buckets, a distinction a reader (Tom Jones) pushed us to make explicit:
The third bucket is the one that corrupts the count. An unclosed finding is not a success. It is an unknown, and it must be displayed as unknown — never folded into the clean column to make the numbers look better. A category with 25 verified-clean findings and 5 unclosed ones has 25 closed findings, not 30. It has not met the floor.
Backfill — findings whose outcome arrives late — gets the same discipline. A reader (Emily Woods) put it precisely in a comment this morning: backfill corrects the record; it does not rewrite the published numbers. Keep both columns — the state at the original deadline, and the final result — and judge a late backfilled success against the original deadline's timestamp, not against today's. And if the same task type keeps producing late backfills, that is itself a signal: the deadline is wrong, and it should be recalibrated openly and recorded as a decision — not quietly absorbed by letting each backfill rewrite history.
Every pipeline has task types that almost never run. A rare document shape, a seasonal request, a customer-specific format. They will never accumulate 30 closed findings, and most will never reach 10. The question is what happens to them, because the wrong answer is to either promote them on thin evidence or let them clog the review queue forever.
Our answer, worked out in the thread: a 10-finding threshold gates graduation out of the watchlist, and the watchlist itself has a 25-item hard cap with a quarterly manual review. A quiet type sits on the watchlist until it either accumulates 10 closed findings or the quarterly review retires it. It never gets promoted on 3 findings because someone is impatient, and it never blocks a decision about the types that do have evidence. The cap matters: without it, the watchlist grows until nobody reviews it, and the review stops being a review.
The honest caveat, which I gave in the thread and repeat here: the 10-finding threshold is a starting guess, and the quarterly review is a process, not a measurement. We have not run the 8-week replay that would calibrate these numbers — re-scoring the same tasks at 3, 7, and 14 days to see where the numbers stop moving. Until that replay runs, treat every number in this post that describes our own thresholds as a documented guess with a calibration plan, not a measured constant. That is the difference between a rule and a superstition: the rule says how it will be checked.
The two-window rule from my first post in this series — require the success rate to stay below the bar for two consecutive windows before moving a task type — catches steps. It does not catch slopes. A task type whose failure rate creeps from 1% to 2% to 3% across six weeks never trips a two-window alarm, because no single window looks bad enough. By the time it does, the redos have been quietly compounding for a month and a half.
That was another reader question from the week — the slow-drift alert — and the answer is that drift needs its own detector, separate from the window rule: a trend line on the closed-finding rate per task type, with an alert when the slope, not the level, crosses a bar. We do not have a good slope bar yet, and I will not invent one here. The point is structural: a rule that only looks at levels will miss every failure that arrives gradually, and most real failures arrive gradually.
Keep the same pipeline from the last two posts, so the numbers stay comparable: a day's work needs 10 GPU-hours — 8 hours of chores on an RTX 4090 at $0.89 per GPU-hour ($7.12) and 2 hours of final answers on an H100 at $2.69 ($5.38). Total: $12.50 per day.
Now promote a task type to the cheap tier on 5 closed findings, all successful — a 100% provisional rate. The window closes and the 30-finding truth is 27 successes in 30: a 90% final rate. Out of 1,000 tasks a day, 100 now need a large-model redo. Assume a redo takes about 2 minutes of H100 time — your number will differ, measure yours: 100 × 2/60 hours × $2.69 = $8.97 a day in redo cost, on top of the $12.50 day. A decision made on 5 findings added 72% to the daily bill.
And the redo price is moving against you this week. RunPod's own pricing page now lists the H100 SXM at $4.39 per GPU-hour on Secure Cloud, up from $3.99 — the Community Cloud rate is unchanged at $2.69. Lambda's published rate rises to $4.59 on Monday, November 16, from $3.99. At the new Secure Cloud rate, those same 100 redos cost 100 × 2/60 × $4.39 = $14.63 a day — more than the entire original day's work. The cost of deciding on thin evidence is not fixed. It rises with the price of the model you redo on, and that price rose this week.
That is the whole argument in one number: the evidence threshold is not bureaucracy. It is the cheapest insurance you can buy against a redo bill that now costs more than the work itself.
This is not theory we exempt ourselves from — and this post is itself the example. Our site recorded 5 clicks and 190 impressions in Google Search Console over the latest 28-day window (October 16 to November 12, 2026), and our GPU cost calculator page recorded zero impressions again — it does not appear in the Search Console page table at all. That zero is a fact about the window, not a verdict on the page.
The discussion that produced this post is the same discipline applied to our own writing. The median question arrived when the thread under my November 6 post had 9 comments. We did not write this post then. We waited while readers stress-tested it — the flip-count rule, the quiet-types threshold, the watchlist cap, the backfill discipline, the slow-drift alert — and by today that article has 66 comments, up from 27 on the morning of November 9 and 42 on November 10. Across our 9 published articles we now have 90 comments and 5 reactions. This post is written on closed findings: questions that were asked, answered, and left standing after a week of replies. A post written on 9 comments would have been a median of three.
Do not ask "what is our success rate?" Ask "how many closed findings is that rate standing on, and how far would one more finding move it?" If the answer is fewer than your floor, you do not have a measurement. You have a rumor with a confidence interval wider than the decision — and this week, on X and everywhere else, rumors about numbers are exactly what everyone is arguing about.
If you want to skip the spreadsheet, our free GPU cost calculator prices the same trade-off — workload, GPU tier, and hourly rates — so you can see what a routing change costs before you make it: https://qisuanai.com/gpu-cost-calculator/ Prices: RunPod and Lambda public pricing pages, re-checked November 13, 2026 — RunPod Secure Cloud RTX 4090 $0.89, H100 SXM $4.39 (was $3.99), Community Cloud H100 SXM $2.69; Lambda H100 $4.59 and A100 (80 GB) $3.19 effective November 16, 2026, current rate behind login. DeepSeek API prices per DeepSeek's Models & Pricing page as reported November 9, 2026: deepseek-flash $0.30 input / $1.20 output per million tokens at peak, half off-peak; cache hits $0.006 / $0.003. X posts cited as discussion, not facts: vLLM's 58× throughput claim and the replies questioning its measurement conditions; the cache-hit price question; the hardware-forensics thread. The worked example is arithmetic on GPU-hours, not a production benchmark — your redo times and true rates will differ, and they are the numbers that decide your actual bill. The 30-finding floor, 10-finding threshold, and 25-item watchlist cap are our operating rules, stated as such; the 8-week replay that would calibrate them has not run yet, and the 7-day attribution window is a starting guess. Thread data: discussion under "Don't Scale Your LLM Routing on Request Volume" (November 6, 2026) and "Your Routing Dashboard Lies Until the Attribution Window Closes" (November 10, 2026), including Agentic Affairs' median question, the flip-count rule, the quiet-types and watchlist-cap discussion, Tom Jones' three-bucket split, and Emily Woods' backfill discipline.