Your eval monitor fired on four days this week. At your sample size, that was the most likely count

A developer's analysis of LLM evaluation monitoring reveals that alert thresholds on eval scores are hypothesis tests whose false-alarm rates are often ignored. With 150 judge scores per hour and a 92% baseline, a monitor set to alert below 88% has a 53% chance of firing on any given day, making four alert days in a week the most likely outcome—pure noise, not regressions. The developer argues that window size, determined by traffic and sampling rate, is the critical parameter that tools like Langfuse expose but that users rarely tune, and that reducing sampling from 100% to 10% widens confidence intervals by a factor of about 3.2.

A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate drops below 88 percent. Their baseline is around 92. The monitor paged on four separate days last week. Four investigations, four shrugs, four "it recovered on its own." Nobody chose a false-alarm rate for that monitor. But it has one, and it is not small. With roughly 150 judge scores an hour, a fresh window each check, and a true pass rate of 0.92, the chance that at least one hourly check dips below 0.88 at some point in a day is 53 percent. That makes the expected number of alert days in a week 3.7, and the single most likely weekly count exactly four. Their monitor was not detecting regressions. It was sampling noise on a schedule, and the on-call rotation was the readout. An alert threshold on an eval score is a hypothesis test. Every tool that ships eval monitoring runs that test for you. What none of the configuration surfaces I read this month asks you for is the one number the test depends on: how many scores are in the window. The setup is ordinary binomial arithmetic, which is what makes it checkable. Each check looks at a fresh window of n judge verdicts. The true pass rate is p. The monitor fires when the observed rate falls below a threshold t. Then: python import math from scipy.stats import binom def alpha per check n, p0=0.92, t=0.88 : """P false alarm : windowed rate < t although nothing changed.""" k = math.ceil t n - 1 largest count that fires return binom.cdf k, n, p0 def p fire today n, checks=24, kw : a = alpha per check n, kw return 1 - 1 - a checks tumbling windows, independent checks def power per check n, p1=0.85, t=0.88 : k = math.ceil t n - 1 return binom.cdf k, n, p1 P one check catches a real drop Run that for a 0.88 threshold against a 0.92 baseline, across realistic window sizes, and add the probability that a single check catches a genuine regression to 0.85: | Scores per window | Fires per check on noise | Fires some time today | Expected alert days per week | Catches a real drop to 0.85, per check | |---|---|---|---|---| | 25 | 13.5% | 96.9% | 6.8 | 52.9% | | 50 | 10.2% | 92.4% | 6.5 | 63.9% | | 100 | 5.6% | 74.8% | 5.2 | 75.3% | | 150 | 3.1% | 53.3% | 3.7 | 81.9% | | 200 | 1.8% | 35.0% | 2.5 | 86.3% | | 400 | 0.2% | 4.9% | 0.3 | 94.9% | Read the 25-per-window row twice. That monitor pages nearly every day on pure noise, and a single check still misses a real seven-point regression about half the time. It manages to be trigger-happy and insensitive simultaneously, which is what happens when the threshold is chosen as a round number four points under baseline instead of as a quantile of anything. The window size is not an abstract parameter. It is your traffic times your sampling rate. Sample 10 percent of 500 traces an hour and n is 50. The sampling knob every tool gives you for cost control is also, silently, the monitor's sensitivity knob: cutting sampling from 100 percent to 10 percent widens the score's confidence interval by a factor of about 3.2, the square root of ten. None of this is exotic. It is the standard behaviour of binomial proportions, catalogued in detail in Brown, Cai and DasGupta's "Interval Estimation for a Binomial Proportion" Statistical Science, 2001 . So the question I brought to the current tooling was not "who has alerting." Most do. It was three narrower questions. This decides n, so it decides everything above. Langfuse runs judge evaluators at ingest time on observations that match your filters, with a sampling percentage you set; matching observations enter an evaluation queue and are scored asynchronously. Filters cover observation type, trace name, tags, user, session and metadata, so the scored population is well defined, and the docs are explicit that sampling exists to manage evaluation cost. LangSmith does the equivalent through automation rules: a filter selects runs, a sampling rate decides what fraction of the filtered runs the evaluator sees, and the scores attach back to the traces. Opik's online evaluation rules score live production traces with LLM-as-judge metrics and write results back as feedback scores on each trace. Future AGI's continuous eval tasks take the same two controls; a forward-only cursor means history is never backfilled. Phoenix evaluates production traces when you run evals over them by SDK or in the app; for continuously scheduled scoring of live traffic its own docs point you to Arize AX, the commercial platform. DeepEval sits at the same boundary from the other side: the framework is offline-first, and production scoring is the companion Confident AI platform's job. Langfuse has the most developed vocabulary here of the set I read. Monitors watch numeric, categorical or boolean scores; you set an operator, an alert threshold and an optional warning threshold over a lookback window, and route through Slack, webhooks or GitHub Actions. It is also the only config I found that makes you decide what an empty window means: treat missing data as zero, hold the previous severity, record NO DATA silently, or page after sustained NO DATA. How many monitors you can create depends on plan tier; the features themselves do not. LangSmith alerts on five metric types, run count, cost, errors, latency and feedback score, that last one being where online eval results live, and it routes to Slack, PagerDuty, Dynatrace or any webhook. Future AGI's monitors take a static or percentage-change threshold, or learn one from the historical mean, with separate warning and critical levels and a check frequency in minutes. Opik writes rule scores onto traces and tracks them on dashboards; its online-rules documentation describes score computation and display, and no notification configuration appears on that page. Phoenix routes threshold-based triggers on production traffic to Arize AX, per its own docs. I did not audit AX's alert configuration, nor Confident AI's, so the two commercial companions stay out of the comparison below. I read three alerting configuration surfaces end to end this month: Langfuse's monitor settings, LangSmith's alert settings and Future AGI's monitor model. In all three, the vocabulary is drawn from the same short list: a value, a direction, a window, sometimes a check frequency. None of the three has a field for the number of scores the window is expected to contain, and none derives the threshold from one. The window is specified in time, five minutes, an hour, a day, and how many scores fall inside it is whatever traffic and sampling happen to produce. Two of the threshold types deserve a specific caution. A percentage-change threshold compares two noisy window estimates, and the difference of two independent windowed rates carries roughly twice the variance of either one, so at small n it is noisier than the static threshold it replaces. And an auto or anomaly threshold that learns the historical mean solves a real problem, baseline drift, but the alert it fires is still a point-estimate comparison; learning where the baseline sits is not the same as knowing how far a healthy window wanders from it. To be precise about scope: this is what the configuration surfaces expose as of early August 2026, from each vendor's public docs and, where the code is public, its source tree. Any of them could ship a sample-size-aware policy tomorrow, and the gap is easy to work around today, which is the last section. You do not need any vendor to fix this. Invert the binomial: decide the false-alarm rate you can staff, then compute the threshold your window size implies. python from scipy.stats import binom def threshold from n n, alpha, p0=0.92 : """Largest rate threshold whose per-check false alarm stays <= alpha.""" k = int binom.ppf alpha, n, p0 if binom.cdf k, n, p0 alpha: k -= 1 return k / n | Scores per window | Page below 1% per check | Warn below 10% per check | |---|---|---| | 25 | 0.720 | 0.800 | | 50 | 0.800 | 0.840 | | 100 | 0.840 | 0.870 | | 150 | 0.860 | 0.887 | | 200 | 0.870 | 0.890 | | 400 | 0.885 | 0.900 | Two things fall out. First, at 25 scores a window, an honest 1-percent page threshold is 0.72, a full sixteen points below the 0.88 the team "felt" was right; the small-n rows are exactly where intuition overshoots most. Second, the pairing maps cleanly onto the warning-plus-critical structure that Langfuse and Future AGI both expose and that LangSmith approximates with two rules: put the warning at the 10 percent quantile and the page at the 1 percent quantile, and both levels inherit a false-alarm ceiling you actually chose. If your platform only takes time-based windows, fix the count instead: check every N scores rather than every hour. A count-based window makes n a constant, which makes the table above exact instead of approximate, and it stops quiet hours from paging you simply because n collapsed overnight. Is 0.92 a magic baseline? No. Every number above recomputes for your baseline and your window in the four lines of code shown; the shape of the conclusion, thresholds must move with n, survives any realistic parameter choice. My monitor uses a rolling window checked every five minutes. Does the daily 53 percent still apply? Not directly, and the direction matters: the five-minute checks include the hourly windows among them, so the daily probability is at least the 53 percent, not less. Plugging 288 checks into the independence formula overstates it badly, though; overlapping windows are correlated, the per-check false alarm is unchanged, and correlated checks cluster their false alarms into the same bad hour. Why not just require two consecutive breaches? That is a legitimate sample-size-aware policy, and on tumbling windows it squares the per-check false-alarm rate. You pay in detection delay, one extra check period, and in power against short-lived regressions. It is the cheapest fix on this page; the quantile threshold is the principled one. Auto thresholds that learn the baseline mean already read the score history. The same history contains everything needed to learn the window-to-window variance and set the alert line at a chosen quantile of it, which would make the monitor's false-alarm rate a configured property instead of an accident of traffic. None of the docs I read says whether any of the auto modes does this today. If someone from one of these teams can point me at a sample-size-aware or variance-aware alert policy in their product, mine is exactly the kind of dashboard it would quiet, and I will happily run the comparison again. Sources for the tool claims, all read in the first week of August 2026: Langfuse docs, "Monitors and Alerts" and the LLM-as-a-judge evaluator pages; LangSmith docs, "Online evaluations" and "Alerts"; Opik repository README and its online evaluation rules doc comet-ml/opik ; Phoenix repository docs, llm-evals page Arize-ai/phoenix ; DeepEval repository README confident-ai/deepeval ; Future AGI repository, tracer models eval task.py and monitor.py future-agi/future-agi . The binomial arithmetic is standard; see Brown, Cai and DasGupta, "Interval Estimation for a Binomial Proportion," Statistical Science 16 2 , 2001.