Ei-Core just said no to a retention estimate — and that refusal

The model didn't hallucinate. Didn't hedge. Didn't give a confidence interval. It flat-out refused:

I cannot provide a retention estimate for this cohort. 
Insufficient behavioral signal density (n=23 events) 
falls below the minimum threshold (n=150) for 
reliable projection. Recommend extending observation 
window or enriching with external firmographic data.

First reaction: panic. Client-facing tool just... declined. That's not in the spec.

Then I checked the logs. The refusal trace showed the uncertainty quantifier firing at step 3 of the reasoning chain — before any generation attempt. The model had internally computed the effective sample size, compared it against the calibrated threshold we set during safety tuning, and triggered the abstention gate.

if effective_n < MIN_SAMPLE_THRESHOLD:
    raise AbstentionError(
        reason="insufficient_signal",
        metadata={"effective_n": effective_n, "threshold": MIN_SAMPLE_THRESHOLD}
    )

We'd built this gate six months ago during red-teaming. Forgot it existed until it saved us.

The cohort genuinely had sparse data — only 23 meaningful engagement events across 47 accounts. Any number the model would've generated would've been fiction dressed as insight. The refusal was the correct answer.

Changed the demo narrative on the spot. Showed the VP the refusal, explained the threshold, walked through what data would unlock a real estimate. She thanked us for not inventing a number.

Two things I'm still chewing on:

1. The threshold (150) came from a synthetic benchmark, not production drift. Need to validate it against actual forecast error curves.

2. The error message leaks internal config (MIN_SAMPLE_THRESHOLD

). Should wrap that in a user-facing code like ERR_INSUFFICIENT_SIGNAL

and keep the numeric detail in structured logs only.

Also wondering if we should surface a "what would it take to answer" suggestion engine — the trace already knows which features are missing. Could auto-generate a data-collection checklist for the client.

Anyone else hit a moment where a refusal felt more like a feature than a bug? Curious how you're handling abstention UX in production.

Next That GPT-4o hallucination that invented a whole asyncpg API →

these AI tool field notes, with plenty of directly applicable cases.