Every LLM answer is a snapshot of the model's training data. Ask for a price, a current model name, a CEO, or a fine cap and you get something that was true β stated with exactly the same confidence as something that still is. Retrieval doesn't fix this on its own: RAG grounds an answer in documents, but nobody checks whether the document the answer leans on is two years old.
FreshGauge is a Bifrost gateway plugin that does that check on the way out. Every completion that passes through the gateway gets its factual claims extracted, verified against live web evidence, scored by authority-weighted recency, and then acted on β logged, corrected in flight, or pre-empted with a dated brief before the model is even called.
This is a write-up of how it's built, the decisions that mattered, and the traps that cost real debugging time.
You could build this as a wrapper around one SDK. It breaks the moment a team uses three providers, streams tokens, or fails over between models. A gateway sees all of that by construction: one process, one config, every request.
Bifrost's plugin ABI gives exactly the hooks needed:
PostLLMHook β the response, after the provider call. Where the audit is born.HTTPTransportPreHook β every HTTP request to the gateway. Where the audit API lives.
Because HTTPTransportPreHook runs after Bifrost's auth, the plugin's admin routes inherit virtual-key auth for free. There is no second auth system to get wrong.
ββββββββββββββββββββββββββββββββββββββββββββββββ
client ββPOSTββββββββββΆβ Bifrost gateway (:8080) β
/v1/chat/completions β β
β HTTPTransportPreHook βββΆ /audit, /stats, β¦ β
β β (auth'd by VK) β
β βΌ β
β provider call (OpenAI / Anthropic / X / β¦) β
β β β
β PostLLMHook β
ββββββββββ¬ββββββββββββββββββββββββββββββββββββββ
β 1. extract claims (β€8)
β 2. correct in flight? βββ ledger read (0 credits)
β 3. dedupe by request/content
βΌ
ββββββββββββββββββββββββ
β auditor (2 workers) β
β per claim: β
β Tavily search βββββΌβββΆ api.tavily.com (1β2 credits)
β fetch + quote β
β stance: +1 / β1 / 0β
β vote = authΒ·e^(βage/Ο)
ββββββββββββ¬ββββββββββββ
βΌ
ββββββββββββββββββββββββ
β SQLite ledger β
β audits Β· claims Β· β
β evidence Β· metrics β
ββββββββββββ¬ββββββββββββ
βΌ
GET /audit/{id}.html (flat paper report)
GET /stats (corrections, grounded)
Three modes sit on one engine:
| Mode | When | Cost | What the user sees |
|---|---|---|---|
shadow |
after the answer | Tavily credits | nothing; the ledger only |
correct |
inside PostLLMHook |
0 credits (ledger read) | dated correction block appended |
brief |
in HTTPTransportPreHook |
1 Tavily search | answer born current |
The cost asymmetry is the point. Detection needs the web. Enforcement does not β it needs the memory of what was already detected.
The extractor pulls sentences that make checkable assertions about the world and classifies them:
const (
Pricing Category = "pricing"
APIVersion Category = "api_version"
Leadership Category = "leadership"
Regulation Category = "regulation"
Limits Category = "limits"
)
Categories are not cosmetic. Each one carries a confidence half-life, Ο, that decides how fast evidence decays:
func Tau(c extract.Category) float64 {
switch c {
case extract.Pricing:
// A five-week-old vendor price page is still current. At tau=30 a 39-day-old
// page scores 0.27 and fails MinVote, which reports a plainly stale price as
// UNVERIFIABLE β abstention where the evidence is good enough to decide is a
// worse error than a slightly generous half-life.
return 120
case extract.Limits:
return 180
case extract.APIVersion:
return 90
case extract.Leadership:
return 365
case extract.Regulation:
return 730
}
return 180
}
A statute moving on a decade timescale and a price sheet moving on a quarterly one cannot share a decay constant. That table is the whole domain model, and it is the difference between a detector that argues and one that abstains.
Most "is this stale?" implementations count sources. Counting is wrong: ten 2023 blog posts should not outvote one vendor pricing page published last month.
So each piece of evidence casts a weighted vote:
// Recency decays from 1.0 at the publish/fetch date to ~0.37 one tau later.
func Recency(e Evidence, cat extract.Category, now time.Time) float64 {
date := e.Published
if date.IsZero() {
date = e.Fetched
}
if date.IsZero() || date.After(now) {
return 1
}
ageDays := now.Sub(date).Hours() / 24
return math.Exp(-ageDays / Tau(cat))
}
// Vote is the authority-weighted recency used for every decision.
func Vote(e Evidence, cat extract.Category, now time.Time) float64 {
if e.Stance == 0 {
return 0
}
return clamp01(e.Authority) * Recency(e, cat, now)
}
Decisions follow from that single scale:
bestCon >= MinVote (0.35) bestCon >= bestSup β CONTRADICTEDbestSup >= MinVote β SUPPORTED
And the freshness score is not a vibe β it's recomputable by hand from the ledger:
f := 100 * (1 - math.Min(1, bestCon))
For the recorded demo: the vendor page (acme.com/pricing, authority 1.0, published 2026-10-02) says $35 and contradicts; a 2024 Medium post (authority 0.4) supports the old $20. The contradiction's vote wins by an order of magnitude, and the score is 28/100 β recent enough to be decisive, old enough that the claim it contradicts is plainly stale.
The first live run against anthropic.com flagged $2 / million input tokens as CONTRADICTED. The page did say $15 / million input tokens β for a different model, in a multi-model table.
Naive comparison of "a number appeared near the claim" turns every pricing table into a contradiction factory. The fix gates comparison on three cheap invariants before any number is compared:
subjectOverlap, sameSubject). Claude Sonnet 5 β Claude Sonnet 4. per million input tokens β per million output tokens β per month (priceKind, taggedPrices).
And the scan is line-first: candidate evidence is the single line containing the number, not the whole page. After the fix, the same page and the same single Tavily search produce $15/M input β CONTRADICTED and $2/M input β SUPPORTED. There's a regression test for exactly that pair.
response_mode: "correct" rewrites the answer before the caller sees it. The correction is drawn entirely from the ledger β the plugin's own memory of contradictions it has already established β so it costs zero Tavily credits.
claims := extract.Claims(text, p.cfg.MaxClaims)
// In-flight correction runs before the audit is enqueued, and only ever reads the local
// ledger. Order matters: the audit below must never see its own correction block as a
// new claim, or every corrected answer would spawn fresh work for text the plugin wrote
// itself.
if !streaming && p.cfg.ResponseMode == "correct" {
p.correctInFlight(resp.ChatResponse, claims)
}
Two details earn their comments:
stripCorrection). Otherwise the block the plugin appended becomes a claim on the next audit and the plugin audits itself forever.normalizeClaim collapses punctuation and casing so the same fact expressed twice hits the same row; priors are capped by correction_max_age_hours (24h default) so a correction never cites evidence the operator considers expired.
The output is appended under a fixed marker, three lines max by default:
---
**Freshness check (FreshGauge).**
- Correction: "Acme Pro costs $20 per month" is stale. https://acme.com/pricing says
"Pro β $35 per user / month" (authority 1.00, published 2026-10-02).
/stats exposes corrections and grounded_requests next to audit totals, so "the gateway changed answers" is a measurable event rather than a claim in a README.
grounding_mode: "brief" runs in HTTPTransportPreHook, only for freshness-sensitive questions ( looksFreshnessSensitive), inside a time box ( grounding_timeout_ms), and injects a short block of dated evidence into the request:
Current evidence (verified 2026-11-21):
- acme.com/pricing (2026-10-02): "Pro β $35 per user / month"
The model answers from that instead of from its weights. One search, opt-in, and it shows up on /stats as grounded_requests.
Stream chunks carry a Delta, not a Message. Reaching for Choices[0].Message dereferences a nil embedded pointer β inside a hook, that surfaces as a recovered panic in the host log and a silently missing audit, which is the worst failure mode: it looks like it works.
The hook detects the stream case and assembles deltas per request:
switch {
case c0.ChatNonStreamResponseChoice != nil:
text = stripCorrection(messageText(resp.ChatResponse))
case c0.ChatStreamResponseChoice != nil:
streaming = true
text = p.streamText(ctx, c0) // buffers deltas until finish_reason
}
Streams are audited once, at finish, and never mutated β there is no message object to amend, so correct is non-streaming by design. Buffers are reaped on completion, and there's a test that feeds a synthetic SSE sequence through the hook to keep it that way.
Deduplication also needs care: the host does not always put a request ID on the context, so the fallback is content-<hash>-<minute>. Two failover attempts producing identical prose collapse into one audit; a genuine repeat a day later gets a fresh one with fresh evidence dates.
Bifrost plugins are Go .so files loaded into the host process. That imposes an exact-match ABI: same Go toolchain and an identical module graph, replace directives included. A one-character drift in a transitive dependency and the host refuses to load the plugin β with an error that does not name the offending package.
What worked:
go1.27.3) and Bifrost ( core v1.15.2, transports v2.5.1). replace for mcp-go exactly:
replace github.com/mark3labs/mcp-go => github.com/maximhq/mcp-go v0.0.0-20261102181017-b7da3bc145a1
.so loads before touching logic.
Runtime config lives in <app-dir>/config.json; the plugin reads it in Init(raw any) and never breaks a completion when its own storage fails:
auditID, cerr := p.st.CreateAudit(model, requestID, claims)
if cerr != nil {
return resp, bifrostErr, nil // never break a completion because the ledger failed
}
A freshness check that takes down the gateway is worse than no freshness check.
Mounted through HTTPTransportPreHook, so it inherits gateway auth:
GET /health
GET /stats # audits, claims, contradictions, corrections, grounded
GET /audits # recent list
GET /audit/{id} # JSON
GET /audit/{id}.html # flat paper report, printable
POST /audit # { "text": "β¦" } β { "audit_id": "β¦", "status": "pending" }
GET /expiring # corrections about to age out of the ledger window
curl -s localhost:8091/audit -d '{"text":"Acme Pro costs $20 per month."}' | jq -r .audit_id
The HTML report is deliberately unglamorous: paper background, hairline rules, one block per claim with the decisive evidence quote, its authority, its publish date, and the arithmetic that produced the score.
A detector is only as good as its labeled set, and the temptation is to label after seeing predictions. The bench avoids that:
bench/claims.csv β one claim per row with is_true_now, label_url, label_quote, label_date, adversarial. bench/run.py posts each claim to POST /audit against live Tavily; bench/score.py compares predicted to label and reports accuracy plus per-verdict precision/recall.
The interesting failure mode is not "wrong verdict on a stale claim" β it's "confident verdict on a claim whose subject the search never actually resolved." Abstention is a feature, and the report says UNVERIFIABLE rather than inventing certainty.
authority Γ e^(βage/Ο) per category, with an explicit abstain band.
The whole thing is a few thousand lines of Go: extractor, scorer, Tavily client, SQLite ledger, auditor workers, an HTML report, and two hooks.
Code & more: https://www.dailybuild.xyz/project/321-fresh-guage