{"slug": "show-hn-driftguard-response-drift-detection-for-langgraph-agents", "title": "Show HN: DriftGuard – response drift detection for LangGraph agents", "summary": "DriftGuard, a new open-source tool for detecting response drift in LangGraph agents, was released on Hacker News. It uses embedding-based comparison to flag when an LLM strays from its intended domain without requiring ground-truth labels or separate classifiers. The tool provides adaptive thresholds and supports integration as a guardrail or monitoring callback.", "body_md": "Embedding-based response drift detection for LangChain agents.\n\nDetects when an LLM starts answering outside its intended domain (a legal assistant drifting into cooking advice, a medical chatbot wandering into finance) without ground-truth labels or a separate classifier.\n\n[How it works](#how-it-works)[Installation](#installation)[Quick start](#quick-start)[Integration patterns](#integration-patterns)[LangGraph guardrail](#langgraph-guardrail)[Async support](#async-support)[Alert sinks](#alert-sinks)[Multi-topic corpora](#multi-topic-corpora-clustering)[Domain auditing](#domain-auditing)[Building a corpus with FPS](#building-a-corpus-with-fps)[Distribution-level detection](#distribution-level-detection-windowed)[Visualisation](#visualisation)[Persisting a corpus](#persisting-a-corpus)[DriftResult reference](#driftresult-reference)[Development](#development)\n\n**Build a reference corpus** from representative on-topic texts.**Embed each LLM response** with the same model.**Compare** using two complementary signals:*Centroid distance*: how close is the response to the centre of the corpus (or its nearest cluster)?*Nearest-neighbour distance*: is the response close to at least one reference text?\n\n**Flag drift** when both signals agree the response is far from the reference domain.\n\nUsing both signals reduces false positives: a paraphrase that sits slightly off the centroid is rescued when it's still close to a known reference text.\n\nThe threshold for each signal is **adaptive**: the 5th percentile of within-corpus similarity scores, so ~95% of reference texts clear it with no manual tuning.\n\n```\ngit clone https://github.com/vinerya/driftguard.git\ncd driftguard\npip install -r requirements.txt\npip install -e .\n```\n\nRequires Python ≥ 3.9. The only runtime dependencies are `langchain-core`\n\nand `numpy`\n\n.\n\nOptional extras:\n\n```\npip install -e \".[viz]\"        # matplotlib + scikit-learn for corpus.plot()\npip install langgraph          # LangGraph guardrail nodes\npython\nfrom driftguard import ReferenceCorpus, DriftDetector\nfrom langchain_openai import OpenAIEmbeddings\n\nembeddings = OpenAIEmbeddings()\n\n# 1. Build the reference corpus from representative on-topic texts\ncorpus = ReferenceCorpus(embeddings_model=embeddings)\ncorpus.add_texts([\n    \"tort law\", \"contract formation\", \"negligence standard\",\n    \"criminal intent\", \"due process rights\",\n])\n\n# 2. Create the detector\ndetector = DriftDetector(corpus=corpus)\n\n# 3. Check a response\nresult = detector.check(\"habeas corpus\")\nprint(result.is_drift)                # False (on-topic)\nprint(result.centroid_similarity)     # e.g. 0.91\nprint(result.max_reference_similarity)# e.g. 0.95\nprint(result.threshold)               # e.g. 0.87\n\nresult = detector.check(\"best pasta recipe\")\nprint(result.is_drift)                # True (off-topic)\n```\n\nAttach to any LangChain LLM or chat model. Runs on every response without interrupting the pipeline; use for monitoring, logging, or metrics.\n\n``` python\nfrom driftguard import DriftCallbackHandler, AlertManager\n\nalerts = AlertManager(sinks=[\"log\"])\nhandler = DriftCallbackHandler(detector=detector, alerts=alerts)\n\nllm = ChatOpenAI(callbacks=[handler])\nresponse = llm.invoke(\"What is the recipe for tiramisu?\")\n# Drift is logged as a WARNING; the response still returns normally.\n\nprint(handler.history[-1].is_drift)   # True\n```\n\nInsert as a step in a LangChain chain. Raises `DriftError`\n\non drift; passes the text through unchanged otherwise.\n\n``` python\nfrom driftguard import DriftRunnable, DriftError\nfrom langchain_core.output_parsers import StrOutputParser\n\ndrift = DriftRunnable(detector=detector)\nchain = llm | StrOutputParser() | drift.as_guard()\n\ntry:\n    result = chain.invoke(\"What is the recipe for tiramisu?\")\nexcept DriftError as e:\n    print(f\"Blocked: centroid_sim={e.result.centroid_similarity:.3f} \"\n          f\"< threshold={e.result.threshold:.3f}\")\n```\n\nAnnotates the chain output with drift metadata without halting. Useful when you want to observe drift but let the response through for the user to see.\n\n```\nchain = llm | StrOutputParser() | drift.as_passthrough()\noutput = chain.invoke(\"habeas corpus\")\n# {\"output\": \"Habeas corpus is a legal right...\", \"drift\": DriftResult(...)}\nprint(output[\"drift\"].is_drift)       # False\n```\n\n`driftguard`\n\nships a first-class LangGraph integration. The node and routing helpers are plain callables that match LangGraph's expected signatures, no LangGraph import inside the library itself, so the module loads fine even if LangGraph isn't installed.\n\n``` python\nfrom langgraph.graph import StateGraph\nfrom typing import Any\nfrom typing_extensions import TypedDict\n\nfrom driftguard.langgraph import drift_node, route_on_drift\n\nclass AgentState(TypedDict):\n    query: str\n    response: str\n    drift: Any          # holds the DriftResult written by the drift node\n\ngraph = StateGraph(AgentState)\n\ngraph.add_node(\"llm\", call_llm)                     # writes state[\"response\"]\ngraph.add_node(\"drift_check\", drift_node(detector)) # reads \"response\", writes \"drift\"\ngraph.add_node(\"fallback\", handle_fallback)\ngraph.add_node(\"respond\", finalize)\n\ngraph.set_entry_point(\"llm\")\ngraph.add_edge(\"llm\", \"drift_check\")\ngraph.add_conditional_edges(\n    \"drift_check\",\n    route_on_drift,                 # returns \"drift\" or \"ok\"\n    {\"drift\": \"fallback\", \"ok\": \"respond\"},\n)\n\napp = graph.compile()\n```\n\n**Custom state key**: if your LLM node writes to a key other than `\"response\"`\n\n:\n\n```\ngraph.add_node(\"drift_check\", drift_node(detector, text_key=\"output\"))\n```\n\n**Async graphs**: swap `drift_node`\n\nfor `adrift_node`\n\n:\n\n``` python\nfrom driftguard.langgraph import adrift_node\n\ngraph.add_node(\"drift_check\", adrift_node(detector))\n```\n\n**Custom route labels**: use `make_route_on_drift`\n\nwhen your edge map uses different names:\n\n``` python\nfrom driftguard.langgraph import make_route_on_drift\n\nrouter = make_route_on_drift(on_drift=\"blocked\", on_ok=\"continue\")\ngraph.add_conditional_edges(\n    \"drift_check\", router, {\"blocked\": \"fallback\", \"continue\": \"respond\"}\n)\n```\n\nEvery public method has an async counterpart:\n\n```\nawait corpus.aadd_texts([\"tort law\", \"negligence\"])\nresult = await detector.acheck(\"contract formation\")\n```\n\n`AsyncDriftCallbackHandler`\n\nmirrors `DriftCallbackHandler`\n\nfor async LangChain pipelines.\n\n`AlertManager`\n\ndispatches drift alerts to one or more sinks simultaneously:\n\n``` python\nfrom driftguard import AlertManager\n\nalerts = AlertManager(sinks=[\n    \"log\",                                       # WARNING via Python logging\n    \"https://your-service.example/webhook\",      # POST JSON payload\n    lambda result: my_queue.put(result),         # arbitrary sync or async callable\n])\n```\n\nPass an `AlertManager`\n\ninstance to `DriftCallbackHandler`\n\n, `DriftRunnable`\n\n, or the LangGraph nodes; all accept one via the `alerts`\n\nargument.\n\nWhen your reference corpus spans several distinct topics, a single global centroid produces false positives for texts that are on-topic but far from the average. Set `n_clusters`\n\nto partition the corpus into groups; each query is then compared to its nearest cluster rather than the global centre.\n\n```\ncorpus = ReferenceCorpus(embeddings_model=embeddings, n_clusters=2)\ncorpus.add_texts([\n    # Legal cluster\n    \"tort law\", \"contract formation\", \"negligence\",\n    # Medical cluster\n    \"malpractice\", \"diagnosis\", \"clinical trial\",\n])\n\ndetector = DriftDetector(corpus=corpus)\n\ndetector.check(\"habeas corpus\").is_drift   # False (routes to legal cluster)\ndetector.check(\"prognosis\").is_drift       # False (routes to medical cluster)\ndetector.check(\"pasta recipe\").is_drift    # True  (far from both clusters)\n```\n\nClustering uses numpy k-means internally with no extra dependencies.\n\nThe `Auditor`\n\nclass runs drift detection over a batch of historical responses and returns a structured report: pass rate, score distribution, flagged outliers. Use it before deployment to validate your corpus, after incidents to understand what went wrong, or in CI to catch domain regressions between prompt versions.\n\n``` python\nfrom driftguard import Auditor\n\nauditor = Auditor(detector)\nreport = auditor.run(production_responses)\n\nprint(f\"Pass rate:  {report.pass_rate:.1%}\")\nprint(f\"Drift rate: {report.drift_rate:.1%}\")\nprint(f\"Flagged:    {report.flagged} / {report.total}\")\n```\n\n**Export the report** for a compliance doc or CI artifact:\n\n```\nreport.to_json()             # structured JSON string\nopen(\"report.html\", \"w\").write(report.to_html())  # self-contained HTML report\n```\n\nThe HTML report includes a summary dashboard, centroid similarity distribution (p5 → p95), and a table of all flagged responses with their scores.\n\n**Async**: all responses are checked concurrently:\n\n```\nreport = await auditor.arun(production_responses)\n```\n\nDetect domain shift between prompt versions, model upgrades, or dataset changes:\n\n```\ncomparison = corpus_v1.compare(corpus_v2)\n\nprint(f\"Centroid shift: {comparison.centroid_shift:.4f}\")  # cosine distance\nprint(f\"Threshold delta: {comparison.threshold_delta:+.4f}\")\nprint(f\"Significant: {comparison.is_significant}\")         # shift > 0.05\n```\n\nA `centroid_shift`\n\nabove 0.05 (configurable via `significant_shift_threshold`\n\n) means the two corpora represent meaningfully different domains, worth investigating before swapping one for the other.\n\nHand-picking reference texts is tedious and easy to get wrong. `ReferenceCorpus.from_texts()`\n\naccepts a large pool of candidates and uses **Farthest Point Sampling** to automatically select the `n`\n\nmost coverage-maximising texts; each new selection is the one farthest (in cosine distance) from all already-chosen texts.\n\n```\n# 500 example legal responses; pick the 30 most diverse ones.\ncorpus = ReferenceCorpus.from_texts(\n    candidates=my_500_legal_responses,\n    embeddings_model=embeddings,\n    n=30,\n)\n```\n\nThe result is a fully initialised `ReferenceCorpus`\n\nready for use with `DriftDetector`\n\n. An async variant is also available:\n\n```\ncorpus = await ReferenceCorpus.afrom_texts(candidates, embeddings_model=embeddings, n=30)\n```\n\nPer-response checks are sensitive to one-off anomalies. `WindowedDriftDetector`\n\naccumulates a sliding window of responses and checks whether the *window's* embedding distribution has shifted from the reference. Two signals can trigger drift:\n\n**Centroid shift**: the window's mean embedding has moved away from the reference.** Drift fraction**: more than`drift_fraction_threshold`\n\n(default 30%) of recent responses are individually off-topic.\n\n``` python\nfrom driftguard import WindowedDriftDetector\n\nwd = WindowedDriftDetector(corpus=corpus, window_size=20, drift_fraction_threshold=0.3)\n\nfor response in llm_responses:\n    result = wd.update(response)\n    if result is None:\n        continue   # window still filling\n    if result.is_drift:\n        print(f\"Window drift detected: \"\n              f\"centroid_sim={result.window_centroid_similarity:.3f}, \"\n              f\"drift_fraction={result.drift_fraction:.0%}\")\n```\n\n`result`\n\nis a `WindowDriftResult`\n\nreturned on every call once the window is full. Use `on_drift`\n\nfor async-friendly callbacks:\n\n```\nwd = WindowedDriftDetector(corpus=corpus, on_drift=lambda r: alert_queue.put(r))\n```\n\nAsync usage mirrors the sync API:\n\n```\nresult = await wd.aupdate(response)\n```\n\n`corpus.plot()`\n\nprojects the reference corpus into 2D via t-SNE and optionally overlays texts colour-coded by drift status, useful for debugging false positives and tuning `threshold_percentile`\n\n.\n\n```\npip install driftguard[viz]   # adds matplotlib + scikit-learn\ncorpus.plot(check_texts=[\"habeas corpus\", \"pasta recipe\", \"clinical trial\"])\n```\n\nBlue circles are reference texts; green triangles are on-topic detections; red X markers are flagged as drift.\n\nFor more control, call `plot_corpus`\n\ndirectly:\n\n``` python\nfrom driftguard.viz import plot_corpus\nimport matplotlib.pyplot as plt\n\nfig, ax = plt.subplots(figsize=(10, 7))\nplot_corpus(corpus, check_texts=probe_texts, ax=ax)\nplt.show()\n```\n\nSave a trained corpus to disk and reload it on the next run, no need to re-embed reference texts every time.\n\n```\ncorpus.save(\"legal_corpus\")\n# writes legal_corpus.npz  (embeddings, centroid, thresholds, cluster data)\n#        legal_corpus.texts.json  (original texts)\n\nloaded = ReferenceCorpus(embeddings_model=embeddings)\nloaded.load(\"legal_corpus\")\n```\n\nCluster data (centroids, per-cluster thresholds) is persisted alongside the embeddings.\n\nEvery call to `detector.check()`\n\nor `detector.acheck()`\n\nreturns a frozen `DriftResult`\n\n:\n\n| Field | Type | Description |\n|---|---|---|\n`is_drift` |\n`bool` |\n`True` when both centroid and NN signals indicate drift |\n`centroid_similarity` |\n`float` |\nCosine similarity to the nearest cluster (or global) centroid |\n`max_reference_similarity` |\n`float` |\nCosine similarity to the closest individual reference text |\n`threshold` |\n`float` |\nAdaptive centroid threshold for this check |\n`nn_threshold` |\n`float` |\nAdaptive nearest-neighbour threshold |\n`text` |\n`str` |\nThe checked text |\n`timestamp` |\n`float` |\nUnix timestamp |\n`metadata` |\n`dict` |\nAny kwargs passed to `check()` , e.g. `run_id` |\n\n`DriftError`\n\n(raised by `as_guard()`\n\n) exposes the full `DriftResult`\n\non its `.result`\n\nattribute.\n\n`WindowedDriftDetector.update()`\n\nreturns a `WindowDriftResult`\n\nonce the window is full:\n\n| Field | Type | Description |\n|---|---|---|\n`is_drift` |\n`bool` |\n`True` when centroid or fraction signal fires |\n`window_centroid_similarity` |\n`float` |\nCosine similarity of window centroid to reference |\n`drift_fraction` |\n`float` |\nFraction of window responses individually flagged |\n`window_size` |\n`int` |\nNumber of responses in the window |\n`threshold` |\n`float` |\nReference threshold used for centroid check |\n`drift_fraction_threshold` |\n`float` |\nConfigured fraction threshold |\n`timestamp` |\n`float` |\nUnix timestamp |\n\n```\npip install -e \".[dev]\"\npytest\n```\n\nAll tests use deterministic `FakeEmbeddings`\n\n, no API key or network access required.\n\nMIT", "url": "https://wpnews.pro/news/show-hn-driftguard-response-drift-detection-for-langgraph-agents", "canonical_source": "https://github.com/vinerya/driftGuard", "published_at": "2026-06-29 16:04:55+00:00", "updated_at": "2026-06-29 16:21:04.321352+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "developer-tools", "natural-language-processing"], "entities": ["DriftGuard", "LangGraph", "LangChain", "OpenAI", "Hacker News"], "alternates": {"html": "https://wpnews.pro/news/show-hn-driftguard-response-drift-detection-for-langgraph-agents", "markdown": "https://wpnews.pro/news/show-hn-driftguard-response-drift-detection-for-langgraph-agents.md", "text": "https://wpnews.pro/news/show-hn-driftguard-response-drift-detection-for-langgraph-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-driftguard-response-drift-detection-for-langgraph-agents.jsonld"}}