{"slug": "your-ai-agents-are-failing-silently-here-s-how-to-catch-it", "title": "Your AI Agents Are Failing Silently — Here's How to Catch It", "summary": "A developer built Drift, an open-source tool that hooks into LangChain agent execution to detect silent failures such as wrong tool sequences, latency spikes, and output drift using statistical anomaly detection. Drift applies three detectors—latency/token SPC, sequence anomaly detection, and output drift detection—and can be added with three lines of code. The tool caught a 76σ latency spike, a novel delete_file call, and an 18x token count increase in a demo.", "body_md": "Last month I ran hundreds of LangChain agent calls in production. Some of them silently failed by using wrong tool sequences, latency spikes, or even hallucinated outputs. My logs showed zero errors. No exceptions. No warnings.\n\nThe agent just did the wrong thing, quietly.\n\nTraditional monitoring tools weren't built for this. Datadog can tell you a function threw an exception. It can't tell you your agent called *delete_file* when it's never done that before, or that your LLM is suddenly generating 10x more tokens than usual, or that output quality has been slowly degrading over the last 500 runs.\n\nSo I built Drift.\n\n**What Drift Does**\n\nDrift hooks into your agent's execution and applies statistical anomaly detection to the event stream in real time. Three detectors run simultaneously:\n\n**Latency & token SPC** — Uses rolling z-scores to flag when a tool call or LLM response takes significantly longer or uses significantly more tokens than its baseline. Catches hung API calls, runaway generation, and upstream provider issues.\n\n**Sequence anomaly detection** — Builds a Markov transition matrix of tool-call sequences and flags when the agent takes a path that's never or rarely been seen. Catches agents going off-script, skipping required steps, or making dangerous tool calls they've never made before.\n\n**Output drift detection** — Tracks output length, vocabulary diversity, and structural patterns over time. Flags when outputs shift significantly from baseline. Catches hallucination drift, prompt injection effects, and gradual quality degradation.\n\n**Three Lines to Add It**\n\n`bash`\n\npip install drift-detection\n\npython\n\n`from drift import DriftGuard\n\nfrom drift.callbacks.langchain import DriftCallbackHandler\n\nguard = DriftGuard(on_anomaly=lambda a: print(f\"🚨 {a}\"))\n\nhandler = DriftCallbackHandler(guard)\n\nagent.run(\"your query\", callbacks=[handler])\n\nguard.report()`\n\nDRIFT DEMO — Agent Anomaly Detection\n\n[Phase 1] Building baseline with 20 normal agent runs...\n\n✓ 120 events processed, 0 anomalies\n\n[Phase 2] Injecting anomalies...\n\n--- Injecting: Latency spike on 'search_web' ---\n\n🚨 [CRITICAL] latency_spike: 'search_web' latency is 2500.0ms,\n\n76.3σ above mean (200.7ms ± 30.1)\n\n--- Injecting: Novel tool sequence (search_web → delete_file) ---\n\n🚨 [HIGH] sequence_anomaly: Novel transition: 'search_web' → 'delete_file'\n\n(never observed; known transitions: ['parse_document'])\n\n--- Injecting: Output drift on 'write_response' ---\n\n🚨 [CRITICAL] token_anomaly: 'write_response' token_count is 2000 tokens,\n\n66.5σ above mean (107.8 ± 28.5)\n\n🚨 [HIGH] output_drift: output_length is 3014 chars, 43.3σ above baseline\n\n🚨 [MEDIUM] output_drift: novel structure 'code_block' (seen: ['plain_text'])`\n\nThe latency spike at 76σ. The *delete_file* call never seen before. The token count 18x baseline. All caught automatically.\n\n**Design Decisions**\n\n**What's Coming**\n\nTry It\n\n`bashpip install drift-detection`\n\nGitHub: [GitHub: dombinic/Drift](https://github.com/dombinic/Drift)\n\nStar it if it's useful. Open an issue if you want a feature — I read everything.", "url": "https://wpnews.pro/news/your-ai-agents-are-failing-silently-here-s-how-to-catch-it", "canonical_source": "https://dev.to/dombinic/your-ai-agents-are-failing-silently-heres-how-to-catch-it-3gca", "published_at": "2026-06-13 16:44:27+00:00", "updated_at": "2026-06-13 17:15:18.343315+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning", "large-language-models", "ai-safety"], "entities": ["LangChain", "Datadog", "Drift", "GitHub", "dombinic"], "alternates": {"html": "https://wpnews.pro/news/your-ai-agents-are-failing-silently-here-s-how-to-catch-it", "markdown": "https://wpnews.pro/news/your-ai-agents-are-failing-silently-here-s-how-to-catch-it.md", "text": "https://wpnews.pro/news/your-ai-agents-are-failing-silently-here-s-how-to-catch-it.txt", "jsonld": "https://wpnews.pro/news/your-ai-agents-are-failing-silently-here-s-how-to-catch-it.jsonld"}}