Four Traps in MCP Health Checking: What Broke My Overnight Batches A developer building automation with Claude Code has documented four traps in MCP health checking that broke overnight batches, including failures from rate limits and service outages. The engineer solved the problem by implementing a health-check hook that records failures outside the context window, using a file-based cache to persist server health status across context compactions. This approach has prevented overnight batch failures and improved the reliability of their autonomous environment, which generates ¥1.2M/month in revenue. An MCP server dying at 2 a.m. used to mean waking up to a log full of connection refused and zero work done. Building a health-check hook that records failures outside the context window fixed that — my overnight batches haven't been killed by an MCP outage since. When I first started automating with Claude Code, there was one failure mode I hated more than any other: a batch job kicked off at night, stopped dead by an MCP server timeout. I'd wake up to a pile of connection refused logs and nothing having advanced since the previous evening. For SNS post automation that's survivable, but the time it stopped a client deliverable generation run, I genuinely panicked. MCP is the mechanism that lets Claude Code use browser operations, DB lookups, external API calls, and so on as tools. From Claude's side it just looks like calling a tool named something like mcp obsidian search , but underneath, communication with a local process or a remote HTTP server is running. When that server stops responding for whatever reason, Claude Code returns the tool call as an error and the whole flow of the session jams up. The problem wasn't only "why did it stop." There were cases where it was stopped by a 429 rate limit , yet it would re-call 30 seconds later → another 429 → stopped again, looping forever. There were also cases where a 503 service temporarily unavailable was judged the same as a 401 expired auth , pointlessly running a re-authentication flow. Unless you vary the strategy per status code, it doesn't matter how fast a model you use — it's wasted. Maintaining an autonomous environment at ¥1.2M/month revenue, what I noticed is that the time spent on "mechanisms that keep things from stopping" has a higher long-term ROI than the code added to increase earnings. Back at ¥600K/month I thought "more tasks means more earnings." After being laid off and dropping to zero, my thinking changed while rebuilding it from scratch. Adding tasks doesn't help if the environment is unstable — throughput pins to a ceiling. Conversely, killing a single infrastructure-level problem raises the completion rate across all existing tasks. The MCP health check is the classic example of this. By wiring ~/.claude/scripts/hooks/mcp-health-check.js into a hook, an HTTP probe runs before Claude Code calls a tool, and depending on the response status it dispatches to "block immediately," "retry after backoff," or "run a reconnect command and re-probe." The verdict is persisted to ~/.claude/mcp-health-cache.json , so even when context is compacted, the health record carries over. When a Claude Code session runs for a long time, the conversation history gets compacted. Even if there was information in the past saying "this server was down," that doesn't survive into the post-compaction context. The result is waste: repeatedly attempting tool calls against a server already known to be unhealthy, and receiving an error each time. A file-based cache is independent of the context. ~/.claude/mcp-health-cache.json doesn't disappear no matter how much the session is compacted. When the health-check hook runs on the next turn, it loads the previous state from the file, and until nextRetryAt has passed it blocks immediately without even re-probing. The idea of holding state outside the context window is what matters fundamentally. Some people think, "why not just do error handling in Claude's prompt?" I actually tried it. A system prompt saying "if this tool errors, try another approach" works reasonably well for a one-off error. But in a situation where the MCP server is down and failures happen back to back, the model burns a huge number of tokens on "trying." And on the next turn, it goes right back to calling the same server. Writing the fact that "the server is dead" outside the context and blocking at the hook level is overwhelmingly cleaner. Another common misconception is "MCP's own retry settings are enough." The MCP protocol has transport-layer retries, but it has no feature for reading status codes and switching strategy. 429 and 503 call for different backoff durations, and 401/403 are cases that need re-authentication rather than a retry. Implementing this dispatch at the application layer is what this hook is for. mcp-health-check.js responds to two kinds of Claude Code hook events. It's written verbatim in the comment at the top of the code lines 7–12 . - PreToolUse: probe MCP server health before MCP tool execution - PostToolUseFailure: mark unhealthy servers, attempt reconnect, and re-probe PreToolUse is called before the tool is executed. Here it fires a probe, checks whether the server is alive, and decides whether to allow execution or block with exit code 2. PostToolUseFailure is called after a tool returns an error. It parses the error text to identify the failure pattern, marks the server unhealthy, and attempts a reconnect. Claude Code が mcp ツールを呼ぶ │ ▼ PreToolUse フック起動 ┌──────────────────────────────────────────────────────┐ │ mcp-health-check.js │ │ │ │ ① mcp-health-cache.json を読む │ │ status=healthy かつ expiresAt が未来？ │ │ YES ─────────────────────────────────────────→ │ exit 0 │ NO ↓ │ （ツール実行へ） │ │ │ ② nextRetryAt が未来（unhealthy クールダウン中）？ │ │ YES → ブロック ──────────────────────────────→ │ exit 2 │ NO ↓ │ （ツールをスキップ） │ │ │ ③ プローブ実行 │ │ HTTPサーバー → GET リクエスト（5秒タイムアウト） │ │ stdioサーバー → プロセス起動（5秒生存確認） │ │ │ │ レスポンスのステータスコード判定 │ │ ┌──────────────────────────────────────────┐ │ │ │ ECONNREFUSED / ENOTFOUND / タイムアウト │──→ │ │ │ → 即 markUnhealthy & exit 2 │ │ │ ├──────────────────────────────────────────┤ │ │ │ 401 / 403 / 429 / 503 │──→ │ │ │ → reconnect コマンドを実行 │ │ │ │ → 成功すれば再プローブ │ │ │ │ → 再プローブ OK → markHealthy & exit 0 │ │ │ │ → 再プローブ NG → markUnhealthy & exit 2│ │ │ ├──────────────────────────────────────────┤ │ │ │ 200 系 / 400 / 401 / 403 / 405 / 406 │ │ │ │（"到達できた"証明として healthy 扱い） │──→ │ exit 0 │ └──────────────────────────────────────────┘ │ │ │ │ ④ 状態を mcp-health-cache.json に書き出す │ └──────────────────────────────────────────────────────┘ │ ▼ ツール実行後にエラーが出た場合 ┌──────────────────────────────────────────────────────┐ │ PostToolUseFailure フック │ │ エラーテキストを FAILURE PATTERNS と照合 │ │ failureCode 特定 → markUnhealthy → reconnect試行 │ │ → 再プローブ OK なら markHealthy │ └──────────────────────────────────────────────────────┘ Reading the actual code, the design intent shows up in the numbers lines 22–26 . js const DEFAULT TTL MS = 2 60 1000; // 2分 const DEFAULT TIMEOUT MS = 5000; // 5秒 const DEFAULT BACKOFF MS = 30 1000; // 30秒（初回バックオフ） const MAX BACKOFF MS = 10 60 1000; // 10分（上限） A TTL of 2 minutes is the trade-off between "I don't want to run a probe every single time" and "I don't want to hold a stale result too long." In scenarios where Claude Code calls tools back to back, hitting the same server multiple times within 2 minutes is not unusual. With a cache, there's no need to run an HTTP probe each time, and latency drops. The 5-second timeout is the threshold for confirming that a local stdio server "can start as a process." Rather than startup completion, it judges normality by the process "staying alive for 5 seconds" the timer logic in probeCommandServer , lines 428–472 . Even for a heavy server that takes more than 5 seconds to start, this can be adjusted with the environment variable ECC MCP HEALTH TIMEOUT MS . The backoff calculation in the markUnhealthy function lines 213–229 is compressed into one line. js const nextRetryDelay = Math.min backoffBase 2 Math.max failureCount - 1, 0 , MAX BACKOFF MS ; backoffBase defaults to 30 seconds. When failureCount is 1, 2 0 = 1 gives 30 seconds; the second time 2 1 = 2 gives 60 seconds; the third 120 seconds; the fourth 240 seconds, doubling each time, capping out at a maximum of 600 seconds 10 minutes . It waits 30 seconds after the first failure, and if it hasn't recovered there, widens the interval to 1 minute, 2 minutes, 4 minutes — which avoids the situation of endlessly firing pointless probes at a downed server. The definition of HEALTHY HTTP CODES line 32 looks odd at first glance. js const HEALTHY HTTP CODES = new Set 200, 201, 202, 204, 301, 302, 303, 304, 307, 308, 400, 401, 403, 405, 406 ; Some 400-level codes are treated as "healthy." The reason is written in the code's comment lines 29–32 . // The preflight HTTP probe only checks reachability; it does not have access to // Claude Code's stored OAuth bearer token. Treat auth-gated responses as // reachable so the real MCP client can attempt the authenticated call. A // Streamable HTTP MCP server can also return 406 to a bare GET that omits // Accept: text/event-stream; that still proves the endpoint is alive. The preflight probe is a GET request with no OAuth token. An endpoint that requires authentication returns 401 or 403, but that means it is "correctly rejecting an unauthenticated request" — evidence that the server is alive. 406 is a normal rejection response to a request without the Accept: text/event-stream header. These are all codes that prove "it was reachable." Meanwhile, the set targeted for reconnect is a subset line 33 . js const RECONNECT STATUS CODES = new Set 401, 403, 429, 503 ; 401 and 403 are treated as "reachable" in the probe context, but when they come back after an actual tool call , the meaning is different. When the detectFailureCode function is called in PostToolUseFailure, a 401/403 in the error message is taken as "authentication failure" and becomes the trigger for a reconnect. The key point is that the probe verdict and the post-failure verdict operate at different layers. MCP errors don't necessarily come back as HTTP status codes. Errors from stdio servers come back as text. FAILURE PATTERNS lines 34–40 is the mechanism that identifies the failure type from that text via regular expressions. js const FAILURE PATTERNS = { code: 401, pattern: /\b401\b|unauthori sz ed|auth ?:entication ?\s+ ?:failed|expired|invalid /i }, { code: 403, pattern: /\b403\b|forbidden|permission denied/i }, { code: 429, pattern: /\b429\b|rate limit|too many requests/i }, { code: 503, pattern: /\b503\b|service unavailable|overloaded|temporarily unavailable/i }, { code: 'transport', pattern: /ECONNREFUSED|ENOTFOUND|EAI AGAIN|timed? out|socket hang up|connection ?:failed|lost|reset|closed /i } ; From error messages containing strings like "unauthorized," "auth expired," "rate limit," or "ECONNREFUSED," it automatically identifies the failure code. The transport code isn't an HTTP status — it's the case where the network connection itself is cut. This is a state of "can't even reach the server," where backoff takes priority over reconnect. ~/.claude/mcp-health-cache.json is written out by the saveState function lines 95–102 , formatted with JSON.stringify. The actual file has a structure like this. { "version": 1, "servers": { "obsidian": { "status": "healthy", "checkedAt": 1753666200000, "expiresAt": 1753666320000, "failureCount": 0, "lastError": null, "lastFailureCode": null, "nextRetryAt": 1753666200000, "lastRestoredAt": 1753666200000, "source": "~/.claude/settings.json" }, "agentmemory": { "status": "unhealthy", "checkedAt": 1753665900000, "expiresAt": 1753665900000, "failureCount": 3, "lastError": "ECONNREFUSED 127.0.0.1:3001", "lastFailureCode": "transport", "nextRetryAt": 1753666020000, "lastRestoredAt": null } } } It's a two-stage gate: if expiresAt is in the future, pass without re-probing; if nextRetryAt is in the future, block without even retrying. The more failureCount accumulates, the longer the backoff stretches, so a repeatedly-down server gets progressively wider attempt intervals. lastRestoredAt is a record of "when it recovered," usable for after-the-fact uptime analysis. The source field is the path to the config file. The configPaths function lines 54–72 searches in the order: the current directory's .claude.json → the current directory's .claude/settings.json → home's .claude.json → home's .claude/settings.json , and the first one found is used. Even when you have per-project MCP configuration, the correct config is referenced. The reconnectCommand function lines 516–527 reads the command from environment variables. js const key = ECC MCP RECONNECT ${String serverName .toUpperCase .replace / ^A-Z0-9 /g, ' ' } ; const command = process.env key || process.env.ECC MCP RECONNECT COMMAND || ''; If the server name is agentmemory , it looks for an environment variable named ECC MCP RECONNECT AGENTMEMORY . If not found, it uses the global fallback ECC MCP RECONNECT COMMAND . If the command string contains {server} , it is expanded to the server name lines 524–526 . For example, if you use PM2 as your process manager, you can configure it like this. export ECC MCP RECONNECT COMMAND="pm2 restart {server}" With this, when the agentmemory server returns a 401 or 503, pm2 restart agentmemory runs automatically, and tool execution is allowed only after a re-probe confirms it's normal. When a reconnect succeeds, reconnect-command is recorded in markHealthy 's restoredBy field lines 607–609 . At this point you should have a grasp of the hook's overall shape. In the next section I'll dig into the actual stumbling points and the traps that are easy to fall into during configuration. The first thing the hook does is identify "which MCP server is this call for." The extractMcpTarget function lines 133–167 handles that. js if toolName.startsWith 'mcp ' { return null; } const segments = toolName.slice 5 .split ' ' ; if segments.length < 2 || segments 0 { return null; } return { server: segments 0 , tool: segments.slice 1 .join ' ' }; Given a name like mcp obsidian search notes , slice 5 makes it obsidian search notes , and split ' ' makes it 'obsidian', 'search', 'notes' . segments 0 is the server name, and the rest joined with double underscores is the tool name. That said, parsing the tool name is strictly a fallback. It looks for an explicit server field first lines 135–145 . js const explicitServer = input.server || input.mcp server || input.tool input?.server || input.tool input?.mcp server || input.tool input?.connector || null; The reason it walks multiple paths is that the hook event's schema can have fields in shifted positions depending on the Claude Code version and connection method HTTP/stdio . Furthermore, if JSON parsing fails and the truncated flag is set, extractMcpTargetFromRaw lines 169–179 applies a regular expression to the raw string to extract the same information. It's a design that doesn't give up even when parsing fails. handlePreToolUse lines 567–630 looks simple, but two cache checks run in succession. // 第1ゲート: healthy かつキャッシュ有効 → プローブなしで通過 if previous.status === 'healthy' && Number previous.expiresAt || 0 now { return { rawInput, exitCode: 0, logs }; } // 第2ゲート: unhealthy かつクールダウン中 → プローブなしでブロック if previous.status === 'unhealthy' && Number previous.nextRetryAt || 0 now { logs.push MCPHealthCheck ${target.server} is marked unhealthy until ${new Date previous.nextRetryAt .toISOString }; skipping ${target.tool || 'tool'} ; return { rawInput, exitCode: shouldFailOpen ? 0 : 2, logs }; } "healthy and within TTL" passes, "unhealthy and in backoff" is blocked, and only in neither case unhealthy but backoff expired, or the cache is empty is a probe executed. In sessions that call tools at high frequency, almost all calls pass at the first gate, so HTTP request latency is effectively zero. The conditions that trigger a reconnect need similar care lines 601–603 . js let reconnect = { attempted: false, success: false, reason: 'probe failed' }; if probe.failureCode || previous.status === 'unhealthy' { reconnect = attemptReconnect target.server ; A reconnect doesn't run just because "the probe failed." It requires either probe.failureCode a status code for which reconnect is enabled or the condition "already unhealthy since last time." Running a reconnect command every time when you can't even connect due to ECONNREFUSED is wasteful, but when the server exists yet is unusable — as with 429 or 503 — actively attempting to reconnect is worthwhile; that judgment is embedded here. For an HTTP server, firing a GET with requestHttp tells you what you need. How do you check a stdio server? The approach in probeCommandServer lines 301–481 is distinctive. Actually start the process and watch it for 5 seconds. If it stays alive, it's healthy; if it exits before then, it's not. js timer = setTimeout = { // タイムアウト到達 = 5秒間プロセスが生存 → 正常 // ただし: ロードされたマシンではexitイベントがタイマーより // わずかに遅れて届くことがある。高速クラッシュを見逃さないため // プロセスの状態を再確認する if child.exitCode == null || child.signalCode == null { attemptFinish { ok: false, statusCode: child.exitCode, reason: stderr.trim || process exited before handshake ... } ; return; } // SIGTERMで終了させ、200ms後にSIGKILLで確実に始末する child.kill 'SIGTERM' ; setTimeout = { try { child.kill 'SIGKILL' ; } catch { / ignore / } }, 200 .unref?. ; attemptFinish { ok: true, statusCode: null, reason: ${serverName} accepted a new stdio process } ; }, timeoutMs ; As the comment says, the "fast-crashing server" problem really is a nuisance. If the process exits right after startup, the exit event and the timer callback can arrive at nearly the same moment. If the timer runs first, it would return ok: true without a child.exitCode check. As a countermeasure, the timer callback re-checks child.exitCode == null , and if it's already dead, drops it to ok: false . Windows behavior is worked out too lines 329–334, 438–461 . A command without an extension, like npx , needs to resolve to npx.cmd on Windows, and on top of that, from Node 18.20 onward, executing .cmd via a shell was restricted as part of the CVE-2024-27980 fix. The code lines up fallbacks of command.cmd / command.exe / command.bat in a candidates array, and tries the next candidate in order on ENOENT. It also executes .cmd / .bat files via a shell, but has a safety valve that refuses shell execution if the command string contains shell metacharacters &|< ^% etc. line 339, UNSAFE SHELL CHARS . failureSummary lines 231–244 is quietly important. js const pieces = typeof input.error === 'string' ? input.error : '', typeof input.message === 'string' ? input.message : '', typeof input.tool response === 'string' ? input.tool response : '', typeof output === 'string' ? output : '', typeof output?.output === 'string' ? output.output : '', typeof output?.stderr === 'string' ? output.stderr : '', typeof input.tool input?.error === 'string' ? input.tool input.error : '' .filter Boolean ; return pieces.join '\n' ; Which field the MCP error text lands in differs by server implementation and Claude Code version. The case where it's in error , the case where it's in tool response , the case where it's in output.stderr — I've encountered all of them in practice. By passing the string joined from all fields with pieces.join '\n' to detectFailureCode , the regular expressions can catch the error text no matter which field it's in. shouldFailOpen lines 557–559 is a one-liner over an environment variable. function shouldFailOpen { return /^ 1|true|yes $/i.test String process.env.ECC MCP HEALTH FAIL OPEN || '' ; } It's the setting that lets through tool calls to unhealthy servers. You use it during development when you want to "have the hook running but not blocking, just to observe." In PreToolUse , it returns exit 0 pass instead of exit 2 block . The exitCode for PostToolUseFailure is always 0 line 677 . This is an important design point: the post-failure hook's purpose is recording state , not blocking tool execution. Changing the exit code of a tool call that has already failed is meaningless; it only writes logs and updates the cache, handing the state over to the next PreToolUse . From wiring this mechanism into my own environment to having it stable, I got stuck in three situations. I'll write the symptom, the cause, and the fix in order. Symptom : The agentmemory server remained in the cache with status: healthy , yet actual tool calls returned ECONNREFUSED . The health check looked meaningless. Cause : A stdio server probe only confirms that "the process stays alive for 5 seconds." The process starts, gets terminated with SIGTERM after 5 seconds, and then when the real tool call arrives, the server starts again from zero. But in a real startup, it takes time to complete the MCP handshake JSON message exchange over stdio . A different error was occurring during that handshake. Specifically, a server that tried to start without the environment variable ANTHROPIC API KEY being set was crashing right after startup. At probe time it stayed alive for 5 seconds so it got recorded as healthy , but during the actual handshake it would crash a few seconds in. Fix : Check first whether the server process's environment variables are all in place. mcp-health-check.js merges config.env into the process's environment variables when starting lines 306–309 . Confirming that the same environment variables as the production startup are in the config file's env section is the first order of business. Only after understanding that "healthy in the probe" and "healthy in the tool call" are strictly different verdicts did the debugging direction finally become clear. Symptom : The Obsidian remote server temporarily returned 503s overnight. Checking the next morning, the server itself had long since recovered, but nextRetryAt in mcp-health-cache.json pointed to a future time and it stayed blocked. Cause : Because 503 is included in RECONNECT STATUS CODES , a reconnect is attempted when a 503 comes in. But I had set neither ECC MCP RECONNECT OBSIDIAN nor ECC MCP RECONNECT COMMAND . Without a reconnect command, attemptReconnect returns attempted: false line 531 , only markUnhealthy runs, and failureCount accumulates. Three 503s make failureCount=3 , and backoff is 30 2 2 = 120 seconds . Five more over the next 30 minutes makes failureCount=5 and 30 2 4 = 480 seconds . As a dozen-plus errors piled up overnight, the backoff was pinned at the ceiling of 600 seconds 10 minutes . Even after the server recovered, a probe only ran every 10 minutes, and all tool calls in between were blocked. Fix : Either configure a reconnect command, or set ECC MCP HEALTH BACKOFF MS to a small value e.g. 10000 = 10 seconds to lower the backoff base. I set ECC MCP RECONNECT COMMAND=echo noop a do-nothing dummy in my launchd plist, which gets it treated as "reconnect attempted" and suppresses failureCount accumulation. The proper approach is to manage the service with PM2 or systemd and configure a real reconnect command. export ECC MCP RECONNECT OBSIDIAN="pm2 restart obsidian-mcp" Once this setting was in, the cycle started working: a 503 comes in → PM2 restarts → a re-probe confirms normal → immediate return to healthy . Symptom : When starting Claude Code in a particular project directory, the MCP health check logged "No MCP config found for obsidian" and skipped the probe, yet the tool call itself ran. The hook appeared to be functioning while not actually functioning. Cause : The configPaths function lines 54–72 searches the current directory's config file first. return path.join cwd, '.claude.json' , path.join cwd, '.claude', 'settings.json' , // ← これが先に見つかる path.join home, '.claude.json' , path.join home, '.claude', 'settings.json' ; That project had a .claude/settings.json , and it defined only project-specific MCP servers like playwright . resolveServerConfig 'obsidian' doesn't stop at the point of finding the first file in list order — if that file doesn't have obsidian , it moves on to the next file. In readJsonFile 's loop, if data?.mcpServers?. serverName is null it returns null and continues the loop — but what actually caused the problem was a case where I had pointed the environment variable ECC MCP CONFIG PATH at the home config file and there was a typo in that path. 誤（タイポあり） export ECC MCP CONFIG PATH="/Users/~/.claude/settings.json" 正 export ECC MCP CONFIG PATH="${HOME}/.claude/settings.json" path.resolve returned the nonexistent path /Users/~/.claude/settings.json , readJsonFile returned null, and it was treated as config not found. Because setting ECC MCP CONFIG PATH completely overrides the normal search paths lines 55–61 , a typo makes all server configuration disappear. Fix : Use an absolute path for ECC MCP CONFIG PATH . Tilde expansion is something the shell does; Node's path.resolve does not do it. When writing it in a launchd plist, ~ is not expanded, so you need to write it as /Users/your-home-directory , ${HOME} via a shell script , or the equivalent of os.homedir . php < -- launchd plist では ~ が展開されない --