{"slug": "four-traps-in-mcp-health-checking-what-broke-my-overnight-batches", "title": "Four Traps in MCP Health Checking: What Broke My Overnight Batches", "summary": "A developer building automation with Claude Code has documented four traps in MCP health checking that broke overnight batches, including failures from rate limits and service outages. The engineer solved the problem by implementing a health-check hook that records failures outside the context window, using a file-based cache to persist server health status across context compactions. This approach has prevented overnight batch failures and improved the reliability of their autonomous environment, which generates ¥1.2M/month in revenue.", "body_md": "An MCP server dying at 2 a.m. used to mean waking up to a log full of `connection refused`\n\nand zero work done. Building a health-check hook that records failures *outside* the context window fixed that — my overnight batches haven't been killed by an MCP outage since.\n\nWhen I first started automating with Claude Code, there was one failure mode I hated more than any other: a batch job kicked off at night, stopped dead by an MCP server timeout. I'd wake up to a pile of `connection refused`\n\nlogs and nothing having advanced since the previous evening. For SNS post automation that's survivable, but the time it stopped a client deliverable generation run, I genuinely panicked.\n\nMCP is the mechanism that lets Claude Code use browser operations, DB lookups, external API calls, and so on as tools. From Claude's side it just looks like calling a tool named something like `mcp__obsidian__search`\n\n, but underneath, communication with a local process or a remote HTTP server is running. When that server stops responding for whatever reason, Claude Code returns the tool call as an error and the whole flow of the session jams up.\n\nThe problem wasn't only \"why did it stop.\" There were cases where it was stopped by a 429 (rate limit), yet it would re-call 30 seconds later → another 429 → stopped again, looping forever. There were also cases where a 503 (service temporarily unavailable) was judged the same as a 401 (expired auth), pointlessly running a re-authentication flow. Unless you vary the strategy per status code, it doesn't matter how fast a model you use — it's wasted.\n\nMaintaining an autonomous environment at ¥1.2M/month revenue, what I noticed is that the time spent on \"mechanisms that keep things from stopping\" has a higher long-term ROI than the code added to increase earnings.\n\nBack at ¥600K/month I thought \"more tasks means more earnings.\" After being laid off and dropping to zero, my thinking changed while rebuilding it from scratch. Adding tasks doesn't help if the environment is unstable — throughput pins to a ceiling. Conversely, killing a single infrastructure-level problem raises the completion rate across all existing tasks.\n\nThe MCP health check is the classic example of this. By wiring `~/.claude/scripts/hooks/mcp-health-check.js`\n\ninto a hook, an HTTP probe runs before Claude Code calls a tool, and depending on the response status it dispatches to \"block immediately,\" \"retry after backoff,\" or \"run a reconnect command and re-probe.\" The verdict is persisted to `~/.claude/mcp-health-cache.json`\n\n, so even when context is compacted, the health record carries over.\n\nWhen a Claude Code session runs for a long time, the conversation history gets compacted. Even if there was information in the past saying \"this server was down,\" that doesn't survive into the post-compaction context. The result is waste: repeatedly attempting tool calls against a server already known to be unhealthy, and receiving an error each time.\n\nA file-based cache is independent of the context. `~/.claude/mcp-health-cache.json`\n\ndoesn't disappear no matter how much the session is compacted. When the health-check hook runs on the next turn, it loads the previous state from the file, and until `nextRetryAt`\n\nhas passed it blocks immediately without even re-probing. The idea of holding state *outside the context window* is what matters fundamentally.\n\nSome people think, \"why not just do error handling in Claude's prompt?\" I actually tried it. A system prompt saying \"if this tool errors, try another approach\" works reasonably well for a one-off error. But in a situation where the MCP server is down and failures happen back to back, the model burns a huge number of tokens on \"trying.\" And on the next turn, it goes right back to calling the same server. Writing the fact that \"the server is dead\" outside the context and blocking at the hook level is overwhelmingly cleaner.\n\nAnother common misconception is \"MCP's own retry settings are enough.\" The MCP protocol has transport-layer retries, but it has no feature for reading status codes and switching strategy. 429 and 503 call for different backoff durations, and 401/403 are cases that need re-authentication rather than a retry. Implementing this dispatch at the application layer is what this hook is for.\n\n`mcp-health-check.js`\n\nresponds to two kinds of Claude Code hook events. It's written verbatim in the comment at the top of the code (lines 7–12).\n\n```\n- PreToolUse: probe MCP server health before MCP tool execution\n- PostToolUseFailure: mark unhealthy servers, attempt reconnect, and re-probe\n```\n\n`PreToolUse`\n\nis called before the tool is executed. Here it fires a probe, checks whether the server is alive, and decides whether to allow execution or block with exit code 2. `PostToolUseFailure`\n\nis called after a tool returns an error. It parses the error text to identify the failure pattern, marks the server unhealthy, and attempts a reconnect.\n\n```\nClaude Code が mcp__* ツールを呼ぶ\n        │\n        ▼ PreToolUse フック起動\n┌──────────────────────────────────────────────────────┐\n│            mcp-health-check.js                       │\n│                                                      │\n│  ① mcp-health-cache.json を読む                      │\n│     status=healthy かつ expiresAt が未来？            │\n│     YES ─────────────────────────────────────────→  │ exit 0\n│     NO  ↓                                            │ （ツール実行へ）\n│                                                      │\n│  ② nextRetryAt が未来（unhealthy クールダウン中）？   │\n│     YES → ブロック ──────────────────────────────→  │ exit 2\n│     NO  ↓                                            │ （ツールをスキップ）\n│                                                      │\n│  ③ プローブ実行                                       │\n│     HTTPサーバー → GET リクエスト（5秒タイムアウト）  │\n│     stdioサーバー → プロセス起動（5秒生存確認）       │\n│                                                      │\n│     レスポンスのステータスコード判定                   │\n│     ┌──────────────────────────────────────────┐    │\n│     │ ECONNREFUSED / ENOTFOUND / タイムアウト   │──→ │\n│     │  → 即 markUnhealthy & exit 2             │    │\n│     ├──────────────────────────────────────────┤    │\n│     │ 401 / 403 / 429 / 503                    │──→ │\n│     │  → reconnect コマンドを実行               │    │\n│     │  → 成功すれば再プローブ                   │    │\n│     │  → 再プローブ OK → markHealthy & exit 0  │    │\n│     │  → 再プローブ NG → markUnhealthy & exit 2│    │\n│     ├──────────────────────────────────────────┤    │\n│     │ 200 系 / 400 / 401 / 403 / 405 / 406    │    │\n│     │（\"到達できた\"証明として healthy 扱い）    │──→ │ exit 0\n│     └──────────────────────────────────────────┘    │\n│                                                      │\n│  ④ 状態を mcp-health-cache.json に書き出す            │\n└──────────────────────────────────────────────────────┘\n        │\n        ▼ ツール実行後にエラーが出た場合\n┌──────────────────────────────────────────────────────┐\n│ PostToolUseFailure フック                             │\n│   エラーテキストを FAILURE_PATTERNS と照合            │\n│   failureCode 特定 → markUnhealthy → reconnect試行  │\n│   → 再プローブ OK なら markHealthy                   │\n└──────────────────────────────────────────────────────┘\n```\n\nReading the actual code, the design intent shows up in the numbers (lines 22–26).\n\n``` js\nconst DEFAULT_TTL_MS    = 2 * 60 * 1000;   // 2分\nconst DEFAULT_TIMEOUT_MS = 5000;            // 5秒\nconst DEFAULT_BACKOFF_MS = 30 * 1000;       // 30秒（初回バックオフ）\nconst MAX_BACKOFF_MS     = 10 * 60 * 1000; // 10分（上限）\n```\n\nA TTL of 2 minutes is the trade-off between \"I don't want to run a probe every single time\" and \"I don't want to hold a stale result too long.\" In scenarios where Claude Code calls tools back to back, hitting the same server multiple times within 2 minutes is not unusual. With a cache, there's no need to run an HTTP probe each time, and latency drops.\n\nThe 5-second timeout is the threshold for confirming that a local stdio server \"can start as a process.\" Rather than startup completion, it judges normality by the process \"staying alive for 5 seconds\" (the timer logic in `probeCommandServer`\n\n, lines 428–472). Even for a heavy server that takes more than 5 seconds to start, this can be adjusted with the environment variable `ECC_MCP_HEALTH_TIMEOUT_MS`\n\n.\n\nThe backoff calculation in the `markUnhealthy`\n\nfunction (lines 213–229) is compressed into one line.\n\n``` js\nconst nextRetryDelay = Math.min(\n  backoffBase * (2 ** Math.max(failureCount - 1, 0)),\n  MAX_BACKOFF_MS\n);\n```\n\n`backoffBase`\n\ndefaults to 30 seconds. When `failureCount`\n\nis 1, `2 ** 0 = 1`\n\ngives 30 seconds; the second time `2 ** 1 = 2`\n\ngives 60 seconds; the third 120 seconds; the fourth 240 seconds, doubling each time, capping out at a maximum of 600 seconds (10 minutes). It waits 30 seconds after the first failure, and if it hasn't recovered there, widens the interval to 1 minute, 2 minutes, 4 minutes — which avoids the situation of endlessly firing pointless probes at a downed server.\n\nThe definition of HEALTHY_HTTP_CODES (line 32) looks odd at first glance.\n\n``` js\nconst HEALTHY_HTTP_CODES = new Set([\n  200, 201, 202, 204,\n  301, 302, 303, 304, 307, 308,\n  400, 401, 403, 405, 406\n]);\n```\n\nSome 400-level codes are treated as \"healthy.\" The reason is written in the code's comment (lines 29–32).\n\n```\n// The preflight HTTP probe only checks reachability; it does not have access to\n// Claude Code's stored OAuth bearer token. Treat auth-gated responses as\n// reachable so the real MCP client can attempt the authenticated call. A\n// Streamable HTTP MCP server can also return 406 to a bare GET that omits\n// Accept: text/event-stream; that still proves the endpoint is alive.\n```\n\nThe preflight probe is a GET request with no OAuth token. An endpoint that requires authentication returns 401 or 403, but that means it is \"correctly rejecting an unauthenticated request\" — evidence that the server is alive. 406 is a normal rejection response to a request without the `Accept: text/event-stream`\n\nheader. These are all codes that prove \"it was reachable.\"\n\nMeanwhile, the set targeted for reconnect is a subset (line 33).\n\n``` js\nconst RECONNECT_STATUS_CODES = new Set([401, 403, 429, 503]);\n```\n\n401 and 403 are treated as \"reachable\" in the probe context, but when they come back **after an actual tool call**, the meaning is different. When the `detectFailureCode`\n\nfunction is called in PostToolUseFailure, a 401/403 in the error message is taken as \"authentication failure\" and becomes the trigger for a reconnect. The key point is that the probe verdict and the post-failure verdict operate at different layers.\n\nMCP errors don't necessarily come back as HTTP status codes. Errors from stdio servers come back as text. `FAILURE_PATTERNS`\n\n(lines 34–40) is the mechanism that identifies the failure type from that text via regular expressions.\n\n``` js\nconst FAILURE_PATTERNS = [\n  { code: 401, pattern: /\\b401\\b|unauthori[sz]ed|auth(?:entication)?\\s+(?:failed|expired|invalid)/i },\n  { code: 403, pattern: /\\b403\\b|forbidden|permission denied/i },\n  { code: 429, pattern: /\\b429\\b|rate limit|too many requests/i },\n  { code: 503, pattern: /\\b503\\b|service unavailable|overloaded|temporarily unavailable/i },\n  { code: 'transport', pattern: /ECONNREFUSED|ENOTFOUND|EAI_AGAIN|timed? out|socket hang up|connection (?:failed|lost|reset|closed)/i }\n];\n```\n\nFrom error messages containing strings like \"unauthorized,\" \"auth expired,\" \"rate limit,\" or \"ECONNREFUSED,\" it automatically identifies the failure code. The `transport`\n\ncode isn't an HTTP status — it's the case where the network connection itself is cut. This is a state of \"can't even reach the server,\" where backoff takes priority over reconnect.\n\n`~/.claude/mcp-health-cache.json`\n\nis written out by the `saveState`\n\nfunction (lines 95–102), formatted with JSON.stringify. The actual file has a structure like this.\n\n```\n{\n  \"version\": 1,\n  \"servers\": {\n    \"obsidian\": {\n      \"status\": \"healthy\",\n      \"checkedAt\": 1753666200000,\n      \"expiresAt\": 1753666320000,\n      \"failureCount\": 0,\n      \"lastError\": null,\n      \"lastFailureCode\": null,\n      \"nextRetryAt\": 1753666200000,\n      \"lastRestoredAt\": 1753666200000,\n      \"source\": \"~/.claude/settings.json\"\n    },\n    \"agentmemory\": {\n      \"status\": \"unhealthy\",\n      \"checkedAt\": 1753665900000,\n      \"expiresAt\": 1753665900000,\n      \"failureCount\": 3,\n      \"lastError\": \"ECONNREFUSED 127.0.0.1:3001\",\n      \"lastFailureCode\": \"transport\",\n      \"nextRetryAt\": 1753666020000,\n      \"lastRestoredAt\": null\n    }\n  }\n}\n```\n\nIt's a two-stage gate: if `expiresAt`\n\nis in the future, pass without re-probing; if `nextRetryAt`\n\nis in the future, block without even retrying. The more `failureCount`\n\naccumulates, the longer the backoff stretches, so a repeatedly-down server gets progressively wider attempt intervals. `lastRestoredAt`\n\nis a record of \"when it recovered,\" usable for after-the-fact uptime analysis.\n\nThe `source`\n\nfield is the path to the config file. The `configPaths`\n\nfunction (lines 54–72) searches in the order: the current directory's `.claude.json`\n\n→ the current directory's `.claude/settings.json`\n\n→ home's `.claude.json`\n\n→ home's `.claude/settings.json`\n\n, and the first one found is used. Even when you have per-project MCP configuration, the correct config is referenced.\n\nThe `reconnectCommand`\n\nfunction (lines 516–527) reads the command from environment variables.\n\n``` js\nconst key = `ECC_MCP_RECONNECT_${String(serverName).toUpperCase().replace(/[^A-Z0-9]/g, '_')}`;\nconst command = process.env[key] || process.env.ECC_MCP_RECONNECT_COMMAND || '';\n```\n\nIf the server name is `agentmemory`\n\n, it looks for an environment variable named `ECC_MCP_RECONNECT_AGENTMEMORY`\n\n. If not found, it uses the global fallback `ECC_MCP_RECONNECT_COMMAND`\n\n. If the command string contains `{server}`\n\n, it is expanded to the server name (lines 524–526).\n\nFor example, if you use PM2 as your process manager, you can configure it like this.\n\n```\nexport ECC_MCP_RECONNECT_COMMAND=\"pm2 restart {server}\"\n```\n\nWith this, when the `agentmemory`\n\nserver returns a 401 or 503, `pm2 restart agentmemory`\n\nruns automatically, and tool execution is allowed only after a re-probe confirms it's normal. When a reconnect succeeds, `reconnect-command`\n\nis recorded in `markHealthy`\n\n's `restoredBy`\n\nfield (lines 607–609).\n\nAt this point you should have a grasp of the hook's overall shape. In the next section I'll dig into the actual stumbling points and the traps that are easy to fall into during configuration.\n\nThe first thing the hook does is identify \"which MCP server is this call for.\" The `extractMcpTarget`\n\nfunction (lines 133–167) handles that.\n\n``` js\nif (!toolName.startsWith('mcp__')) {\n  return null;\n}\n\nconst segments = toolName.slice(5).split('__');\nif (segments.length < 2 || !segments[0]) {\n  return null;\n}\n\nreturn {\n  server: segments[0],\n  tool: segments.slice(1).join('__')\n};\n```\n\nGiven a name like `mcp__obsidian__search_notes`\n\n, `slice(5)`\n\nmakes it `obsidian__search_notes`\n\n, and `split('__')`\n\nmakes it `['obsidian', 'search', 'notes']`\n\n. `segments[0]`\n\nis the server name, and the rest joined with double underscores is the tool name.\n\nThat said, parsing the tool name is strictly a fallback. It looks for an explicit `server`\n\nfield first (lines 135–145).\n\n``` js\nconst explicitServer = input.server\n  || input.mcp_server\n  || input.tool_input?.server\n  || input.tool_input?.mcp_server\n  || input.tool_input?.connector\n  || null;\n```\n\nThe reason it walks multiple paths is that the hook event's schema can have fields in shifted positions depending on the Claude Code version and connection method (HTTP/stdio). Furthermore, if JSON parsing fails and the `truncated`\n\nflag is set, `extractMcpTargetFromRaw`\n\n(lines 169–179) applies a regular expression to the raw string to extract the same information. It's a design that doesn't give up even when parsing fails.\n\n`handlePreToolUse`\n\n(lines 567–630) looks simple, but two cache checks run in succession.\n\n```\n// 第1ゲート: healthy かつキャッシュ有効 → プローブなしで通過\nif (previous.status === 'healthy' && Number(previous.expiresAt || 0) > now) {\n  return { rawInput, exitCode: 0, logs };\n}\n\n// 第2ゲート: unhealthy かつクールダウン中 → プローブなしでブロック\nif (previous.status === 'unhealthy' && Number(previous.nextRetryAt || 0) > now) {\n  logs.push(\n    `[MCPHealthCheck] ${target.server} is marked unhealthy until ${new Date(previous.nextRetryAt).toISOString()}; skipping ${target.tool || 'tool'}`\n  );\n  return { rawInput, exitCode: shouldFailOpen() ? 0 : 2, logs };\n}\n```\n\n\"healthy and within TTL\" passes, \"unhealthy and in backoff\" is blocked, and only in neither case (unhealthy but backoff expired, or the cache is empty) is a probe executed. In sessions that call tools at high frequency, almost all calls pass at the first gate, so HTTP request latency is effectively zero.\n\nThe conditions that trigger a reconnect need similar care (lines 601–603).\n\n``` js\nlet reconnect = { attempted: false, success: false, reason: 'probe failed' };\nif (probe.failureCode || previous.status === 'unhealthy') {\n  reconnect = attemptReconnect(target.server);\n```\n\nA reconnect doesn't run just because \"the probe failed.\" It requires either `probe.failureCode`\n\n(a status code for which reconnect is enabled) or the condition \"already unhealthy since last time.\" Running a reconnect command every time when you can't even connect due to ECONNREFUSED is wasteful, but when the server exists yet is unusable — as with 429 or 503 — actively attempting to reconnect is worthwhile; that judgment is embedded here.\n\nFor an HTTP server, firing a GET with `requestHttp`\n\ntells you what you need. How do you check a stdio server? The approach in `probeCommandServer`\n\n(lines 301–481) is distinctive.\n\n**Actually start the process and watch it for 5 seconds. If it stays alive, it's healthy; if it exits before then, it's not.**\n\n``` js\ntimer = setTimeout(() => {\n  // タイムアウト到達 = 5秒間プロセスが生存 → 正常\n\n  // ただし: ロードされたマシンではexitイベントがタイマーより\n  // わずかに遅れて届くことがある。高速クラッシュを見逃さないため\n  // プロセスの状態を再確認する\n  if (child.exitCode !== null || child.signalCode !== null) {\n    attemptFinish({\n      ok: false,\n      statusCode: child.exitCode,\n      reason: stderr.trim() || `process exited before handshake (...)`\n    });\n    return;\n  }\n\n  // SIGTERMで終了させ、200ms後にSIGKILLで確実に始末する\n  child.kill('SIGTERM');\n  setTimeout(() => {\n    try { child.kill('SIGKILL'); } catch { /* ignore */ }\n  }, 200).unref?.();\n\n  attemptFinish({\n    ok: true,\n    statusCode: null,\n    reason: `${serverName} accepted a new stdio process`\n  });\n}, timeoutMs);\n```\n\nAs the comment says, the \"fast-crashing server\" problem really is a nuisance. If the process exits right after startup, the `exit`\n\nevent and the timer callback can arrive at nearly the same moment. If the timer runs first, it would return `ok: true`\n\nwithout a `child.exitCode`\n\ncheck. As a countermeasure, the timer callback re-checks `child.exitCode !== null`\n\n, and if it's already dead, drops it to `ok: false`\n\n.\n\nWindows behavior is worked out too (lines 329–334, 438–461). A command without an extension, like `npx`\n\n, needs to resolve to `npx.cmd`\n\non Windows, and on top of that, from Node 18.20 onward, executing `.cmd`\n\nvia a shell was restricted as part of the CVE-2024-27980 fix. The code lines up fallbacks of `command.cmd`\n\n/ `command.exe`\n\n/ `command.bat`\n\nin a `candidates`\n\narray, and tries the next candidate in order on ENOENT. It also executes `.cmd`\n\n/`.bat`\n\nfiles via a shell, but has a safety valve that refuses shell execution if the command string contains shell metacharacters (`&|<>^%()`\n\netc.) (line 339, `UNSAFE_SHELL_CHARS`\n\n).\n\n`failureSummary`\n\n(lines 231–244) is quietly important.\n\n``` js\nconst pieces = [\n  typeof input.error === 'string' ? input.error : '',\n  typeof input.message === 'string' ? input.message : '',\n  typeof input.tool_response === 'string' ? input.tool_response : '',\n  typeof output === 'string' ? output : '',\n  typeof output?.output === 'string' ? output.output : '',\n  typeof output?.stderr === 'string' ? output.stderr : '',\n  typeof input.tool_input?.error === 'string' ? input.tool_input.error : ''\n].filter(Boolean);\n\nreturn pieces.join('\\n');\n```\n\nWhich field the MCP error text lands in differs by server implementation and Claude Code version. The case where it's in `error`\n\n, the case where it's in `tool_response`\n\n, the case where it's in `output.stderr`\n\n— I've encountered all of them in practice. By passing the string joined from all fields with `pieces.join('\\n')`\n\nto `detectFailureCode`\n\n, the regular expressions can catch the error text no matter which field it's in.\n\n`shouldFailOpen`\n\n(lines 557–559) is a one-liner over an environment variable.\n\n```\nfunction shouldFailOpen() {\n  return /^(1|true|yes)$/i.test(String(process.env.ECC_MCP_HEALTH_FAIL_OPEN || ''));\n}\n```\n\nIt's the setting that **lets through** tool calls to unhealthy servers. You use it during development when you want to \"have the hook running but not blocking, just to observe.\" In `PreToolUse`\n\n, it returns `exit 0`\n\n(pass) instead of `exit 2`\n\n(block).\n\nThe exitCode for `PostToolUseFailure`\n\nis always `0`\n\n(line 677). This is an important design point: the post-failure hook's purpose is **recording state**, not blocking tool execution. Changing the exit code of a tool call that has already failed is meaningless; it only writes logs and updates the cache, handing the state over to the next `PreToolUse`\n\n.\n\nFrom wiring this mechanism into my own environment to having it stable, I got stuck in three situations. I'll write the symptom, the cause, and the fix in order.\n\n**Symptom**: The `agentmemory`\n\nserver remained in the cache with `status: healthy`\n\n, yet actual tool calls returned `ECONNREFUSED`\n\n. The health check looked meaningless.\n\n**Cause**: A stdio server probe only confirms that \"the process stays alive for 5 seconds.\" The process starts, gets terminated with SIGTERM after 5 seconds, and then when the real tool call arrives, the server starts again from zero. But in a real startup, it takes time to complete the MCP handshake (JSON message exchange over stdio). A different error was occurring during that handshake.\n\nSpecifically, a server that tried to start without the environment variable `ANTHROPIC_API_KEY`\n\nbeing set was crashing right after startup. At probe time it stayed alive for 5 seconds so it got recorded as `healthy`\n\n, but during the actual handshake it would crash a few seconds in.\n\n**Fix**: Check first whether the server process's environment variables are all in place. `mcp-health-check.js`\n\nmerges `config.env`\n\ninto the process's environment variables when starting (lines 306–309). Confirming that the same environment variables as the production startup are in the config file's `env`\n\nsection is the first order of business. Only after understanding that \"healthy in the probe\" and \"healthy in the tool call\" are strictly different verdicts did the debugging direction finally become clear.\n\n**Symptom**: The Obsidian remote server temporarily returned 503s overnight. Checking the next morning, the server itself had long since recovered, but `nextRetryAt`\n\nin `mcp-health-cache.json`\n\npointed to a future time and it stayed blocked.\n\n**Cause**: Because 503 is included in `RECONNECT_STATUS_CODES`\n\n, a reconnect is attempted when a 503 comes in. But I had set neither `ECC_MCP_RECONNECT_OBSIDIAN`\n\nnor `ECC_MCP_RECONNECT_COMMAND`\n\n. Without a reconnect command, `attemptReconnect`\n\nreturns `attempted: false`\n\n(line 531), only `markUnhealthy`\n\nruns, and `failureCount`\n\naccumulates. Three 503s make `failureCount=3`\n\n, and backoff is `30 * (2 ** 2) = 120 seconds`\n\n. Five more over the next 30 minutes makes `failureCount=5`\n\nand `30 * (2 ** 4) = 480 seconds`\n\n. As a dozen-plus errors piled up overnight, the backoff was pinned at the ceiling of 600 seconds (10 minutes). Even after the server recovered, a probe only ran every 10 minutes, and all tool calls in between were blocked.\n\n**Fix**: Either configure a reconnect command, or set `ECC_MCP_HEALTH_BACKOFF_MS`\n\nto a small value (e.g. `10000`\n\n= 10 seconds) to lower the backoff base. I set `ECC_MCP_RECONNECT_COMMAND=echo noop`\n\n(a do-nothing dummy) in my launchd plist, which gets it treated as \"reconnect attempted\" and suppresses failureCount accumulation. The proper approach is to manage the service with PM2 or systemd and configure a real reconnect command.\n\n```\nexport ECC_MCP_RECONNECT_OBSIDIAN=\"pm2 restart obsidian-mcp\"\n```\n\nOnce this setting was in, the cycle started working: a 503 comes in → PM2 restarts → a re-probe confirms normal → immediate return to `healthy`\n\n.\n\n**Symptom**: When starting Claude Code in a particular project directory, the MCP health check logged \"No MCP config found for obsidian\" and skipped the probe, yet the tool call itself ran. The hook appeared to be functioning while not actually functioning.\n\n**Cause**: The `configPaths`\n\nfunction (lines 54–72) searches the current directory's config file first.\n\n```\nreturn [\n  path.join(cwd, '.claude.json'),\n  path.join(cwd, '.claude', 'settings.json'),   // ← これが先に見つかる\n  path.join(home, '.claude.json'),\n  path.join(home, '.claude', 'settings.json')\n];\n```\n\nThat project had a `.claude/settings.json`\n\n, and it defined only project-specific MCP servers (like `playwright`\n\n). `resolveServerConfig('obsidian')`\n\ndoesn't stop at the point of finding the first file in list order — if that file doesn't have `obsidian`\n\n, it moves on to the next file. In `readJsonFile`\n\n's loop, if `data?.mcpServers?.[serverName]`\n\nis null it returns `null`\n\nand continues the loop — but what actually caused the problem was a case where I had pointed the environment variable `ECC_MCP_CONFIG_PATH`\n\nat the home config file and there was a typo in that path.\n\n```\n# 誤（タイポあり）\nexport ECC_MCP_CONFIG_PATH=\"/Users/~/.claude/settings.json\"\n\n# 正\nexport ECC_MCP_CONFIG_PATH=\"${HOME}/.claude/settings.json\"\n```\n\n`path.resolve`\n\nreturned the nonexistent path `/Users/~/.claude/settings.json`\n\n, `readJsonFile`\n\nreturned null, and it was treated as config not found. Because setting `ECC_MCP_CONFIG_PATH`\n\ncompletely overrides the normal search paths (lines 55–61), a typo makes all server configuration disappear.\n\n**Fix**: Use an absolute path for `ECC_MCP_CONFIG_PATH`\n\n. Tilde expansion is something the shell does; Node's `path.resolve`\n\ndoes not do it. When writing it in a launchd plist, `~`\n\nis not expanded, so you need to write it as `/Users/your-home-directory`\n\n, `${HOME}`\n\n(via a shell script), or the equivalent of `os.homedir()`\n\n.\n\n``` php\n<!-- launchd plist では ~ が展開されない -->\n<key>ECC_MCP_CONFIG_PATH</key>\n<string>/Users/lily/.claude/settings.json</string>\n```\n\nWriting the actual home directory path literally is the most reliable.\n\n**Symptom**: The hook was working correctly and `unhealthy`\n\nwas recorded in the cache. Yet tool calls went through and errors flowed into Claude's context. It produced the token waste of receiving the same error 20 times in one session.\n\n**Cause**: I had forgotten to remove `ECC_MCP_HEALTH_FAIL_OPEN=1`\n\n, which I'd set during debugging, from my launchd plist. As long as this setting exists, `shouldFailOpen()`\n\nreturns `true`\n\nand all tool calls to unhealthy servers become `exit 0`\n\n(pass). The hook is running but blocking nothing.\n\nSo there was an inconsistency where looking at `mcp-health-cache.json`\n\nshowed `status: unhealthy`\n\nrecorded, and yet tool calls went through.\n\n**Fix**: Treat `ECC_MCP_HEALTH_FAIL_OPEN`\n\nas a debug-only flag and never write it in a permanent launchd plist. I made myself an operational rule to delete it as soon as debugging is over. To check whether the hook is blocking as intended, the quickest route is to read `~/.claude/mcp-health-cache.json`\n\ndirectly and, in a state where a server with `status: unhealthy`\n\nexists, confirm that the `skipping`\n\nwording appears in the hook's stderr log when a tool call runs.\n\n```\n# フックのログをリアルタイムで確認（stderrはClaude Codeのhookログに流れる）\ntail -f ~/.claude/logs/hooks.log | grep MCPHealthCheck\n```\n\nAfter getting past these four sticking points, the environment stabilized. Now it's normal for a batch running overnight to have finished 100 items by the time I wake up. Since introducing this mechanism, a batch has never once been stopped by an MCP server going down.\n\nThe previous section covered four sticking points in detail. Here I'll enumerate, as a list, the traps I actually hit or that are easy to overlook. Ones that overlap with P2 are omitted.\n\n`CLAUDE_HOOK_EVENT_NAME`\n\nisn't passed, all calls are processed as PreToolUseLine 704 of the code has `const eventName = process.env.CLAUDE_HOOK_EVENT_NAME || 'PreToolUse';`\n\n. If this environment variable is missing, event determination is pinned to PreToolUse. There are cases where a hook you meant to register as PostToolUseFailure was actually running as PreToolUse. The first step in isolating this is to dump `process.env`\n\nto stderr and check whether Claude Code is passing ENV correctly.\n\n`truncated`\n\nflag is set, and unless fail-open is on it becomes an immediate block`MAX_STDIN = 1024 * 1024`\n\n(line 22) is that limit. If, on a large tool call that stuffs an entire file's contents into tool_input, the hook input exceeds this value, it tries to identify the target server with the parse still incomplete. If the target can be identified, you can raise the limit with `ECC_HOOK_INPUT_MAX_BYTES`\n\n, but if it can't be parsed completely, it's blocked with `exit 2`\n\nunless `fail-open`\n\nis on (lines 695–701). Without knowing this behavior, it looks like the mysterious phenomenon of \"the hook falsely blocks only certain tools.\"\n\n`saveState`\n\nis designed to swallow errors, so write failures continue silently`saveState`\n\nat lines 95–102 catches all errors with try/catch and, as the comment says — \"Never block the hook on state persistence errors.\" — silently continues. Disk full, insufficient permissions, the `~/.claude/`\n\ndirectory being gone: in every case the hook keeps returning exit 0 or exit 2. But because the cache isn't written out, healthy verdicts disappear after the TTL and a probe runs every time. If you feel like \"for some reason an HTTP probe runs every time,\" start with a disk check via `df -h ~/.claude/`\n\nand a permission check via `ls -la ~/.claude/mcp-health-cache.json`\n\n.\n\n`resolveServerConfig`\n\n(lines 181–197) scans all files in configPaths() order and, at the point the server name is found, records that path in the `source`\n\nfield. If a server of the same name is defined both in a project-specific `.claude/settings.json`\n\nand in home's `~/.claude/settings.json`\n\n, the project side wins. You can check whether an unintended config is being used via the `source`\n\nfield in `~/.claude/mcp-health-cache.json`\n\n. 80% of \"I fixed the home config but it isn't reflected\" symptoms are this.\n\n`attemptReconnect`\n\n(lines 528–555) executes with `spawnSync(command, { shell: true, ... })`\n\n. Command chains containing `&&`\n\nor `;`\n\nwork fine, but command substitutions like `$(date)`\n\ncan get expanded unintentionally. Keeping the value of `ECC_MCP_RECONNECT_COMMAND`\n\nto a simple single command like `pm2 restart {server}`\n\nis safest. Using the `{server}`\n\nplaceholder (lines 524–526) saves you from having to set a separate environment variable per server.\n\n`markUnhealthy`\n\naccumulates `previous.failureCount + 1`\n\nevery time (line 215). Meanwhile, `markHealthy`\n\nforce-resets to `failureCount: 0`\n\n(lines 200–210). The problem is that when no reconnect command is configured and the backoff pins at the ceiling (600 seconds), you're made to wait 10 minutes until the next re-probe. All tool calls in that window are blocked. Even after a long-unstable server recovers, recovery is only confirmed every 10 minutes. The fastest way to clear this state is to manually delete the target server's entry from the cache file.\n\n``` python\n  python3 -c \"\n  import json, pathlib\n  p = pathlib.Path.home() / '.claude/mcp-health-cache.json'\n  s = json.loads(p.read_text())\n  s['servers'].pop('agentmemory', None)\n  p.write_text(json.dumps(s, indent=2))\n  print('reset done')\n  \"\n```\n\n`probeCommandServer`\n\n(lines 306–309) merges `config.env`\n\ninto `process.env`\n\nat process startup. In an interactive zsh session, `ANTHROPIC_API_KEY`\n\nis automatically inherited from `.zshrc`\n\n, but Claude started from launchd doesn't have the login shell's environment variables. Most cases of the symptom \"works when I try it by hand, fails in the overnight batch\" have their cause here. The fundamental countermeasure is to explicitly enumerate the required API keys in every server's `config.env`\n\nsection.\n\nBecause 429 is included in `RECONNECT_STATUS_CODES`\n\n, a reconnect runs when the probe returns 429 (line 33). Even if the reconnect command executes, if the underlying problem (rate limiting) isn't resolved, the re-probe also returns 429, `markUnhealthy`\n\nruns, and `failureCount`\n\nincreases. It's a vicious circle where the probe itself consumes the rate limit. To prevent this, you can either set `ECC_MCP_HEALTH_TTL_MS`\n\nlonger to reduce probe frequency, or change the target server's probe URL to a health endpoint that doesn't require authentication.\n\nThe return value of `handlePostToolUseFailure`\n\nis always `exitCode: 0`\n\n(line 677). This is correct by design — returning a block for a \"tool call that has already failed\" is too late. This hook's job is \"writing state so the next PreToolUse blocks,\" and the exit code has no meaning. Most reports of \"the PostToolUseFailure hook is running but nothing gets blocked\" are resolved by checking whether the cache is being read correctly on the next PreToolUse.\n\nOver six months of supporting a ¥1.2M/month autonomous environment with this hook, here are 12 things I stick to.\n\n**1. Always configure a reconnect command, even a dummy one**\n\nWhen RECONNECT_STATUS_CODES errors pile up without a reconnect command, `failureCount`\n\nsnowballs and the backoff pins at the ceiling. Access to an already-recovered server stays blocked for a long time. Under PM2 management, configure a real one; otherwise even just `echo noop`\n\ngets it treated as \"reconnect attempted\" and suppresses failureCount accumulation.\n\n```\n# PM2管理下なら本物を\nexport ECC_MCP_RECONNECT_COMMAND=\"pm2 restart {server}\"\n\n# とりあえず蓄積を止めるだけなら\nexport ECC_MCP_RECONNECT_COMMAND=\"echo noop\"\n```\n\n**2. Don't write ECC_MCP_HEALTH_FAIL_OPEN in your launchd plist**\n\nWhile this variable is set, `shouldFailOpen()`\n\n(lines 557–559) returns `true`\n\nand all tool calls to unhealthy servers become `exit 0`\n\n(pass). The hook looks like it's working while blocking nothing. Treat it as a debug-only flag and make an operational rule to remove it as soon as debugging is done.\n\n**3. Write an absolute path in ECC_MCP_CONFIG_PATH**\n\n`~`\n\nis not expanded in a launchd plist. Writing `/Users/your-home-directory-name/.claude/settings.json`\n\nliterally is the most reliable. `path.resolve`\n\nis a Node feature and does not perform the shell's tilde expansion (lines 55–61). A typo makes all server configuration disappear.\n\n**4. Register aliases for cache inspection, reset, and log monitoring**\n\nSo you can move immediately when a problem occurs, put aliases in `.zshrc`\n\n.\n\n```\nalias mcp-health=\"python3 -m json.tool ~/.claude/mcp-health-cache.json\"\nalias mcp-reset=\"echo '{\\\"version\\\":1,\\\"servers\\\":{}}' > ~/.claude/mcp-health-cache.json\"\nalias mcp-logs=\"tail -f ~/.claude/logs/hooks.log | grep MCPHealthCheck\"\n```\n\n**5. Explicitly list every API key a stdio server needs in config.env**\n\nClaude started from launchd doesn't have the login shell's environment variables. Enumerating all required keys in each server's `env`\n\nsection in `~/.claude/settings.json`\n\nmakes interactive behavior and batch behavior match.\n\n**6. Tune the TTL to match session usage frequency**\n\nThe default of 2 minutes (`DEFAULT_TTL_MS = 2 * 60 * 1000`\n\n, line 23) is a general-purpose value. In environments that call one server at high frequency, like overnight batches, extending it to 5–10 minutes greatly reduces probe count and raises throughput. During debugging, shrinking it to 30 seconds makes reproducing problems faster.\n\n```\n# 夜間バッチ環境\nexport ECC_MCP_HEALTH_TTL_MS=600000   # 10分\n\n# デバッグ中\nexport ECC_MCP_HEALTH_TTL_MS=30000    # 30秒\n```\n\n**7. Match the backoff base value to the server's characteristics**\n\nThe default 30 seconds assumes a local process restart. For an external API server that takes 1–5 minutes to recover, raising it to `ECC_MCP_HEALTH_BACKOFF_MS=60000`\n\n(1 minute) makes the recovery-check interval match reality. Conversely, if a local stdio server recovers in a few seconds, 10 seconds is plenty.\n\n**8. Manage important servers with PM2**\n\nWhen PM2's auto-restart combines with this hook's reconnect → re-probe cycle, recovery from transient failures becomes fully automated. The cycle is: the server crashes → PM2 restarts it → the next PreToolUse hook calls the reconnect command → PM2 returns success as \"already started\" → the re-probe confirms healthy → tool execution is allowed. Since moving to this combination, cases of \"the server was dead the next morning\" have dropped to zero.\n\n**9. Periodically check config-file resolution results via the source field**\n\nThe `source`\n\nfield on each entry in `~/.claude/mcp-health-cache.json`\n\nshows which config file the server configuration was read from. After adding a `.claude/settings.json`\n\nto a project directory, use this field to confirm the intended config is being used. Most of \"I fixed the home config but it doesn't work\" is resolved here.\n\n**10. Build the habit of watching the hook's logs in real time**\n\n`emitLogs`\n\n(lines 561–565) outputs logs to stderr. Because they flow into Claude Code's hook log, you can check them with `tail -f`\n\n. If the word `skipping`\n\nappears, blocking is working. If `connection restored`\n\nappears, the reconnect succeeded. With the habit of reading the logs, you can confirm in seconds whether the hook is behaving as intended.\n\n**11. Centralize hook registration in the home config**\n\nYou can write hooks in a project's `.claude/settings.json`\n\ntoo, but I strongly recommend writing the health-check hook only in home's `~/.claude/settings.json`\n\n. Scattering it per project creates holes like \"the hook doesn't take effect only in that project.\" To receive uniform protection across all projects, treat the home config as canonical.\n\n**12. When failureCount grows abnormally large, manually delete the entry to reset it**\n\nIf `markHealthy`\n\nis called it automatically resets to `failureCount: 0`\n\n, but when the backoff is too long and the path to markHealthy is closed off, manual deletion is fastest. Registering the python3 one-liner above as an alias lets you reset a specific server's state in 10 seconds.\n\nAfter being laid off and dropping to ¥0/month revenue, the thing I feared most in rebuilding my autonomous environment from scratch was the scenario of \"the batch I started at night is completely wiped out by morning.\" What most reliably prevents that isn't a stronger model or better-polished prompts — it's a mechanism that detects failures and records them outside the context.\n\nIf I summarize the design of `mcp-health-check.js`\n\nin one line: \"don't lose health information even when context is compacted.\" `mcp-health-cache.json`\n\nlives outside the session. The backoff calculation is handled by code. The model doesn't have to reason about \"this server might be down.\" The tokens and attention saved can go to the actual task.\n\nThe reason the ¥1.2M/month environment keeps *being* ¥1.2M/month is that the overnight batches don't fall over before morning. And they don't fall over because this hook is quietly doing its job.\n\nThe implementation is 721 lines, but the core comes down to three points.\n\n`MAX_BACKOFF_MS`\n\n's 10 minutes prevents total abandonmentOnce you understand these three points, you'll be able to tune it and isolate problems on your own.\n\nI've put the full picture of the mechanism, the breakdown of the ¥1.2M/month, and the 30-day procedure into a paid note.\n\n📕 [Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート](https://note.com/bokuwalily/n/n849b3a07784a)\n\n*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.\n\nFollow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*", "url": "https://wpnews.pro/news/four-traps-in-mcp-health-checking-what-broke-my-overnight-batches", "canonical_source": "https://dev.to/bokuwalily/four-traps-in-mcp-health-checking-what-broke-my-overnight-batches-1187", "published_at": "2026-08-22 00:00:06+00:00", "updated_at": "2026-08-22 00:13:59.156815+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "artificial-intelligence"], "entities": ["Claude Code", "MCP", "Obsidian"], "alternates": {"html": "https://wpnews.pro/news/four-traps-in-mcp-health-checking-what-broke-my-overnight-batches", "markdown": "https://wpnews.pro/news/four-traps-in-mcp-health-checking-what-broke-my-overnight-batches.md", "text": "https://wpnews.pro/news/four-traps-in-mcp-health-checking-what-broke-my-overnight-batches.txt", "jsonld": "https://wpnews.pro/news/four-traps-in-mcp-health-checking-what-broke-my-overnight-batches.jsonld"}}