{"slug": "i-upgraded-my-json-failure-library-to-sqlite-now-my-ai-agents-share-crash-fixes", "title": "I Upgraded My JSON Failure Library to SQLite. Now My AI Agents Share Crash Fixes Across Machines.", "summary": "A developer upgraded their AI agents' JSON failure library to SQLite to solve concurrent write corruption and enable cross-machine crash fix sharing. The migration uses a single SQLite file with proper locking and indexes, allowing agents on different machines to access the same failure database via a shared network drive.", "body_md": "The JSON failure library worked great — until my agents started running on multiple machines. Suddenly, crash fixes logged on one server were invisible to agents on another. Worse, concurrent writes from parallel agents corrupted the JSON file.\n\nI needed a real database. Here's how I upgraded to SQLite without rewriting the whole library.\n\nSQLite is the perfect upgrade path. It's a single file, supports concurrent readers, and handles writes with proper locking. Here's the migration:\n\n``` python\nimport sqlite3\nimport time\nimport platform\nfrom pathlib import Path\nfrom typing import Optional\n\nDB_PATH = Path.home() / \".mcp\" / \"failures.db\"\n\ndef init_db():\n    \"\"\"Create the failures table if it doesn't exist.\"\"\"\n    DB_PATH.parent.mkdir(parents=True, exist_ok=True)\n    conn = sqlite3.connect(str(DB_PATH), timeout=30)\n    conn.execute(\"\"\"\n        CREATE TABLE IF NOT EXISTS failures (\n            id INTEGER PRIMARY KEY AUTOINCREMENT,\n            task TEXT NOT NULL,\n            error TEXT NOT NULL,\n            attempted_fix TEXT NOT NULL,\n            result TEXT NOT NULL,\n            timestamp REAL NOT NULL,\n            env TEXT\n        )\n    \"\"\")\n    conn.execute(\"CREATE INDEX IF NOT EXISTS idx_task_result ON failures(task, result)\")\n    conn.execute(\"CREATE INDEX IF NOT EXISTS idx_timestamp ON failures(timestamp)\")\n    conn.commit()\n    conn.close()\n\ndef log_failure(task: str, error: str, attempted_fix: str, result: str, env: str = None):\n    \"\"\"Log a failure to the SQLite database.\"\"\"\n    conn = sqlite3.connect(str(DB_PATH), timeout=30)\n    conn.execute(\n        \"INSERT INTO failures (task, error, attempted_fix, result, timestamp, env) VALUES (?, ?, ?, ?, ?, ?)\",\n        (task, error, attempted_fix, result, time.time(), env)\n    )\n    conn.commit()\n    conn.close()\n\ndef check_failure_library(task: str, env: str = None) -> Optional[dict]:\n    \"\"\"Check if this task has a known failure with a verified fix.\"\"\"\n    conn = sqlite3.connect(str(DB_PATH), timeout=30)\n    conn.row_factory = sqlite3.Row\n    query = \"SELECT * FROM failures WHERE task = ? AND result = 'fixed'\"\n    params = [task]\n    if env:\n        query += \" AND (env = ? OR env IS NULL)\"\n        params.append(env)\n    query += \" ORDER BY timestamp DESC LIMIT 1\"\n    row = conn.execute(query, params).fetchone()\n    conn.close()\n    return dict(row) if row else None\n```\n\nThe key changes from the JSON version:\n\n`idx_task_result`\n\nmakes the lookup fast even with thousands of entries`timeout=30`\n\nprevents \"database is locked\" errors on WindowsSQLite solves three problems the JSON library couldn't:\n\n**Concurrent writes** — SQLite uses file-level locking. Multiple agents can read simultaneously, and writes are serialized automatically. No more corrupted JSON from race conditions. On Windows, SQLite uses `msvcrt`\n\ninternally for file locking; on Unix, it uses `fcntl`\n\n. You don't need to implement platform-specific locking yourself.\n\n**Cross-machine sharing** — Put the SQLite file on a shared network drive (NFS, SMB), and every agent on every machine reads the same failure library. No sync needed. This is the killer feature: a fix logged on your CI server is immediately available to agents running on your laptop.\n\n**Query flexibility** — With SQL, you can filter by environment, date range, or result type. The JSON library required loading the entire file into memory and filtering in Python. With 10,000+ entries, that difference is measurable.\n\nMoving from JSON to SQLite is a one-time migration. Here's how I did it without downtime:\n\n``` python\ndef migrate_json_to_sqlite(json_path: Path, db_path: Path):\n    \"\"\"One-time migration from JSON failures to SQLite.\"\"\"\n    if not json_path.exists():\n        init_db()\n        return\n\n    import json\n    init_db()\n\n    failures = json.loads(json_path.read_text())\n    conn = sqlite3.connect(str(db_path), timeout=30)\n    for f in failures:\n        conn.execute(\n            \"INSERT INTO failures (task, error, attempted_fix, result, timestamp, env) VALUES (?, ?, ?, ?, ?, ?)\",\n            (f[\"task\"], f[\"error\"], f[\"attempted_fix\"], f[\"result\"], f.get(\"timestamp\", time.time()), f.get(\"env\"))\n        )\n    conn.commit()\n    conn.close()\n\n    # Rename old JSON file as backup\n    json_path.rename(json_path.with_suffix(\".json.bak\"))\n```\n\nRun this once at startup. After migration, the JSON file is backed up but no longer used.\n\nHere's the updated MCPToolHandler with SQLite integration:\n\n``` python\nclass MCPToolHandler:\n    def __init__(self):\n        init_db()\n        self.env = f\"{platform.system()}|{platform.machine()}\"\n\n    async def execute_tool(self, tool_name: str, args: dict):\n        # Check failure library before executing\n        known_failure = check_failure_library(tool_name, env=self.env)\n        if known_failure:\n            # Apply the verified fix\n            args = self._apply_fix(tool_name, args, known_failure)\n        else:\n            args = args  # No known failure, use original args\n\n        try:\n            result = await self._call_tool(tool_name, args)\n            return result\n        except Exception as e:\n            # Log the failure\n            attempted_fix = self._suggest_fix(tool_name, e)\n            log_failure(tool_name, str(e), attempted_fix, \"pending\", env=self.env)\n\n            # Try the fix\n            try:\n                fixed_args = self._apply_fix(tool_name, args, known_failure)\n                result = await self._call_tool(tool_name, fixed_args)\n                log_failure(tool_name, str(e), attempted_fix, \"fixed\", env=self.env)\n                return result\n            except Exception as e2:\n                log_failure(tool_name, str(e), attempted_fix, \"failed\", env=self.env)\n                raise\n```\n\nThe guard `if known_failure:`\n\nprevents passing `None`\n\nto `_apply_fix`\n\n. The `log_failure`\n\ncall only marks `\"fixed\"`\n\nafter the retry succeeds — never before verification.\n\n```\nconn = sqlite3.connect(str(DB_PATH), timeout=30)\nconn.execute(\"PRAGMA journal_mode=WAL\")\n```\n\n**Network drives**: SQLite works on NFS/SMB but can be slow. For high-write scenarios, use a proper database server (PostgreSQL) instead. The `timeout=30`\n\nparameter is critical on network filesystems where lock acquisition can be delayed.\n\n**File size growth**: Failures accumulate. Add a cleanup job:\n\n``` python\ndef prune_old_failures(days=30):\n    cutoff = time.time() - days * 86400\n    conn = sqlite3.connect(str(DB_PATH), timeout=30)\n    conn.execute(\"DELETE FROM failures WHERE timestamp < ?\", (cutoff,))\n    conn.commit()\n    conn.close()\n```\n\n**Cross-platform locking**: SQLite handles file locking internally, but on Windows you may need to set `timeout=30`\n\nto avoid \"database is locked\" errors under heavy concurrent writes. SQLite uses `msvcrt.locking()`\n\non Windows and `fcntl.flock()`\n\non Unix — you don't need to implement this yourself, but the timeout prevents indefinite hangs.\n\n**VACUUM for performance**: After pruning old entries, reclaim disk space:\n\n```\nconn = sqlite3.connect(str(DB_PATH), timeout=30)\nconn.execute(\"VACUUM\")\nconn.commit()\nconn.close()\n```\n\nHave you migrated from file-based to database-based state in your agent infrastructure? What's your approach for cross-machine failure sharing — a shared database, a message queue, or something else entirely?\n\nI'm curious whether SQLite is the right choice for teams with dozens of agents, or if a proper database server becomes necessary at that scale.\n\nDrop a comment below — I read every response.", "url": "https://wpnews.pro/news/i-upgraded-my-json-failure-library-to-sqlite-now-my-ai-agents-share-crash-fixes", "canonical_source": "https://dev.to/chenyuan20509/i-upgraded-my-json-failure-library-to-sqlite-now-my-ai-agents-share-crash-fixes-across-machines-43hb", "published_at": "2026-07-24 16:22:27+00:00", "updated_at": "2026-07-24 16:33:57.685176+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning"], "entities": ["SQLite"], "alternates": {"html": "https://wpnews.pro/news/i-upgraded-my-json-failure-library-to-sqlite-now-my-ai-agents-share-crash-fixes", "markdown": "https://wpnews.pro/news/i-upgraded-my-json-failure-library-to-sqlite-now-my-ai-agents-share-crash-fixes.md", "text": "https://wpnews.pro/news/i-upgraded-my-json-failure-library-to-sqlite-now-my-ai-agents-share-crash-fixes.txt", "jsonld": "https://wpnews.pro/news/i-upgraded-my-json-failure-library-to-sqlite-now-my-ai-agents-share-crash-fixes.jsonld"}}