{"slug": "i-added-retry-logic-to-my-sqlite-failure-library-here-s-the-exponential-backoff", "title": "I Added Retry Logic to My SQLite Failure Library. Here's the Exponential Backoff Pattern That Works.", "summary": "A developer added retry logic with exponential backoff and jitter to an SQLite failure library to handle 'database is locked' errors under concurrent writes. The solution combines a retry wrapper, connection pooling, and WAL journal mode to reduce write contention in production.", "body_md": "My SQLite failure library started throwing \"database is locked\" errors under concurrent load. Multiple agents writing failures simultaneously caused write contention. I needed retry logic — but not the dumb kind.\n\nThe problem: SQLite's default timeout is 5 seconds. Under heavy concurrent writes, agents timeout before the lock is released. The fix is a retry wrapper with exponential backoff and jitter:\n\n``` python\nimport sqlite3\nimport time\nimport random\nfrom typing import Optional\n\nDB_PATH = Path.home() / \".mcp\" / \"failures.db\"\n\ndef execute_with_retry(func, max_retries=5, base_delay=0.1):\n    \"\"\"Execute a database function with exponential backoff retry.\"\"\"\n    for attempt in range(max_retries):\n        try:\n            return func()\n        except sqlite3.OperationalError as e:\n            if \"database is locked\" not in str(e):\n                raise\n            if attempt == max_retries - 1:\n                raise\n            # Exponential backoff with jitter\n            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.05)\n            time.sleep(delay)\n    return func()\n\ndef log_failure(task: str, error: str, attempted_fix: str, result: str, env: str = None):\n    \"\"\"Log a failure with retry logic.\"\"\"\n    def _insert():\n        conn = sqlite3.connect(str(DB_PATH), timeout=30)\n        conn.execute(\n            \"INSERT INTO failures (task, error, attempted_fix, result, timestamp, env) VALUES (?, ?, ?, ?, ?, ?)\",\n            (task, error, attempted_fix, result, time.time(), env)\n        )\n        conn.commit()\n        conn.close()\n\n    execute_with_retry(_insert, max_retries=5, base_delay=0.1)\n```\n\nThe key parameters:\n\nExponential backoff works because it spreads out retry attempts. When multiple agents hit the same lock simultaneously, they don't all retry at the same time. The jitter ensures they retry at slightly different intervals.\n\nThe timeout parameter is the first line of defense — SQLite itself waits for the lock before raising an error. The retry wrapper handles the case where the timeout expires.\n\nI also added a connection pool to reduce the number of simultaneous connections:\n\n``` python\nclass ConnectionPool:\n    def __init__(self, db_path, pool_size=5):\n        self.db_path = db_path\n        self.pool_size = pool_size\n        self._pool = []\n\n    def get(self):\n        if self._pool:\n            return self._pool.pop()\n        return sqlite3.connect(str(self.db_path), timeout=30)\n\n    def put(self, conn):\n        if len(self._pool) < self.pool_size:\n            self._pool.append(conn)\n        else:\n            conn.close()\n\npool = ConnectionPool(DB_PATH)\n\ndef log_failure(task: str, error: str, attempted_fix: str, result: str, env: str = None):\n    conn = pool.get()\n    try:\n        conn.execute(\n            \"INSERT INTO failures (task, error, attempted_fix, result, timestamp, env) VALUES (?, ?, ?, ?, ?, ?)\",\n            (task, error, attempted_fix, result, time.time(), env)\n        )\n        conn.commit()\n    finally:\n        pool.put(conn)\nconn = sqlite3.connect(str(DB_PATH), timeout=30)\nconn.execute(\"PRAGMA journal_mode=WAL\")\n```\n\n**Busy timeout vs retry**: SQLite's `timeout`\n\nparameter handles the first wait. The retry wrapper handles cases where the timeout expires. Don't rely on just one.\n\n**Deadlock detection**: If you see consistent \"database is locked\" errors even with retries, you may have a deadlock. Add logging to identify which operations are holding locks:\n\n``` python\nimport traceback\ndef log_failure_with_deadlock_detection(...):\n    try:\n        _insert()\n    except sqlite3.OperationalError as e:\n        if \"database is locked\" in str(e):\n            print(f\"Deadlock detected: {traceback.format_exc()}\")\n        raise\n```\n\nHere's the complete retry + pool + WAL setup I'm running in production:\n\n``` python\nimport sqlite3\nimport time\nimport random\nfrom pathlib import Path\n\nDB_PATH = Path.home() / \".mcp\" / \"failures.db\"\n\ndef init_db():\n    conn = sqlite3.connect(str(DB_PATH), timeout=30)\n    conn.execute(\"PRAGMA journal_mode=WAL\")\n    conn.execute(\"PRAGMA busy_timeout=30000\")\n    conn.execute(\"\"\"\n        CREATE TABLE IF NOT EXISTS failures (\n            id INTEGER PRIMARY KEY AUTOINCREMENT,\n            task TEXT NOT NULL,\n            error TEXT NOT NULL,\n            attempted_fix TEXT,\n            result TEXT NOT NULL,\n            timestamp REAL NOT NULL,\n            env TEXT\n        )\n    \"\"\")\n    conn.execute(\"CREATE INDEX IF NOT EXISTS idx_task ON failures(task)\")\n    conn.execute(\"CREATE INDEX IF NOT EXISTS idx_timestamp ON failures(timestamp)\")\n    conn.commit()\n    conn.close()\n\nclass RetryConfig:\n    def __init__(self, max_retries=5, base_delay=0.1, max_delay=2.0, jitter=True):\n        self.max_retries = max_retries\n        self.base_delay = base_delay\n        self.max_delay = max_delay\n        self.jitter = jitter\n\ndef execute_with_retry(func, config=None):\n    if config is None:\n        config = RetryConfig()\n\n    last_error = None\n    for attempt in range(config.max_retries):\n        try:\n            return func()\n        except sqlite3.OperationalError as e:\n            last_error = e\n            if \"database is locked\" not in str(e):\n                raise\n            if attempt == config.max_retries - 1:\n                raise\n\n            delay = min(config.base_delay * (2 ** attempt), config.max_delay)\n            if config.jitter:\n                delay += random.uniform(0, 0.05)\n            time.sleep(delay)\n\n    raise last_error\n\nclass ConnectionPool:\n    def __init__(self, db_path, pool_size=5):\n        self.db_path = db_path\n        self.pool_size = pool_size\n        self._pool = []\n\n    def get(self):\n        if self._pool:\n            return self._pool.pop()\n        conn = sqlite3.connect(str(self.db_path), timeout=30)\n        conn.execute(\"PRAGMA journal_mode=WAL\")\n        conn.execute(\"PRAGMA busy_timeout=30000\")\n        return conn\n\n    def put(self, conn):\n        if len(self._pool) < self.pool_size:\n            self._pool.append(conn)\n        else:\n            conn.close()\n\npool = ConnectionPool(DB_PATH)\n\ndef log_failure(task: str, error: str, attempted_fix: str, result: str, env: str = None):\n    def _insert():\n        conn = pool.get()\n        try:\n            conn.execute(\n                \"INSERT INTO failures (task, error, attempted_fix, result, timestamp, env) VALUES (?, ?, ?, ?, ?, ?)\",\n                (task, error, attempted_fix, result, time.time(), env)\n            )\n            conn.commit()\n        finally:\n            pool.put(conn)\n\n    execute_with_retry(_insert)\n\n# Initialize on startup\ninit_db()\n```\n\nThis is what I run across 3 concurrent agents. Zero \"database is locked\" errors since deployment.\n\nHave you dealt with SQLite \"database is locked\" errors in production? What's your retry strategy — exponential backoff, fixed intervals, or something else? I'm curious whether a connection pool actually helps or if it just adds complexity.\n\nDrop a comment below — I read every response.", "url": "https://wpnews.pro/news/i-added-retry-logic-to-my-sqlite-failure-library-here-s-the-exponential-backoff", "canonical_source": "https://dev.to/chenyuan20509/i-added-retry-logic-to-my-sqlite-failure-library-heres-the-exponential-backoff-pattern-that-works-2bf6", "published_at": "2026-07-24 17:43:00+00:00", "updated_at": "2026-07-24 18:03:46.228923+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["SQLite"], "alternates": {"html": "https://wpnews.pro/news/i-added-retry-logic-to-my-sqlite-failure-library-here-s-the-exponential-backoff", "markdown": "https://wpnews.pro/news/i-added-retry-logic-to-my-sqlite-failure-library-here-s-the-exponential-backoff.md", "text": "https://wpnews.pro/news/i-added-retry-logic-to-my-sqlite-failure-library-here-s-the-exponential-backoff.txt", "jsonld": "https://wpnews.pro/news/i-added-retry-logic-to-my-sqlite-failure-library-here-s-the-exponential-backoff.jsonld"}}