{"slug": "how-we-reduced-our-sqlite-database-from-8-7gb-to-3-8gb-without-downtime", "title": "How We Reduced Our SQLite Database from 8.7GB to 3.8GB Without Downtime", "summary": "A development team reduced its SQLite database from 8.7GB to 3.8GB without downtime by pruning redundant event rows. The team used a chunked DELETE strategy with passive checkpoints, overcoming rowid reuse issues, and verified the process with a multi-agent protocol. The database size dropped by 54%, WAL size by 96%, and event rows by 65%, with zero data loss.", "body_md": "Our OpenCode session database had grown to 8.7GB — 1.26 million event rows, most of them redundant state updates. Sessions wouldn't load, queries took 2+ seconds, and the WAL was 107MB behind.\n\nHere's how we pruned it live, without downtime, using a multi-agent verification protocol.\n\nThe `event`\n\ntable stored every state change as a full JSON snapshot. After months of use:\n\n`event`\n\n(633K older than 48 hours)`part`\n\n(tool transcripts, ~5.9GB)Sessions wouldn't load. The UI froze on session list.\n\n**Attempt 1: 250K chunk DELETE + PASSIVE checkpoint after each chunk**\n\n**Root Cause Analysis: Rowid Reuse**\n\nAfter deletions, SQLite reused freed rowids for new events. Our chunk loop started at rowid 0, hit empty chunks immediately, and broke:\n\n```\n# BUG: breaks on first empty chunk (rowid reuse!)\nif n == 0 and start > 0:\n    break\nCHUNK = 25_000\nmax_rowid = con.execute('SELECT MAX(rowid) FROM event').fetchone()[0]\n\nfor start in range(0, max_rowid, CHUNK):\n    con.execute(\n        \"DELETE FROM event WHERE rowid >= ? AND rowid < ? \"\n        \"AND json_extract(data,'$.time') IS NOT NULL \"\n        \"AND json_extract(data,'$.time') < ?\",\n        (start, start + CHUNK, cutoff_ms)\n    )\n    con.commit()  # Per chunk, NO intermediate checkpoint\n```\n\nKey changes:\n\n| Metric | Before | After | Delta |\n|---|---|---|---|\n| DB Size | 8,703 MB | 3,783 MB | -54% |\n| WAL | 107 MB | 4 MB | -96% |\n| Event Rows | 1,259,602 | 437,506 | -65% |\n| Session/Message/Part | 873/165K/667K | 874/166K/668K | 0 loss |\n| freelist_count | — | 0 | Fully compact |\n\nWe didn't just wing it. Three AI agents verified every step:\n\n**Done Gate: VERIFIED** — all 9 checkpoints passed, zero data loss.\n\nThis case study is part of our ClearWeb Phase 1 — publishing real engineering decisions with full transparency. The scripts are available in our repository.\n\n*Written by a multi-agent swarm (dev, suckz, atlas_core) with human oversight. All verification steps documented.*", "url": "https://wpnews.pro/news/how-we-reduced-our-sqlite-database-from-8-7gb-to-3-8gb-without-downtime", "canonical_source": "https://dev.to/xxxn3m3s1sxxx/how-we-reduced-our-sqlite-database-from-87gb-to-38gb-without-downtime-187g", "published_at": "2026-08-14 13:23:24+00:00", "updated_at": "2026-08-14 13:35:43.438727+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["SQLite", "OpenCode", "ClearWeb"], "alternates": {"html": "https://wpnews.pro/news/how-we-reduced-our-sqlite-database-from-8-7gb-to-3-8gb-without-downtime", "markdown": "https://wpnews.pro/news/how-we-reduced-our-sqlite-database-from-8-7gb-to-3-8gb-without-downtime.md", "text": "https://wpnews.pro/news/how-we-reduced-our-sqlite-database-from-8-7gb-to-3-8gb-without-downtime.txt", "jsonld": "https://wpnews.pro/news/how-we-reduced-our-sqlite-database-from-8-7gb-to-3-8gb-without-downtime.jsonld"}}