{"slug": "my-coding-agent-s-memory-hit-22-gb-in-git-the-bug-one-line-per-message", "title": "My coding agent's memory hit 22 GB in git. The bug? One line per message.", "summary": "A developer building Roam, a tool that syncs coding-agent sessions across machines via a private git repo, traced a 22 GB repository and failed GitHub pushes to per-message headings that rewrote on every capture, making every line of a 9 MB diary file appear changed and forcing git to store 1,425 full copies. The fix removed the message-count suffix from headings and split each session into append-only chunk files, cutting stored growth from over 10x to under 2x in a 15-capture test, and a related JavaScript substring-retention bug in search was fixed by copying strings, dropping index build memory from over 4 GB to 1.2 GB in 78 seconds.", "body_md": "So I build this tool called Roam. It saves my coding agent sessions (OpenCode, Claude Code, Codex, whatever I'm using that day) into a private git repo, so when I switch from my PC to my laptop I can just pick up where I left off. That's the whole idea.\n\nTwo weeks ago my sync just stopped. I went and checked the repo behind it. 22 GB. GitHub was timing out on every single push.\n\nAnd see, I don't have 22 GB of conversations. Nobody does. So where did it come from? That's what this post is about, and how I fixed it.\n\nEvery project had one diary file, `LOG.md`. Each agent session was a section inside it, and every message in the session had a heading like this:\n\n```\n### [3/57] user\n```\n\nMeaning message 3 of 57. Every time Roam captured the session again (when the agent goes idle, or every 10 minutes), it rewrote that whole section with the latest transcript.\n\nNow, git doesn't store \"the lines that changed\". It stores the whole file version every time. Usually that's fine, files are small, git compresses, nobody cares.\n\nTwo things stacked on top of each other.\n\n**First, the file was big.** One of my long running projects had a diary of around 9 MB.\n\n**Second, every capture changed every heading.** When message 58 comes in, `[3/57]` becomes `[3/58]`. And `[4/57]` becomes `[4/58]`. Every heading in the session changes, every single time. So even git's compression can't save you, because the whole 9 MB file is \"different\" on every save.\n\nThis is what I actually measured on my relay:\n\n| Size of one project's diary | ~9 MB | \n| Full copies of it stored in git | **1,425** | \n| Copies stored in one single day | 296 | \n| Total repo size | **22 GB** | \n\nThen GitHub started rejecting pushes with timeouts (HTTP 408). And the worst part? My tool kept saying \"saved\", because the local commit worked. The push didn't. So sync was silently dead for two days before I even noticed. That one's on me.\n\nThat \"of 57\" part was honestly just decoration. Nobody needed it. Remove it and a message heading never changes once it's written:\n\n```\n### [3] user\n```\n\nTiny change. But now old lines are byte for byte the same between captures, which is exactly what git wants.\n\nInstead of rewriting one big file, every session gets its own folder now:\n\n```\nsessions/<session-id>/\n  head.md        # title, keywords, summary (small, can change)\n  state.json     # what has been captured so far\n  c-000001.md    # chunk 1: messages 1-12\n  c-000002.md    # chunk 2: messages 13-20\n  ...\n```\n\nA capture only writes a new chunk with the messages that are actually new. Old chunks are never touched again, so git stores each message once. That's it. When a session hits 64 chunks they get merged into one so the folder doesn't explode. And the readers rebuild the same diary as before, so search and resume didn't need to change at all.\n\nI wrote a test for this: capture one growing session 15 times and measure what git actually stores. Before the fix it was more than 10x the size of the session. After, under 2x, and most of that is just git's own bookkeeping.\n\nThe whole 22 GB thing went unnoticed because a failed push looked exactly like a successful one. So now:\n\n`roam capture` exits with an error\nFunny part. Later I found basically the same bug in search. To show results, my search index kept small strings like section titles and keywords, cut out of each diary file. And in JavaScript, a small string cut out of a big one can keep the whole big string alive in memory. So those tiny titles were quietly holding every diary file in RAM. Building the index for my 4.8 GB history went past 4 GB of memory.\n\nCopying those strings properly, and reading old diaries one file at a time instead of as one giant string, brought it down to **1.2 GB in 78 seconds**. Search now takes 0.6 s for a rare word and around 5 s for a really common one. It used to take 47 s.\n\nRoam is free to try on Windows: [https://roam-rho-gold.vercel.app](https://roam-rho-gold.vercel.app) (docs and releases are here: [https://github.com/DeepxD-code/roam-releases](https://github.com/DeepxD-code/roam-releases)). If you've used git as a database and run into the same thing, tell me how you dealt with it, I'd really like to know.\n\n*I wrote this with help from an AI assistant. Every number is from my own measurements.*", "url": "https://wpnews.pro/news/my-coding-agent-s-memory-hit-22-gb-in-git-the-bug-one-line-per-message", "canonical_source": "https://dev.to/deepxdcode/my-coding-agents-memory-hit-22-gb-in-git-the-bug-one-line-per-message-33mb", "published_at": "2026-11-21 21:04:55+00:00", "updated_at": "2026-11-21 21:12:02.381278+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["Roam", "GitHub", "OpenCode", "Claude Code", "Codex"], "also_reported_by": [], "takeaway": "Append-only chunk files plus immutable message headings cut git storage growth from over 10x to under 2x of session size across 15 captures, and copying substrings before indexing cut search memory from over 4 GB to 1.2 GB.", "recommendation": {"advice": "Store agent session transcripts as append-only chunk files with headings that never change, and make failed pushes exit non-zero so sync failures aren't silent.", "playbook": "coding-agents", "playbook_url": "https://wpnews.pro/playbooks/coding-agents"}, "alternates": {"html": "https://wpnews.pro/news/my-coding-agent-s-memory-hit-22-gb-in-git-the-bug-one-line-per-message", "markdown": "https://wpnews.pro/news/my-coding-agent-s-memory-hit-22-gb-in-git-the-bug-one-line-per-message.md", "text": "https://wpnews.pro/news/my-coding-agent-s-memory-hit-22-gb-in-git-the-bug-one-line-per-message.txt", "jsonld": "https://wpnews.pro/news/my-coding-agent-s-memory-hit-22-gb-in-git-the-bug-one-line-per-message.jsonld"}}