{"slug": "taming-the-cadvisor-leak-and-cleaning-up-llm-garbage", "title": "Taming the cadvisor leak and cleaning up LLM garbage", "summary": "Poindexter developers fixed a memory leak in cadvisor that caused an OOM cascade in the WSL2 VM, and patched a content generation bug where LLMs were pulling audit logs and session transcripts as context. The team also improved error handling for truncated LLM JSON and CLI network flukes, and added automated tests for onboarding documentation.", "body_md": "*What we shipped on 2026-07-01*\n\nWe spent most of today fighting an OOM cascade that nearly took down the WSL2 VM, starting with our attempt to cap cadvisor memory so it couldn’t starve the system (PR #2019). The proximate cause was a slow leak–cadvisor climbed to ~4 GB over 19 hours, eroding headroom until routine workload spikes triggered `[Errno 12] Cannot allocate memory`\n\nacross the board. We initially capped it at `512m`\n\n, but live measurement proved that too tight; a fresh restart immediately climbed to ~490 MB for the initial cgroup scan and metric registry, causing immediate OOM restarts. We’ve since raised the cap to `1g`\n\n(PR #2021), giving us 2x headroom over the measured working set while still keeping it 4x below the runaway peak that crashed the pipeline.\n\nWhile the infra was stabilizing, we caught a disturbing trend in our content generation where two tasks produced complete garbage–one essentially writing a Wikipedia article on punctuation marks and another leaking audit logs (PR #2016). The culprit wasn’t RAG pollution in the way we suspected, but a latent defect in `two_pass_writer._embed_and_fetch_snippets`\n\n. It was querying the entire embeddings table without a `source_table`\n\nfilter, meaning session transcripts and brain ops-logs were being pulled as context for content drafts. We’ve now scoped those snippets strictly to content sources and stripped out prompt-echo preambles.\n\nWe also tightened up the topic sweep pipeline after a `run_niche_topic_sweep`\n\njob failed due to truncated LLM JSON (PR #2020). The triage model hallucinated an explanation about podcast scripts, but the reality was simpler: the model hit a length cap and stopped mid-object. We’ve updated `llm_final_score`\n\nto handle `JSONDecodeError`\n\nby logging a warning and degrading gracefully to embedding pre-rank rather than bubbling the error up and killing the entire job.\n\nOn the CLI side, we chased a phantom OAuth credential error that was actually a network fluke (PR #2017). The “No CLI OAuth credentials configured” message was masking a `ConnectionResetError: [WinError 64]`\n\ncaused by the WSL2 host port-proxy resetting SSL handshakes with Postgres. We’ve separated these reports so we aren’t hunting for missing keys when the network is just flaky. We also fixed a `ModuleNotFoundError`\n\nwhere `pipeline resume`\n\nand `regen`\n\ncrashed outside the repo root because the editable install didn’t include the `brain/`\n\ndirectory in its `.pth`\n\nfile (PR #2011).\n\nFinally, we addressed some embarrassing drift in our public onboarding docs (PR #2018). A fresh `poindexter setup`\n\nwas hitting broken commands and un-pullable model defaults. To stop this from happening again, we added `test_package_metadata`\n\n, which parses the README’s fenced blocks and asserts that every command resolves against the Click command tree.\n\nThe VM is stable now, and the writer is finally staying on topic. We’re still tuning the balance between LLM autonomy and strict filtering, but the system is no longer eating itself from the inside out.\n\n*Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.*", "url": "https://wpnews.pro/news/taming-the-cadvisor-leak-and-cleaning-up-llm-garbage", "canonical_source": "https://www.gladlabs.io/posts/taming-the-cadvisor-leak-and-cleaning-up-llm-garba-3361e7c5", "published_at": "2026-07-01 20:01:06+00:00", "updated_at": "2026-07-13 21:09:17.201796+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Poindexter", "cadvisor", "WSL2", "Postgres", "Click"], "alternates": {"html": "https://wpnews.pro/news/taming-the-cadvisor-leak-and-cleaning-up-llm-garbage", "markdown": "https://wpnews.pro/news/taming-the-cadvisor-leak-and-cleaning-up-llm-garbage.md", "text": "https://wpnews.pro/news/taming-the-cadvisor-leak-and-cleaning-up-llm-garbage.txt", "jsonld": "https://wpnews.pro/news/taming-the-cadvisor-leak-and-cleaning-up-llm-garbage.jsonld"}}