{"slug": "debugging-a-python-memory-leak-that-was-actually-a-measurement-bug-ru-maxrss-vs", "title": "Debugging a Python \"Memory Leak\" That Was Actually a Measurement Bug (ru_maxrss vs VmRSS)", "summary": "A developer debugging a Python pipeline for building a RAG knowledge base discovered that a perceived memory leak was actually a measurement bug. The developer was using `ru_maxrss` from `resource.getrusage`, which returns the peak resident set size since process start, not the current RSS. After switching to reading `VmRSS` from `/proc/self/status`, the logs showed memory was stable at around 1.8GB after model load, and the developer also optimized encoding by batching chunks.", "body_md": "I was running a pipeline for building a RAG knowledge base: crawl web articles, split them into chunks, create embeddings, and push them into Qdrant.\n\nThen my memory usage logs started looking like this:\n\n```\n  50/1287 pushed... (RSS 1594MB)\n  ⚠️ RSS 1594MB exceeded the 1024MB limit — aborting\n```\n\nEvery time I resumed the process, the RSS number kept climbing. I was convinced the embedding loop was leaking memory. I sprinkled `gc.collect()`\n\ncalls around, added more `del`\n\nstatements — nothing helped. The number never went down. Not once.\n\nSpoiler: **nothing was leaking except my measurement method.**\n\nHere's the RSS measurement code I started with:\n\n``` php\nimport resource\n\ndef rss_mb() -> int:\n    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024\n```\n\nAt a glance, it looks like it returns the current RSS.\n\nIt doesn't. `ru_maxrss`\n\nis the **maximum resident set size since the process started**. It's a peak value, a high-water mark.\n\nOnce your process touches a memory peak — even for a moment — this number will never go down, no matter how much memory you free afterwards.\n\nBecause it's a peak value, using it to monitor current memory leads you straight into misdiagnosis.\n\nEven when memory is actually being freed, your logs look like this:\n\n```\nRSS 1200MB\nRSS 1500MB\nRSS 1800MB\nRSS 1800MB\nRSS 1800MB\n```\n\nLooking at this log, memory appears to grow monotonically. In reality it only says \"at some point, this process used 1800MB.\"\n\nThe consequences:\n\n`gc.collect()`\n\nappears to do nothing`del`\n\nappears to free nothingThat last one is exactly what hit me.\n\nFor the *current* RSS on Linux, read `VmRSS`\n\nfrom `/proc/self/status`\n\n:\n\n``` php\ndef rss_mb() -> int:\n    # VmRSS = current RSS\n    # ru_maxrss is a peak value — don't use it for live monitoring\n    try:\n        with open(\"/proc/self/status\") as f:\n            for line in f:\n                if line.startswith(\"VmRSS:\"):\n                    return int(line.split()[1]) // 1024  # kB -> MB\n    except OSError:\n        # Fallback for non-Linux / unreadable /proc\n        import resource\n        return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024\n\n    return 0\n```\n\nAfter the fix, the logs looked like this:\n\n```\n  850/1105 pushed... (RSS 1974MB)\n  900/1105 pushed... (RSS 1809MB)\n  950/1105 pushed... (RSS 1811MB)\n  1000/1105 pushed... (RSS 1862MB)\n```\n\nThe number goes up *and down* now. It turned out memory was **stable at around 1.8GB after model load**. No explosion. No leak. It had been fine the whole time.\n\nThere was a second problem hiding underneath.\n\nRight after loading the embedding model, RSS was already around 1.6GB. The model was `intfloat/multilingual-e5-base`\n\n— that's just what it costs to hold it in memory.\n\nSo my 1GB RSS limit was dead on arrival. Two mistakes stacked on top of each other:\n\n`ru_maxrss`\n\nwas a current valueBefore suspecting a memory leak, check your baseline.\n\nWhile investigating, I found another inefficiency — chunks were being encoded one at a time:\n\n```\n# Before: one encode() call per chunk\nfor r in batch:\n    vector = model.encode(prefix + r[\"text\"], normalize_embeddings=True)\n```\n\nI switched to batched encoding, 50 chunks at a time:\n\n``` python\nimport gc\nimport torch\n\nPUSH_BATCH = 50\nprefix = \"passage: \"  # required prefix for e5-family models (indexing side)\n\nfor batch_start in range(0, len(rows), PUSH_BATCH):\n    batch = rows[batch_start : batch_start + PUSH_BATCH]\n    texts = [prefix + r[\"text\"] for r in batch]\n\n    with torch.no_grad():\n        vectors = model.encode(\n            texts,\n            normalize_embeddings=True,\n            batch_size=PUSH_BATCH,\n        )\n\n    for r, vector in zip(batch, vectors):\n        upsert(\n            collection=collection,\n            vector=vector.tolist(),\n            payload=payload_of(r),\n        )\n\n    del vectors, texts\n    gc.collect()\n\n    if rss_mb() > RSS_LIMIT_MB:\n        print(\"RSS limit exceeded — aborting. Remaining chunks resume next run.\")\n        break\n```\n\nBatching amortizes the tokenize/forward overhead, and throughput improved significantly.\n\nAlso note `torch.no_grad()`\n\n: inference doesn't need gradient buffers, and forgetting this is a way to get a *real* memory leak.\n\n`intfloat/multilingual-e5-base`\n\nis an e5-family model, and e5 models expect a prefix on every text.\n\nIndexing side uses `\"passage: \"`\n\n:\n\n```\nprefix = \"passage: \"\ntexts = [prefix + r[\"text\"] for r in batch]\n```\n\nQuery side uses `\"query: \"`\n\n:\n\n```\nquery_vector = model.encode(\n    \"query: \" + query,\n    normalize_embeddings=True,\n)\n```\n\nSkip the prefixes and embeddings still get created — your search quality just quietly degrades.\n\nWhen the RSS limit is exceeded, the process stops instead of failing outright — and it can resume where it left off.\n\nThe trick: record a `qdrant_id`\n\non every chunk that's been pushed. On the next run, only process the ones that haven't been:\n\n```\nSELECT *\nFROM chunks\nWHERE qdrant_id IS NULL\nORDER BY id;\n```\n\nRAG ingestion jobs tend to grow in size, so building them \"interruption-first\" from the start pays off quickly.\n\n| Symptom | Root cause | Fix |\n|---|---|---|\n| RSS appears to grow monotonically |\n`ru_maxrss` is a peak, not a current value |\nRead `VmRSS` from `/proc/self/status`\n|\n`gc.collect()` doesn't lower the number |\nYou're looking at a high-water mark | Measure current RSS separately |\n| RSS is high right after model load | That's the embedding model itself | Check the post-load baseline |\n| Embedding is slow | One `encode()` call per chunk |\nBatch encode with `batch_size`\n|\n| Interruption means starting over | Push state isn't persisted | Reprocess only `qdrant_id IS NULL`\n|\n\nBefore suspecting a memory leak, check whether the value you're measuring is a **current value or a peak value**.\n\nMy mistake was reading the name `ru_maxrss`\n\nand assuming \"current RSS\". It literally stands for *maximum* resident set size. The `max`\n\nwas right there.\n\nMemory wasn't exploding — I was watching a high-water mark and calling it a leak. Get the measurement wrong and you'll chase bugs that don't exist.\n\nSuspect your metrics before you suspect your memory.\n\nThat was the real takeaway.\n\n*This is an English version of my Japanese article on Zenn.*", "url": "https://wpnews.pro/news/debugging-a-python-memory-leak-that-was-actually-a-measurement-bug-ru-maxrss-vs", "canonical_source": "https://dev.to/flipslidersand/debugging-a-python-memory-leak-that-was-actually-a-measurement-bug-rumaxrss-vs-vmrss-46c1", "published_at": "2026-07-09 15:24:10+00:00", "updated_at": "2026-07-09 15:35:41.496963+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["Qdrant", "intfloat/multilingual-e5-base"], "alternates": {"html": "https://wpnews.pro/news/debugging-a-python-memory-leak-that-was-actually-a-measurement-bug-ru-maxrss-vs", "markdown": "https://wpnews.pro/news/debugging-a-python-memory-leak-that-was-actually-a-measurement-bug-ru-maxrss-vs.md", "text": "https://wpnews.pro/news/debugging-a-python-memory-leak-that-was-actually-a-measurement-bug-ru-maxrss-vs.txt", "jsonld": "https://wpnews.pro/news/debugging-a-python-memory-leak-that-was-actually-a-measurement-bug-ru-maxrss-vs.jsonld"}}