# Debugging a Python "Memory Leak" That Was Actually a Measurement Bug (ru_maxrss vs VmRSS) > Source: > Published: 2026-07-09 15:24:10+00:00 I was running a pipeline for building a RAG knowledge base: crawl web articles, split them into chunks, create embeddings, and push them into Qdrant. Then my memory usage logs started looking like this: ``` 50/1287 pushed... (RSS 1594MB) ⚠️ RSS 1594MB exceeded the 1024MB limit — aborting ``` Every time I resumed the process, the RSS number kept climbing. I was convinced the embedding loop was leaking memory. I sprinkled `gc.collect()` calls around, added more `del` statements — nothing helped. The number never went down. Not once. Spoiler: **nothing was leaking except my measurement method.** Here's the RSS measurement code I started with: ``` php import resource def rss_mb() -> int: return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024 ``` At a glance, it looks like it returns the current RSS. It doesn't. `ru_maxrss` is the **maximum resident set size since the process started**. It's a peak value, a high-water mark. Once your process touches a memory peak — even for a moment — this number will never go down, no matter how much memory you free afterwards. Because it's a peak value, using it to monitor current memory leads you straight into misdiagnosis. Even when memory is actually being freed, your logs look like this: ``` RSS 1200MB RSS 1500MB RSS 1800MB RSS 1800MB RSS 1800MB ``` Looking at this log, memory appears to grow monotonically. In reality it only says "at some point, this process used 1800MB." The consequences: `gc.collect()` appears to do nothing`del` appears to free nothingThat last one is exactly what hit me. For the *current* RSS on Linux, read `VmRSS` from `/proc/self/status` : ``` php def rss_mb() -> int: # VmRSS = current RSS # ru_maxrss is a peak value — don't use it for live monitoring try: with open("/proc/self/status") as f: for line in f: if line.startswith("VmRSS:"): return int(line.split()[1]) // 1024 # kB -> MB except OSError: # Fallback for non-Linux / unreadable /proc import resource return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024 return 0 ``` After the fix, the logs looked like this: ``` 850/1105 pushed... (RSS 1974MB) 900/1105 pushed... (RSS 1809MB) 950/1105 pushed... (RSS 1811MB) 1000/1105 pushed... (RSS 1862MB) ``` The number goes up *and down* now. It turned out memory was **stable at around 1.8GB after model load**. No explosion. No leak. It had been fine the whole time. There was a second problem hiding underneath. Right after loading the embedding model, RSS was already around 1.6GB. The model was `intfloat/multilingual-e5-base` — that's just what it costs to hold it in memory. So my 1GB RSS limit was dead on arrival. Two mistakes stacked on top of each other: `ru_maxrss` was a current valueBefore suspecting a memory leak, check your baseline. While investigating, I found another inefficiency — chunks were being encoded one at a time: ``` # Before: one encode() call per chunk for r in batch: vector = model.encode(prefix + r["text"], normalize_embeddings=True) ``` I switched to batched encoding, 50 chunks at a time: ``` python import gc import torch PUSH_BATCH = 50 prefix = "passage: " # required prefix for e5-family models (indexing side) for batch_start in range(0, len(rows), PUSH_BATCH): batch = rows[batch_start : batch_start + PUSH_BATCH] texts = [prefix + r["text"] for r in batch] with torch.no_grad(): vectors = model.encode( texts, normalize_embeddings=True, batch_size=PUSH_BATCH, ) for r, vector in zip(batch, vectors): upsert( collection=collection, vector=vector.tolist(), payload=payload_of(r), ) del vectors, texts gc.collect() if rss_mb() > RSS_LIMIT_MB: print("RSS limit exceeded — aborting. Remaining chunks resume next run.") break ``` Batching amortizes the tokenize/forward overhead, and throughput improved significantly. Also note `torch.no_grad()` : inference doesn't need gradient buffers, and forgetting this is a way to get a *real* memory leak. `intfloat/multilingual-e5-base` is an e5-family model, and e5 models expect a prefix on every text. Indexing side uses `"passage: "` : ``` prefix = "passage: " texts = [prefix + r["text"] for r in batch] ``` Query side uses `"query: "` : ``` query_vector = model.encode( "query: " + query, normalize_embeddings=True, ) ``` Skip the prefixes and embeddings still get created — your search quality just quietly degrades. When the RSS limit is exceeded, the process stops instead of failing outright — and it can resume where it left off. The trick: record a `qdrant_id` on every chunk that's been pushed. On the next run, only process the ones that haven't been: ``` SELECT * FROM chunks WHERE qdrant_id IS NULL ORDER BY id; ``` RAG ingestion jobs tend to grow in size, so building them "interruption-first" from the start pays off quickly. | Symptom | Root cause | Fix | |---|---|---| | RSS appears to grow monotonically | `ru_maxrss` is a peak, not a current value | Read `VmRSS` from `/proc/self/status` | `gc.collect()` doesn't lower the number | You're looking at a high-water mark | Measure current RSS separately | | RSS is high right after model load | That's the embedding model itself | Check the post-load baseline | | Embedding is slow | One `encode()` call per chunk | Batch encode with `batch_size` | | Interruption means starting over | Push state isn't persisted | Reprocess only `qdrant_id IS NULL` | Before suspecting a memory leak, check whether the value you're measuring is a **current value or a peak value**. My mistake was reading the name `ru_maxrss` and assuming "current RSS". It literally stands for *maximum* resident set size. The `max` was right there. Memory wasn't exploding — I was watching a high-water mark and calling it a leak. Get the measurement wrong and you'll chase bugs that don't exist. Suspect your metrics before you suspect your memory. That was the real takeaway. *This is an English version of my Japanese article on Zenn.*