Debugging a Python "Memory Leak" That Was Actually a Measurement Bug (ru_maxrss vs VmRSS)

A developer debugging a Python pipeline for building a RAG knowledge base discovered that a perceived memory leak was actually a measurement bug. The developer was using `ru_maxrss` from `resource.getrusage`, which returns the peak resident set size since process start, not the current RSS. After switching to reading `VmRSS` from `/proc/self/status`, the logs showed memory was stable at around 1.8GB after model load, and the developer also optimized encoding by batching chunks.

I was running a pipeline for building a RAG knowledge base: crawl web articles, split them into chunks, create embeddings, and push them into Qdrant. Then my memory usage logs started looking like this: 50/1287 pushed... RSS 1594MB ⚠️ RSS 1594MB exceeded the 1024MB limit — aborting Every time I resumed the process, the RSS number kept climbing. I was convinced the embedding loop was leaking memory. I sprinkled gc.collect calls around, added more del statements — nothing helped. The number never went down. Not once. Spoiler: nothing was leaking except my measurement method. Here's the RSS measurement code I started with: php import resource def rss mb - int: return resource.getrusage resource.RUSAGE SELF .ru maxrss // 1024 At a glance, it looks like it returns the current RSS. It doesn't. ru maxrss is the maximum resident set size since the process started . It's a peak value, a high-water mark. Once your process touches a memory peak — even for a moment — this number will never go down, no matter how much memory you free afterwards. Because it's a peak value, using it to monitor current memory leads you straight into misdiagnosis. Even when memory is actually being freed, your logs look like this: RSS 1200MB RSS 1500MB RSS 1800MB RSS 1800MB RSS 1800MB Looking at this log, memory appears to grow monotonically. In reality it only says "at some point, this process used 1800MB." The consequences: gc.collect appears to do nothing del appears to free nothingThat last one is exactly what hit me. For the current RSS on Linux, read VmRSS from /proc/self/status : php def rss mb - int: VmRSS = current RSS ru maxrss is a peak value — don't use it for live monitoring try: with open "/proc/self/status" as f: for line in f: if line.startswith "VmRSS:" : return int line.split 1 // 1024 kB - MB except OSError: Fallback for non-Linux / unreadable /proc import resource return resource.getrusage resource.RUSAGE SELF .ru maxrss // 1024 return 0 After the fix, the logs looked like this: 850/1105 pushed... RSS 1974MB 900/1105 pushed... RSS 1809MB 950/1105 pushed... RSS 1811MB 1000/1105 pushed... RSS 1862MB The number goes up and down now. It turned out memory was stable at around 1.8GB after model load . No explosion. No leak. It had been fine the whole time. There was a second problem hiding underneath. Right after loading the embedding model, RSS was already around 1.6GB. The model was intfloat/multilingual-e5-base — that's just what it costs to hold it in memory. So my 1GB RSS limit was dead on arrival. Two mistakes stacked on top of each other: ru maxrss was a current valueBefore suspecting a memory leak, check your baseline. While investigating, I found another inefficiency — chunks were being encoded one at a time: Before: one encode call per chunk for r in batch: vector = model.encode prefix + r "text" , normalize embeddings=True I switched to batched encoding, 50 chunks at a time: python import gc import torch PUSH BATCH = 50 prefix = "passage: " required prefix for e5-family models indexing side for batch start in range 0, len rows , PUSH BATCH : batch = rows batch start : batch start + PUSH BATCH texts = prefix + r "text" for r in batch with torch.no grad : vectors = model.encode texts, normalize embeddings=True, batch size=PUSH BATCH, for r, vector in zip batch, vectors : upsert collection=collection, vector=vector.tolist , payload=payload of r , del vectors, texts gc.collect if rss mb RSS LIMIT MB: print "RSS limit exceeded — aborting. Remaining chunks resume next run." break Batching amortizes the tokenize/forward overhead, and throughput improved significantly. Also note torch.no grad : inference doesn't need gradient buffers, and forgetting this is a way to get a real memory leak. intfloat/multilingual-e5-base is an e5-family model, and e5 models expect a prefix on every text. Indexing side uses "passage: " : prefix = "passage: " texts = prefix + r "text" for r in batch Query side uses "query: " : query vector = model.encode "query: " + query, normalize embeddings=True, Skip the prefixes and embeddings still get created — your search quality just quietly degrades. When the RSS limit is exceeded, the process stops instead of failing outright — and it can resume where it left off. The trick: record a qdrant id on every chunk that's been pushed. On the next run, only process the ones that haven't been: SELECT FROM chunks WHERE qdrant id IS NULL ORDER BY id; RAG ingestion jobs tend to grow in size, so building them "interruption-first" from the start pays off quickly. | Symptom | Root cause | Fix | |---|---|---| | RSS appears to grow monotonically | ru maxrss is a peak, not a current value | Read VmRSS from /proc/self/status | gc.collect doesn't lower the number | You're looking at a high-water mark | Measure current RSS separately | | RSS is high right after model load | That's the embedding model itself | Check the post-load baseline | | Embedding is slow | One encode call per chunk | Batch encode with batch size | | Interruption means starting over | Push state isn't persisted | Reprocess only qdrant id IS NULL | Before suspecting a memory leak, check whether the value you're measuring is a current value or a peak value . My mistake was reading the name ru maxrss and assuming "current RSS". It literally stands for maximum resident set size. The max was right there. Memory wasn't exploding — I was watching a high-water mark and calling it a leak. Get the measurement wrong and you'll chase bugs that don't exist. Suspect your metrics before you suspect your memory. That was the real takeaway. This is an English version of my Japanese article on Zenn.