I was running a pipeline for building a RAG knowledge base: crawl web articles, split them into chunks, create embeddings, and push them into Qdrant.
Then my memory usage logs started looking like this:
50/1287 pushed... (RSS 1594MB)
⚠️ RSS 1594MB exceeded the 1024MB limit — aborting
Every time I resumed the process, the RSS number kept climbing. I was convinced the embedding loop was leaking memory. I sprinkled gc.collect()
calls around, added more del
statements — nothing helped. The number never went down. Not once.
Spoiler: nothing was leaking except my measurement method.
Here's the RSS measurement code I started with:
import resource
def rss_mb() -> int:
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024
At a glance, it looks like it returns the current RSS.
It doesn't. ru_maxrss
is the maximum resident set size since the process started. It's a peak value, a high-water mark.
Once your process touches a memory peak — even for a moment — this number will never go down, no matter how much memory you free afterwards.
Because it's a peak value, using it to monitor current memory leads you straight into misdiagnosis.
Even when memory is actually being freed, your logs look like this:
RSS 1200MB
RSS 1500MB
RSS 1800MB
RSS 1800MB
RSS 1800MB
Looking at this log, memory appears to grow monotonically. In reality it only says "at some point, this process used 1800MB."
The consequences:
gc.collect()
appears to do nothingdel
appears to free nothingThat last one is exactly what hit me.
For the current RSS on Linux, read VmRSS
from /proc/self/status
:
def rss_mb() -> int:
try:
with open("/proc/self/status") as f:
for line in f:
if line.startswith("VmRSS:"):
return int(line.split()[1]) // 1024 # kB -> MB
except OSError:
import resource
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024
return 0
After the fix, the logs looked like this:
850/1105 pushed... (RSS 1974MB)
900/1105 pushed... (RSS 1809MB)
950/1105 pushed... (RSS 1811MB)
1000/1105 pushed... (RSS 1862MB)
The number goes up and down now. It turned out memory was stable at around 1.8GB after model load. No explosion. No leak. It had been fine the whole time.
There was a second problem hiding underneath.
Right after the embedding model, RSS was already around 1.6GB. The model was intfloat/multilingual-e5-base
— that's just what it costs to hold it in memory.
So my 1GB RSS limit was dead on arrival. Two mistakes stacked on top of each other:
ru_maxrss
was a current valueBefore suspecting a memory leak, check your baseline.
While investigating, I found another inefficiency — chunks were being encoded one at a time:
for r in batch:
vector = model.encode(prefix + r["text"], normalize_embeddings=True)
I switched to batched encoding, 50 chunks at a time:
import gc
import torch
PUSH_BATCH = 50
prefix = "passage: " # required prefix for e5-family models (indexing side)
for batch_start in range(0, len(rows), PUSH_BATCH):
batch = rows[batch_start : batch_start + PUSH_BATCH]
texts = [prefix + r["text"] for r in batch]
with torch.no_grad():
vectors = model.encode(
texts,
normalize_embeddings=True,
batch_size=PUSH_BATCH,
)
for r, vector in zip(batch, vectors):
upsert(
collection=collection,
vector=vector.tolist(),
payload=payload_of(r),
)
del vectors, texts
gc.collect()
if rss_mb() > RSS_LIMIT_MB:
print("RSS limit exceeded — aborting. Remaining chunks resume next run.")
break
Batching amortizes the tokenize/forward overhead, and throughput improved significantly.
Also note torch.no_grad()
: inference doesn't need gradient buffers, and forgetting this is a way to get a real memory leak.
intfloat/multilingual-e5-base
is an e5-family model, and e5 models expect a prefix on every text.
Indexing side uses "passage: "
:
prefix = "passage: "
texts = [prefix + r["text"] for r in batch]
Query side uses "query: "
:
query_vector = model.encode(
"query: " + query,
normalize_embeddings=True,
)
Skip the prefixes and embeddings still get created — your search quality just quietly degrades.
When the RSS limit is exceeded, the process stops instead of failing outright — and it can resume where it left off.
The trick: record a qdrant_id
on every chunk that's been pushed. On the next run, only process the ones that haven't been:
SELECT *
FROM chunks
WHERE qdrant_id IS NULL
ORDER BY id;
RAG ingestion jobs tend to grow in size, so building them "interruption-first" from the start pays off quickly.
| Symptom | Root cause | Fix |
|---|---|---|
| RSS appears to grow monotonically | ||
ru_maxrss is a peak, not a current value |
||
Read VmRSS from /proc/self/status |
||
gc.collect() doesn't lower the number |
||
| You're looking at a high-water mark | Measure current RSS separately | |
| RSS is high right after model load | That's the embedding model itself | Check the post-load baseline |
| Embedding is slow | One encode() call per chunk |
|
Batch encode with batch_size |
||
| Interruption means starting over | Push state isn't persisted | Reprocess only qdrant_id IS NULL |
Before suspecting a memory leak, check whether the value you're measuring is a current value or a peak value.
My mistake was reading the name ru_maxrss
and assuming "current RSS". It literally stands for maximum resident set size. The max
was right there.
Memory wasn't exploding — I was watching a high-water mark and calling it a leak. Get the measurement wrong and you'll chase bugs that don't exist.
Suspect your metrics before you suspect your memory.
That was the real takeaway.
This is an English version of my Japanese article on Zenn.