RAGpipeline. When your vector embeddings grow into the millions, keeping everything in-memory via HNSW (Hierarchical Navigable Small World) becomes financially unsustainable. The trade-off is always latency versus budget, but the gap between in-memory and on-disk indexing is closing.
The Memory Wall #
HNSW is the gold standard for speed, but it's a memory hog because it stores the entire graph structure in RAM. If you're hitting OOM (Out of Memory) errors or your cloud bill is spiking, you have to move toward Approximate Nearest Neighbor (ANN) indexes that leverage SSDs.
Comparing Indexing Strategies #
HNSW (In-Memory): Ultra-low latency, high RAM requirement. Best for small to medium datasets where milliseconds matter.SPANN (Disk-optimized): Uses a compressed centroid-based approach to reduce memory footprint while keeping a secondary index on disk. It hits a sweet spot between speed and cost.DiskANN (On-Disk): Specifically designed for SSDs. It minimizes RAM usage by storing the majority of the graph on disk, using a small in-memory cache for routing.
Practical Implementation Logic #
If you are building a real-world AI workflow, the decision tree usually looks like this:
-
Under 1M vectors: Stick with HNSW. The operational simplicity outweighs the cost.
-
1M to 10M vectors: Look into product quantization (PQ) to compress vectors within HNSW, or transition to SPANN.
-
10M+ vectors: DiskANN is the only sane choice to avoid spending thousands on high-memory instances.
The biggest hurdle when switching to on-disk indexes is the "tail latency." You'll see a jump from sub-10ms responses to 50-100ms because of the SSD I/O. For most RAG applications, this is a negligible price to pay for a 90% reduction in infrastructure costs.
For those deploying from scratch, I recommend testing your recall rate after switching to on-disk indexes. Compression and disk-based retrieval can introduce a slight drop in accuracy, so you'll need to tune your efSearch
parameters to balance precision and speed.
Next Confusion Matrix: Why "90% Accuracy" is Often a Lie →