In the landscape of high-scale AI, many architects fall into the “RAM Trap”: the expensive conviction that a billion-vector search system requires a professional-grade server cluster groaning under terabytes of physical memory. This belief stems from a legacy mindset where performance is equated strictly with memory residency. However, as we move into the era of multi-tier storage architectures, the brute-force approach of keeping everything in RAM is becoming both economically and operationally unsustainable.
To build a sustainable system, we must challenge our fundamental assumptions about data priority. Does every part of a vector-search system actually deserve fast memory? By distinguishing between the data required for initial discovery and the data required for final precision, we can build systems that scale to billions of vectors without requiring a blank check for infrastructure.
This post is about the actual decisions that go into a memory-efficient vector search system: what belongs in RAM, what can live on disk, and what representation you use at each stage of retrieval. It’s built around Qdrant, and it’s backed by a real benchmark, not just architecture diagrams. The benchmark runs on 105,126 real H&M product embeddings (384 dimensions, from Qdrant’s own published hm_ecommerce_products dataset) with synthetic price/availability/geography metadata layered on top for filtering. The memory math is extrapolated from there to a billion vectors, and every claim about actual recall or latency numbers in this post is a measured result, not a projection. I’ll flag the few things that are extrapolated.
Say you’re running a search for an ecommerce marketplace with a billion product embeddings. The obvious instinct is to throw it all in RAM, both vectors,index, everything because RAM is fast and disk is scary. Then you price it out and the instinct changes fast.
An ecommerce marketplace with a billion product embeddings, each one at 384 dimensions, stored as float32.
The arithmetic is simple. Each vector costs:
384 dimensions × 4 bytes = 1,536 bytes per vector
Scale that to a billion vectors:
1,536 bytes × 1,000,000,000 ≈ 1.54 TB
These figures represent only the “naked” vectors. A production-ready environment faces significant “invisible” costs:
Each of those adds its own RAM bill, and none of them are optional in a real deployment.
So the obvious question: do you actually need 1.5+ TB (or, at the more commonly cited 768-dimension embedding size, closer to 3 TB) of RAM just to search a billion vectors?
The answer is no, but only if you design the storage and retrieval architecture on purpose. Billion-scale vector search isn’t really a RAM-sizing exercise. It’s a decision problem: what has to live in RAM, what can live on disk, and what representation should each stage of retrieval actually use. The rest of this post works through that decision, one lever at a time, then tests the results on a real benchmark.
Given that requirements continuously evolve, why should we be forced into binary tradeoffs? A truly adaptable system should allow seamless customization to meet a product’s specific demands.This is where in my case, I came across Qdrant.
Qdrant is a natural fit for this problem because it treats memory placement as a per-component decision rather than an all-or-nothing switch; vectors, quantized vectors, the HNSW graph, and payload indexes can each be independently placed in RAM or on disk, which is exactly the kind of control a billion-vector system needs. Its quantization isn’t a separate compression step bolted on afterward; it’s integrated with the retrieval path itself, supporting native rescoring against original vectors with a tunable oversampling factor. On-disk storage goes through mmap and the OS page cache deliberately, making “RAM as a cache, not the dataset” a real, testable architectural pattern rather than a workaround. And critically for a filtered-search use case like ecommerce, Qdrant builds filtering into the HNSW graph itself, with ACORN available for the harder case of multiple high-cardinality filters; instead of treating filtering as a bolt-on post-processing step. Together, these make Qdrant less a place to store vectors and more a toolkit for deliberately trading off RAM, disk, latency, and recall.
Qdrant gives you three storage choices that matter here:
In-memory vectors. Everything sits in RAM. This is the fastest option, and it’s also the most memory-hungry. The disk is touched only for persistence.
Memory-mapped (on-disk) vectors. Qdrant always stores vectors in a memory-mapped file on disk; the question is whether they’re also pulled into RAM. mmap’d files aren’t loaded into RAM directly — they go through the OS page cache, so frequently accessed pages end up resident in memory even though the “official” storage location is disk. With enough RAM available, this can get close to in-memory performance because the page cache does the same job RAM would.
On-disk HNSW. Qdrant can also place the index itself on disk rather than just the vectors. This cuts RAM usage further, but graph traversal can now trigger disk I/O, so the speed of your underlying storage starts to matter in a way it didn’t before. Qdrant’s own guidance is to treat this as an aggressive RAM-saving move, not a default performance configuration.
That gives us the first important distinction for the rest of this piece: “on disk” doesn’t mean “never in memory.” mmap lets the operating system decide, dynamically, which parts of your dataset deserve to be cached — which is a very different mental model from a hard RAM/disk split.
Now put actual numbers against the billion-vector scenario, at the dimensionality this benchmark actually uses (384d), across the representations Qdrant supports.
These are raw vector sizes only. They don’t include the HNSW graph, payload indexes, or payload data, all of which need their own RAM budget on top of this table. But the shape of the table is already the point: moving from float32 to binary is roughly a 32x reduction in the vector storage bill before you’ve touched indexing or filtering at all. That’s why compression becomes the first lever worth pulling, and it’s the subject of the next section.
Qdrant’s current documentation describes four quantization methods: Scalar, Binary, Product, and TurboQuant, each sitting at a different point on the compression/accuracy trade-off curve.
The basic idea is the same across all of them: instead of storing every vector component as a 32-bit float, store a compressed approximation instead.
None of these are free; every one of them removes information that was present in the original float32 vector. Which raises the actual engineering question this section exists to ask: how much compression can you apply before recall becomes unacceptable? That’s an empirical question, not a documentation question, and it’s what Section 20 of this post actually measures.
One more thing worth knowing about here even before running numbers: Qdrant supports asymmetric quantization — storing vectors as binary but scoring incoming queries with scalar quantization instead. This keeps the RAM footprint close to pure binary while giving noticeably better precision, which matters most in memory-constrained or disk-I/O-bound deployments. It wasn’t part of this benchmark, but it’s a relevant knob if binary quantization’s recall loss turns out to be a problem for your workload.
Compressing vectors is only half the story. The more interesting move is what Qdrant lets you do with the compressed and original vectors together.
Qdrant can store quantized vectors alongside the original, full-precision ones. That opens up a specific retrieval pattern:
Qdrant explicitly supports rescoring candidates against their original, uncompressed vectors after an initial quantized search. The idea behind this is worth stating plainly, because it reframes the whole compression question: you don’t need full precision for every comparison in the dataset. You need full precision only for the small number of candidates that might actually make it into the final result set.
That’s a very different cost structure than “compress everything and accept whatever recall you get.” It means the compression decision and the retrieval-path decision aren’t the same decision — you can be aggressive with quantization precisely because you’re not relying on it alone to produce the final ranking. Section 21 measures exactly how much recall this recovers, and it turns out to be substantial.
Putting the last two sections together gives Qdrant’s specifically documented hybrid setup: original vectors on disk, quantized vectors in RAM.
This is deliberately different from just “quantize everything” or just “put everything on disk.” The quantized vectors; small, fast to scan; stay resident in RAM where the bulk of the search work happens. The original, full-precision vectors — large, but only needed for a handful of candidates per query — live on disk and get pulled in only when rescoring actually needs them.
The economics here is the whole point: keeping every full-precision vector in RAM for a billion-vector collection is the ~1.4 TB (at 384d) or ~3 TB (at 768d) number from Section 3. Keeping only the quantized vectors in RAM, with originals on disk and touched only for the rescoring step, is potentially an order of magnitude cheaper; without giving up the accuracy that rescoring recovers. Sections 19–21 test whether that promise holds up on the actual benchmark data, and where it starts to cost you.
Section 2 introduced mmap in passing. It’s worth slowing down here, because it’s the mechanism that makes everything in Section 6 actually work.
With Qdrant’s mmap storage, the access path looks like this:
Application
↕
Page cache
↕
NVMe
The vector dataset doesn’t need to be resident in physical RAM to be usable. Instead, the operating system’s page cache holds whatever’s been accessed recently, and cold data gets pulled in from disk only when something actually asks for it. Qdrant’s own documentation states this plainly: with sufficient RAM, mmap-backed storage can get almost as fast as pure in-memory storage, because the page cache ends up doing the same job.
A few pieces make this work, and they’re worth naming individually because they show up again later in the filtering and HNSW-on-disk discussions:
The important nuance, stated as plainly as Qdrant states it: mmap doesn’t make disk access as fast as RAM. What it does is let you use RAM more selectively:spending it on the working set that actually gets queried, instead of the entire dataset regardless of access pattern. That’s a meaningfully different claim than “disk is basically free now,” and it’s worth testing rather than taking on faith — which is exactly what the mmap-specific benchmark later in this post does.
Stack Sections 6 and 7 together and there isn’t just one “memory-efficient” setup; there’s a spectrum, and where you land on it is a real trade-off between RAM footprint and I/O exposure.
Now the graph itself is off RAM too. This is the most memory-frugal option, and also the one where I/O costs stop being theoretical.
Qdrant’s own documentation explicitly states that pushing both vectors and HNSW onto disk can cut RAM substantially, but graph traversal itself may now require I/O, which is a fundamentally different cost than the “only touch disk for rescoring” pattern in Configuration B. In this benchmark, these three map directly onto what got built and measured as Configs A, B/C, and D (Section 18 covers naming); the latency deltas between them, especially the jump from B to D, are one of the more concrete results this post has.
It’s tempting to treat vector compression as the whole memory story. It isn’t. HNSW :- the graph index Qdrant uses for dense-vector search, has its own memory footprint, and at a billion-vector scale, it stops being a rounding error.
The parameters that matter most:
Every one of these knobs pushes graph size up or down independently of whatever you did with vector compression. That’s the point worth underlining here: compressing the vectors does not automatically solve the entire memory problem. A collection can have beautifully compressed, binary-quantized vectors and still be memory-heavy if the HNSW graph on top of it is large and fully RAM-resident. The two costs are separate line items, and treating them as one is how memory budgets end up wrong in practice.
Section 8’s Configuration C put the HNSW graph on disk alongside the original vectors. It’s worth pulling that apart on its own, because the trade-off is sharper than the vector-storage decision.
The benefit is straightforward: RAM requirements drop dramatically, since the graph, which can be a meaningful chunk of total memory at scale is no longer resident.
The cost is where it gets interesting. Graph traversal is inherently a sequence of “look at this node, then jump to its neighbors” operations, and when those neighbors aren’t in RAM, each jump can trigger disk I/O. That makes:
This is exactly why Qdrant recommends fast NVMe specifically for this configuration, a graph traversal that has to wait on spinning disk or slow network storage for every hop is going to feel very different from the one backed by NVMe.
The central engineering question here isn’t “can I save RAM by doing this”; you obviously can. It’s: is the RAM saved worth the additional I/O it costs? That’s not answerable in the abstract; it depends on your latency budget and your storage. Config D in this benchmark is exactly this setup, and its latency numbers (Section 19 onward) give one concrete answer for one specific hardware/dataset combination, not a universal one.
Zoom out from pure vector storage for a moment. Real search queries in ecommerce are almost never “find things similar to this vector” in isolation; they come with structured constraints attached.
Take a realistic query: “Find running shoes similar to this product, under ₹10,000, size 9, in stock, and deliverable to Mumbai.”
The vector search here isn’t operating over the whole collection anymore. It’s implicitly scoped by:
category = running shoes
price <= 10,000
size = 9
inventory > 0
location = Mumbai
Qdrant’s payload indexes make this kind of filtering efficient, and more importantly for a billion-vector system, they can influence how the query planner approaches the search in the first place. That’s the framing worth carrying forward: filtering isn’t a separate concern bolted onto vector search. It changes the effective size of the search space, and therefore the actual cost of retrieval. A query with a highly selective filter is a fundamentally cheaper problem than one with no filter at all, if the engine is built to take advantage of that, which is exactly what the next two sections are about.
Here’s the problem with naïve filtering on top of an ANN graph: a sufficiently restrictive filter can wreck ordinary HNSW traversal, because many of a node’s graph neighbors simply won’t satisfy the filter. Follow enough dead-end edges and the search stalls before it ever reaches the true nearest matches — even though nothing is technically wrong with the graph.
Traditional (post-filter):
HNSW search → Candidates → Filter
Qdrant’s filter-aware approach:
HNSW traversal + Payload index → Filter-aware candidate traversal
Qdrant’s answer, documented as “filterable HNSW,” is to bake the fix into the graph itself rather than filtering after the fact. When a payload field is indexed, Qdrant walks its values and adds extra HNSW edges between points that share a value in that field — so a query filtered to that value still has a connected graph to traverse, instead of hitting a scattered set of islands.
This isn’t free. Those extra edges get added per indexed field, not per combination of fields, and they cost real build time. Qdrant’s own published benchmark on a one-million-point collection saw index build time go from about 116 seconds with no extra edges to 507–650 seconds with them, roughly 4.4x–5.6x longer. And there’s a size cap: a payload value shared by too many points (roughly a fifth of the collection or more, depending on graph density) gets skipped entirely, on the theory that the main graph should already keep that many points connected without help.
The framing that matters for the rest of this post: filtering → search efficiency → memory → billion-scale architecture are not four separate topics. They’re one connected decision, and filterable HNSW is Qdrant’s default answer to it.
Push the query harder: “Find size-9 trail-running shoes under ₹10,000, in stock in Mumbai, from brands I’ve purchased before.” Now there are multiple filters stacked together, and that’s exactly where filterable HNSW’s per-field edges start to fall short — because those extra edges are built per field, not per combination, a two-filter intersection can land somewhere no single field’s edges actually cover.
This is where Qdrant’s ACORN mechanism comes in. Rather than repairing the graph at index time, ACORN repairs traversal at query time: when direct neighbors of a node have been filtered out, it looks one hop further, at neighbors of neighbors; instead of giving up. This recovers accuracy on exactly the cases filterable HNSW’s static edges miss, at the cost of extra work per query. It’s opt-in per query via the acorn search parameter, so enabling it doesn’t require rebuilding anything.
Qdrant’s own benchmark work on this (a separate one-million-vector test, not this project’s dataset) is a useful reference point for what ACORN actually buys you: on single-field filters, filterable HNSW alone already got very close to full recall for most selectivities, and ACORN mattered most specifically on the fields that got no extra edges — because their values were too common to qualify — and on two-field intersections, where extra edges from either individual field didn’t cover the combined constraint. On a 4% double-filter intersection, for instance, plain filterable HNSW recall dropped to the 60–70% range while ACORN recovered it close to 100%, at several times the latency. At very low selectivity (a fraction of a percent matching), Qdrant’s planner tends to skip the graph entirely and read straight from the payload index instead, which turned out to be the cheapest and most accurate path in that regime.
That last point is worth sitting with, because it previews something this project’s own filtering benchmark ran into directly: at high enough selectivity, or at small enough scale, the graph-repair mechanisms may simply have nothing to fix. Section 23 of this post covers what happened when this dataset’s own high-selectivity filter tier was tested against ACORN and the result wasn’t the clean “ACORN helps” story the documentation-level discussion might suggest.
The trade-off worth carrying forward either way: filter accuracy vs. search performance, and the right answer depends on filter selectivity, dataset scale, and how many strict filters get combined at once; not a fixed rule.
Filtering isn’t free just because it makes search faster. Qdrant’s payload indexes, the structures that make filterable HNSW and fast filter matching possible, consume their own memory and disk space, and Qdrant’s documentation is direct about the implication: index the fields you actually filter on, not everything in the payload.
That’s a genuine trade-off, not a formality:
More indexes
↓
Faster filtering
↓
More memory + disk
In practice this means someone has to decide, deliberately, which payload fields earn an index. A field nobody filters on is pure memory cost with zero retrieval benefit — and at scale, “index everything just in case” is exactly the kind of decision that quietly erodes all the RAM savings won earlier from quantization and mmap.
This is a real decision this project had to make, not just a documentation point. The raw H&M dataset carries 33 columns. Only 8 of them — product_id, title, description, category, brand, price, availability, geography — are actually used for filtering or display, and only those made it into the payload. The other 25 columns exist in the source data but aren’t stored in Qdrant at all, precisely because indexing or storing fields nobody queries costs memory for no retrieval benefit.
Here’s a detail that runs against the usual instinct at a billion-vector scale, where “just use approximate nearest neighbor search” feels like the obvious answer.
Qdrant’s query planner can choose to skip HNSW entirely and fall back to a full scan, when the filtered subset of the collection is estimated to be small enough. This is decided per query, based on the estimated size of the data satisfying the filter condition, against a configurable threshold (full_scan_threshold, measured in kilobytes of vector data).
Why this makes sense once you think about it: ANN search exists to avoid scanning the entire dataset. If a filter has already cut the candidate set down to a few hundred or a few thousand vectors, scanning those directly can be cheaper than traversing a graph structure built to handle a search space many orders of magnitude larger. The graph’s overhead, hopping between nodes, checking edges doesn’t pay for itself once there’s barely anything left to search.
This is also why “use HNSW because you have a billion vectors” is an incomplete rule. The real rule is closer to: use HNSW when the effective search space is large, and let the planner fall back to brute force when a filter has already done most of the work for you. It’s a small detail, but it’s the kind of thing that separates a system that’s just running ANN everywhere from one that’s actually reasoning about the query it’s serving.
Pull Sections 2–15 together and a single mental model falls out of all of it.
The goal was never to fit the entire database into RAM. The goal is to fit the right working set into RAM — the quantized vectors that carry most of the search load, the hot HNSW structures, the payload indexes actually used for filtering — and let mmap and the OS page cache handle everything colder.
Qdrant’s own capacity-planning guidance makes this explicit: frequently accessed data should stay in memory, and the rest can be offloaded to disk without the system falling over. It’s a reframe worth stating plainly, because it’s the thesis the entire first half of this post has been building toward: billion-vector search isn’t a RAM-sizing problem. It’s a working-set problem.
That’s the last of the conceptual groundwork. From here, the post moves into what actually got built and measured.
Time to move from architecture to what was actually built and measured. You can refer to the code and scripts here.
GitHub - vatsala-singh/Billion-Vector-Search-System
The benchmark runs on 105,126 real products from Qdrant’s own published Qdrant/hm_ecommerce_products dataset — real H&M titles, descriptions, categories, and precomputed 384-dimension BGE-small embeddings. This was a deliberate choice over generating fully synthetic data: real product text and real embeddings produce a realistic similarity structure that synthetic data can’t easily fake, and the dataset was already sized and formatted for exactly this kind of benchmark.
Three fields the schema needs — price, availability, and geography — don’t exist in any real-world source for this catalog, so those were generated synthetically and layered on top. Everything else (title, description, category, brand, embedding) is the genuine dataset.
Two schema decisions are worth stating explicitly, because they shaped everything downstream:
The embedding column itself needed a correction along the way: the dataset’s readme documents the column as bge_embedding, but the actual column in the live dataset is dense_embedding. This is confirmed against the HuggingFace dataset viewer and fixed before any collection was built; a small thing, but the kind of detail that silently breaks an entire pipeline if it’s caught late.
The payload was narrowed from the dataset’s 33 raw columns down to the 8 actually used for filtering or display, per the memory reasoning in Section 14: product_id, title, description, category, brand, price, availability, geography.
Query resolution had its own bug worth flagging here since it affects every result downstream: the first attempt resolved queries by matching on title, but many H&M products share a title across different colorways or sizes so the same title could silently map to different products between runs, producing inconsistent query counts run to run. This is fixed by joining on product_id, which is guaranteed unique, both when generating queries and when resolving them for benchmarking. Every recall and latency number in this post reflects that fix.
Five configurations were built, in strict order, changing one variable at a time — the same discipline the earlier architecture sections argued for, applied for real:
This maps directly onto the three-configuration comparison from Section 8: A is “everything in RAM,” B/C is the quantized-in-RAM-with-originals-on-disk hybrid from Section 6, and D is the aggressive disk-backed setup from Section 10. Building them in this exact order — one variable per step — is what makes the results in the next few sections actually comparable to each other, instead of conflating “quantization changed things” with “storage location also changed things” in the same measurement.
Two indexing bugs surfaced during this build phase and are worth stating here because they explain why the numbers in this post are trustworthy rather than just plausible-looking:
First, indexed_vectors_count got stuck at 0 out of 105,126 on the first attempt. The cause was a misunderstanding of indexing_threshold=0 — it was assumed this would force immediate indexing, but it actually disables indexing entirely. This was fixed by removing it at collection-creation time and instead lowering the threshold to a small non-zero value (1,000) after upload.
Second, and more consequential: before that fix, roughly 6,000 leftover points per collection stayed unindexed and were quietly brute-force searched alongside the indexed ones — which inflates recall and corrupts latency, since exact search and approximate search were getting blended into a single reported number without anyone intending that. This was fixed by adding a wait_for_indexing() step that polls until indexed_vectors_count matches points_count and status is green, before any benchmark is allowed to run. Every collection below reports indexed_vectors_count: 105126 — fully indexed, not partially.
With Configs A and B fully indexed, here’s what full precision vs. scalar vs. binary quantization actually looked like, at 100 queries per collection:
A couple of things worth pausing on here, because they’re not quite what the documentation-level story predicts.
Recall@10 for BQ came in higher than SQ (0.978 vs. 0.928). Section 4 framed binary quantization as the lossier of the two, and it generally is — but that comparison here is doing something specific: rescore=False was passed explicitly for both collections in this table, and BQ’s recall gap tends to show up specifically
The rescore=False finding is worth its own paragraph, because it changed what “no rescoring” actually meant in this benchmark. Early on, quantization_params=None (i.e., not setting the parameter at all) was assumed to be equivalent to explicitly setting rescore=False. It isn’t. On the identical BQ collection, leaving the parameter unset produced a materially better recall number than explicitly disabling rescoring — a gap between 0.742 and 0.978 recall@10 depending on which of the two you did. Left implicit, the benchmark had been silently reporting a much-better-than-real “no rescoring” number for BQ in an earlier pass. The table above uses the corrected, explicit rescore=False — forced for quantized collections directly in the benchmark CLI — so 0.978 recall@10 for BQ is the honest number, not the inflated one.
Latency roughly doubled for both quantized collections versus baseline (3.08ms → ~7.8ms p50). This runs against the usual pitch for quantization — smaller vectors, faster comparisons — and it’s worth being straight about it rather than smoothing it over: at this specific scale (105K vectors, single machine), the quantized collections were slower, not faster, on raw latency. This is almost certainly an artifact of scale and machine-level noise rather than a real property of quantization — binary quantization’s whole performance case rests on cheap bitwise operations at scale that a 105K-vector collection may simply not exercise enough to show. It’s flagged here rather than explained away, because a benchmark that only reports the numbers that match the documentation’s story isn’t a real benchmark.
The core signal does hold up cleanly: scalar quantization gives roughly 4x compression with a moderate accuracy cost, and binary quantization compresses further with a real accuracy trade-off that,as the next section shows,mostly disappears once rescoring is turned back on.
Section 5 made the case, at the documentation level, that you don’t need full precision for every comparison,just for the small set of candidates that might make the final result. This is where that claim gets tested against real numbers.
The sweep varies oversampling — how many extra candidates get pulled before rescoring trims back down to the final top-K — from no rescoring at all up to 8x, on both SQ and BQ collections:
**Scalar quantization:**| Setting | Recall@10 | Recall@100 | p50 | p95 | p99 || ----- | ----- | ----- | ----- | ----- | ----- || No rescoring | 0.928 | 0.9566 | 2.65 ms | 4.86 ms | 24.48 ms || Rescore, 1x | 0.990 | 0.9996 | 2.65 ms | 3.98 ms | 4.34 ms || Rescore, 2x | 0.990 | 0.9998 | 2.46 ms | 3.60 ms | 5.13 ms || Rescore, 4x | 0.988 | 0.9998 | 2.91 ms | 4.15 ms | 4.47 ms || Rescore, 8x | 0.988 | 0.9994 | 3.54 ms | 4.92 ms | 5.11 ms |**Binary quantization:**| Setting | Recall@10 | Recall@100 | p50 | p95 | p99 || ----- | ----- | ----- | ----- | ----- | ----- || No rescoring | 0.742 | 0.6448 | 4.65 ms | 6.43 ms | 10.41 ms || Rescore, 1x | 0.978 | 0.9264 | 2.29 ms | 3.36 ms | 3.80 ms || Rescore, 2x | 0.982 | 0.9776 | 2.43 ms | 3.22 ms | 3.71 ms || Rescore, 4x | 0.984 | 0.9942 | 2.39 ms | 3.65 ms | 5.17 ms || Rescore, 8x | 0.988 | 0.9986 | 3.30 ms | 4.45 ms | 5.10 ms |
The headline number: BQ recall@10 goes from 0.742 to 0.978 with just 1x oversampling turned on — a single extra pass of candidates rescored against the original vectors recovers almost all of binary quantization’s accuracy loss. That’s the core claim from Section 5 holding up under an actual measurement, not just a plausible architecture diagram.
Two things worth noting beyond the headline number:
Returns diminish fast. Recall@10 for BQ barely moves between 2x and 8x oversampling (0.982 → 0.988), while Recall@100 keeps climbing more noticeably (0.9776 → 0.9986) — rescoring more candidates mostly helps the long tail of results, not the top handful, which already gets fixed almost immediately.
Rescoring didn’t just fix recall; it made latency better, not worse. This is the part that runs against the naive intuition that rescoring is a “pay accuracy back with latency” trade. Compare BQ’s no-rescoring p99 latency (10.41ms) against 1x rescoring’s p99 (3.80ms): rescoring is faster on the tail, not slower. The likely explanation is that without rescoring, some queries were falling back to more expensive comparison paths internally; with a small, tightly bounded rescoring step against a compact candidate set, the tail gets more predictable rather than less. Either way, on this dataset there’s essentially no reason to skip rescoring — it improved both recall and worst-case latency simultaneously.
Put together with Section 4’s framing: this is the practical version of “you don’t need full precision for every comparison.” A 1x-oversampled rescore against the original vectors — touching only a small multiple of the final result count, not the whole candidate pool — recovered nearly all of binary quantization’s lost accuracy, at no latency cost. That’s the actual case for the quantize-then-rescore architecture from Section 6, measured rather than asserted.
Now let’s come to the storage-location question from Sections 6–10, measured rather than argued. All numbers below are without rescoring, same 100-query set, same fully-indexed collections.
**Scalar quantization, across storage configs:**| Config | Recall@10 | p50 | p95 | p99 || ----- | ----- | ----- | ----- | ----- || B — RAM only | 0.928 | 7.77 ms | 11.69 ms | 20.93 ms || C — originals on disk | 0.914 | 5.94 ms | 11.35 ms | 30.13 ms || D — originals \+ HNSW on disk | 0.920 | 6.15 ms | 10.20 ms | 25.69 ms |**Binary quantization, across storage configs:**| Config | Recall@10 | p50 | p95 | p99 || ----- | ----- | ----- | ----- | ----- || B — RAM only | 0.978 | 7.81 ms | 11.81 ms | 19.33 ms || C — originals on disk | 0.982 | 14.83 ms | 23.87 ms | 47.14 ms || D — originals \+ HNSW on disk | 0.984 | 2.45 ms | 4.22 ms | 6.72 ms |
Two very different stories in these two tables, and both deserve honesty rather than a tidy narrative.
Scalar quantization roughly matches what Section 7’s mmap theory predicts. Recall stays essentially flat across B, C, and D (0.928 → 0.914 → 0.920 — within noise for a 100-query sample). Median latency (p50) doesn’t get meaningfully worse moving vectors and even HNSW to disk — consistent with the session’s own read of this result: without rescoring, search never actually touches the original vectors, so putting them on disk is largely invisible at the median. Where the disk cost does show up is the tail: p99 climbs from 20.93ms (B) to 30.13ms ( C )once original vectors move to disk, which lines up with Section 7’s page-fault story — most queries hit the warm page cache, but the unlucky ones pay a real disk-read cost.
Binary quantization tells a much messier story, and it’s worth being upfront about why. The BQ-on-disk row got noticeably slower — p50 nearly doubling to 14.83ms — while the BQ-disk-plus-HNSW-disk row (D) came back faster than the RAM-only baseline (2.45ms vs. 7.81ms p50). That second result — disk-backed HNSW outperforming RAM-only HNSW — isn’t architecturally plausible on its face, and it wasn’t treated as a real finding. It’s the specific symptom of a benchmarking bug caught during this project: back-to-back runs were biasing latency numbers through the OS page cache, because whichever collection got benchmarked most recently (or most repeatedly) inherited a warmer cache than the ones benchmarked before it. The fix was adding an untimed warm-up pass ahead of every timed run, specifically to prevent one collection’s numbers from riding on another’s residual cache state. Rather than quietly re-running until the numbers looked reasonable, this result is flagged here as what it is: a measurement artifact, not a claim that disk-backed HNSW beats RAM.
The honest takeaway from this pair of tables: the mmap claim from Section 7 holds up for scalar quantization at this scale — flat recall, tail-latency cost from disk reads, nothing more dramatic. The binary quantization numbers in this configuration need to be treated with more caution and, ideally, rerun cleanly with the warm-up fix applied consistently before being trusted for anything beyond “something about cache-state ordering was going on here.”
Section 10 raised the natural follow-up question: at what point does storage latency actually overwhelm the benefit of putting HNSW on disk? The honest answer here is that this benchmark can’t say — it ran on a single machine with one storage tier, so there was no NVMe-vs-slower-SSD-vs-network-storage comparison to run. Section 21’s Config D numbers show what one specific storage class looks like; they don’t say anything about where the crossover point is against something slower. That’s a real open question this project didn’t have the infrastructure to answer, not a claim being quietly skipped over.
Section 13 set up the promise: filterable HNSW handles most filter shapes well, and ACORN is supposed to recover accuracy on the shapes it doesn’t — high-cardinality, multi-filter, high-selectivity queries. Here’s what actually happened when that was tested on this dataset.
Filters were built across selectivity tiers, from broad (L1) down to highly restrictive (L3, and later an even tighter L4):
That L3 result runs directly against the intuition Section 13 built up from Qdrant’s own published ACORN benchmark, where high-selectivity and multi-filter combinations were exactly where ACORN earned its keep. The likely explanation isn’t that ACORN doesn’t work,it’s scale. Qdrant’s own published ACORN benchmark shows the effect clearly at roughly 5 million vectors and ~4% selectivity. This benchmark runs at ~105K vectors. At that much smaller scale, filterable HNSW’s index-time edges may already preserve enough connectivity that there’s simply no broken graph for ACORN’s second-hop traversal to repair — the effect Section 13’s own referenced benchmark shows may genuinely require more vectors, or a tighter selectivity, than this dataset provides. An L4 “extreme selectivity” tier (tighter price headroom plus a fifth filter field) was added specifically to test whether the ACORN effect would show up at all in this dataset before concluding it’s simply out of reach at this scale — that result was not yet reviewed at the time of writing, and is left here as an open item rather than a filled-in number.
One filtering bug is worth including here as its own finding, because it’s a good example of how a benchmark can look reasonable while quietly measuring the wrong thing. The first attempt at building high-selectivity filters sampled category, brand, and price fields independently at random. That produced a filter tier where, on average, only 0.2 matching candidates existed out of 105,126 — because category and brand aren’t actually independent in a real product catalog, so random independent sampling mostly generated filter combinations that don’t correspond to any real product at all. The fix was to derive each filter from an actual product row’s own category, brand, and price (with some headroom on price), which guarantees at least one real match while staying genuinely restrictive. The 0.47% L3 selectivity figure above reflects that fix — the earlier, broken version of this filter tier would have produced numbers that looked like a system failure when it was actually a bad test-data generator.
Also worth stating plainly rather than glossing over: at this dataset’s scale, absolute latency for filtered queries is often sub-2ms, which is likely below the noise floor for reliably distinguishing ACORN’s per-query overhead from ordinary measurement jitter. That’s a scale limitation worth naming explicitly — it’s not something more query repetitions alone would fix, since the signal being measured may simply be smaller than the noise at this collection size.
Pulling everything from Sections 2–23 together, here’s the architecture this benchmark actually supports, not as a universal prescription, but as one reasonable answer to the trade-offs measured above:
This is the shape that Section 20’s rescoring numbers and Section 21’s scalar-quantization mmap numbers actually justify: quantized vectors and the hot parts of the index stay in RAM because that’s where the bulk of search cost lives; original vectors sit on disk because rescoring only needs to touch a small, bounded set of them per query, not the whole collection.
Worth restating plainly: this is a reasonable architecture given what this benchmark measured, on this dataset, at this scale — not the correct architecture for every deployment. A workload with a much tighter latency budget, or a much larger collection where the disk-backed HNSW question in Section 21 actually gets tested cleanly, could reasonably land somewhere else on this same spectrum.
A practical decision framework, grounded in what Section 21 actually showed rather than stated as a rule of thumb:
Keep HNSW in RAM when:
Put HNSW on disk when:
Qdrant’s own guidelines treat putting both vectors and HNSW on disk as a more aggressive RAM-saving configuration, not a default performance posture — and the measured numbers here back that framing up. The scalar quantization results support the “this is workable” case cleanly. The binary quantization results from the same config were compromised by the cache-ordering bug described in Section 21, so — in the spirit of not overstating what was actually shown — the honest answer for BQ specifically on disk-backed HNSW is “re-run this cleanly before trusting a number,” not a confident recommendation either way.
Section 4 laid out what each method is documented to do. Here’s how that guidance holds up against what actually got measured:
Scalar quantization — a good starting point when you want substantial compression with a comparatively small accuracy hit. That held up: ~4x compression, recall@10 of 0.928 without rescoring, and 0.99 with even 1x oversampling.
Binary quantization — useful when memory reduction matters more than anything else, particularly on suitable high-dimensional distributions. Also held up, with a caveat worth repeating: without rescoring, BQ’s Recall@100 in particular took a real hit (0.6448) — this is not a method to run without rescoring turned on, based on what Section 20 showed. With 1x rescoring, the accuracy case becomes very strong.
Product quantization — documented as useful when minimizing memory is the overriding concern, at the cost of more complexity and a bigger accuracy trade-off. Not built in this project — it was explicitly optional given the time available, and there’s no measured data here to report on it one way or the other.
TurboQuant — worth evaluating when the embedding model and quality requirements make its higher compression attractive. Also not built here, for the same reason.
The one method this project can’t speak to from direct measurement is exactly the one Qdrant’s documentation frames as the “if minimizing memory is the overriding concern” option — which is a fair gap to flag rather than paper over with confident-sounding guidance.
It’s tempting, after all of the above, to summarize this whole post as “minimize RAM.” That’s incomplete, and worth rejecting explicitly. A system running 200GB of RAM against very slow NVMe can genuinely be worse than one running 500GB against fast storage — Section 21’s disk-latency numbers are a small-scale preview of exactly that trade-off. Similarly, a system chasing 99% recall at enormous infrastructure cost can be a worse decision than one settling for 98.5% recall at a fraction of the cost — which is essentially the choice Section 19 and 20 hand you directly: BQ with 1x rescoring gets extremely close to full-precision recall at a fraction of the RAM footprint.
The actual optimization target is:
Cost ↔ RAM ↔ storage ↔ latency ↔ recall ↔ throughput
One honest gap to flag here: this project didn’t get to computing the cost side of that equation directly — translating the RAM and disk footprints measured above into an actual $/GB comparison across configurations. That’s a natural next step on top of everything measured so far, not something this benchmark set out to answer.
Go back to where this started: a billion product embeddings, and the instinct to just throw everything in RAM.
The answer this post actually supports isn’t a single trick. It’s a hierarchy, and each piece of it did something specific and measurable:
HNSW → avoid exhaustive search over the whole collection.Filtering → reduce the effective search space before vector search even runs.Quantization → make the vector representation dramatically smaller — 4x with scalar, more with binary — measured directly in Section 19.mmap → let the dataset exceed physical RAM, with the OS page cache doing the work of deciding what stays hot — confirmed for scalar quantization in Section 21.RAM-resident quantized vectors → keep the representation that carries most of the search load fast and close.Original vectors on disk → retain full-precision data economically, touched only when rescoring actually needs it.Rescoring → recover the accuracy approximate retrieval gave up — and, on this dataset, at essentially no latency cost, which was the most concrete positive surprise in the entire benchmark.Disk-backed HNSW, when it’s worth it → push memory requirements lower still, accepting the I/O cost that comes with it — a trade-off this benchmark could measure but not fully validate at this scale.
At a billion-vector scale, the question was never really “how do I fit everything into RAM.” It’s “which parts of my retrieval pipeline actually deserve RAM, and which can be efficiently backed by storage” — and that’s not a question with one universal answer. It’s a question a real benchmark, run on real data, has to actually answer for your specific dataset, filters, and latency budget. This one did that at 105K vectors. The next honest step is doing it again at a scale where the disk-backed HNSW and ACORN questions this post had to leave open finally get resolved.
Building a Billion-Vector Search System Without Putting Everything in RAM was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.