MemWeaver – a vector DB with predictable memory usage

MemWeaver, a temporal vector database with predictable memory usage, calculates memory cost as a function of vectors, dimensions, and M before deployment, with 1M vectors at dim=128 and M=16 requiring approximately 640 MB. The database integrates time as a structural relevance dimension, ranking recent content higher without explicit timestamp filters, and includes a cold tier for older vectors to bound memory costs. It is built for use cases like observability, news search, code search, and knowledge bases, and has been evaluated on the SIFT1M benchmark.

A temporal vector database with predictable memory usage. Memory cost is a function of vectors, dimensions, and M — known before deployment, not discovered in production. memory ≈ n × node size For dim=128, M=16: node size = 640 bytes , so 1M vectors requires approximately 640 MB — calculable before any data is inserted. Most vector databases have two operational problems: Memory is unpredictable. You cannot know your memory cost before you deploy. You provision generously, monitor closely, and scale reactively. Search is temporally blind. A document from last week and a document from three years ago are treated identically. Recency is either ignored or handled by timestamp filters that don't affect ranking. MemWeaver is designed around two opposite principles: Memory is a first-class design constraint , not a runtime surprise. Time is a first-class dimension of relevance , not a filter applied after the fact. MemWeaver's temporal architecture makes it a natural fit for domains where recency affects relevance. Observability and incident search Semantic search over logs, traces, and alerts benefits directly from temporal relevance. During an incident, engineers care about what happened in the last hour, not three weeks ago. MemWeaver surfaces recent signals first without explicit timestamp filters in every query. News and content search An article from last week is more relevant than the same article from three years ago for most queries. Temporal relevance is structural in MemWeaver — recent content naturally ranks ahead of semantically equivalent older content. Code search Large codebases accumulate years of commits. Semantic search over code benefits directly from temporal relevance — recent implementations reflect current architecture, while older code may represent deprecated patterns or refactored approaches. MemWeaver surfaces current code first, keeping stale results in cold storage without deleting them. Memory predictability matters here: embedding servers indexing active repositories need stable memory budgets, not runtime surprises at commit time. Knowledge bases with retention policies Enterprise knowledge bases accumulate content over years. Recent documentation, recent decisions, and recent discussions are more likely to reflect current reality. MemWeaver's cold tier moves older content to disk or S3 automatically, keeping memory costs bounded as the corpus grows. Cost-sensitive deployments MemWeaver's arena allocator makes memory usage predictable before deployment — no runtime surprises. The cold tier lets older vectors live on local SSD 5x memory latency or S3, reducing infrastructure cost without sacrificing search quality or recall. Radar is a semantic knowledge search system built on MemWeaver, indexing articles from RSS feeds and Common Crawl with temporal relevance. Search query: "startup funding" Recent articles surface naturally ahead of older ones. No explicit timestamp filtering — temporal relevance is structural, not bolted on. RSS feeds / Common Crawl CDX ↓ FetchArticlesFromRSS / FetchArticlesForDomain ↓ EmbedBatch embed-svc — sentence-transformers ↓ StoreText → Postgres metadata + local disk article bodies ↓ BatchInsertMemWeaver HNSW vector index ↓ Search API → UI Each stage is an independent Temporal activity — crash-safe, retriable, observable. Text storage is outside MemWeaver; the search API returns vector id values resolved against Postgres metadata. Evaluated on SIFT1M 1M vectors, dim=128 using two hardware configurations: AWS m5d.4xlarge Intel Xeon, AVX-512, 16 vCPUs, 64 GB RAM — arena vs naive comparison Apple M-series, 6 performance cores — parallel insertion and TimeBucket benchmarks Configuration: M=16, M MAX0=32, k=10. Compiled with target-cpu=native . Recommended production configuration: ef construction=100 + capacity-aware, ef search=200. | Implementation | Build time | Search 50 queries | Speedup | |---|---|---|---| | Arena MemWeaver | 187s | 20.4ms | — | | Naive Vec-based | 322s | 34.9ms | 1.72x slower | Arena allocation is 1.72x faster to build and 1.71x faster to query. Collocating each vector with its HNSW edges means graph traversal loads vector and edges in one cache access instead of two. The arena implementation maintains a constant ~8 MB peak overhead throughout 1M vector insertion. The naive Vec-based implementation shows reallocation spikes of 21.6 MB at 270k vectors and 35.7 MB at 530k vectors. The arena implementation also uses 25% less total RSS at 1M vectors 608 MB growth vs 759 MB . M=16, M MAX0=32, ef search=100, k=10, 10,000 queries. | Config | Build 1 thread | Build 6 threads | Speedup | Mean recall@10 | Min recall | |---|---|---|---|---|---| | Naive Vec-based | 195s | — | — | 0.965 | 0.300 | | ef construction=40 | 124s | 70s | 1.77x | 0.965 | 0.300 | | ef construction=100 | 253s | 112s | 2.26x | 0.961 | 0.000 | | ef construction=100 + capacity-aware | 273s | 103s | 2.65x | 0.979 | 0.400 | | TimeBucket n=4, ef construction=100 + capacity-aware | — | 83.5s | — | 0.9947 | 0.400 | Throughput: single thread 7,420 vec/s → two-phase 4 threads 12,624 vec/s, 1.70x speedup . Recommended production configuration: ef construction=100 + capacity-aware + 6 threads — 103s build faster than ef=40 single-thread at 124s , mean recall 0.979 better than ef=40 at 0.965 , no zero-recall queries. Why ef construction=100 standard degrades quality: more candidates triggers aggressive diversity pruning, creating eviction cascades that isolate sparse-region nodes 13 zero-recall queries . Capacity-aware neighbor selection penalizes high-degree nodes, preventing cascades. Why higher ef construction gives better parallel speedup: more work per insertion means a larger parallel fraction and smaller sequential fraction Amdahl's law . ef=40: 1.77x speedup; ef=100: 2.65x speedup. SIFT1M, k=10, 10,000 queries, ef construction=100 + capacity-aware. | ef search | Threads | QPS | p50 | p99 | Recall@10 | Min recall | |---|---|---|---|---|---|---| | 100 | 1 | 2,908 | 0.332ms | 0.871ms | 0.979 | 0.400 | | 100 | 6 | 17,021 | 0.351ms | 0.512ms | 0.979 | 0.400 | | 200 | 1 | 1,761 | 0.572ms | 0.800ms | 0.994 | 0.600 | | 200 | 6 | 9,916 | 0.588ms | 0.813ms | 0.994 | 0.600 | Search scales near-linearly — 97% efficiency at 6 threads for ef=100, 5.6x at ef=200. Both run on Apple M-series, SIFT1M, M=16, ef construction=100, k=10. Qdrant build time includes upsert 13.4s + background indexing 19.2s = 32.6s total. | System | Build | ef search | QPS 6t | p50 | p99 | Recall@10 | Min recall | |---|---|---|---|---|---|---|---| | MemWeaver | 60s | 100 | ~10,400 | ~0.570ms | ~0.810ms | 0.988 | 0.400 | MemWeaver | 103s | 200 | 9,916 | 0.588ms | 0.813ms | 0.994 | 0.600 | | Qdrant | 32.6s | 100 | 6,158 | 0.951ms | 1.495ms | 0.995 | 0.500 | At equivalent recall 0.994 vs 0.995 , MemWeaver is 1.6x higher throughput and 1.6x lower latency than Qdrant. MemWeaver min recall 0.600 is also better than Qdrant 0.500 at this operating point. Qdrant builds 1.8x faster 32.6s vs 60s , reflecting lower ef construction defaults and scalar quantization which reduces vector size 4x. MemWeaver uses full float32 throughout, prioritizing recall quality at equivalent parameters. Qdrant's QPS plateaus after 4 concurrent tasks 5,866 → 6,158 due to server overhead. MemWeaver is an embedded library — parallel search scales directly with thread count with no server layer. After swap out 308 arena blocks, 208ms , searching directly from Mac SSD, ef construction=100, ef search=100: | Threads | QPS | p50 | p99 | vs hot tier | |---|---|---|---|---| | 1 | 508 | 1.851ms | 2.810ms | 5.7x slower | | 2 | 748 | 2.421ms | 5.308ms | 7.8x slower | | 4 | 1,000 | 3.631ms | 7.029ms | 11.6x slower | | 6 | 588 | 10.689ms | 14.119ms | 29x slower | 6 threads exceeds Mac SSD random read bandwidth — latency collapses. 4 threads is the optimal cold tier configuration. Recall is identical to hot tier 0.979 — tiering is a pure performance/memory tradeoff with zero correctness cost. Production NVMe ~3GB/s vs Mac SSD ~1.5GB/s is expected to improve cold tier QPS ~2x. SIFT1M, k=10, 10,000 queries, ef construction=100 + capacity-aware, restored from disk. | Threads | QPS | p50 | p95 | p99 | Recall mean | Min recall | |---|---|---|---|---|---|---| | 1 | 496 | 2.077ms | 2.560ms | 2.736ms | 0.9947 | 0.400 | | 2 | 985 | 2.039ms | 2.510ms | 2.697ms | 0.9947 | 0.400 | | 4 | 1,909 | 2.083ms | 2.567ms | 2.761ms | 0.9947 | 0.400 | | 6 | 2,802 | 2.135ms | 2.660ms | 2.906ms | 0.9947 | 0.400 | Hot, cold, and restored phases produce identical recall phases match=true — tiering across the full hot→cold→restored cycle has zero correctness cost. The higher latency vs single HNSW 2.1ms vs 0.57ms reflects merging results across 4 independent buckets. The read-heavy neighbor search phase parallelized across threads is separated from the write-heavy edge update phase applied sequentially . Upper-level beam search ef=5 at layers 1+ replaces greedy descent, escaping local optima. See docs/parallel insertion.md /daib/mem weaver/blob/main/docs/parallel insertion.md for the full design. The min=0.300 across all configurations reflects two hard queries in sparse regions of SIFT1M — a fundamental HNSW characteristic at M=16, ef search=100, not an implementation issue. Full writeup: docs/cold-load-benchmark-findings.md /daib/mem weaver/blob/main/docs/cold-load-benchmark-findings.md . SIFT1M, 1M vectors / 128 dim, page cache dropped between build and load sudo purge for genuine cold-disk numbers.MemWeaver's HNSW does an eager, whole-index bulk load; LanceDB is lazy/mmap-backed and pages in only what a query touches. That architectural difference initially favored LanceDB on cold time-to-first-answer — but closing the gap between MemWeaver's on-disk format and its in-memory, queryable form rather than trying to make loading itself faster reversed the result: | MemWeaver HNSW | LanceDB auto partitions | LanceDB 1 partition | | |---|---|---|---| | Cold time-to-first-answer initial, file-consolidated, CRC32 verified | ~248 ms | ~212 ms | ~264 ms | Cold time-to-first-answer current | ~130 ms | ~212 ms | ~264 ms | | Warm p50 latency | 0.25–0.34 ms | 1.4–5.2 ms | 1.4–5.2 ms | | Warm peak QPS | ~14,000–15,400 | ~1,220–1,320 | ~1,220–1,320 | MemWeaver now wins on both cold time-to-first-answer and warm steady-state. Two changes closed the gap: File consolidation — 308 per-block files → one file, cutting raw cold I/O by 37% ~200ms → ~126ms . Batching the per-block mmap allocation itself was also tried and measured no benefit page-fault-in cost dominates regardless of call count — the win is entirely from fewer open / read round-trips, not from allocation batching. CRC32 scoped correctly, not skipped globally. Verification stays on for the blob/S3-restore path, where corruption risk from network transfer is real. It's removed only from the trusted local-disk restore path, where re-verifying bytes the process just wrote itself added cost ~79–128ms without addressing a real risk. The underlying principle: the cost of loading from disk was never really about disk speed — it was about the distance between the on-disk representation and the runnable, in-memory data structure. When that distance is zero the on-disk bytes are the queryable structure, addressed by relocatable offsets rather than absolute pointers , cold load approaches the physical floor of moving bytes. LanceDB's lazy/mmap model minimizes this by only ever touching what a query needs; MemWeaver's eager model now minimizes it by eliminating the reconstruction step entirely, so paying for the whole index up front costs barely more than reading it off disk. Bottom line: MemWeaver now leads on cold time-to-first-answer as well as on warm steady-state throughput ~10–50x . LanceDB's lazy/mmap model remains architecturally distinct and may still be preferable for workloads with extremely large indexes relative to available memory, where paging in only the touched fraction of a much larger-than-RAM index is the deciding factor rather than raw cold-start latency. A note on comparability: part of MemWeaver's ~130ms figure reflects skipping CRC32 verification on the trusted local-disk restore path see above . Checking Lance's own source directly lance-core , lance-io , lance-file , lance-table , lance-encoding confirms it performs no equivalent CRC32/checksum-based integrity check when reading its file format from disk either — the CRC crates present in its dependency tree are transitive S3 network-transfer checksums via the AWS SDK, general compression libraries , not part of Lance's own read path. So both numbers reflect each system's own default trust model for local-disk reads, and that model happens to match on this specific point. MemWeaver partitions vectors into time buckets. Recent vectors live in hot buckets in-memory HNSW . Older vectors are demoted to cold buckets disk or S3 . Hot buckets RAM : recent vectors, fast HNSW search Cold buckets disk : older vectors, on-demand search S3 cold tier: archived buckets, restored on demand Search spans all buckets. Results are merged and re-ranked with recency weighting — vectors in recent buckets score higher for equivalent semantic distance. // Demote a bucket to disk index.swap out bucket seq .await?; // Restore from disk for search index.swap in bucket seq .await?; // Recall is identical after swap cycle — bit-perfect restore validated Bit-perfect restore is validated: recall is identical before and after a hot→cold→hot cycle. Cold buckets can be archived to S3 or GCS, Azure Blob via object store . On search, cold buckets are temporarily read from object storage without full restoration. s3://your-bucket/{BLOB PREFIX}/ ├── catalog.json ← live collection registry └── {collection}/ ├── collection.json ← index config + WAL high-water mark ├── wal/ │ └── 00000000000000000001.wal ← binary WAL entry insert batch └── seq {n}/ ├── bucket meta.json ← commit pointer → snap {T}/ └── snap {T}/ ├── block 0.arena ← arena bytes + CRC32 ├── levels.bin ← node IDs and level assignments └── manifest.json ← HNSW entry point and max layer See docs/snapshot-format.md and docs/crash-recovery.md for the full format specification. MemWeaver scales along two dimensions: Collection sharding write scaling : Large collections are partitioned across nodes. Each node owns a subset of time buckets. Consistent hashing routes inserts to the correct shard. Search results are merged and re-ranked at query time across shards. Stateless reader nodes read scaling : Cold buckets are served from S3 by any reader node — no local state required. S3 is the source of truth. Adding reader nodes scales search throughput linearly. Hot buckets are replicated to reader nodes on demand. ┌─────────────┐ │ Search API │ └──────┬──────┘ │ ┌────────────┼────────────┐ ▼ ▼ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Reader 0 │ │ Reader 1 │ │ Reader N │ ← stateless, S3-backed └──────────┘ └──────────┘ └──────────┘ │ │ │ └────────────┼────────────┘ ▼ ┌─────────────┐ │ S3 │ ← source of truth for cold tier └─────────────┘ This is currently in progress — see Current Status current-status . | Variable | Default | Description | |---|---|---| LISTEN ADDR | 0.0.0.0:50051 | gRPC listen address | DATA DIR | ./data | Local directory for on-disk bucket files | BLOB BUCKET | — | Blob storage bucket S3/GCS/Azure ; required for WAL and snapshots | BLOB REGION | us-east-1 | Storage region | BLOB PREFIX | mem-weaver | Key prefix inside the bucket | WAL UPLOAD INTERVAL MS | 200 | How often the WAL uploader flushes to blob storage | SNAPSHOT INTERVAL SECS | — | How often arena snapshots are taken; unset disables snapshots | SNAPSHOT MIN DIRTY VECTORS | 0 | Minimum new vectors per collection before a snapshot fires; 0 = always snapshot dirty buckets. WAL replay is cheap for small counts — a value around 10000 avoids frequent small uploads while still bounding recovery time | service MemWeaver { rpc BatchInsert BatchInsertRequest returns BatchInsertResponse ; rpc Search SearchRequest returns SearchResponse ; rpc CreateCollection CreateCollectionRequest returns CreateCollectionResponse ; } message InsertItem { repeated float vector = 1; uint64 timestamp = 2; // unix seconds uint64 vector id = 3; // caller-assigned } message SearchRequest { repeated float query = 1; uint32 k = 2; uint32 ef = 3; optional uint64 time range start = 4; // optional temporal filter optional uint64 time range end = 5; } message SearchHit { uint64 vector id = 1; // caller resolves to text via their own store float distance = 2; } MemWeaver returns vector id values. Text storage is the caller's responsibility — use Postgres, SQLite, Cassandra, or S3 depending on your scale. This keeps MemWeaver focused and composable with infrastructure you already run. MemWeaver stores each vector adjacent to its HNSW edges in a single arena block: vector 1 | edges 1 | vector 2 | edges 2 | ... | vector n | edges n Cache locality during graph traversal: accessing a node's vector and its edges is a single cache line read. NodeId u32 : block index: 14 bits | offset 3: 18 bits - Arena size: 2MB. 8-byte alignment saves 3 bits. - Result: 14 bits block index + 18 bits offset → 32 GB addressable . - Node lookup: two pointer hops block array + offset arithmetic . For typical configurations, node sizes are naturally cache-line aligned: dim=128, M=16: node size = 512 vector + 128 edges = 640 bytes = 64 × 10 ✓ max level per node is stored in levels.bin alongside vector id , not collocated in the arena. It is not on the critical path during HNSW traversal. This keeps node size a clean multiple of 64 bytes and enables efficient cold storage. memory ≈ n × node size node size = dim × 4 + M MAX0 × 4 + higher layer nodes × M × 4 For dim=128, M=16, M MAX0=32: node size ≈ 640 bytes 1M vectors ≈ 640 MB 10M vectors ≈ 6.4 GB 100M vectors ≈ 64 GB Known before deployment. Not discovered in production. Download SIFT1M dataset wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz tar -xzf sift.tar.gz Set the path to the extracted sift/ directory export SIFT1M BASE PATH=/path/to/sift cargo sift1m-recall-hnsw Runs the HNSW recall cases plus hot and direct-on-disk QPS measurements. To run only the recall comparison: SIFT1M RECALL N BASE=1000000 \ SIFT1M RECALL N QUERIES=10000 \ SIFT1M PARALLEL THREADS=6 \ cargo test --release -p index --test sift1m hnsw recall \ sift1m hnsw recall vs bruteforce -- --nocapture Set SIFT1M PARALLEL THREADS to benchmark a specific parallel-insertion thread count. Without it, the recall test runs both four- and six-thread cases. SIFT1M RECALL N BASE=1000000 \ SIFT1M RECALL N QUERIES=10000 \ SIFT1M PARALLEL THREADS=6 \ cargo test --release -p index --test sift1m hnsw recall \ sift1m hnsw recall vs bruteforce -- --nocapture SIFT1M RECALL N BASE=1000000 \ SIFT1M RECALL N QUERIES=10000 \ cargo test --release -p index --test sift1m memory \ sift1m memory -- --nocapture This uses the standard SIFT corpus and HNSW construction environment variables and reports RSS and peak RSS after corpus decoding and every 10,000 inserted vectors. Set SIFT1M HNSW BENCH VARIANT=naive or both to choose the heap-backed implementation as well; set SIFT1M HNSW MEM INSERT STEP to change the reporting interval. cargo test --release -p index --test sift1m time bucket recall \ sift1m time bucket recall vs bruteforce performance -- --nocapture This builds a time-bucket index, measures recall in hot, direct-on-disk, and restored phases, then reports QPS and latency for hot and direct-on-disk search at 1, 2, 4, and 6 concurrent query threads. The performance test defaults to 1,000,000 base vectors, 10,000 queries, four insertion threads, and four time buckets. Override these for a shorter run: SIFT1M RECALL N BASE=1000000 \ SIFT1M RECALL N QUERIES=10000 \ SIFT1M PARALLEL THREADS=6 \ SIFT1M TIME BUCKET COUNTS=4 \ cargo test --release -p index --test sift1m time bucket recall \ sift1m time bucket recall vs bruteforce performance -- --nocapture Set SIFT1M PARALLEL THREADS=1 to use sequential insertion. The test reports recall statistics but does not fail on a recall threshold. MEM WEAVER S3 BUCKET=your-bucket \ MEM WEAVER S3 REGION=us-east-1 \ MEM WEAVER S3 PROFILE=default \ cargo test --release -p index --test sift1m time bucket recall \ sift1m time bucket s3 recall vs bruteforce -- --nocapture This test uses 10,000 base vectors and 10 queries by default. It uploads each cold bucket, evicts and deletes its local files, restores from S3 to a fresh directory, and then measures the restored index. It returns early when S3 credentials are unavailable. If Qdrant is running locally, the comparison test includes it automatically: macOS Apple Silicon wget https://github.com/qdrant/qdrant/releases/latest/download/qdrant-aarch64-apple-darwin.tar.gz tar -xzf qdrant-aarch64-apple-darwin.tar.gz Linux x86 64 wget https://github.com/qdrant/qdrant/releases/latest/download/qdrant-x86 64-unknown-linux-musl.tar.gz tar -xzf qdrant-x86 64-unknown-linux-musl.tar.gz ./qdrant & Run comparison — detects Qdrant automatically SIFT1M RECALL N BASE=1000000 \ SIFT1M RECALL N QUERIES=10000 \ cargo test --release -p index --test sift1m qdrant recall \ sift1m qdrant recall vs bruteforce -- --nocapture If Qdrant is not running, the test skips the comparison gracefully. Core index: - HNSW index with arena allocator - Collocated node layout vector + edges - NodeId encoding for O 1 node lookup - SIMD distance computation AVX2/AVX-512 via wide crate - Benchmarked on SIFT1M Temporal relevance: - Time-bucketed multi-HNSW - Configurable aging policy - Recency-weighted search with pluggable distance adjustment Hot/cold tiering: - Explicit hot/cold tiering — swap out / swap in - Bit-perfect restore — identical recall after swap cycle - Cold bucket search via temporary file read - S3/GCS/Azure cold tier via object store Persistence and crash recovery format /daib/mem weaver/blob/main/docs/snapshot-format.md · recovery /daib/mem weaver/blob/main/docs/crash-recovery.md · tests /daib/mem weaver/blob/main/docs/crash-recovery-tests.md : - Metadata storage levels.bin, manifest.json - Periodic arena snapshots to blob storage versioned staging, CRC32 - Write-Ahead Log — inserts acked only after WAL entry confirmed in blob storage - Crash recovery — restores from snapshot then replays WAL; cross-bucket consistent - Catalog-driven recovery — only live collections restored, stale snapshots ignored - Idempotent inserts — duplicate vector id silently skipped; BatchInsertResponse lists accepted vectors only - Dirty-block tracking — only modified arena blocks are re-uploaded; cold buckets skipped entirely - File-consolidated cold-load path one file instead of 308 per-block files - CRC32 scoped to blob/S3-restore only; skipped on trusted local-disk restores API: - gRPC API BatchInsert, Search, CreateCollection Performance: - Parallel index construction two-phase batch insertion, 1.77-2.65x speedup on 6 cores - Capacity-aware neighbor selection eliminates eviction cascades at high ef construction - Upper-level beam search ef=5 at layers 1+, eliminates local optima traps - Memory controller — automatic eviction policy Cold tier optimization: - HNSW → IVF conversion at bucket seal time - Per-cluster HNSW for cold tier reader nodes Scale: - Collection sharding across nodes write scaling - Stateless reader nodes for horizontal read scaling S3-backed Benchmarks: - Search performance — 17,021 QPS at 6 threads, near-linear scaling - Qdrant comparison benchmarks - Cold-load comparison vs. LanceDB — MemWeaver now leads on both cold and warm MIT