A temporal vector database with predictable memory usage. Memory cost is a function of vectors, dimensions, and M — known before deployment, not discovered in production.
memory ≈ n × node_size
For dim=128, M=16: node_size = 640 bytes
, so 1M vectors requires approximately 640 MB — calculable before any data is inserted.
Most vector databases have two operational problems:
Memory is unpredictable. You cannot know your memory cost before you deploy. You provision generously, monitor closely, and scale reactively.
Search is temporally blind. A document from last week and a document from three years ago are treated identically. Recency is either ignored or handled by timestamp filters that don't affect ranking.
MemWeaver is designed around two opposite principles:
Memory is a first-class design constraint, not a runtime surprise.** Time is a first-class dimension of relevance**, not a filter applied after the fact.
MemWeaver's temporal architecture makes it a natural fit for domains where recency affects relevance.
Observability and incident search
Semantic search over logs, traces, and alerts benefits directly from temporal relevance. During an incident, engineers care about what happened in the last hour, not three weeks ago. MemWeaver surfaces recent signals first without explicit timestamp filters in every query.
News and content search
An article from last week is more relevant than the same article from three years ago for most queries. Temporal relevance is structural in MemWeaver — recent content naturally ranks ahead of semantically equivalent older content.
Code search
Large codebases accumulate years of commits. Semantic search over code benefits directly from temporal relevance — recent implementations reflect current architecture, while older code may represent deprecated patterns or refactored approaches. MemWeaver surfaces current code first, keeping stale results in cold storage without deleting them. Memory predictability matters here: embedding servers indexing active repositories need stable memory budgets, not runtime surprises at commit time.
Knowledge bases with retention policies
Enterprise knowledge bases accumulate content over years. Recent documentation, recent decisions, and recent discussions are more likely to reflect current reality. MemWeaver's cold tier moves older content to disk or S3 automatically, keeping memory costs bounded as the corpus grows.
Cost-sensitive deployments
MemWeaver's arena allocator makes memory usage predictable before deployment — no runtime surprises. The cold tier lets older vectors live on local SSD (5x memory latency) or S3, reducing infrastructure cost without sacrificing search quality or recall.
Radar is a semantic knowledge search system built on MemWeaver, indexing articles from RSS feeds and Common Crawl with temporal relevance.
Search query: "startup funding"
Recent articles surface naturally ahead of older ones. No explicit timestamp filtering — temporal relevance is structural, not bolted on.
RSS feeds / Common Crawl CDX
↓
FetchArticlesFromRSS / FetchArticlesForDomain
↓
EmbedBatch (embed-svc — sentence-transformers)
↓
StoreText → Postgres (metadata) + local disk (article bodies)
↓
BatchInsertMemWeaver (HNSW vector index)
↓
Search API → UI
Each stage is an independent Temporal activity — crash-safe, retriable, observable. Text storage is outside MemWeaver; the search API returns vector_id
values resolved against Postgres metadata.
Evaluated on SIFT1M (1M vectors, dim=128) using two hardware configurations:
AWS m5d.4xlarge(Intel Xeon, AVX-512, 16 vCPUs, 64 GB RAM) — arena vs naive comparison** Apple M-series, 6 performance cores**— parallel insertion and TimeBucket benchmarks
Configuration: M=16, M_MAX0=32, k=10. Compiled with target-cpu=native
. Recommended production configuration: ef_construction=100 + capacity-aware, ef_search=200.
| Implementation | Build time | Search (50 queries) | Speedup |
|---|---|---|---|
| Arena (MemWeaver) | 187s | 20.4ms | — |
| Naive (Vec-based) | 322s | 34.9ms | 1.72x slower |
Arena allocation is 1.72x faster to build and 1.71x faster to query. Collocating each vector with its HNSW edges means graph traversal loads vector and edges in one cache access instead of two.
The arena implementation maintains a constant ~8 MB peak overhead throughout 1M vector insertion. The naive Vec-based implementation shows reallocation spikes of 21.6 MB at 270k vectors and 35.7 MB at 530k vectors.
The arena implementation also uses 25% less total RSS at 1M vectors (608 MB growth vs 759 MB).
M=16, M_MAX0=32, ef_search=100, k=10, 10,000 queries.
| Config | Build (1 thread) | Build (6 threads) | Speedup | Mean recall@10 | Min recall |
|---|---|---|---|---|---|
| Naive (Vec-based) | 195s | — | — | 0.965 | 0.300 |
| ef_construction=40 | 124s | 70s | 1.77x | 0.965 | 0.300 |
| ef_construction=100 | 253s | 112s | 2.26x | 0.961 | 0.000 |
| ef_construction=100 + capacity-aware | 273s | 103s | |||
| 2.65x | |||||
| 0.979 | |||||
| 0.400 | |||||
| TimeBucket n=4, ef_construction=100 + capacity-aware | — | 83.5s | — | 0.9947 | |
| 0.400 |
Throughput: single thread 7,420 vec/s → two-phase 4 threads 12,624 vec/s, 1.70x speedup.
Recommended production configuration: ef_construction=100 + capacity-aware + 6 threads — 103s build (faster than ef=40 single-thread at 124s), mean recall 0.979 (better than ef=40 at 0.965), no zero-recall queries.
Why ef_construction=100 standard degrades quality: more candidates triggers aggressive diversity pruning, creating eviction cascades that isolate sparse-region nodes (13 zero-recall queries). Capacity-aware neighbor selection penalizes high-degree nodes, preventing cascades.
Why higher ef_construction gives better parallel speedup: more work per insertion means a larger parallel fraction and smaller sequential fraction (Amdahl's law). ef=40: 1.77x speedup; ef=100: 2.65x speedup.
SIFT1M, k=10, 10,000 queries, ef_construction=100 + capacity-aware.
| ef_search | Threads | QPS | p50 | p99 | Recall@10 | Min recall |
|---|---|---|---|---|---|---|
| 100 | 1 | 2,908 | 0.332ms | 0.871ms | 0.979 | 0.400 |
| 100 | 6 | 17,021 | 0.351ms | 0.512ms | 0.979 | 0.400 |
| 200 | 1 | 1,761 | 0.572ms | 0.800ms | 0.994 | 0.600 |
| 200 | 6 | 9,916 | 0.588ms | 0.813ms | 0.994 | 0.600 |
Search scales near-linearly — 97% efficiency at 6 threads for ef=100, 5.6x at ef=200.
Both run on Apple M-series, SIFT1M, M=16, ef_construction=100, k=10. Qdrant build time includes upsert (13.4s) + background indexing (19.2s) = 32.6s total.
| System | Build | ef_search | QPS (6t) | p50 | p99 | Recall@10 | Min recall |
|---|---|---|---|---|---|---|---|
| MemWeaver | 60s | 100 | ~10,400 | ~0.570ms | ~0.810ms | 0.988 | 0.400 |
| MemWeaver | |||||||
| 103s | |||||||
| 200 | |||||||
| 9,916 | |||||||
| 0.588ms | |||||||
| 0.813ms | |||||||
| 0.994 | |||||||
| 0.600 | |||||||
| Qdrant | 32.6s | 100 | 6,158 | 0.951ms | 1.495ms | 0.995 | 0.500 |
At equivalent recall (0.994 vs 0.995), MemWeaver is 1.6x higher throughput and 1.6x lower latency than Qdrant. MemWeaver min recall (0.600) is also better than Qdrant (0.500) at this operating point.
Qdrant builds 1.8x faster (32.6s vs 60s), reflecting lower ef_construction defaults and scalar quantization which reduces vector size 4x. MemWeaver uses full float32 throughout, prioritizing recall quality at equivalent parameters.
Qdrant's QPS plateaus after 4 concurrent tasks (5,866 → 6,158) due to server overhead. MemWeaver is an embedded library — parallel search scales directly with thread count with no server layer.
After swap_out (308 arena blocks, 208ms), searching directly from Mac SSD, ef_construction=100, ef_search=100:
| Threads | QPS | p50 | p99 | vs hot tier |
|---|---|---|---|---|
| 1 | 508 | 1.851ms | 2.810ms | 5.7x slower |
| 2 | 748 | 2.421ms | 5.308ms | 7.8x slower |
| 4 | 1,000 | |||
| 3.631ms | ||||
| 7.029ms | ||||
| 11.6x slower | ||||
| 6 | 588* | 10.689ms | 14.119ms | 29x slower |
*6 threads exceeds Mac SSD random read bandwidth — latency collapses. 4 threads is the optimal cold tier configuration.
Recall is identical to hot tier (0.979) — tiering is a pure performance/memory tradeoff with zero correctness cost. Production NVMe (~3GB/s vs Mac SSD ~1.5GB/s) is expected to improve cold tier QPS ~2x.
SIFT1M, k=10, 10,000 queries, ef_construction=100 + capacity-aware, restored from disk.
| Threads | QPS | p50 | p95 | p99 | Recall mean | Min recall |
|---|---|---|---|---|---|---|
| 1 | 496 | 2.077ms | 2.560ms | 2.736ms | 0.9947 | 0.400 |
| 2 | 985 | 2.039ms | 2.510ms | 2.697ms | 0.9947 | 0.400 |
| 4 | 1,909 | 2.083ms | 2.567ms | 2.761ms | 0.9947 | 0.400 |
| 6 | 2,802 | |||||
| 2.135ms | ||||||
| 2.660ms | ||||||
| 2.906ms | ||||||
| 0.9947 | ||||||
| 0.400 |
Hot, cold, and restored phases produce identical recall (phases_match=true
) — tiering across the full hot→cold→restored cycle has zero correctness cost. The higher latency vs single HNSW (2.1ms vs 0.57ms) reflects merging results across 4 independent buckets.
The read-heavy neighbor search phase (parallelized across threads) is separated from the write-heavy edge update phase (applied sequentially). Upper-level beam search (ef=5 at layers 1+) replaces greedy descent, escaping local optima. See docs/parallel_insertion.md for the full design.
The min=0.300 across all configurations reflects two hard queries in sparse regions of SIFT1M — a fundamental HNSW characteristic at M=16, ef_search=100, not an implementation issue.
Full writeup: docs/cold-load-benchmark-findings.md. SIFT1M, 1M vectors / 128 dim, page cache dropped between build and load (
sudo purge
) for genuine cold-disk numbers.MemWeaver's HNSW does an eager, whole-index bulk load; LanceDB is lazy/mmap-backed and pages in only what a query touches. That architectural difference initially favored LanceDB on cold time-to-first-answer — but closing the gap between MemWeaver's on-disk format and its in-memory, queryable form (rather than trying to make itself faster) reversed the result:
| MemWeaver (HNSW) | LanceDB (auto partitions) | LanceDB (1 partition) | |
|---|---|---|---|
| Cold time-to-first-answer (initial, file-consolidated, CRC32 verified) | ~248 ms | ~212 ms | ~264 ms |
| Cold time-to-first-answer (current) | |||
| ~130 ms | |||
| ~212 ms | ~264 ms | ||
| Warm p50 latency | 0.25–0.34 ms | 1.4–5.2 ms | 1.4–5.2 ms |
| Warm peak QPS | ~14,000–15,400 | ~1,220–1,320 | ~1,220–1,320 |
MemWeaver now wins on both cold time-to-first-answer and warm steady-state. Two changes closed the gap:
File consolidation— 308 per-block files → one file, cutting raw cold I/O by 37% (~200ms → ~126ms). Batching the per-block mmap allocation itself was also tried and measured no benefit (page-fault-in cost dominates regardless of call count) — the win is entirely from feweropen()
/read()
round-trips, not from allocation batching.CRC32 scoped correctly, not skipped globally. Verification stays on for the blob/S3-restore path, where corruption risk from network transfer is real. It's removed only from the trusted local-disk restore path, where re-verifying bytes the process just wrote itself added cost (~79–128ms) without addressing a real risk.
The underlying principle: the cost of from disk was never really about disk speed — it was about the distance between the on-disk representation and the runnable, in-memory data structure. When that distance is zero (the on-disk bytes are the queryable structure, addressed by relocatable offsets rather than absolute pointers), cold load approaches the physical floor of moving bytes. LanceDB's lazy/mmap model minimizes this by only ever touching what a query needs; MemWeaver's eager model now minimizes it by eliminating the reconstruction step entirely, so paying for the whole index up front costs barely more than reading it off disk.
Bottom line: MemWeaver now leads on cold time-to-first-answer as well as on warm steady-state throughput (~10–50x). LanceDB's lazy/mmap model remains architecturally distinct and may still be preferable for workloads with extremely large indexes relative to available memory, where paging in only the touched fraction of a much larger-than-RAM index is the deciding factor rather than raw cold-start latency.
A note on comparability: part of MemWeaver's ~130ms figure reflects skipping CRC32 verification on the trusted local-disk restore path (see above). Checking Lance's own source directly (lance-core
, lance-io
, lance-file
, lance-table
, lance-encoding
) confirms it performs no equivalent CRC32/checksum-based integrity check when reading its file format from disk either — the CRC crates present in its dependency tree are transitive (S3 network-transfer checksums via the AWS SDK, general compression libraries), not part of Lance's own read path. So both numbers reflect each system's own default trust model for local-disk reads, and that model happens to match on this specific point.
MemWeaver partitions vectors into time buckets. Recent vectors live in hot buckets (in-memory HNSW). Older vectors are demoted to cold buckets (disk or S3).
Hot buckets (RAM): recent vectors, fast HNSW search
Cold buckets (disk): older vectors, on-demand search
S3 cold tier: archived buckets, restored on demand
Search spans all buckets. Results are merged and re-ranked with recency weighting — vectors in recent buckets score higher for equivalent semantic distance.
// Demote a bucket to disk
index.swap_out(bucket_seq).await?;
// Restore from disk for search
index.swap_in(bucket_seq).await?;
// Recall is identical after swap cycle — bit-perfect restore validated
Bit-perfect restore is validated: recall is identical before and after a hot→cold→hot cycle.
Cold buckets can be archived to S3 (or GCS, Azure Blob via object_store
). On search, cold buckets are temporarily read from object storage without full restoration.
s3://your-bucket/{BLOB_PREFIX}/
├── catalog.json ← live collection registry
└── {collection}/
├── collection.json ← index config + WAL high-water mark
├── wal/
│ └── 00000000000000000001.wal ← binary WAL entry (insert batch)
└── seq_{n}/
├── bucket_meta.json ← commit pointer → snap_{T}/
└── snap_{T}/
├── block_0.arena ← arena bytes + CRC32
├── levels.bin ← node IDs and level assignments
└── manifest.json ← HNSW entry point and max layer
See docs/snapshot-format.md
and docs/crash-recovery.md
for the full format specification.
MemWeaver scales along two dimensions:
Collection sharding (write scaling):
Large collections are partitioned across nodes. Each node owns a subset of time buckets. Consistent hashing routes inserts to the correct shard. Search results are merged and re-ranked at query time across shards.
Stateless reader nodes (read scaling):
Cold buckets are served from S3 by any reader node — no local state required. S3 is the source of truth. Adding reader nodes scales search throughput linearly. Hot buckets are replicated to reader nodes on demand.
┌─────────────┐
│ Search API │
└──────┬──────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Reader 0 │ │ Reader 1 │ │ Reader N │ ← stateless, S3-backed
└──────────┘ └──────────┘ └──────────┘
│ │ │
└────────────┼────────────┘
▼
┌─────────────┐
│ S3 │ ← source of truth for cold tier
└─────────────┘
This is currently in progress — see Current Status.
| Variable | Default | Description |
|---|---|---|
LISTEN_ADDR |
||
0.0.0.0:50051 |
||
| gRPC listen address | ||
DATA_DIR |
||
./data |
||
| Local directory for on-disk bucket files | ||
BLOB_BUCKET |
||
| — | Blob storage bucket (S3/GCS/Azure); required for WAL and snapshots | |
BLOB_REGION |
||
us-east-1 |
||
| Storage region | ||
BLOB_PREFIX |
||
mem-weaver |
||
| Key prefix inside the bucket | ||
WAL_UPLOAD_INTERVAL_MS |
||
200 |
||
| How often the WAL up flushes to blob storage | ||
SNAPSHOT_INTERVAL_SECS |
||
| — | How often arena snapshots are taken; unset disables snapshots | |
SNAPSHOT_MIN_DIRTY_VECTORS |
||
0 |
||
Minimum new vectors per collection before a snapshot fires; 0 = always snapshot dirty buckets. WAL replay is cheap for small counts — a value around 10000 avoids frequent small uploads while still bounding recovery time |
service MemWeaver {
rpc BatchInsert(BatchInsertRequest) returns (BatchInsertResponse);
rpc Search(SearchRequest) returns (SearchResponse);
rpc CreateCollection(CreateCollectionRequest) returns (CreateCollectionResponse);
}
message InsertItem {
repeated float vector = 1;
uint64 timestamp = 2; // unix seconds
uint64 vector_id = 3; // caller-assigned
}
message SearchRequest {
repeated float query = 1;
uint32 k = 2;
uint32 ef = 3;
optional uint64 time_range_start = 4; // optional temporal filter
optional uint64 time_range_end = 5;
}
message SearchHit {
uint64 vector_id = 1; // caller resolves to text via their own store
float distance = 2;
}
MemWeaver returns vector_id
values. Text storage is the caller's responsibility — use Postgres, SQLite, Cassandra, or S3 depending on your scale. This keeps MemWeaver focused and composable with infrastructure you already run.
MemWeaver stores each vector adjacent to its HNSW edges in a single arena block:
[ vector_1 | edges_1 | vector_2 | edges_2 | ... | vector_n | edges_n ]
Cache locality during graph traversal: accessing a node's vector and its edges is a single cache line read.
NodeId (u32): [ block_index: 14 bits | offset >> 3: 18 bits ]
- Arena size: 2MB. 8-byte alignment saves 3 bits.
- Result: 14 bits block index + 18 bits offset → 32 GB addressable. - Node lookup: two pointer hops (block array + offset arithmetic).
For typical configurations, node sizes are naturally cache-line aligned:
dim=128, M=16:
node_size = 512 (vector) + 128 (edges) = 640 bytes = 64 × 10 ✓
max_level
per node is stored in levels.bin
alongside vector_id
, not collocated in the arena. It is not on the critical path during HNSW traversal. This keeps node size a clean multiple of 64 bytes and enables efficient cold storage.
memory ≈ n × node_size
node_size = dim × 4 + M_MAX0 × 4 + higher_layer_nodes × M × 4
For dim=128, M=16, M_MAX0=32:
node_size ≈ 640 bytes
1M vectors ≈ 640 MB
10M vectors ≈ 6.4 GB
100M vectors ≈ 64 GB
Known before deployment. Not discovered in production.
wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
tar -xzf sift.tar.gz
export SIFT1M_BASE_PATH=/path/to/sift
cargo sift1m-recall-hnsw
Runs the HNSW recall cases plus hot and direct-on-disk QPS measurements. To run only the recall comparison:
SIFT1M_RECALL_N_BASE=1000000 \
SIFT1M_RECALL_N_QUERIES=10000 \
SIFT1M_PARALLEL_THREADS=6 \
cargo test --release -p index --test sift1m_hnsw_recall \
sift1m_hnsw_recall_vs_bruteforce -- --nocapture
Set SIFT1M_PARALLEL_THREADS
to benchmark a specific parallel-insertion thread count. Without it, the recall test runs both four- and six-thread cases.
SIFT1M_RECALL_N_BASE=1000000 \
SIFT1M_RECALL_N_QUERIES=10000 \
SIFT1M_PARALLEL_THREADS=6 \
cargo test --release -p index --test sift1m_hnsw_recall \
sift1m_hnsw_recall_vs_bruteforce -- --nocapture
SIFT1M_RECALL_N_BASE=1000000 \
SIFT1M_RECALL_N_QUERIES=10000 \
cargo test --release -p index --test sift1m_memory \
sift1m_memory -- --nocapture
This uses the standard SIFT corpus and HNSW construction environment variables
and reports RSS and peak RSS after corpus decoding and every 10,000 inserted
vectors. Set SIFT1M_HNSW_BENCH_VARIANT=naive
or both
to choose the
heap-backed implementation as well; set SIFT1M_HNSW_MEM_INSERT_STEP
to change the reporting interval.
cargo test --release -p index --test sift1m_time_bucket_recall \
sift1m_time_bucket_recall_vs_bruteforce_performance -- --nocapture
This builds a time-bucket index, measures recall in hot, direct-on-disk, and restored phases, then reports QPS and latency for hot and direct-on-disk search at 1, 2, 4, and 6 concurrent query threads.
The performance test defaults to 1,000,000 base vectors, 10,000 queries, four insertion threads, and four time buckets. Override these for a shorter run:
SIFT1M_RECALL_N_BASE=1000000 \
SIFT1M_RECALL_N_QUERIES=10000 \
SIFT1M_PARALLEL_THREADS=6 \
SIFT1M_TIME_BUCKET_COUNTS=4 \
cargo test --release -p index --test sift1m_time_bucket_recall \
sift1m_time_bucket_recall_vs_bruteforce_performance -- --nocapture
Set SIFT1M_PARALLEL_THREADS=1
to use sequential insertion. The test reports recall statistics but does not fail on a recall threshold.
MEM_WEAVER_S3_BUCKET=your-bucket \
MEM_WEAVER_S3_REGION=us-east-1 \
MEM_WEAVER_S3_PROFILE=default \
cargo test --release -p index --test sift1m_time_bucket_recall \
sift1m_time_bucket_s3_recall_vs_bruteforce -- --nocapture
This test uses 10,000 base vectors and 10 queries by default. It uploads each cold bucket, evicts and deletes its local files, restores from S3 to a fresh directory, and then measures the restored index. It returns early when S3 credentials are unavailable.
If Qdrant is running locally, the comparison test includes it automatically:
wget https://github.com/qdrant/qdrant/releases/latest/download/qdrant-aarch64-apple-darwin.tar.gz
tar -xzf qdrant-aarch64-apple-darwin.tar.gz
wget https://github.com/qdrant/qdrant/releases/latest/download/qdrant-x86_64-unknown-linux-musl.tar.gz
tar -xzf qdrant-x86_64-unknown-linux-musl.tar.gz
./qdrant &
SIFT1M_RECALL_N_BASE=1000000 \
SIFT1M_RECALL_N_QUERIES=10000 \
cargo test --release -p index --test sift1m_qdrant_recall \
sift1m_qdrant_recall_vs_bruteforce -- --nocapture
If Qdrant is not running, the test skips the comparison gracefully.
Core index:
- HNSW index with arena allocator
- Collocated node layout (vector + edges)
- NodeId encoding for O(1) node lookup
- SIMD distance computation (AVX2/AVX-512 via
wide
crate) - Benchmarked on SIFT1M
Temporal relevance:
- Time-bucketed multi-HNSW
- Configurable aging policy
- Recency-weighted search with pluggable distance adjustment
Hot/cold tiering:
- Explicit hot/cold tiering — swap_out / swap_in
- Bit-perfect restore — identical recall after swap cycle
- Cold bucket search via temporary file read
- S3/GCS/Azure cold tier via
object_store
Persistence and crash recovery (format · recovery · tests):
- Metadata storage (levels.bin, manifest.json)
- Periodic arena snapshots to blob storage (versioned staging, CRC32)
- Write-Ahead Log — inserts acked only after WAL entry confirmed in blob storage
- Crash recovery — restores from snapshot then replays WAL; cross-bucket consistent
- Catalog-driven recovery — only live collections restored, stale snapshots ignored
- Idempotent inserts — duplicate
vector_id
silently skipped;BatchInsertResponse
lists accepted vectors only - Dirty-block tracking — only modified arena blocks are re-uploaded; cold buckets skipped entirely
- File-consolidated cold-load path (one file instead of 308 per-block files)
- CRC32 scoped to blob/S3-restore only; skipped on trusted local-disk restores
API:
- gRPC API (BatchInsert, Search, CreateCollection)
Performance:
- Parallel index construction (two-phase batch insertion, 1.77-2.65x speedup on 6 cores)
- Capacity-aware neighbor selection (eliminates eviction cascades at high ef_construction)
- Upper-level beam search (ef=5 at layers 1+, eliminates local optima traps)
- Memory controller — automatic eviction policy
Cold tier optimization:
- HNSW → IVF conversion at bucket seal time
- Per-cluster HNSW for cold tier reader nodes
Scale:
- Collection sharding across nodes (write scaling)
- Stateless reader nodes for horizontal read scaling (S3-backed)
Benchmarks:
- Search performance — 17,021 QPS at 6 threads, near-linear scaling
- Qdrant comparison benchmarks
- Cold-load comparison vs. LanceDB — MemWeaver now leads on both cold and warm
MIT