SetrixDB: a set engine in Go — exact set intersection over IDs (and where it loses)

A developer built SetrixDB, an exact set engine in Go that performs presence checks and intersections over uint64 IDs using a from-scratch Minimal Perfect Hash Function (CHD v2) and AVX-512-accelerated bitset AND operations. Benchmarks on a 2 vCPU AMD EPYC server show the structure using 0.5 bytes per key versus 22.3 bytes for a Go map, with exact results verified externally against sort and comm on datasets including 25 million movie ratings and 19.2 million Wikipedia titles. The writeup explicitly documents where the approach loses, such as Roaring bitmaps beating it on random 64-bit IDs.

“Given an ID, is it in this list?” and “which IDs are in both lists at the same time?” These look like textbook exercises. But when those lists hold millions or billions of elements and must answer in microseconds — in a faceted filter, a permission check, a pre-filter of candidates for an LLM — the answer stops being trivial. This article is about one specific primitive: an exact set engine over uint64 IDs , with measured, reproducible numbers — and a dedicated section on where it loses to an established library. It is not about replacing databases; it is about an operation that usually gets left open. A lot of modern software spends its time crossing lists of identifiers : In all of these, what matters is exact presence and exact intersection over IDs — not payloads. Generic structures map , joins, sorted scans solve it — just not optimally : they carry pointers, indirections and comparisons you don't need when the data is the number. The core choice: always work with uint64 IDs . A set is a pile of uint64 ; an intersection is an AND . Everything is arithmetic. Before a set can exist, I need dense, collision-free identifiers. My first keygen was a positional hash — a simple arithmetic formula. It collided badly: on 200k short alphanumeric tokens, "Oa" and "0b" landed on the same ID. The fix was implementing a Minimal Perfect Hash Function CHD v2, from scratch : If this article has one takeaway, it is this: measure the collision rate on the real corpus is the step almost everyone skips — and it changes the whole architecture. uint64 ID. AND runs in vpandq + vpopcntq via cgo, with builtin cpu supports and a scalar fallback — the same binary runs anywhere. 1/ N+1 of the IDs . Environment all measurements : reference server — 2 vCPU AMD EPYC Zen4, AVX-512 , 3.8 GB RAM, Go 1.22 + gcc for cgo . Date: 09/2026. | Structure | memory | speed | exact? | |---|---|---|---| | map uint64 Go | 22.3 B/key | 133.3M ops/s | yes | | SetrixDB MPHF CHD v2 | 0.5 B/key structure | ~118 ns/lookup | yes | | Bloom filter 1% false positive | 1.2 B/key | 23.6M ops/s | no | Speed parity with map , at 2.2× less memory — and exact , unlike a probabilistic filter. | Strategy | dense IDs denso32 | random 64-bit IDs aleat64 | |---|---|---| | Sorted merge SetrixDB | 9.2 ms | 11.3 ms | | Roaring compressed bitmap | 148 µs | 523 ms | | Hash join map | 91.6 ms | 94.9 ms | | Bitset AND pure Go | 29 µs | — | | Bitset AND AVX-512 | 6 µs | — | Let me be explicit, because a comparison without context is misleading: universe/8 . That's where aleat64 case, Roaring took 523 ms — but that's because it was designed for a different regime. The point is not "I always win"; it's So where does it win? In the opposite regime: a dense universe that fits in RAM , large sets, exact intersection on the hot path. That's exactly what a real-data test showed ↓ I ran three public datasets and checked every result externally sort + comm . 1,067,371 real sale lines UK, 2009–2011 . Query "United Kingdom AND Q4/2011 AND price ≥ 5" → 22,701 rows in 823 µs . Independent check: 22,701. Identical. enwiki-latest-all-titles-in-ns0 : 19,264,252 titles. "multi-word AND starts with s " → 1,408,399 in 9.5 ms ; "multi-word AND United " → 38,602 in 8.0 ms . Verified: identical. 25,000,095 real ratings; derived facets genre, decade, score . Sets with 10.9M and 12.4M members. Three queries, all externally verified: | Query | Result | |---|---| | Drama AND 2000s AND rating ≥ 4 | 1,634,027 | | Drama AND rating ≥ 4 | 6,096,563 | | Comedy AND rating ≥ 4 AND 2000s | 965,677 | And here is the number I like most — because it is about picking the right representation. On the same 25M-ID universe, with sets of tens of millions: | Path | memory/set | latency A∩B | |---|---|---| | Sorted list merge | 87.7 MB | 80.4 ms | | Dense bitset AVX-512 | 2 MB | 227 µs | Same exact result, ~350× faster and ~43× smaller . When the universe is dense and fits in memory, the bitset isn't just the fastest — it's the most economical too. It IS an embeddable set engine , in Go, that answers exact presence and exact intersection over uint64 IDs, with a SIMD kernel, sharding and a cluster mode. It coexists with your current database: your data stays where it is; SetrixDB sits beside it as an index/pre-filter . It is NOT a relational, columnar, NoSQL or vector database. It doesn't do SQL, joins or similarity. And — importantly — it stores sets of IDs, not payloads . v0.1.0 . "Why not just use CRoaring/Roaring?" Because Roaring is excellent — and it is the right answer when the universe doesn't fit in RAM or is very sparse. SetrixDB targets another point: native Go, embeddable, dense universe that fits in RAM , with MPHF in the keygen and sharding/cluster built in. If your case is Roaring's case, use "Why not a map /Bloom filter?" A map stores pointers and is ~44× fatter per key 22.3 vs 0.5 B/key here . Bloom is smaller but it errs 1% false positive — in permissions, erring toward "can see" is unacceptable. "Does MPHF handle insert/delete?" No. It's built for a set. Mutable loads require a rebuild or the sparse mode . It's a conscious trade for O 1 lookup at ~4 bits/key. "What about Go's GC on the hot path?" The bitsets are contiguous uint64 , allocated once; the hot loop doesn't allocate. For DMA NPU there's UnsafePtr + pinning — with the caveat of keeping the buffer alive. "Isn't this just 'bitset with AVX-512'?" Partly, yes — and that's fine: bitset + SIMD is a solid, well-known base. What the project adds is the package : a collision-free keygen, adaptive representation dense/sparse/hybrid , sharding/cluster, and the "stored sets" mode only the name travels over the network . SetrixDB is open source Apache-2.0 . If the next wave isn't about storing more , but about deciding faster — and if set operations deserve a dedicated, exact, vectorized engine beside what you already use — come test it. Code, reproducible benchmarks and a quickstart: https://github.com/setrixdb/setrixdb https://github.com/setrixdb/setrixdb Run the benchmarks, open an issue, and tell me where the numbers don't add up. SetrixDB — the arithmetic set engine. Sets. In microseconds. On any chip. Beside your database. License: Apache-2.0 · Copyright 2026 SetrixDB.