{"slug": "setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses", "title": "SetrixDB: a set engine in Go — exact set intersection over IDs (and where it loses)", "summary": "A developer built SetrixDB, an exact set engine in Go that performs presence checks and intersections over uint64 IDs using a from-scratch Minimal Perfect Hash Function (CHD v2) and AVX-512-accelerated bitset AND operations. Benchmarks on a 2 vCPU AMD EPYC server show the structure using 0.5 bytes per key versus 22.3 bytes for a Go map, with exact results verified externally against sort and comm on datasets including 25 million movie ratings and 19.2 million Wikipedia titles. The writeup explicitly documents where the approach loses, such as Roaring bitmaps beating it on random 64-bit IDs.", "body_md": "“Given an ID, is it in this list?” and “which IDs are in both lists at the same time?” These look like\n\ntextbook exercises. But when those lists hold **millions or billions** of elements and must answer in\n\n**microseconds** — in a faceted filter, a permission check, a pre-filter of candidates for an LLM — the\n\nanswer stops being trivial.\n\nThis article is about one specific primitive: an **exact set engine over `uint64` IDs**, with measured,\n\nreproducible numbers — and a dedicated section on **where it loses** to an established library. It is\n\nnot about replacing databases; it is about an operation that usually gets left open.\n\nA lot of modern software spends its time crossing **lists of identifiers**:\n\nIn all of these, what matters is **exact presence** and **exact intersection** over IDs — not\n\npayloads. Generic structures (`map`, joins, sorted scans) solve it — just not *optimally*: they carry\n\npointers, indirections and comparisons you don't need when the data **is** the number.\n\nThe core choice: always work with **`uint64` IDs**. A set is a pile of `uint64`; an intersection is an\n\n`AND`. **Everything is arithmetic.**\n\nBefore a set can exist, I need dense, collision-free identifiers. My first keygen was a *positional hash* — a simple arithmetic formula. It collided badly: on 200k short alphanumeric tokens, \n\n`\"Oa\"` and `\"0b\"` landed on the same ID.\nThe fix was implementing a **Minimal Perfect Hash Function** (CHD v2, from scratch):\n\nIf this article has one takeaway, it is this: **measure the collision rate on the real corpus**\n\nis the step almost everyone skips — and it changes the whole architecture.\n\n`uint64` ID.`AND` runs in `vpandq` + `vpopcntq`) via cgo, with `__builtin_cpu_supports`) and a scalar fallback — the same binary runs anywhere.` 1/(N+1)` of the IDs).\n**Environment (all measurements):** reference server — **2 vCPU AMD EPYC (Zen4, AVX-512), 3.8 GB RAM, Go 1.22 (+ gcc for cgo)**. Date: 09/2026.\n\n| Structure | memory | speed | exact? | \n|---|---|---|---|\n| `map[uint64]` (Go) | 22.3 B/key | 133.3M ops/s | yes | \n| **SetrixDB (MPHF CHD v2)** | **0.5 B/key** (structure) | ~118 ns/lookup | **yes** | \n| Bloom filter (1% false positive) | 1.2 B/key | 23.6M ops/s | **no** | \n\nSpeed parity with `map`, at **2.2× less memory** — and **exact**, unlike a probabilistic filter.\n\n| Strategy | dense IDs ( `denso32` ) | random 64-bit IDs ( `aleat64` ) | \n|---|---|---|\n| Sorted merge (SetrixDB) | 9.2 ms | 11.3 ms | \n| Roaring (compressed bitmap) | **148 µs** | 523 ms | \n| Hash join ( `map` ) | 91.6 ms | 94.9 ms | \n| Bitset `AND` (pure Go) | 29 µs | — | \n| **Bitset `AND` (AVX-512)** | **6 µs** | — | \n\nLet me be explicit, because a comparison without context is misleading:\n\n`universe/8`). That's where `aleat64` case, Roaring took 523 ms — but that's because it was\ndesigned for a different regime. The point is not \"I always win\"; it's **So where does it win?** In the opposite regime: **a dense universe that fits in RAM**, large sets,\n\nexact intersection on the hot path. That's exactly what a real-data test showed ↓\n\nI ran three public datasets and checked **every result externally** (`sort` + `comm`).\n\n**1,067,371** real sale lines (UK, 2009–2011). Query \"United Kingdom **AND** Q4/2011 **AND** price ≥\n\n5\" → **22,701 rows** in **823 µs**. Independent check: 22,701. Identical.\n\n`enwiki-latest-all-titles-in-ns0`: **19,264,252** titles. \"multi-word **AND** starts with `s`\" →\n\n**1,408,399** in **9.5 ms**; \"multi-word **AND** `United`\" → **38,602** in **8.0 ms**. Verified: identical.\n\n**25,000,095** real ratings; derived facets (genre, decade, score). Sets with **10.9M** and **12.4M**\n\nmembers. Three queries, all externally verified:\n\n| Query | Result | \n|---|---|\n| Drama **AND** 2000s**AND** rating ≥ 4 | **1,634,027** | \n| Drama **AND** rating ≥ 4 | **6,096,563** | \n| Comedy **AND** rating ≥ 4**AND** 2000s | **965,677** | \n\nAnd here is **the number I like most** — because it is about picking the right representation. On the\n\nsame 25M-ID universe, with sets of tens of millions:\n\n| Path | memory/set | latency (A∩B) | \n|---|---|---|\n| Sorted list merge | 87.7 MB | 80.4 ms | \n| **Dense bitset (AVX-512)** | **2 MB** | **227 µs** | \n\nSame exact result, **~350× faster and ~43× smaller**. When the universe is dense and fits in memory,\n\nthe bitset isn't just the fastest — it's the most **economical** too.\n\n**It IS** an **embeddable set engine**, in Go, that answers **exact presence** and **exact intersection**\n\nover `uint64` IDs, with a SIMD kernel, sharding and a cluster mode. It **coexists** with your current\n\ndatabase: your data stays where it is; SetrixDB sits beside it as an **index/pre-filter**.\n\n**It is NOT** a relational, columnar, NoSQL or vector database. It doesn't do SQL, joins or\n\nsimilarity. And — importantly — **it stores sets of IDs, not payloads**.\n\n`v0.1.0`).\n**\"Why not just use CRoaring/Roaring?\"**\n\nBecause Roaring is excellent — and it is the right answer when the universe doesn't fit in RAM or is\n\nvery sparse. SetrixDB targets another point: **native Go, embeddable, dense universe that fits in RAM**, with MPHF in the keygen and sharding/cluster built in. If your case is Roaring's case, use\n\n**\"Why not a `map`/Bloom filter?\"**\n\nA `map` stores pointers and is ~44× fatter per key (22.3 vs 0.5 B/key here). Bloom is smaller but it\n\n**errs** (1% false positive) — in permissions, erring toward \"can see\" is unacceptable.\n\n**\"Does MPHF handle insert/delete?\"**\n\nNo. It's built for a set. Mutable loads require a rebuild (or the sparse mode). It's a conscious\n\ntrade for `O(1)` lookup at ~4 bits/key.\n\n**\"What about Go's GC on the hot path?\"**\n\nThe bitsets are contiguous `[]uint64`, allocated once; the hot loop doesn't allocate. For DMA (NPU)\n\nthere's `UnsafePtr` + pinning — with the caveat of keeping the buffer alive.\n\n**\"Isn't this just 'bitset with AVX-512'?\"**\n\nPartly, yes — and that's fine: bitset + SIMD is a solid, well-known base. What the project adds is the\n\n**package**: a collision-free keygen, adaptive representation (dense/sparse/hybrid), sharding/cluster,\n\nand the \"stored sets\" mode (only the *name* travels over the network).\n\nSetrixDB is **open source (Apache-2.0)**. If the next wave isn't about *storing more*, but about\n\n**deciding faster** — and if set operations deserve a dedicated, exact, vectorized engine beside what\n\nyou already use — come test it.\n\n**Code, reproducible benchmarks and a quickstart:** [https://github.com/setrixdb/setrixdb](https://github.com/setrixdb/setrixdb)\n\nRun the benchmarks, open an issue, and tell me where the numbers don't add up.\n\n**SetrixDB — the arithmetic set engine.**\n\n*Sets. In microseconds. On any chip. Beside your database.*\n\n*License: Apache-2.0 · Copyright 2026 SetrixDB.*", "url": "https://wpnews.pro/news/setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses", "canonical_source": "https://dev.to/tgosoul/setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses-39dm", "published_at": "2026-09-15 21:40:11+00:00", "updated_at": "2026-09-15 22:07:14.021142+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "mlops"], "entities": ["SetrixDB", "Go", "AMD EPYC", "Roaring", "Wikipedia", "AVX-512"], "alternates": {"html": "https://wpnews.pro/news/setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses", "markdown": "https://wpnews.pro/news/setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses.md", "text": "https://wpnews.pro/news/setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses.txt", "jsonld": "https://wpnews.pro/news/setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses.jsonld"}}