# SetrixDB: a set engine in Go — exact set intersection over IDs (and where it loses)

> Source: <https://dev.to/tgosoul/setrixdb-a-set-engine-in-go-exact-set-intersection-over-ids-and-where-it-loses-39dm>
> Published: 2026-09-15 21:40:11+00:00

“Given an ID, is it in this list?” and “which IDs are in both lists at the same time?” These look like

textbook exercises. But when those lists hold **millions or billions** of elements and must answer in

**microseconds** — in a faceted filter, a permission check, a pre-filter of candidates for an LLM — the

answer stops being trivial.

This article is about one specific primitive: an **exact set engine over `uint64` IDs**, with measured,

reproducible numbers — and a dedicated section on **where it loses** to an established library. It is

not about replacing databases; it is about an operation that usually gets left open.

A lot of modern software spends its time crossing **lists of identifiers**:

In all of these, what matters is **exact presence** and **exact intersection** over IDs — not

payloads. Generic structures (`map`, joins, sorted scans) solve it — just not *optimally*: they carry

pointers, indirections and comparisons you don't need when the data **is** the number.

The core choice: always work with **`uint64` IDs**. A set is a pile of `uint64`; an intersection is an

`AND`. **Everything is arithmetic.**

Before a set can exist, I need dense, collision-free identifiers. My first keygen was a *positional hash* — a simple arithmetic formula. It collided badly: on 200k short alphanumeric tokens, 

`"Oa"` and `"0b"` landed on the same ID.
The fix was implementing a **Minimal Perfect Hash Function** (CHD v2, from scratch):

If this article has one takeaway, it is this: **measure the collision rate on the real corpus**

is the step almost everyone skips — and it changes the whole architecture.

`uint64` ID.`AND` runs in `vpandq` + `vpopcntq`) via cgo, with `__builtin_cpu_supports`) and a scalar fallback — the same binary runs anywhere.` 1/(N+1)` of the IDs).
**Environment (all measurements):** reference server — **2 vCPU AMD EPYC (Zen4, AVX-512), 3.8 GB RAM, Go 1.22 (+ gcc for cgo)**. Date: 09/2026.

| Structure | memory | speed | exact? | 
|---|---|---|---|
| `map[uint64]` (Go) | 22.3 B/key | 133.3M ops/s | yes | 
| **SetrixDB (MPHF CHD v2)** | **0.5 B/key** (structure) | ~118 ns/lookup | **yes** | 
| Bloom filter (1% false positive) | 1.2 B/key | 23.6M ops/s | **no** | 

Speed parity with `map`, at **2.2× less memory** — and **exact**, unlike a probabilistic filter.

| Strategy | dense IDs ( `denso32` ) | random 64-bit IDs ( `aleat64` ) | 
|---|---|---|
| Sorted merge (SetrixDB) | 9.2 ms | 11.3 ms | 
| Roaring (compressed bitmap) | **148 µs** | 523 ms | 
| Hash join ( `map` ) | 91.6 ms | 94.9 ms | 
| Bitset `AND` (pure Go) | 29 µs | — | 
| **Bitset `AND` (AVX-512)** | **6 µs** | — | 

Let me be explicit, because a comparison without context is misleading:

`universe/8`). That's where `aleat64` case, Roaring took 523 ms — but that's because it was
designed for a different regime. The point is not "I always win"; it's **So where does it win?** In the opposite regime: **a dense universe that fits in RAM**, large sets,

exact intersection on the hot path. That's exactly what a real-data test showed ↓

I ran three public datasets and checked **every result externally** (`sort` + `comm`).

**1,067,371** real sale lines (UK, 2009–2011). Query "United Kingdom **AND** Q4/2011 **AND** price ≥

5" → **22,701 rows** in **823 µs**. Independent check: 22,701. Identical.

`enwiki-latest-all-titles-in-ns0`: **19,264,252** titles. "multi-word **AND** starts with `s`" →

**1,408,399** in **9.5 ms**; "multi-word **AND** `United`" → **38,602** in **8.0 ms**. Verified: identical.

**25,000,095** real ratings; derived facets (genre, decade, score). Sets with **10.9M** and **12.4M**

members. Three queries, all externally verified:

| Query | Result | 
|---|---|
| Drama **AND** 2000s**AND** rating ≥ 4 | **1,634,027** | 
| Drama **AND** rating ≥ 4 | **6,096,563** | 
| Comedy **AND** rating ≥ 4**AND** 2000s | **965,677** | 

And here is **the number I like most** — because it is about picking the right representation. On the

same 25M-ID universe, with sets of tens of millions:

| Path | memory/set | latency (A∩B) | 
|---|---|---|
| Sorted list merge | 87.7 MB | 80.4 ms | 
| **Dense bitset (AVX-512)** | **2 MB** | **227 µs** | 

Same exact result, **~350× faster and ~43× smaller**. When the universe is dense and fits in memory,

the bitset isn't just the fastest — it's the most **economical** too.

**It IS** an **embeddable set engine**, in Go, that answers **exact presence** and **exact intersection**

over `uint64` IDs, with a SIMD kernel, sharding and a cluster mode. It **coexists** with your current

database: your data stays where it is; SetrixDB sits beside it as an **index/pre-filter**.

**It is NOT** a relational, columnar, NoSQL or vector database. It doesn't do SQL, joins or

similarity. And — importantly — **it stores sets of IDs, not payloads**.

`v0.1.0`).
**"Why not just use CRoaring/Roaring?"**

Because Roaring is excellent — and it is the right answer when the universe doesn't fit in RAM or is

very sparse. SetrixDB targets another point: **native Go, embeddable, dense universe that fits in RAM**, with MPHF in the keygen and sharding/cluster built in. If your case is Roaring's case, use

**"Why not a `map`/Bloom filter?"**

A `map` stores pointers and is ~44× fatter per key (22.3 vs 0.5 B/key here). Bloom is smaller but it

**errs** (1% false positive) — in permissions, erring toward "can see" is unacceptable.

**"Does MPHF handle insert/delete?"**

No. It's built for a set. Mutable loads require a rebuild (or the sparse mode). It's a conscious

trade for `O(1)` lookup at ~4 bits/key.

**"What about Go's GC on the hot path?"**

The bitsets are contiguous `[]uint64`, allocated once; the hot loop doesn't allocate. For DMA (NPU)

there's `UnsafePtr` + pinning — with the caveat of keeping the buffer alive.

**"Isn't this just 'bitset with AVX-512'?"**

Partly, yes — and that's fine: bitset + SIMD is a solid, well-known base. What the project adds is the

**package**: a collision-free keygen, adaptive representation (dense/sparse/hybrid), sharding/cluster,

and the "stored sets" mode (only the *name* travels over the network).

SetrixDB is **open source (Apache-2.0)**. If the next wave isn't about *storing more*, but about

**deciding faster** — and if set operations deserve a dedicated, exact, vectorized engine beside what

you already use — come test it.

**Code, reproducible benchmarks and a quickstart:** [https://github.com/setrixdb/setrixdb](https://github.com/setrixdb/setrixdb)

Run the benchmarks, open an issue, and tell me where the numbers don't add up.

**SetrixDB — the arithmetic set engine.**

*Sets. In microseconds. On any chip. Beside your database.*

*License: Apache-2.0 · Copyright 2026 SetrixDB.*