Rayforce

Rayforce, a pure C17 embeddable engine with zero dependencies, fuses columnar analytics and graph traversals into a single operation DAG that executes as morsel-driven bytecode. The engine features a multi-pass optimizer, radix-partitioned hash joins sized for L2 cache, and a Lisp-like query language called Rayfall with a REPL interface. Rayforce supports graph algorithms including BFS, PageRank, and Louvain through double-indexed CSR storage, with all operations executing without malloc calls.

Columnar analytics and graph traversal in one fused pipeline. Rayforce is a pure C17 zero-dependency embeddable engine where columnar analytics and graph traversals share a single operation DAG, pass through a multi-pass optimizer, and execute as fused morsel-driven bytecode. No malloc. make debug build ASan + UBSan make release optimized build make test run full test suite ./rayforce start the Rayfall REPL Rayforce ships with Rayfall — a Lisp-like query language with a rich set of builtins. The REPL prompt is ‣ : ‣ set t table Symbol Side Qty list AAPL GOOG MSFT AAPL GOOG Buy Sell Buy Sell Buy 100 200 150 300 250 ‣ select {from:t by: Symbol Qty: sum Qty } +--------+----------------------------+ | Symbol | Qty | | sym | i64 | +--------+----------------------------+ | AAPL | 400 | | GOOG | 450 | | MSFT | 150 | +-------------------------------------+ | 3 rows 3 shown 2 columns 2 shown | +-------------------------------------+ ‣ pivot t 'Symbol 'Side 'Qty sum +--------+-----+----------------------+ | Symbol | Buy | Sell | | sym | i64 | i64 | +--------+-----+----------------------+ | AAPL | 100 | 300 | | GOOG | 250 | 200 | | MSFT | 150 | 0 | +-------------------------------------+ | 3 rows 3 shown 3 columns 3 shown | +-------------------------------------+ Headers: include/rayforce.h /RayforceDB/rayforce/blob/master/include/rayforce.h types, memory, atoms, vectors, tables, symbols , src/ops/ops.h DAG construction, opcodes, optimizer, executor, graph algorithms , src/mem/heap.h allocator lifecycle . include <rayforce.h include "mem/heap.h" include "ops/ops.h" int main void { ray heap init ; ray sym init ; / Build a table / int64 t regions = {0, 0, 1, 1, 2, 2}; int64 t amounts = {100, 200, 150, 300, 175, 225}; ray t reg = ray vec from raw RAY I64, regions, 6 ; ray t amt = ray vec from raw RAY I64, amounts, 6 ; ray t tbl = ray table new 2 ; tbl = ray table add col tbl, ray sym intern "region", 6 , reg ; tbl = ray table add col tbl, ray sym intern "amount", 6 , amt ; ray release reg ; ray release amt ; / Group by region, sum amounts / ray graph t g = ray graph new tbl ; ray op t keys = { ray scan g, "region" }; uint16 t agg ops = { OP SUM }; ray op t agg ins = { ray scan g, "amount" }; ray op t grp = ray group g, keys, 1, agg ops, agg ins, 1 ; ray t result = ray execute g, grp ; if result && RAY IS ERR result ray release result ; ray graph free g ; ray release tbl ; ray sym destroy ; ray heap destroy ; } Build — Construct a lazy DAG: scans, filters, joins, aggregations, window functions, graph traversals. Nothing executes yet. Optimize — Multi-pass rewriting: type inference → constant folding → SIP → factorize → predicate pushdown → filter reorder → projection pushdown → partition pruning → fusion → DCE. Execute — Fused morsel-driven bytecode processes 1024-element chunks that stay L1-resident. Radix-partitioned hash joins size partitions to fit L2. Thread pool dispatches morsels in parallel. Execution engine - Lazy operation DAG — nothing runs until ray execute - Multi-pass optimizer with sideways information passing - Fused morsel-driven bytecode — element-wise ops merged into single-pass chunks - Radix-partitioned hash joins sized for L2 cache - Thread pool with parallel morsel dispatch Graph engine - Double-indexed CSR storage forward + reverse , mmap support - BFS, DFS, Dijkstra, A , PageRank, Louvain, Betweenness, LFTJ, and more - Factorized execution avoids materializing cross-products - SIP propagates selection bitmaps backward through expand chains Rayfall language - Arithmetic, string, aggregation, joins, higher-order, I/O builtins - Lambdas compile lazily to bytecode, run in computed-goto VM select / update / pivot bridge to the DAG optimizer at runtime Memory - Buddy allocator with slab cache — O 1 for ~90% of allocations - Thread-local arenas, lock-free allocation, COW ref counting - No system allocator — ray alloc / ray free for everything Vector search - Multi-metric HNSW index cosine / L2 / inner-product with save/load - Rayfall builtins: cos-dist / l2-dist / inner-prod / norm / knn and the HNSW lifecycle hnsw-build / ann / hnsw-save / hnsw-load / hnsw-free / hnsw-info - Filter-aware ANN via select ... where ... nearest ann handle query take k - Iterative streaming scan: the where predicate is pushed into HNSW's beam loop so rejected candidates don't consume result slots Storage - Columnar files with mmap, splayed tables, date-partitioned tables - CSV reader with parallel mmap parse, type inference, null handling include/rayforce.h Single public header src/mem/ Buddy allocator, slab cache, arena, COW src/core/ Type system, platform abstraction, runtime src/vec/ Vector, list, string, selection bitmap ops src/table/ Table, symbol intern table src/store/ Column files, CSR, splayed/parted tables, HNSW src/ops/ DAG, optimizer, fused executor, LFTJ src/io/ CSV reader/writer parallel mmap src/lang/ Rayfall parser, evaluator, bytecode VM src/app/ REPL, terminal, pretty-printer test/ Test suites examples/rfl/ Rayfall example scripts examples/ C API examples website/ Documentation site GitHub Pages Full docs: rayforcedb.github.io/rayforce https://rayforcedb.github.io/rayforce/ Quick Start https://rayforcedb.github.io/rayforce/docs/quick-start.html — build, REPL, first query Rayfall Language https://rayforcedb.github.io/rayforce/docs/rayfall-syntax.html — syntax and builtins Data Types https://rayforcedb.github.io/rayforce/docs/data-types.html — types and collections Queries https://rayforcedb.github.io/rayforce/docs/queries-select.html — select, joins, pivot, window C API https://rayforcedb.github.io/rayforce/docs/c-api-core.html — full API reference Graph Engine https://rayforcedb.github.io/rayforce/docs/graph-algorithms.html — algorithms Architecture https://rayforcedb.github.io/rayforce/docs/architecture-pipeline.html — DAG, optimizer, memory Rayforce has Python bindings at rayforce-py — contributions welcome. Contributions are welcome. You can help by: - Reporting bugs and requesting features via GitHub Issues https://github.com/RayforceDB/rayforce/issues - Submitting pull requests - Creating example scripts and use cases - Improving documentation Rayforce is jointly developed with and sponsored by Lynx . This partnership has been instrumental in making Rayforce a mature, production-ready engine. Lynx's active involvement in development and their commitment to innovative open-source technologies in the financial sector has enabled Rayforce to reach its full potential.