{"slug": "rayforce", "title": "Rayforce", "summary": "Rayforce, a pure C17 embeddable engine with zero dependencies, fuses columnar analytics and graph traversals into a single operation DAG that executes as morsel-driven bytecode. The engine features a multi-pass optimizer, radix-partitioned hash joins sized for L2 cache, and a Lisp-like query language called Rayfall with a REPL interface. Rayforce supports graph algorithms including BFS, PageRank, and Louvain through double-indexed CSR storage, with all operations executing without malloc calls.", "body_md": "Columnar analytics and graph traversal in one fused pipeline.\n\nRayforce is a pure C17 zero-dependency embeddable engine where columnar analytics and graph traversals share a single operation DAG, pass through a multi-pass optimizer, and execute as fused morsel-driven bytecode. No malloc.\n\n```\nmake            # debug build (ASan + UBSan)\nmake release    # optimized build\nmake test       # run full test suite\n./rayforce      # start the Rayfall REPL\n```\n\nRayforce ships with **Rayfall** — a Lisp-like query language with a rich set\nof builtins. The REPL prompt is `‣`\n\n:\n\n```\n‣ (set t (table [Symbol Side Qty]\n    (list [AAPL GOOG MSFT AAPL GOOG]\n          [Buy Sell Buy Sell Buy]\n          [100 200 150 300 250])))\n\n‣ (select {from:t by: Symbol Qty: (sum Qty)})\n+--------+----------------------------+\n| Symbol |            Qty             |\n|  sym   |            i64             |\n+--------+----------------------------+\n| AAPL   | 400                        |\n| GOOG   | 450                        |\n| MSFT   | 150                        |\n+-------------------------------------+\n| 3 rows (3 shown) 2 columns (2 shown)|\n+-------------------------------------+\n\n‣ (pivot t 'Symbol 'Side 'Qty sum)\n+--------+-----+----------------------+\n| Symbol | Buy |         Sell         |\n|  sym   | i64 |         i64          |\n+--------+-----+----------------------+\n| AAPL   | 100 | 300                  |\n| GOOG   | 250 | 200                  |\n| MSFT   | 150 | 0                    |\n+-------------------------------------+\n| 3 rows (3 shown) 3 columns (3 shown)|\n+-------------------------------------+\n```\n\nHeaders: [ include/rayforce.h](/RayforceDB/rayforce/blob/master/include/rayforce.h) (types, memory, atoms,\nvectors, tables, symbols),\n\n`src/ops/ops.h`\n\n(DAG construction, opcodes,\noptimizer, executor, graph algorithms), `src/mem/heap.h`\n\n(allocator lifecycle).\n\n```\n#include <rayforce.h>\n#include \"mem/heap.h\"\n#include \"ops/ops.h\"\n\nint main(void) {\n    ray_heap_init();\n    ray_sym_init();\n\n    /* Build a table */\n    int64_t regions[] = {0, 0, 1, 1, 2, 2};\n    int64_t amounts[] = {100, 200, 150, 300, 175, 225};\n    ray_t* reg = ray_vec_from_raw(RAY_I64, regions, 6);\n    ray_t* amt = ray_vec_from_raw(RAY_I64, amounts, 6);\n    ray_t* tbl = ray_table_new(2);\n    tbl = ray_table_add_col(tbl, ray_sym_intern(\"region\", 6), reg);\n    tbl = ray_table_add_col(tbl, ray_sym_intern(\"amount\", 6), amt);\n    ray_release(reg); ray_release(amt);\n\n    /* Group by region, sum amounts */\n    ray_graph_t* g = ray_graph_new(tbl);\n    ray_op_t* keys[]    = { ray_scan(g, \"region\") };\n    uint16_t  agg_ops[] = { OP_SUM };\n    ray_op_t* agg_ins[] = { ray_scan(g, \"amount\") };\n    ray_op_t* grp = ray_group(g, keys, 1, agg_ops, agg_ins, 1);\n\n    ray_t* result = ray_execute(g, grp);\n\n    if (result && !RAY_IS_ERR(result)) ray_release(result);\n    ray_graph_free(g);\n    ray_release(tbl);\n    ray_sym_destroy();\n    ray_heap_destroy();\n}\n```\n\n**Build** — Construct a lazy DAG: scans, filters, joins, aggregations, window\nfunctions, graph traversals. Nothing executes yet.\n\n**Optimize** — Multi-pass rewriting: type inference → constant folding → SIP →\nfactorize → predicate pushdown → filter reorder → projection pushdown →\npartition pruning → fusion → DCE.\n\n**Execute** — Fused morsel-driven bytecode processes 1024-element chunks that\nstay L1-resident. Radix-partitioned hash joins size partitions to fit L2.\nThread pool dispatches morsels in parallel.\n\n**Execution engine**\n\n- Lazy operation DAG — nothing runs until\n`ray_execute`\n\n- Multi-pass optimizer with sideways information passing\n- Fused morsel-driven bytecode — element-wise ops merged into single-pass chunks\n- Radix-partitioned hash joins sized for L2 cache\n- Thread pool with parallel morsel dispatch\n\n**Graph engine**\n\n- Double-indexed CSR storage (forward + reverse), mmap support\n- BFS, DFS, Dijkstra, A*, PageRank, Louvain, Betweenness, LFTJ, and more\n- Factorized execution avoids materializing cross-products\n- SIP propagates selection bitmaps backward through expand chains\n\n**Rayfall language**\n\n- Arithmetic, string, aggregation, joins, higher-order, I/O builtins\n- Lambdas compile lazily to bytecode, run in computed-goto VM\n`select`\n\n/`update`\n\n/`pivot`\n\nbridge to the DAG optimizer at runtime\n\n**Memory**\n\n- Buddy allocator with slab cache — O(1) for ~90% of allocations\n- Thread-local arenas, lock-free allocation, COW ref counting\n- No system allocator —\n`ray_alloc`\n\n/`ray_free`\n\nfor everything\n\n**Vector search**\n\n- Multi-metric HNSW index (cosine / L2 / inner-product) with save/load\n- Rayfall builtins:\n`cos-dist`\n\n/`l2-dist`\n\n/`inner-prod`\n\n/`norm`\n\n/`knn`\n\nand the HNSW lifecycle`hnsw-build`\n\n/`ann`\n\n/`hnsw-save`\n\n/`hnsw-load`\n\n/`hnsw-free`\n\n/`hnsw-info`\n\n- Filter-aware ANN via\n`select ... where ... nearest (ann handle query) take k`\n\n- Iterative streaming scan: the\n`where`\n\npredicate is pushed into HNSW's beam loop so rejected candidates don't consume result slots\n\n**Storage**\n\n- Columnar files with mmap, splayed tables, date-partitioned tables\n- CSV reader with parallel mmap parse, type inference, null handling\n\n```\ninclude/rayforce.h         Single public header\nsrc/mem/                    Buddy allocator, slab cache, arena, COW\nsrc/core/                   Type system, platform abstraction, runtime\nsrc/vec/                    Vector, list, string, selection bitmap ops\nsrc/table/                  Table, symbol intern table\nsrc/store/                  Column files, CSR, splayed/parted tables, HNSW\nsrc/ops/                    DAG, optimizer, fused executor, LFTJ\nsrc/io/                     CSV reader/writer (parallel mmap)\nsrc/lang/                   Rayfall parser, evaluator, bytecode VM\nsrc/app/                    REPL, terminal, pretty-printer\ntest/                       Test suites\nexamples/rfl/               Rayfall example scripts\nexamples/                   C API examples\nwebsite/                    Documentation site (GitHub Pages)\n```\n\nFull docs: [rayforcedb.github.io/rayforce](https://rayforcedb.github.io/rayforce/)\n\n[Quick Start](https://rayforcedb.github.io/rayforce/docs/quick-start.html)— build, REPL, first query[Rayfall Language](https://rayforcedb.github.io/rayforce/docs/rayfall-syntax.html)— syntax and builtins[Data Types](https://rayforcedb.github.io/rayforce/docs/data-types.html)— types and collections[Queries](https://rayforcedb.github.io/rayforce/docs/queries-select.html)— select, joins, pivot, window[C API](https://rayforcedb.github.io/rayforce/docs/c-api-core.html)— full API reference[Graph Engine](https://rayforcedb.github.io/rayforce/docs/graph-algorithms.html)— algorithms[Architecture](https://rayforcedb.github.io/rayforce/docs/architecture-pipeline.html)— DAG, optimizer, memory\n\nRayforce has Python bindings at ** rayforce-py** — contributions welcome.\n\nContributions are welcome. You can help by:\n\n- Reporting bugs and requesting features via\n[GitHub Issues](https://github.com/RayforceDB/rayforce/issues) - Submitting pull requests\n- Creating example scripts and use cases\n- Improving documentation\n\nRayforce is jointly developed with and sponsored by ** Lynx**.\n\nThis partnership has been instrumental in making Rayforce a mature, production-ready engine. Lynx's active involvement in development and their commitment to innovative open-source technologies in the financial sector has enabled Rayforce to reach its full potential.", "url": "https://wpnews.pro/news/rayforce", "canonical_source": "https://github.com/RayforceDB/rayforce", "published_at": "2026-06-06 19:01:32+00:00", "updated_at": "2026-06-06 19:17:32.770428+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "ai-products"], "entities": ["Rayforce", "Rayfall"], "alternates": {"html": "https://wpnews.pro/news/rayforce", "markdown": "https://wpnews.pro/news/rayforce.md", "text": "https://wpnews.pro/news/rayforce.txt", "jsonld": "https://wpnews.pro/news/rayforce.jsonld"}}