Skip to content

README.md

Latest commit

 

History

History
292 lines (233 loc) · 11.6 KB

README.md

File metadata and controls

292 lines (233 loc) · 11.6 KB
Raw

Rayforce

Columnar analytics and graph traversal in one fused pipeline.

MIT License Single Header Docs C Zero Dependencies Custom Allocator GitHub Stars

Rayforce is a pure C zero-dependency embeddable engine where columnar analytics and graph traversals share a single operation DAG, pass through a multi-pass optimizer, and execute as morsel-driven bytecode compiled at execution time. No malloc.

Install

Homebrew (macOS & Linux):

brew install rayforcedb/tap/rayforce

Debian / Ubuntu (.deb, x86-64) — grab the .deb from the latest release:

curl -LO https://github.com/RayforceDB/rayforce/releases/download/vX.Y.Z/rayforce_X.Y.Z_amd64.deb
sudo dpkg -i rayforce_X.Y.Z_amd64.deb

Prebuilt tarball (Linux x86-64 / macOS arm64) is also attached to each release. Or build from source below — zero dependencies, just make.

Quick Start

make            # debug build (ASan + UBSan)
make release    # optimized build
make test       # run full test suite
./rayforce      # start the Rayfall REPL

make compiles with clang when CC is not set; make CC=gcc (or CC=gcc in the environment) builds with gcc.

Rayfall REPL

Rayforce ships with Rayfall — a Lisp-like query language with a rich set of builtins. The REPL prompt is ‣:

‣ (set t (table [Symbol Side Qty]
    (list [AAPL GOOG MSFT AAPL GOOG]
          [Buy Sell Buy Sell Buy]
          [100 200 150 300 250])))

‣ (select {from:t by: Symbol Qty: (sum Qty)})
+--------+----------------------------+
| Symbol |            Qty             |
|  SYM   |            I64             |
+--------+----------------------------+
| AAPL   | 400                        |
| GOOG   | 450                        |
| MSFT   | 150                        |
+-------------------------------------+
| 3 rows (3 shown) 2 columns (2 shown)|
+-------------------------------------+

‣ (pivot t 'Symbol 'Side 'Qty sum)
+--------+-----+----------------------+
| Symbol | Buy |         Sell         |
|  SYM   | I64 |         I64          |
+--------+-----+----------------------+
| AAPL   | 100 | 300                  |
| GOOG   | 250 | 200                  |
| MSFT   | 150 | 0Nl                  |
+-------------------------------------+
| 3 rows (3 shown) 3 columns (3 shown)|
+-------------------------------------+

C API

Headers: include/rayforce.h (types, memory, atoms, vectors, tables, symbols), src/ops/ops.h (DAG construction, opcodes, optimizer, executor, graph algorithms), src/mem/heap.h (allocator lifecycle).

#include <rayforce.h>
#include "mem/heap.h"
#include "ops/ops.h"

int main(void) {
    ray_heap_init();
    ray_sym_init();

    /* Build a table */
    int64_t regions[] = {0, 0, 1, 1, 2, 2};
    int64_t amounts[] = {100, 200, 150, 300, 175, 225};
    ray_t* reg = ray_vec_from_raw(RAY_I64, regions, 6);
    ray_t* amt = ray_vec_from_raw(RAY_I64, amounts, 6);
    ray_t* tbl = ray_table_new(2);
    tbl = ray_table_add_col(tbl, ray_sym_intern("region", 6), reg);
    tbl = ray_table_add_col(tbl, ray_sym_intern("amount", 6), amt);
    ray_release(reg); ray_release(amt);

    /* Group by region, sum amounts */
    ray_graph_t* g = ray_graph_new(tbl);
    ray_op_t* keys[]    = { ray_scan(g, "region") };
    uint16_t  agg_ops[] = { OP_SUM };
    ray_op_t* agg_ins[] = { ray_scan(g, "amount") };
    ray_op_t* grp = ray_group(g, keys, 1, agg_ops, agg_ins, 1);

    ray_t* result = ray_execute(g, grp);

    if (result && !RAY_IS_ERR(result)) ray_release(result);
    ray_graph_free(g);
    ray_release(tbl);
    ray_sym_destroy();
    ray_heap_destroy();
}

Embedding

make lib builds the engine as a static library, librayforce.a, with no dependencies beyond libc, libm and pthreads. The build compiles library objects with -ffunction-sections -fdata-sections (the Makefile's LIB_CFLAGS flavour, applied automatically to the .lib.o objects that go into the archive), so no extra flags are needed at this step.

To discard everything your code never reaches, pass -Wl,--gc-sections to the linker on the final link, and -s to strip the result. A DAG-route client, meaning one that builds and executes query plans through the ray_graph_*, ray_scan and ray_execute C API and never calls ray_eval_str or loads .rfl files, never touches the Rayfall parser, compiler, REPL or IPC server, and section GC drops that whole surface (mixing the two routes pulls the parser back in):

make clean && make lib
cc -O2 -Iinclude -Isrc examples/analytics.c librayforce.a \
   -lm -lpthread -Wl,--gc-sections -s -o analytics

The command above is the Linux one. On macOS, use -Wl,-dead_strip in place of -Wl,--gc-sections (the Makefile's platform block does the same for its own release links). On MinGW, link the winsock libraries as well: -lws2_32 -lmswsock -lkernel32 -ladvapi32, with the same section-GC flag the toolchain accepts.

A checkout at the current commit links analytics.c to a 3.47 MB stripped binary, from a 10.22 MB archive; sizes move with the tree, so this is just one measurement, not a bound.

Memory follows consumed-argument conventions documented per function in the headers: arguments marked consumes are released by the call, and returns marked borrowed stay owned by the container. On error paths the library releases the references it holds. Values are COW, managed with ray_retain/ray_release; error objects are owned and freed with ray_error_free.

How It Works

Build — Construct a lazy DAG: scans, filters, joins, aggregations, window functions, graph traversals. Nothing executes yet.

Optimize — Multi-pass rewriting: type inference → constant folding → idiom rewrite → SIP → factorize → predicate pushdown → filter reorder → projection pushdown → partition pruning → DCE.

Execute — Element-wise subtrees are compiled to flat bytecode at execution time and evaluated over 1024-element morsels that stay L1-resident, with no intermediate heap allocation. Radix-partitioned hash joins size partitions to fit L2. Thread pool dispatches morsels in parallel.

Features

Execution engine

  • Lazy operation DAG — nothing runs until ray_execute
  • Multi-pass optimizer with sideways information passing
  • Expression compilation at execution time — element-wise subtrees compiled to flat bytecode, evaluated in 1024-row morsel batches with no intermediate materialization
  • Radix-partitioned hash joins sized for L2 cache
  • Thread pool with parallel morsel dispatch

Graph engine

  • Double-indexed CSR storage (forward + reverse), mmap support
  • BFS, DFS, Dijkstra, A*, PageRank, Louvain, Betweenness, LFTJ, and more
  • Factorized execution avoids materializing cross-products
  • SIP propagates selection bitmaps backward through expand chains

Rayfall language

  • Arithmetic, string, aggregation, joins, higher-order, I/O builtins
  • Lambdas compile lazily to bytecode, run in computed-goto VM
  • select/update/pivot bridge to the DAG optimizer at runtime

Memory

  • Buddy allocator with slab cache — O(1) for ~90% of allocations
  • Thread-local arenas, lock-free allocation, COW ref counting
  • No system allocator — ray_alloc/ray_free for everything

Vector search

  • Multi-metric HNSW index (cosine / L2 / inner-product) with save/load
  • Rayfall builtins: cos-dist / l2-dist / inner-prod / norm / knn and the HNSW lifecycle hnsw-build / ann / hnsw-save / hnsw-load / hnsw-free / hnsw-info
  • Filter-aware ANN via select ... where ... nearest (ann handle query) take k
  • Iterative streaming scan: the where predicate is pushed into HNSW's beam loop so rejected candidates don't consume result slots

Storage

  • Columnar files with mmap, splayed tables, date-partitioned tables
  • CSV reader with parallel mmap parse, type inference, null handling

Project Structure

include/rayforce.h         Single public header
src/mem/                    Buddy allocator, slab cache, arena, COW
src/core/                   Type system, platform abstraction, runtime
src/vec/                    Vector, list, string, selection bitmap ops
src/table/                  Table, symbol intern table
src/store/                  Column files, CSR, splayed/parted tables, HNSW
src/ops/                    DAG, optimizer, fused executor, LFTJ
src/io/                     CSV reader/writer (parallel mmap)
src/lang/                   Rayfall parser, evaluator, bytecode VM
src/app/                    REPL, terminal, pretty-printer
test/                       Test suites
examples/rfl/               Rayfall example scripts
examples/                   C API examples
website/                    Documentation site (GitHub Pages)

Documentation

Full docs: rayforcedb.github.io/rayforce

Python bindings

Rayforce has Python bindings at rayforce-py — contributions welcome.

Contributing

Contributions are welcome. You can help by:

  • Reporting bugs and requesting features via GitHub Issues
  • Submitting pull requests
  • Creating example scripts and use cases
  • Improving documentation

Development Partnership

Rayforce is jointly developed with and sponsored by Lynx.

This partnership has been instrumental in making Rayforce a mature, production-ready engine. Lynx's active involvement in development and their commitment to innovative open-source technologies in the financial sector has enabled Rayforce to reach its full potential.

License

MIT