Skip to content

About

🗂️ jit-aware dataframes

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

705 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TruffleData — Schema-aware data access for GraalVM Truffle

CI Codecov CodSpeed Release Java 25

Bring data contracts into compiled guest code. TruffleData gives Java and Elide embedders schema-aware access to database results, protobuf messages, objects, and sequences through GraalVM Truffle. Known types, presence, nullity, numeric widths, and length constraints inform validation and specialized access.

Get started · Architecture · Contracts and samples · Performance · Native Buffa integration · Bebop (experimental)

Schema result = Schema.of(Field.requiredLong("a"));
try (JdbcCursor rows = JdbcSource.execute(
    connection, "SELECT a FROM t WHERE a > ?", List.of(100L), result, Limits.DEFAULT)) {
  Batch batch;
  while ((batch = rows.nextBatch()) != null) {
    consume(batch); // Immutable rows can outlive the cursor.
  }
}

Adapters supply contracts, optionally informed by a known query or protobuf schema. Ingestion validates the data; Truffle specializes reads. The database executes and plans SQL. TruffleData neither parses SQL nor translates guest callbacks into queries.

Architecture

flowchart LR
    JDBC["JDBC results"] --> Adapt["Adapters + contracts"]
    Proto["Protobuf messages / binary / text"] --> Adapt
    Json["JSON text (simdjson)"] --> Adapt
    Csv["CSV text (strict RFC 4180)"] --> Adapt
    Bebop["Bebop binary (experimental)"] --> Adapt
    Objects["Objects + sequences"] --> Adapt
    Schema["Build-time or runtime schema"] --> Adapt
    Adapt --> Owned["Immutable batches (incl. CSV) / static rows"]
    Adapt --> Borrowed["Scoped Buffa and JSON views"]
    Adapt --> Dynamic["Live dynamic records"]
    Owned --> Access["Interop / DataLibrary / generated getters"]
    Borrowed --> Access
    Dynamic --> Access
    Access --> Guest["Truffle guest execution"]
    Guest -. "Partial evaluation" .-> Code["Compiled specialized reads"]
Loading

Owned rows retain validated facts. Live objects validate mutable values, and borrowed views enforce ownership, thread confinement, and lifetime. Java null stays at annotated ingestion boundaries; present null and missing fields have distinct representations.

Use the JVM or Native Image. The optional Myna/Buffa backend supports JVM FFM and static SVM linkage, with Java fallback. The JSON and YAML backends support the same linkage through native parsers; YAML also offers a pure-Java kernel. CSV runs on both runtimes in pure Java, with an optional native structural-index kernel. Native attachments ship prebuilt for three platforms:

Platform Static archive Shared library LTO
Linux x86-64 (linux-x86_64) .a, fat objects .so ThinLTO with clang/lld from the same archive
Linux AArch64 (linux-aarch64) .a, fat objects .so ThinLTO with clang/lld from the same archive
macOS AArch64 (macos-aarch64) .a, machine code .dylib (@rpath) ThinLTO with ld64.lld from -bitcode.a

See representation and compilation guarantees, ownership and native setup, the JSON backend, YAML backend, and CSV backend.

Warning

Bebop support is experimental. truffledata-bebop encodes and decodes Bebop values in the 2026 indexed wire layout, with a strict profile that accepts one encoding per value and in-place reads for guests. It tracks an upstream alpha (2026.0.0-alpha.6), its strict profile is not frozen, and its API, wire identities and fixtures can change in any release. No performance has been measured. Read the Bebop backend before depending on it.

Choose an artifact

Every artifact is in the dev.elide group, for example dev.elide:truffledata-core.

Artifact Purpose
truffledata-core Schemas, batches, static/dynamic objects, sequences and interop
truffledata-jdbc Driver-independent JDBC ingestion and query contracts
truffledata-jdbc-sqlite Native SQLite integration
truffledata-jdbc-mysql MySQL Connector/J integration
truffledata-jdbc-postgresql pgJDBC integration
truffledata-protobuf-runtime Stock-free protobuf plans, slab codecs and generated-record support
truffledata-protobuf Protobuf host compatibility, runtime schema compilation and contract generation
truffledata-protobuf-buffa Native slab decoding, projections and reusable borrowed sessions
truffledata-protobuf-buffa-bindings Myna bindings, pulled in by the Buffa adapter
truffledata-protobuf-buffa-cells Opt-in: Buffa slab decoding through Myna Truffle cells (-Dtruffledata.buffa.cells=true)
truffledata-cells Myna's Truffle cells runtime, pulled in by the opt-in *-cells routes
truffledata-tape Shared native-tape values, sinks, export, snapshots, sessions, streams and typed records, with the Myna value ABI; pulled in by the JSON adapter
truffledata-tape-jvm JVM FFM tape reader, pulled in by the JSON JVM backend; keep it off native-image classpaths
truffledata-json Native simdjson JSON sessions, messages, streams, typed plans and the static SVM backend
truffledata-json-jvm JVM FFM backend for JSON; keep it off native-image classpaths
truffledata-json-bindings Myna JSON bindings, pulled in by the JSON adapter
truffledata-json-cells Opt-in: the JSON document decode through Myna Truffle cells (-Dtruffledata.json.cells=true)
truffledata-yaml YAML sessions, documents, streams, metadata and typed plans (no native code)
truffledata-yaml-native Static SVM YAML backend; add it to native images that link the YAML archive
truffledata-yaml-jvm YAML JVM FFM backend; keep it off native-image classpaths
truffledata-yaml-bindings Generated native YAML bindings
truffledata-yaml-java Pure-Java YAML kernel (no native library) behind the same sessions
truffledata-yaml-snakeyaml SnakeYAML Engine event source for the Java YAML kernel
truffledata-yaml-cells Opt-in: the native YAML decode path through Myna Truffle cells (-Dtruffledata.yaml.cells=true)
truffledata-csv Strict streaming CSV into immutable string and primitive column batches, typed plans and one-pass export; pure Java
truffledata-csv-native Optional native CSV kernel and static SVM backend
truffledata-csv-jvm CSV JVM FFM provider; keep it off native-image classpaths
truffledata-csv-cells Opt-in: the native CSV kernel through Myna Truffle cells (-Dtruffledata.csv.cells=true)
truffledata-csv-bindings Generated native CSV bindings
truffledata-bebop Experimental. Bebop encoding and decoding (general and strict profiles), bebopc descriptor loading, in-place rows, arrays and column views; pure Java
truffledata-bom Aligned libraries, drivers and runtime dependencies

The runtime baseline is Java 25 and Elide GraalVM 25.5.5-elide.31. Libraries are thin JARs; choose the adapters and guest runtimes your application needs. Driver-specific artifacts bring their upstream drivers transitively.

repositories {
    mavenCentral()
    maven("https://maven.elide.dev")
    maven("https://maven.elide.dev/gvm-elide/25.5.5-elide.31")
}
dependencies {
    implementation(platform("dev.elide:truffledata-bom:0.15.0"))
    implementation("dev.elide:truffledata-core")
    implementation("dev.elide:truffledata-protobuf")
    // Add JDBC integrations or protobuf-buffa as needed.
}

Release 0.15.0 includes source/Javadoc JARs, consumer POMs, Gradle metadata, signatures, checksums, and Buffa, JSON, YAML and CSV native attachments. See publishing and consumption for Maven, Bazel and native linking.

Release 0.15.0 updates the native libraries to Elide toolchain 2026.11.7 and the bindings to Myna 0.20.0. Buffa and the experimental Bebop validator compile Myna’s implementation checks against the generated Rust records. The shared cells runtime accepts host BigInteger arguments for integer coercion. See the 0.15.0 notes.

Release 0.14.0 moves the bindings to Myna 0.19.0: each seam now declares its status codes, flags, limits and code tables once, as generated constants shared by C++, Rust and Java, with typed struct and handle pointers and counted views. It also adds trufflecheck's Truffle and Native Image rules to the build and fixes a borrowed-payload use-after-free on the JVM and six other bugs. See the 0.14.0 notes.

Release 0.13.1 is a fixes-only release: YAML (a rapidyaml undefined-behaviour fuzz finding, handler exceptions, keep-chomped scalars), CSV mapped rows, JDBC BIT columns and an unbalanced tape end(). See the 0.13.1 notes.

Release 0.13.0 adds experimental Bebop support as truffledata-bebop: encoding and decoding in the 2026 indexed wire layout, a strict profile, a bebopc descriptor loader and in-place reads. The other libraries and their natives are built from the same sources as 0.12.0. See the 0.13.0 notes.

Release 0.12.0 moves to Myna 0.18.0 and adopts its ownership model (borrows, resources, status-reporting creates, held borrows) in the opt-in Myna Truffle cells of Buffa, JSON, YAML and CSV; default routes are unchanged. Native attachments and bindings must be upgraded together (new ABI fingerprints). See the 0.12.0 notes.

Release 0.11.0 lands the SVM performance tranche: protobuf full slab decode in about 0.52× stock protobuf-java's time, faster CSV typed/string readback, opt-in lazy CsvText cells, faster JSON snapshot ingest, an opt-in Buffa Myna-cells artifact, and correctness fixes across CSV, tape, Buffa, JSON and YAML. Maven Central remains deferred under #28. See the 0.11.0 notes.

Release 0.10.0 added native YAML APIs, compact JSON readers and streaming, protobuf slab ownership and projections, allocation-free unknown-field queries, CSV input windows/channel mapping, and native CSV SDKs with musl builds. Artifacts are dual MIT/Apache-2.0 licensed and published on Elide Maven and GitHub. Maven Central publication is deferred after an organization quota rejection; see #28. See the 0.10.0 notes.

Release 0.9.0 added prebuilt natives for Linux x86-64, Linux AArch64 and macOS AArch64, with static and shared libraries, Linux FatLTO objects and a separate macOS ThinLTO archive. It uses Myna 0.13.0 and GraalVM 25.5.5-elide.27. See the 0.9.0 notes.

Release 0.8.0 added native JSON (simdjson, scoped guest values, typed plans, sessions and streams), CSV (Commons CSV scope with SIMD structural-index kernels) and a shared tape layer, on Myna 0.11.0 and Elide toolchain 2026.11.5. See the 0.8.0 notes.

Release 0.7.0 added opt-in native-index typed views and cached shared primitive access, with stronger compilation, lifetime and coverage checks. Native ABI 8 requires matching bindings and native libraries. The new shared-consumer measurements show lower Java allocation for native views, but full native ingestion remains 1.86× stock latency in that fixture. Native memory is accounted for separately; these results do not establish an overall performance win.

Performance

Measured on the released 0.12.0 code (c378d6d) on one quiet host (AMD Ryzen Threadripper PRO 9965WX), with every benchmark run in sequence on one pinned CPU. Native Image on Linux x86-64, O2, serial GC, no PGO, every cells flag off. Each bar is the median of fresh processes, and the whiskers show the full range across those processes.

  • JSON: compiled guest reads take 26.8 ns per document, against 64.8 for Jackson and 90.7 for fastjson2. Parse plus ten reads takes 757 ns: level with DSL-JSON, ahead of jsoniter, Jackson and Gson, and 1.16× slower than fastjson2. Number-heavy documents parse 2.15× faster than fastjson2 while allocating about a tenth of its Java heap.
  • Protobuf: slab decode + release takes 311 ns per message against protobuf-java's 623, and 2.38× less time on a large message; reading every field as well takes 532 ns against 669. Compiled guest reads of generated messages take 20.8 ns, less than half the 44.6 ns of direct protobuf-java getters. Interpreted reads remain 3.71× slower than those getters, though 8× faster than protobuf-java objects exposed through interop.
  • CSV: the retained typed batch reads mixed CSV at 0.73 GB/s, 3.2× FastCSV (0.66 GB/s, 3.0×, at 128 MiB), and the index scan without retention reaches 1.14 GB/s. With every field as a String, the owned batch leads FastCSV at 1 MiB (0.49 against 0.41 GB/s) and is about level at 128 MiB (0.40 against 0.39), where the copied batch is 0.92× FastCSV.
  • YAML has no other library in the native image; its card compares TruffleData's APIs.
JSON on Native Image: TruffleData against fastjson2, DSL-JSON, jsoniter, Jackson and Gson Protobuf on Native Image: TruffleData generated messages, slab and session against protobuf-java CSV on Native Image: TruffleData batches and index scan against FastCSV and Jackson CSV
YAML: TruffleData APIs compared YAML on Native Image: five TruffleData ways to consume the same parse

These are workload-specific results on one machine, not guarantees. Java heap figures exclude native memory. Comparisons are within one suite's fixture; the card notes state each fixture's terms. See the methodology and regeneration, every plotted value, raw reports and image and source hashes.

0.10.0 paired SVM charts

The 0.10.0 charts use the completed November 8 paired SVM runs: three fresh processes per image/cell, Linux x86-64, O2, serial GC, and no PGO. The baseline is 9a1dcec; each format's candidate was measured in its own preserved image. These are workload-specific results, not a new timing capture of the combined release. They predate final release fixes and packaging; exact measured sources and images are recorded with the charts.

  • Nested/repeated typed JSON ingest takes 9.5–15.5% less time. Generic JSON ingest is 2.2% slower; its compiled reads remain faster than the measured tree libraries.
  • Protobuf full slab reads take 16.2% less interpreter time / 32.1% less peak time, while direct stock getters remain faster and some cold reads regress.
  • CSV retained-string construction improves 1.5–2.0% in aggregate time. Retained batches and reference consumers have different ownership contracts.

Generic JSON: ingestion, compiled reads, and Java allocation on the same fixture

Protobuf: warm reads versus direct stock and guest interop, with cold-read tradeoffs

Typed JSON, CSV, and protobuf decode controls

Typed JSON ingest gains and the flat/generic control regressions

CSV string fixture changes and retained-batch versus reference-consumer throughput

Protobuf decode, resident consumption, and Java allocation with ownership differences

Whiskers retain every fork's range; they are not confidence intervals. Java heap figures exclude native storage. See the current report and raw evidence, chart methodology and regeneration, and source/image/artifact hashes.

Historical 0.3.0–0.5.0 measurements

The comparison suite measures AOT, interpreter, peak guest JIT, and Java heap allocation on SVM. It compares our generic and generated access APIs with protobuf Java, plus MySQL/Postgres INT4 decoder functions. All engines share one native executable.

Generated getters: interpreter reads against stock protobuf

Generated getters: peak JIT reads against stock protobuf

Generated getters: ingestion Java heap allocation

Generic API comparisons, including MySQL and PostgreSQL

Generic interpreter reads

Generic peak JIT reads

Generic ingestion allocations

Measured implementation c8ef96a, included in 0.4.0, on Elide GraalVM 25.5.5-elide.24. The 88-cell run uses three fresh timing processes and three separate allocation processes per cell. Bars show medians; whiskers show minimum/maximum process medians, not confidence intervals. Lower is better.

  • Buffa nested interpreted reads: 51.79 → 22.58 ns/row with generated getters.
  • Owned generated scalar ingestion: 29.82 ns / 56 bytes, versus stock 67.01 ns / 136 bytes.
  • Buffa scalar ingestion: 232 Java heap bytes through the generic API, 200 through generated getters; the 0.3.0 generic baseline used 320.
  • Primitive-only SQL fixture ingestion: 128 bytes, down from 160.

Stock remains faster in several binary paths. General nested Buffa messages still materialize rows and arrays. Java heap counts exclude native memory and compiler threads; SQL cases measure decoder functions, not servers or complete queries. Buffa binary input starts in unmanaged memory, and stock nested guest reads cross an explicit Truffle boundary. Large owned-byte ingestion was noisy. See the full results and methodology, raw measurements, and measurement run. Earlier runs remain available; these are workload-specific measurements, not performance guarantees.

In development: independent protobuf runtime

Development code separates the descriptor-free runtime from the optional protobuf-java bridge, adds a bounded descriptor-set compiler, and generates concrete scalar records, builders and codecs. Buffa scalar decoding uses fixed native results without allocating an index, and typed messages own their lifetimes directly.

Release 0.5.0 adopts Myna 0.8.0, adds native SIMD Unicode support, and enforces detachment before scoped rows enter owned containers. Measured allocation improvements are 128 → 80 bytes for static-row fixtures and 104 → 24 bytes for compact scalar serialization (stock: 56 bytes).

In the latest repeat, generated scalar parse/project at peak measures 24.9 ns versus stock’s 115.1 ns; Buffa Timestamp ProtoJSON measures 1.18 µs versus 10.26 µs. Generated nested full ingestion is 3.40 µs versus 2.12 µs, and serialization is 1.55 µs versus 1.19 µs. Generic nested decoding remains behind at 14.54 µs. Shared-runner timings vary substantially between runs; both reports and their ranges are retained. These changes ship in 0.5.0. These shared-runner measurements predate the latest dedicated-host profiles and paired optimization results; they are not new release measurements.

Development measurements and remaining gaps

Current development parse/project peak Current development binary serialization

See the current 120-cell run and caveats and the independent runtime API and limits. Concrete generated binary decoders and serializers cover nested/repeated schemas when every reachable message has at most 64 fields. Editions, extensions and MessageSet remain unsupported.

Protobuf and generated access

Provide schemas at build time with the Bazel/protoc contract generator, or compile descriptors at runtime. Sources accept MessageOrBuilder, binary protobuf, ProtoJSON, and protobuf text format; schemaless wire inspection is also available.

Generated Access_ classes bind to a ProtoPlan and expose primitive getters, presence checks, and array/bytes lengths for both owned and borrowed rows. Keep the accessor in a final guest-node field so partial evaluation can fold its plan and field ordinals. Reads retain contract and lifetime checks. See generated getters and executable integration samples.

Embedding JDBC

Schema result = Schema.of(Field.requiredLong("a"));
try (JdbcCursor cursor = JdbcSource.execute(
    connection, "SELECT a FROM t WHERE a > ?", List.of(100L), result, Limits.DEFAULT)) {
  Batch batch;
  while ((batch = cursor.nextBatch()) != null) {
    Batch.Row row = batch.row(0);
    // Retained rows stay valid after advancing or closing the cursor.
  }
}

The supplied connection remains owned by the caller. The cursor owns its prepared statement and result set. SQL parameters use JDBC binding. A null contract requests conservative JDBC metadata inference. Explicit contracts bind by ordinal, so aliases and duplicate database labels are supported without reparsing SQL. Duplicate labels require ordinal access.

The extended overload accepts a Decoder and StatementConfigurer. These are the driver-cooperation points for Elide: configure fetch behavior, implement exact conversions, or supply a result-specific schema. PostgreSQL cursor fetching and MySQL streaming have driver/transaction requirements; the adapter never changes transaction policy on a borrowed connection. Local batching does not bound a driver's internal buffering.

Ingestion validates every contract. There is no unchecked “trusted” escape hatch. A null, missing field, unsupported value, or incompatible numeric conversion fails rather than manufacturing a default. Cursors are single-pass and thread-confined; batch and iterator consumption cannot be mixed. Closing a cursor releases pending rows, but already returned rows remain usable.

Guest access

Elide can put a row or cursor into a guest binding without enabling Java host access. Rows implement member and ordinal access; JDBC cursors implement iterable/iterator messages and an explicit close() operation.

for (const row of rows) {
  consume(row.a);
}
// Close explicitly when stopping early.
rows.close();
for row in rows:
    consume(row.a)

The internal DataLibrary is for cooperating implementations. ReadLongNode provides a primitive read with a bounded schema/ordinal cache. It belongs beneath an adopted Truffle root or parent node; it is not a host-side replacement for the public interop API.

Representations and semantics

Representation Ownership and specialization
Batch Immutable snapshots; primitive INT64 columns, independent null/missing state
StaticLayout Language-owned immutable records; final narrow integer, floating point and boolean properties
DynamicRecord Live DynamicObject view; cached shape-aware access and validation of mutable values
ProtoPlan / ProtoRow Lossless protobuf contracts, separate presence, compact scalars and generated access
Buffa TypedRow Scoped borrowed typed views; inline small integer records and borrowed bytes
JsonMessage / TapeSnapshot Scoped native JSON index or its immutable heap copy; ordered members, exact number tokens
CsvBatch / CsvRow Immutable, unconfined CSV batches over retained input; string spans unescaped on read, primitive INT64/FLOAT64/BOOLEAN columns
DataArray Sequences without a column schema

DataNull is the interop null object; Missing is an internal presence marker. Missing fields raise the appropriate interop access error. ExactNumber preserves unsigned 64-bit integers and exact decimal values without silently rounding.

LOBs, vendor-specific JDBC values, dates/times and arbitrary host objects require explicit adapter policies. Unsupported objects are rejected. RPC execution remains extension work; protobuf text/JSON parsing is supported, and general JSON (JSON backend) and CSV (CSV backend) are in development.

Build and verify

Bazelisk selects the pinned Bazel version. Bazel is the only build system. Builds default to BuildBuddy Linux amd64 remote execution and caching, with pinned JDKs, native toolchains and executor images. Keep credentials outside source control.

bazel build //core:core //jdbc:jdbc //protobuf:protobuf
bazel test //...
bazel test //tests:compiled --nocache_test_results

Gazelle generates source membership; dependency boundaries remain explicit. See Bazel setup for configuration, local execution and target generation.

Reusable CI workflows use pinned actions and runner hardening. They check:

  • JSpecify/NullAway nullness, compiler warnings and positive/negative compiler probes.
  • Contracts, presence, numeric precision, lifetimes, budgets, shape mutation and cache fallback.
  • GraalJS/GraalPy interop, real MySQL/PostgreSQL servers, and native SQLite.
  • JVM and SVM guest compilation, blocklist rejection controls, graph/AST budgets and allocation ceilings.
  • Static native linking, Java fallback, publication consumers, Codecov coverage and JUnit reports.
  • Twenty-eight CodSpeed simulation workloads, separate from the guest-JIT timing suite.

These gates exercise supported paths and enforce regression budgets. They do not certify arbitrary downstream guest code or every deployment configuration. See Truffle checks and scope.

Reference shelf

Reference What it covers
Contracts and samples Schemas, nullity, protobuf presence, generated access and integration examples
Myna/Buffa Native parsing, borrowed views, ownership, JVM and SVM setup
JSON simdjson backend, ownership, number policy, interop and embedder export
YAML Native YAML parsing, metadata, typed plans, sessions, streams and a schema-exact emitter
CSV Strict CSV contract, limits, typed plans, streaming, interop and export
Truffle guarantees Specialization, boundaries, compilation and allocation gates
Untrusted-input testing Fuzz harnesses, sanitizer profiles, corpus replay and coverage gaps
Comparison benchmarks Raw results, charts, caveats and reproduction
Continuous benchmarks CodSpeed simulation cases and methodology
Build / Publishing Bazel, BuildBuddy, Maven artifacts and release operations
TruffleCEL CEL evaluation with native TruffleData integration

License

TruffleData is dual licensed under MIT or Apache 2.0, at your option (MIT OR Apache-2.0).

About

🗂️ jit-aware dataframes

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Used by

Contributors

Languages