Bring data contracts into compiled guest code. TruffleData gives Java and Elide embedders schema-aware access to database results, protobuf messages, objects, and sequences through GraalVM Truffle. Known types, presence, nullity, numeric widths, and length constraints inform validation and specialized access.
Get started · Architecture · Contracts and samples · Performance · Native Buffa integration · Bebop (experimental)
Schema result = Schema.of(Field.requiredLong("a"));
try (JdbcCursor rows = JdbcSource.execute(
connection, "SELECT a FROM t WHERE a > ?", List.of(100L), result, Limits.DEFAULT)) {
Batch batch;
while ((batch = rows.nextBatch()) != null) {
consume(batch); // Immutable rows can outlive the cursor.
}
}Adapters supply contracts, optionally informed by a known query or protobuf schema. Ingestion validates the data; Truffle specializes reads. The database executes and plans SQL. TruffleData neither parses SQL nor translates guest callbacks into queries.
flowchart LR
JDBC["JDBC results"] --> Adapt["Adapters + contracts"]
Proto["Protobuf messages / binary / text"] --> Adapt
Json["JSON text (simdjson)"] --> Adapt
Csv["CSV text (strict RFC 4180)"] --> Adapt
Bebop["Bebop binary (experimental)"] --> Adapt
Objects["Objects + sequences"] --> Adapt
Schema["Build-time or runtime schema"] --> Adapt
Adapt --> Owned["Immutable batches (incl. CSV) / static rows"]
Adapt --> Borrowed["Scoped Buffa and JSON views"]
Adapt --> Dynamic["Live dynamic records"]
Owned --> Access["Interop / DataLibrary / generated getters"]
Borrowed --> Access
Dynamic --> Access
Access --> Guest["Truffle guest execution"]
Guest -. "Partial evaluation" .-> Code["Compiled specialized reads"]
Owned rows retain validated facts. Live objects validate mutable values, and borrowed views enforce ownership, thread confinement, and lifetime. Java null stays at annotated ingestion boundaries; present null and missing fields have distinct representations.
Use the JVM or Native Image. The optional Myna/Buffa backend supports JVM FFM and static SVM linkage, with Java fallback. The JSON and YAML backends support the same linkage through native parsers; YAML also offers a pure-Java kernel. CSV runs on both runtimes in pure Java, with an optional native structural-index kernel. Native attachments ship prebuilt for three platforms:
| Platform | Static archive | Shared library | LTO |
|---|---|---|---|
Linux x86-64 (linux-x86_64) |
.a, fat objects |
.so |
ThinLTO with clang/lld from the same archive |
Linux AArch64 (linux-aarch64) |
.a, fat objects |
.so |
ThinLTO with clang/lld from the same archive |
macOS AArch64 (macos-aarch64) |
.a, machine code |
.dylib (@rpath) |
ThinLTO with ld64.lld from -bitcode.a |
See representation and compilation guarantees, ownership and native setup, the JSON backend, YAML backend, and CSV backend.
Warning
Bebop support is experimental. truffledata-bebop encodes and decodes
Bebop values in the 2026 indexed wire layout, with a
strict profile that accepts one encoding per value and in-place reads for guests. It tracks an
upstream alpha (2026.0.0-alpha.6), its strict profile is not frozen, and its API, wire
identities and fixtures can change in any release. No performance has been measured. Read the
Bebop backend before depending on it.
Every artifact is in the dev.elide group, for example dev.elide:truffledata-core.
| Artifact | Purpose |
|---|---|
truffledata-core |
Schemas, batches, static/dynamic objects, sequences and interop |
truffledata-jdbc |
Driver-independent JDBC ingestion and query contracts |
truffledata-jdbc-sqlite |
Native SQLite integration |
truffledata-jdbc-mysql |
MySQL Connector/J integration |
truffledata-jdbc-postgresql |
pgJDBC integration |
truffledata-protobuf-runtime |
Stock-free protobuf plans, slab codecs and generated-record support |
truffledata-protobuf |
Protobuf host compatibility, runtime schema compilation and contract generation |
truffledata-protobuf-buffa |
Native slab decoding, projections and reusable borrowed sessions |
truffledata-protobuf-buffa-bindings |
Myna bindings, pulled in by the Buffa adapter |
truffledata-protobuf-buffa-cells |
Opt-in: Buffa slab decoding through Myna Truffle cells (-Dtruffledata.buffa.cells=true) |
truffledata-cells |
Myna's Truffle cells runtime, pulled in by the opt-in *-cells routes |
truffledata-tape |
Shared native-tape values, sinks, export, snapshots, sessions, streams and typed records, with the Myna value ABI; pulled in by the JSON adapter |
truffledata-tape-jvm |
JVM FFM tape reader, pulled in by the JSON JVM backend; keep it off native-image classpaths |
truffledata-json |
Native simdjson JSON sessions, messages, streams, typed plans and the static SVM backend |
truffledata-json-jvm |
JVM FFM backend for JSON; keep it off native-image classpaths |
truffledata-json-bindings |
Myna JSON bindings, pulled in by the JSON adapter |
truffledata-json-cells |
Opt-in: the JSON document decode through Myna Truffle cells (-Dtruffledata.json.cells=true) |
truffledata-yaml |
YAML sessions, documents, streams, metadata and typed plans (no native code) |
truffledata-yaml-native |
Static SVM YAML backend; add it to native images that link the YAML archive |
truffledata-yaml-jvm |
YAML JVM FFM backend; keep it off native-image classpaths |
truffledata-yaml-bindings |
Generated native YAML bindings |
truffledata-yaml-java |
Pure-Java YAML kernel (no native library) behind the same sessions |
truffledata-yaml-snakeyaml |
SnakeYAML Engine event source for the Java YAML kernel |
truffledata-yaml-cells |
Opt-in: the native YAML decode path through Myna Truffle cells (-Dtruffledata.yaml.cells=true) |
truffledata-csv |
Strict streaming CSV into immutable string and primitive column batches, typed plans and one-pass export; pure Java |
truffledata-csv-native |
Optional native CSV kernel and static SVM backend |
truffledata-csv-jvm |
CSV JVM FFM provider; keep it off native-image classpaths |
truffledata-csv-cells |
Opt-in: the native CSV kernel through Myna Truffle cells (-Dtruffledata.csv.cells=true) |
truffledata-csv-bindings |
Generated native CSV bindings |
truffledata-bebop |
Experimental. Bebop encoding and decoding (general and strict profiles), bebopc descriptor loading, in-place rows, arrays and column views; pure Java |
truffledata-bom |
Aligned libraries, drivers and runtime dependencies |
The runtime baseline is Java 25 and Elide GraalVM 25.5.5-elide.31.
Libraries are thin JARs; choose the adapters and guest runtimes your application needs.
Driver-specific artifacts bring their upstream drivers transitively.
repositories {
mavenCentral()
maven("https://maven.elide.dev")
maven("https://maven.elide.dev/gvm-elide/25.5.5-elide.31")
}
dependencies {
implementation(platform("dev.elide:truffledata-bom:0.15.0"))
implementation("dev.elide:truffledata-core")
implementation("dev.elide:truffledata-protobuf")
// Add JDBC integrations or protobuf-buffa as needed.
}Release 0.15.0 includes source/Javadoc JARs, consumer POMs, Gradle metadata, signatures, checksums, and Buffa, JSON, YAML and CSV native attachments. See publishing and consumption for Maven, Bazel and native linking.
Release 0.15.0 updates the native libraries to Elide toolchain 2026.11.7 and the bindings
to Myna 0.20.0. Buffa and the experimental Bebop validator compile Myna’s implementation
checks against the generated Rust records. The shared cells runtime accepts host BigInteger
arguments for integer coercion. See the 0.15.0 notes.
Release 0.14.0 moves the bindings to Myna 0.19.0: each seam now declares its status codes, flags, limits and code tables once, as generated constants shared by C++, Rust and Java, with typed struct and handle pointers and counted views. It also adds trufflecheck's Truffle and Native Image rules to the build and fixes a borrowed-payload use-after-free on the JVM and six other bugs. See the 0.14.0 notes.
Release 0.13.1 is a fixes-only release: YAML (a rapidyaml undefined-behaviour fuzz finding, handler
exceptions, keep-chomped scalars), CSV mapped rows, JDBC BIT columns and an unbalanced tape end().
See the 0.13.1 notes.
Release 0.13.0 adds experimental Bebop support as truffledata-bebop:
encoding and decoding in the 2026 indexed wire layout, a strict profile, a bebopc descriptor
loader and in-place reads. The other libraries and their natives are built from the same sources
as 0.12.0. See the 0.13.0 notes.
Release 0.12.0 moves to Myna 0.18.0 and adopts its ownership model (borrows, resources, status-reporting creates, held borrows) in the opt-in Myna Truffle cells of Buffa, JSON, YAML and CSV; default routes are unchanged. Native attachments and bindings must be upgraded together (new ABI fingerprints). See the 0.12.0 notes.
Release 0.11.0 lands the SVM performance tranche: protobuf full slab decode
in about 0.52× stock protobuf-java's time, faster CSV typed/string readback, opt-in lazy CsvText
cells, faster JSON snapshot ingest, an opt-in Buffa Myna-cells artifact, and
correctness fixes across CSV, tape, Buffa, JSON and YAML. Maven Central remains
deferred under #28. See the
0.11.0 notes.
Release 0.10.0 added native YAML APIs, compact JSON readers and streaming, protobuf slab ownership and projections, allocation-free unknown-field queries, CSV input windows/channel mapping, and native CSV SDKs with musl builds. Artifacts are dual MIT/Apache-2.0 licensed and published on Elide Maven and GitHub. Maven Central publication is deferred after an organization quota rejection; see #28. See the 0.10.0 notes.
Release 0.9.0 added prebuilt natives for Linux x86-64, Linux AArch64 and macOS AArch64, with static and shared libraries, Linux FatLTO objects and a separate macOS ThinLTO archive. It uses Myna 0.13.0 and GraalVM 25.5.5-elide.27. See the 0.9.0 notes.
Release 0.8.0 added native JSON (simdjson, scoped guest values, typed plans, sessions and streams), CSV (Commons CSV scope with SIMD structural-index kernels) and a shared tape layer, on Myna 0.11.0 and Elide toolchain 2026.11.5. See the 0.8.0 notes.
Release 0.7.0 added opt-in native-index typed views and cached shared primitive access, with stronger compilation, lifetime and coverage checks. Native ABI 8 requires matching bindings and native libraries. The new shared-consumer measurements show lower Java allocation for native views, but full native ingestion remains 1.86× stock latency in that fixture. Native memory is accounted for separately; these results do not establish an overall performance win.
Measured on the released 0.12.0 code (c378d6d) on one quiet host (AMD Ryzen Threadripper PRO
9965WX), with every benchmark run in sequence on one pinned CPU. Native Image on Linux
x86-64, O2, serial GC, no PGO, every cells flag off. Each bar is the median of fresh
processes, and the whiskers show the full range across those processes.
- JSON: compiled guest reads take 26.8 ns per document, against 64.8 for Jackson and 90.7 for fastjson2. Parse plus ten reads takes 757 ns: level with DSL-JSON, ahead of jsoniter, Jackson and Gson, and 1.16× slower than fastjson2. Number-heavy documents parse 2.15× faster than fastjson2 while allocating about a tenth of its Java heap.
- Protobuf: slab decode + release takes 311 ns per message against protobuf-java's 623, and 2.38× less time on a large message; reading every field as well takes 532 ns against 669. Compiled guest reads of generated messages take 20.8 ns, less than half the 44.6 ns of direct protobuf-java getters. Interpreted reads remain 3.71× slower than those getters, though 8× faster than protobuf-java objects exposed through interop.
- CSV: the retained typed batch reads mixed CSV at 0.73 GB/s, 3.2× FastCSV (0.66 GB/s, 3.0×, at 128 MiB), and the index scan without retention reaches 1.14 GB/s. With every field as a String, the owned batch leads FastCSV at 1 MiB (0.49 against 0.41 GB/s) and is about level at 128 MiB (0.40 against 0.39), where the copied batch is 0.92× FastCSV.
- YAML has no other library in the native image; its card compares TruffleData's APIs.
YAML: TruffleData APIs compared
These are workload-specific results on one machine, not guarantees. Java heap figures exclude native memory. Comparisons are within one suite's fixture; the card notes state each fixture's terms. See the methodology and regeneration, every plotted value, raw reports and image and source hashes.
0.10.0 paired SVM charts
The 0.10.0 charts use the completed November 8 paired SVM runs: three fresh
processes per image/cell, Linux x86-64, O2, serial GC, and no PGO. The baseline is
9a1dcec; each format's candidate was measured in its own preserved image.
These are workload-specific results, not a new timing capture of the combined release.
They predate final release fixes and packaging; exact measured sources and images
are recorded with the charts.
- Nested/repeated typed JSON ingest takes 9.5–15.5% less time. Generic JSON ingest is 2.2% slower; its compiled reads remain faster than the measured tree libraries.
- Protobuf full slab reads take 16.2% less interpreter time / 32.1% less peak time, while direct stock getters remain faster and some cold reads regress.
- CSV retained-string construction improves 1.5–2.0% in aggregate time. Retained batches and reference consumers have different ownership contracts.
Whiskers retain every fork's range; they are not confidence intervals. Java heap figures exclude native storage. See the current report and raw evidence, chart methodology and regeneration, and source/image/artifact hashes.
Historical 0.3.0–0.5.0 measurements
The comparison suite measures AOT, interpreter, peak guest JIT, and Java heap allocation on SVM. It compares our generic and generated access APIs with protobuf Java, plus MySQL/Postgres INT4 decoder functions. All engines share one native executable.
Measured implementation c8ef96a, included in 0.4.0, on Elide GraalVM
25.5.5-elide.24. The 88-cell run uses three fresh timing processes and three separate
allocation processes per cell. Bars show medians; whiskers show minimum/maximum process
medians, not confidence intervals. Lower is better.
- Buffa nested interpreted reads: 51.79 → 22.58 ns/row with generated getters.
- Owned generated scalar ingestion: 29.82 ns / 56 bytes, versus stock 67.01 ns / 136 bytes.
- Buffa scalar ingestion: 232 Java heap bytes through the generic API, 200 through generated getters; the 0.3.0 generic baseline used 320.
- Primitive-only SQL fixture ingestion: 128 bytes, down from 160.
Stock remains faster in several binary paths. General nested Buffa messages still materialize rows and arrays. Java heap counts exclude native memory and compiler threads; SQL cases measure decoder functions, not servers or complete queries. Buffa binary input starts in unmanaged memory, and stock nested guest reads cross an explicit Truffle boundary. Large owned-byte ingestion was noisy. See the full results and methodology, raw measurements, and measurement run. Earlier runs remain available; these are workload-specific measurements, not performance guarantees.
Development code separates the descriptor-free runtime from the optional protobuf-java bridge, adds a bounded descriptor-set compiler, and generates concrete scalar records, builders and codecs. Buffa scalar decoding uses fixed native results without allocating an index, and typed messages own their lifetimes directly.
Release 0.5.0 adopts Myna 0.8.0, adds native SIMD Unicode support, and enforces detachment before scoped rows enter owned containers. Measured allocation improvements are 128 → 80 bytes for static-row fixtures and 104 → 24 bytes for compact scalar serialization (stock: 56 bytes).
In the latest repeat, generated scalar parse/project at peak measures 24.9 ns versus stock’s 115.1 ns; Buffa Timestamp ProtoJSON measures 1.18 µs versus 10.26 µs. Generated nested full ingestion is 3.40 µs versus 2.12 µs, and serialization is 1.55 µs versus 1.19 µs. Generic nested decoding remains behind at 14.54 µs. Shared-runner timings vary substantially between runs; both reports and their ranges are retained. These changes ship in 0.5.0. These shared-runner measurements predate the latest dedicated-host profiles and paired optimization results; they are not new release measurements.
Development measurements and remaining gaps
See the current 120-cell run and caveats and the independent runtime API and limits. Concrete generated binary decoders and serializers cover nested/repeated schemas when every reachable message has at most 64 fields. Editions, extensions and MessageSet remain unsupported.
Provide schemas at build time with the Bazel/protoc contract generator, or compile
descriptors at runtime. Sources accept MessageOrBuilder, binary protobuf, ProtoJSON,
and protobuf text format; schemaless wire inspection is also available.
Generated Access_ classes bind to a ProtoPlan and expose primitive getters, presence
checks, and array/bytes lengths for both owned and borrowed rows. Keep the accessor in
a final guest-node field so partial evaluation can fold its plan and field ordinals.
Reads retain contract and lifetime checks. See generated getters
and executable integration samples.
Schema result = Schema.of(Field.requiredLong("a"));
try (JdbcCursor cursor = JdbcSource.execute(
connection, "SELECT a FROM t WHERE a > ?", List.of(100L), result, Limits.DEFAULT)) {
Batch batch;
while ((batch = cursor.nextBatch()) != null) {
Batch.Row row = batch.row(0);
// Retained rows stay valid after advancing or closing the cursor.
}
}The supplied connection remains owned by the caller. The cursor owns its prepared statement and result set. SQL parameters use JDBC binding. A null contract requests conservative JDBC metadata inference. Explicit contracts bind by ordinal, so aliases and duplicate database labels are supported without reparsing SQL. Duplicate labels require ordinal access.
The extended overload accepts a Decoder and StatementConfigurer. These are the driver-cooperation points for Elide: configure fetch behavior, implement exact conversions, or supply a result-specific schema. PostgreSQL cursor fetching and MySQL streaming have driver/transaction requirements; the adapter never changes transaction policy on a borrowed connection. Local batching does not bound a driver's internal buffering.
Ingestion validates every contract. There is no unchecked “trusted” escape hatch. A null, missing field, unsupported value, or incompatible numeric conversion fails rather than manufacturing a default. Cursors are single-pass and thread-confined; batch and iterator consumption cannot be mixed. Closing a cursor releases pending rows, but already returned rows remain usable.
Elide can put a row or cursor into a guest binding without enabling Java host access. Rows implement member and ordinal access; JDBC cursors implement iterable/iterator messages and an explicit close() operation.
for (const row of rows) {
consume(row.a);
}
// Close explicitly when stopping early.
rows.close();for row in rows:
consume(row.a)The internal DataLibrary is for cooperating implementations. ReadLongNode provides a primitive read with a bounded schema/ordinal cache. It belongs beneath an adopted Truffle root or parent node; it is not a host-side replacement for the public interop API.
| Representation | Ownership and specialization |
|---|---|
Batch |
Immutable snapshots; primitive INT64 columns, independent null/missing state |
StaticLayout |
Language-owned immutable records; final narrow integer, floating point and boolean properties |
DynamicRecord |
Live DynamicObject view; cached shape-aware access and validation of mutable values |
ProtoPlan / ProtoRow |
Lossless protobuf contracts, separate presence, compact scalars and generated access |
Buffa TypedRow |
Scoped borrowed typed views; inline small integer records and borrowed bytes |
JsonMessage / TapeSnapshot |
Scoped native JSON index or its immutable heap copy; ordered members, exact number tokens |
CsvBatch / CsvRow |
Immutable, unconfined CSV batches over retained input; string spans unescaped on read, primitive INT64/FLOAT64/BOOLEAN columns |
DataArray |
Sequences without a column schema |
DataNull is the interop null object; Missing is an internal presence marker.
Missing fields raise the appropriate interop access error. ExactNumber preserves
unsigned 64-bit integers and exact decimal values without silently rounding.
LOBs, vendor-specific JDBC values, dates/times and arbitrary host objects require explicit adapter policies. Unsupported objects are rejected. RPC execution remains extension work; protobuf text/JSON parsing is supported, and general JSON (JSON backend) and CSV (CSV backend) are in development.
Bazelisk selects the pinned Bazel version. Bazel is the only build system. Builds default to BuildBuddy Linux amd64 remote execution and caching, with pinned JDKs, native toolchains and executor images. Keep credentials outside source control.
bazel build //core:core //jdbc:jdbc //protobuf:protobuf
bazel test //...
bazel test //tests:compiled --nocache_test_resultsGazelle generates source membership; dependency boundaries remain explicit. See Bazel setup for configuration, local execution and target generation.
Reusable CI workflows use pinned actions and runner hardening. They check:
- JSpecify/NullAway nullness, compiler warnings and positive/negative compiler probes.
- Contracts, presence, numeric precision, lifetimes, budgets, shape mutation and cache fallback.
- GraalJS/GraalPy interop, real MySQL/PostgreSQL servers, and native SQLite.
- JVM and SVM guest compilation, blocklist rejection controls, graph/AST budgets and allocation ceilings.
- Static native linking, Java fallback, publication consumers, Codecov coverage and JUnit reports.
- Twenty-eight CodSpeed simulation workloads, separate from the guest-JIT timing suite.
These gates exercise supported paths and enforce regression budgets. They do not certify arbitrary downstream guest code or every deployment configuration. See Truffle checks and scope.
| Reference | What it covers |
|---|---|
| Contracts and samples | Schemas, nullity, protobuf presence, generated access and integration examples |
| Myna/Buffa | Native parsing, borrowed views, ownership, JVM and SVM setup |
| JSON | simdjson backend, ownership, number policy, interop and embedder export |
| YAML | Native YAML parsing, metadata, typed plans, sessions, streams and a schema-exact emitter |
| CSV | Strict CSV contract, limits, typed plans, streaming, interop and export |
| Truffle guarantees | Specialization, boundaries, compilation and allocation gates |
| Untrusted-input testing | Fuzz harnesses, sanitizer profiles, corpus replay and coverage gaps |
| Comparison benchmarks | Raw results, charts, caveats and reproduction |
| Continuous benchmarks | CodSpeed simulation cases and methodology |
| Build / Publishing | Bazel, BuildBuddy, Maven artifacts and release operations |
| TruffleCEL | CEL evaluation with native TruffleData integration |
TruffleData is dual licensed under MIT or Apache 2.0, at your option (MIT OR Apache-2.0).