This repository is an experimental Rust catalog-optional implementation of the Open Table Metadata Protocol (OTMP), with incremental metadata writes, authenticated metadata range reads, and a native DataFusion Parquet reader. Materialized SQLite readers remain the exhaustive validation oracle.
The workspace contains five crates:
otmp-protocol: portable protocol values, canonical codecs, validation, and domain-separated hashes;otmp: the catalog-free table runtime, SQLite image implementation, storage seam, staging, and publication state machine;otmp-cli: a local-directory command adapter;otmp-s3: a bounded conditional-put adapter and separate provider-evidence harness.otmp-datafusion: a DataFusion 55.0.0 table provider pinned to an authenticated metadata generation, with typed file pruning and native Parquet execution.
conformance/ contains language-neutral codec/hash fixtures and self-contained
table packages. tests/ contains the subprocess crash-evidence harness and is
not a Rust crate.
cargo run -p otmp-cli -- init TABLE --schema schema.json
cargo run -p otmp-cli -- inspect-file data.parquet
cargo run -p otmp-cli -- append TABLE --manifest append.json
cargo run -p otmp-cli -- status TABLE
cargo run -p otmp-cli -- files TABLE --ref main
cargo run -p otmp-cli -- history TABLE
cargo run -p otmp-cli -- verify TABLEAppend manifests must supply the expected SHA-256 and length. A local source path is an execution input and is excluded from logical intent identity. Command success is JSON on stdout; failures are structured JSON on stderr.
- Specification
- Qualification and reproduction
- Incremental metadata architecture
- DataFusion reader and bounded metadata access
- Catalog integration
- Conformance fixtures
- Transactions and historical reads
- Contributing
This is an evolving pre-release v1 development line. The 0.0.2-alpha
identifiers remain in use; no backward compatibility or fixture migration is
promised before the first official release.