Refresh PedraDB snapshot adapter to latest main and rebenchmark - #20
Draft
jaredLunde wants to merge 7 commits into
Draft
jaredLunde wants to merge 7 commits into
jaredLunde wants to merge 7 commits into
Conversation
Wire a pedradb feature (rocksdb-compat git dep) that implements the same on-disk fold contract as fjall/rocksdb, extend the conformance suite, and compare all three engines in snapshot_backends. Co-authored-by: Jared Lunde <jared.lunde@gmail.com>
Depend on rocksdb-compat directly (import as rocksdb_compat) instead of renaming the package to pedradb. Co-authored-by: Jared Lunde <jared.lunde@gmail.com>
Pin rocksdb-compat to latest Pedra (5b6789c), run backends one-at-a-time when entries ≥ 50M so a ~250 GiB disk can hold a 1B fold, and add progress logging / settle headroom checks / backend selection env knobs. Co-authored-by: Jared Lunde <jared.lunde@gmail.com>
The snapshot adapter in #19 was pinned to 5b6789c. Pedra main has moved 186 commits; keep the same SnapshotStore surface and relock against the current revision so the backend comparison can be rerun. Co-authored-by: Jared Lunde <jared.lunde@gmail.com>
The 1.3x headroom check started a 500M fjall settle on a 254 GiB volume and the rewrite climbed past 150 GiB. Require 2.1x free space, matching the documented peak, so the large-N comparison skips settle instead of filling the disk. Co-authored-by: Jared Lunde <jared.lunde@gmail.com>
main no longer contains 4d5d8c74. That commit and 057e3f61 were removed after the first pin; relock rocksdb-compat to the branch tip before rerunning the snapshot comparison. Co-authored-by: Jared Lunde <jared.lunde@gmail.com>
100M settled finishes on this host: Pedra leads hydrate, settle, and point gets. 500M Pedra does not: anonymous RSS tracks key count and the 16 GiB guest runs out of memory around 160–240M keys. Co-authored-by: Jared Lunde <jared.lunde@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-runs the PedraDB vs RocksDB vs fjall snapshot comparison from #19 against current Pedra main.
rocksdb-compatto60d88f48(mainas of 2026-09-24). Two later commits (057e3f61,4d5d8c74) are not on main and are not pinned.PedraDbSnapshotadapter compiles unchanged. Pedra snapshot-store tests pass (24/24).snapshot_backendsskipssettle()unless free disk can hold a 2× fjall rewrite.Host: 4 vCPU, 16 GiB RAM, 254 GiB ext4, 1 GiB block cache, 200 B values, one run per cell. Fjall and RocksDB do not link Pedra.
100M, settled
500M, unsettled
Settle skipped (fjall rewrite needs ~2× disk).
Pedra's 500M hydrate does not fit in 16 GiB. The OOM killer stopped the unclamped run at 48% (15.4 GiB anonymous RSS). With
PEDRA_STAGE_MAX_BYTES=67108864it was still at 11.8 GiB by 32% and was stopped. RSS tracked key count at roughly 64 B/key.Test plan
cargo test --release --features fjall,rocksdb,pedradb --test snapshot_store pedradb_