Skip to content

Filesystem catalog: don't rely on a directory listing being complete on local disk #531

Open

Description

@winding-lines

morca's local-disk commit detects a clash by listing the metadata directory, and assumes the listing is complete. The spike's Rust catalog lost 10 commits in 40 rounds of 8 writers when it listed with object_store's local walk, and none with one plain readdir. morca (iceberg-mojo) lost none in 40 rounds, but nothing in POSIX guarantees a readdir sees an entry created concurrently. Consider an atomic claim instead: create the next metadata file with O_EXCL (or link(2)) and treat EEXIST as the clash.

Found in the iceberg-rust spike (#520, spikes/iceberg-rust/README.md, PR #529).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions