Skip to content

feat: add xor filter - #208

Open
tisonkun wants to merge 5 commits into
mainfrom
codex/xor-filter
Open

feat: add xor filter#208
tisonkun wants to merge 5 commits into
mainfrom
codex/xor-filter

Conversation

@tisonkun

@tisonkun tisonkun commented Aug 23, 2026

Copy link
Copy Markdown
Member

Summary

  • add immutable XorFilter construction and lookup with 8- and 16-bit fingerprints behind the xor feature
  • support both ordinary Rust Hash values and precomputed 64-bit hashes, with duplicate removal and bounded seeded construction retries
  • add family 22 serialization, defensive deserialization, public API documentation, README/changelog entries, and dedicated behavior/serialization tests
  • add construction and query benchmarks next to the existing sketch benchmarks, with the xorf crate as a reference implementation for comparison

Design

The implementation follows the three-partite peeling construction from Graf and Lemire's xor-filter paper directly, without adding a third-party runtime dependency. XorFilterBuilder::build returns Result because construction is deliberately bounded at 100 seed attempts instead of risking an unbounded loop. The default base construction seed is deterministic, and callers can provide another seed when needed.

Serialization follows the 3-long family 22 layout proposed in apache/datasketches-java#740. Tests use images generated by that Java implementation and verify that Rust produces byte-for-byte identical images for the same values and explicit seed. The initial API intentionally exposes only the 8- and 16-bit variants defined by that portable format.

Deserialization validates the family, version, fingerprint width, hash count, signed lengths, indexable capacity, item count, and payload length before constructing a filter.

Testing

@tisonkun
tisonkun marked this pull request as draft August 23, 2026 20:06
@tisonkun tisonkun mentioned this pull request Aug 26, 2026
9 tasks
# Conflicts:
#	CHANGELOG.md
#	README.md
#	datasketches/Cargo.toml
#	tests-integration/tests/serde_tests/xor.rs
@tisonkun
tisonkun marked this pull request as ready for review September 3, 2026 01:18
@tisonkun

tisonkun commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

FYI @leerho @freakyzoidberg I'm trying to implement XorFilter in Rust as well

Images with 32-bit fingerprints are a Rust-specific extension: the
portable family 22 format only defines 8- and 16-bit fingerprints, and
other DataSketches implementations reject 32-bit images.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant