Repository navigation
perf(store): IRIs keep their namespace once in the dictionary (#152 step 4) - #176
Open
MathiasVDA wants to merge 1 commit into
Open
MathiasVDA wants to merge 1 commit into
MathiasVDA wants to merge 1 commit into
Conversation
An IRI takes the longest namespace (its text up to one of its last 8 slashes or hashes, at least 8 bytes) that the dictionary keeps, else the longest an earlier IRI had, which is then kept in the checkpoint's namespaces.bin. Its key is a 0, the namespace's number and the rest of the IRI, hashed as the full IRI, so lookups, saved indexes and string ranks are unchanged. On the OSM Norway extract the IRI keys take 10.7 MB instead of 29.3 MB and the store 159.5 MB instead of 178 MB, with byte-identical answers. Dictionary pieces that may hold such keys are format 6 or 7, which older binaries refuse. Refs #152
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
This is step 4 of #152 (shrinking the store). IRIs were the largest kind of dictionary key left after packed WKT (#158): 29.3 MB of the 97 MB of keys on OSM Norway, mostly repeated prefixes such as
https://www.openstreetmap.org/way/.The issue proposed HDT-style front coding in sorted buckets. Term ordinals are in insertion order (the spatial index and the routing graphs rely on that), so this PR keeps each namespace once instead:
/or#, and is at least 8 bytes long.…/way/1/geom/trackshares…/way/with…/way/1./saved 5% on Norway, because every…/way/N/geom/was a namespace of its own.namespaces.bin, which is small, checksummed and read to the heap.[Iri][0], then the namespace's number (a varint), then the rest of the IRI. An IRI written in full never has a 0 byte.dict.idx, with an unchangedindex_check) and stores written before this all work unchanged.STRSTARTSprefix ranges andcmp_textscompare IRIs through a two-partText, ordered as its two parts written out, so ORDER BY, MIN and MAX stay in code point order.OXGSDIC6), or format 7 (OXGSDIC7) when they may also hold WKT in binary form. Older binaries refuse both. Formats 1 to 5 still open and grow.How it was tested
dict::iris_keep_their_namespace_once: which keys become compact; lookups by full key and by split key;term; round trips through saved parts; a damaged compact key is refused.dict::two_part_texts_order_as_written_out.ranks::compact_iris_rank_by_their_full_text, with namespaces that sort differently from their IRIs.iris_written_once_per_namespace_read_back_the_same:namespaces.binis reported as corrupt.urn:subjects, so it still simulates a real format-4 piece.cargo test -p tripletrail-store -p tripletrail -p tripletrail-serverpasses. fmt and clippy (-D warnings) are clean on the store and server crates./mnt/project-files/benchmarks/tripletrail-pr152-namespaces/.namespaces.bin: 216 bytes.IRI(CONCAT(…))lookups take the same time.Documentation
docs/store.md: the formats paragraph describes namespaces, the key layout and formats 6 and 7.docs/guide/13-production.mdanddocs/server.md: thenamespacecolumn ofdisk-usagenow counts the IRIs written in full.Refs #152