Skip to content

perf(store): IRIs keep their namespace once in the dictionary (#152 step 4) - #176

Open
MathiasVDA wants to merge 1 commit into
mainfrom
claude/project-thread-04ivyo-16
Open

MathiasVDA wants to merge 1 commit into
mainfrom
claude/project-thread-04ivyo-16

Conversation

@MathiasVDA

@MathiasVDA MathiasVDA commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

What and why

This is step 4 of #152 (shrinking the store). IRIs were the largest kind of dictionary key left after packed WKT (#158): 29.3 MB of the 97 MB of keys on OSM Norway, mostly repeated prefixes such as https://www.openstreetmap.org/way/.

The issue proposed HDT-style front coding in sorted buckets. Term ordinals are in insertion order (the spatial index and the routing graphs rely on that), so this PR keeps each namespace once instead:

  • A namespace is an IRI's text up to one of its last 8 / or #, and is at least 8 bytes long.
  • A new IRI takes the longest namespace the dictionary keeps. Otherwise it takes the longest namespace an earlier IRI had, and that namespace is then kept. So …/way/1/geom/track shares …/way/ with …/way/1.
    • Trying only the last / saved 5% on Norway, because every …/way/N/geom/ was a namespace of its own.
  • An IRI that is alone in its namespace stays written in full.
  • Namespaces are numbered and saved in the checkpoint's namespaces.bin, which is small, checksummed and read to the heap.
  • A compact IRI key is [Iri][0], then the namespace's number (a varint), then the rest of the IRI. An IRI written in full never has a 0 byte.
  • A compact key is hashed as the full IRI, so lookups by text, saved indexes (dict.idx, with an unchanged index_check) and stores written before this all work unchanged.
  • String ranks, STRSTARTS prefix ranges and cmp_texts compare IRIs through a two-part Text, ordered as its two parts written out, so ORDER BY, MIN and MAX stay in code point order.
  • Dictionary pieces that may hold compact IRI keys are format 6 (OXGSDIC6), or format 7 (OXGSDIC7) when they may also hold WKT in binary form. Older binaries refuse both. Formats 1 to 5 still open and grow.

How it was tested

  • Unit tests:
    • dict::iris_keep_their_namespace_once: which keys become compact; lookups by full key and by split key; term; round trips through saved parts; a damaged compact key is refused.
    • dict::two_part_texts_order_as_written_out.
    • ranks::compact_iris_rank_by_their_full_text, with namespaces that sort differently from their IRIs.
  • Store test iris_written_once_per_namespace_read_back_the_same:
    • The answers match Oxigraph, including ORDER BY, STRSTARTS, MIN and MAX on IRIs.
    • They still match after a checkpoint, a read-only open and a read-write reopen.
    • A damaged namespaces.bin is reported as corrupt.
  • Existing magic tests were updated: plain stores are now v6 and render-wkt stores v7. The format-4 test now uses urn: subjects, so it still simulates a real format-4 piece.
  • cargo test -p tripletrail-store -p tripletrail -p tripletrail-server passes. fmt and clippy (-D warnings) are clean on the store and server crates.
  • OSM Norway (1.46M triples, 580,683 IRIs), release build. Evidence is in /mnt/project-files/benchmarks/tripletrail-pr152-namespaces/.
    • Store: 178.0 MB to 159.5 MB (-10.4%). IRI keys: 29.3 MB to 10.7 MB. namespaces.bin: 216 bytes.
    • Load: 2.5 to 2.6 s, against 2.5 to 2.9 s before.
    • IRI-heavy queries: every answer is byte-identical. All 290k IRI pairs out took 0.096 s on the store with IRIs in full and 0.100 to 0.106 s here. ORDER BY of 290k IRIs, STRSTARTS and IRI(CONCAT(…)) lookups take the same time.

Documentation

  • docs/store.md: the formats paragraph describes namespaces, the key layout and formats 6 and 7.
  • docs/guide/13-production.md and docs/server.md: the namespace column of disk-usage now counts the IRIs written in full.
  • CHANGELOG: one line under Changed.

Refs #152

An IRI takes the longest namespace (its text up to one of its last 8 slashes
or hashes, at least 8 bytes) that the dictionary keeps, else the longest an
earlier IRI had, which is then kept in the checkpoint's namespaces.bin. Its
key is a 0, the namespace's number and the rest of the IRI, hashed as the full
IRI, so lookups, saved indexes and string ranks are unchanged. On the OSM
Norway extract the IRI keys take 10.7 MB instead of 29.3 MB and the store
159.5 MB instead of 178 MB, with byte-identical answers.

Dictionary pieces that may hold such keys are format 6 or 7, which older
binaries refuse.

Refs #152
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant