Skip to content

fix(rdf): fix canonicalizer defects in base IRIs, fallback and JSON-LD - #52

Open
jdsika wants to merge 1 commit into
mainfrom
fix/rdf-canonicalize-defects
Open

jdsika wants to merge 1 commit into
mainfrom
fix/rdf-canonicalize-defects

Conversation

@jdsika

@jdsika jdsika commented Oct 14, 2026 •

Copy link
Copy Markdown

Replaces linkml#3987, as requested there. It fixes the canonicalizer defects that the diffable-rdf comparison (linkml#3986, linkml#3987) recorded, in linkml_runtime.utils.rdf_canonicalize itself rather than by delegating to the library. There is no new dependency, and the diff_stable path is unchanged.

Fixes

Defect on main Fix Reference
A base IRI ending in # makes rdflib read relative references back as different IRIs (http://ex.org/d##a) Declare the base without its fragment. On the rdflib fallback, keep the base only if the output reads back with the same absolute IRIs, because rdflib relativizes by string prefix RFC 3986 §5.1, §5.2.2
N-Triples or N-Quads written for a relative IRI or a literal predicate, which no parser reads ValueError. The fallback is taken because pyoxigraph's N-Triples parser refused exactly this text N-Triples 1.1 §2.2
U+2028 in a literal breaks the fallback's line sort The line sort is gone, because the fallback no longer writes line formats —
Fallback Turtle and RDF/XML vary with the hash seed (statement order, ns1/ns2 names) rdflib gets its statements, and names unbound namespaces, in sorted order —
JSON-LD is not canonicalized Serialized through pyoxigraph and indented. On the fallback, the node array is sorted JSON-LD 1.1 §4.3 (non-normative: arrays convey no order by default)
The trailing newline differs by format Exactly one —
A Dataset is silently flattened TypeError for context-aware graphs —
JSON-LD with a base: rdflib relativizes @ids without declaring @base, so they resolve against the reader's location pyoxigraph writes absolute IRIs. On the fallback, the base check drops the base —
TriG on the fallback is written inside a graph named after a random blank node Written as Turtle, which is TriG with every triple in the default graph, as on the pyoxigraph path TriG §2.1
A shared rdf:List tail is written twice on the fallback Fixed in rdflib (RDFLib/rdflib#3504, merged, not yet released). The test is a strict xfail for rdflib < 7.7 —

Behaviour changes

  • nt and nquads raise ValueError for graphs pyoxigraph rejects, instead of returning text that no N-Triples parser accepts.
  • A Dataset or ConjunctiveGraph raises TypeError. All generators and dumpers pass a plain Graph.
  • JSON-LD output has canonical blank-node labels and writes rdf:type as a property rather than as @type. The RDF is the same.

Output

Run Result
gen-owl, gen-shacl, gen-rdf and gen-shex --format rdf on the metamodel and personinfo Turtle byte-identical to main
gen-shacl --include-annotations on the metamodel, the only generator run that reaches the fallback One trailing blank line removed
gen-owl -f json-ld Isomorphic to main's output (5,318 and 1,134 triples), and now identical across processes

Tests

New tests in test_rdf_canonicalize.py, each of which fails on main:

  • An IRI written against a base reads back unchanged, for Turtle, N3, TriG, RDF/XML and JSON-LD, on both paths, with a base ending in # and one ending in /. 12 of these 20 cases fail on main.
  • Cross-process determinism with four hash seeds: Turtle, N3, TriG, RDF/XML and JSON-LD on the fallback, and JSON-LD on the normal path.
  • The line-format refusal, the newline, the Dataset refusal, and the shared list (strict xfail). fix(rdf): delegate canonicalization to the diffable-rdf library linkml/linkml#3987's defect file also passes against this module, except the shared list case, as expected with rdflib 7.6.

Two existing tests change:

  • The unsupported-format test uses pretty-xml, because json-ld is now supported.
  • The cross-process fallback test drops nt, which is now refused, and covers the formats above.

The PR also folds in the fixture cleanup from the linkml#3985 review: test_template_noblack uses mock_missing_import("black"), and mock_black_import is removed.

Local results: runtime suite 1,865 passed; compiler suite 12,393 passed; pre-commit clean.

Upstream and limitations

@jdsika
jdsika force-pushed the fix/rdf-canonicalize-defects branch from a438150 to 1d3952e Compare October 14, 2026 19:10
The comparison with diffable-rdf recorded defects in
linkml_runtime.utils.rdf_canonicalize. Fix them in the module itself:

- Declare a base IRI without its fragment, as RFC 3986 section 5.1 uses
  it. A base ending in "#" made rdflib read relative references back as
  different IRIs. On the rdflib fallback, keep the base only if the
  output reads back with the same absolute IRIs, because rdflib
  relativizes by string prefix.
- Refuse N-Triples and N-Quads with ValueError for graphs pyoxigraph
  rejects. Their N-Triples is what pyoxigraph's parser just refused;
  N-Triples 1.1 section 2.2 admits only absolute IRIs. This removes the
  fallback's line sort, which split literals on U+2028.
- On the fallback, hand rdflib the statements in sorted order and name
  unbound namespaces in sorted order, so Turtle, RDF/XML and JSON-LD no
  longer vary with the hash seed.
- Serialize JSON-LD through pyoxigraph like the other formats, indented.
  rdflib's JSON-LD writer relativized IRIs against a base without
  declaring it, so they resolved against the reader's location.
- Write TriG on the fallback as Turtle, a TriG document with every
  triple in the default graph, where the pyoxigraph path puts them.
  rdflib named the graph after a random blank node.
- End every output with exactly one newline.
- Refuse a Dataset or ConjunctiveGraph instead of flattening it.

rdflib 7.6 and earlier still write an rdf:List cell shared by two lists
once per list on the fallback; that test is a strict xfail until rdflib
releases its fix.

Also replace mock_black_import with the general mock_missing_import
fixture.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant