[rubensworks]

Compiling the same config twice could produce different output. For Comunica's query-sparql config, 5 runs of compileConfig gave 5 different engine-default.js files. This showed up as noise when comparing Comunica's browser bundle sizes between commits (comunica/comunica#1954): a PR without engine changes reported changes of a few bytes in 9 of 10 bundles.

Causes

1. Quad order depended on I/O timing. RdfStreamIncluder fetched and parsed imports in parallel and pushed their quads as soon as they arrived. Values of a list spread over multiple files, such as the actors of Comunica's Runner, therefore came out in a different order on each run, and so did the order of instantiations.
2. Blank node labels depended on parse order. Each parse created a DataFactory with the default df_<global counter>_ prefix, so labels depended on which file started parsing first. These labels end up as variable names in compiled configs.

Changes

• RdfStreamIncluder still starts parsing each import as soon as it is detected. Once the importing stream has ended, it reads the import streams one after another, in the order in which the imports occur. Imports that are not being read yet pause once their stream buffer is full, so quads keep streaming and memory stays bounded. Errors are still emitted right away. This also removes a latent hang: if an import ended before the importing stream flushed, the runningImporters decrement was skipped.
• RdfParserOptions has a new optional blankNodePrefix. If it is set and no dataFactory is given, the parser uses a DataFactory with that prefix. Imports derive their prefix as <parent prefix><import index>_. Labels are always a prefix followed by digits, so labels of different files can't clash.
• ConfigRegistry.register and ComponentRegistry.registerModule assign config_<n>_ and module_<n>_ prefixes in call order. registerAvailableModules registers modules in sorted order, as module discovery order varies between runs.

Behavior changes

• Imported quads now come after the quads of the importing file, one import after another. Before, the order was effectively random. Anything that depends on the order of list values spread over files gets a stable order, but not necessarily the one a previous run happened to produce.
• Blank node labels in compiled configs change from, for example, df_418_1 to module_71_0_1.
• Imports that come later in the order are only parsed up to their stream buffer until it is their turn, so less of their parsing overlaps. For Comunica's query-sparql config, compiling takes about as long as before (5.2 to 5.9 s on master and on this branch).

Verification

• For Comunica, 10 compiles of query-sparql now give identical output, as do all its other engine configs, including the browser config. Two separate installs in different directories also give identical output. Three full cycles of regenerating all engine files and rebuilding all 10 browser bundles gave byte-identical results.
• 9 new tests. Six of them fail on the previous code: import order with a slow first import, nested import order, blank node prefixes for imports, sorted module registration, and prefixes for both registries. One checks that the quads of an import are emitted while it is still being parsed. The other two cover an explicit dataFactory taking precedence and errors not waiting for slow imports.
• yarn lint, yarn build, yarn test:jest (871 tests, 100% coverage) and yarn test:webpack pass.

Not changed

These sources of nondeterminism remain. None of them changed Comunica's compiled output in my runs:

• If several copies of a module with the same major version are installed, ModuleStateBuilder uses whichever copy it processes last, and that order varies.
• Streams of different component files still interleave in the shared object loader. This only affects resources that are described in multiple component files.
• In Turtle files, N3 prefixes labeled blank nodes (_:x) with its own global counter, which rdf-parse doesn't expose.

---

Generated by Claude Code

[coveralls]

Coverage Report for CI Build 42386710967

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage remained the same at 100.0%

Details

• Coverage remained the same as the base build.
• Patch coverage: 20 of 20 lines across 4 files are fully covered (100%).
• No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	1529
Covered Lines:	1529
Line Coverage:	100.0%
Relevant Branches:	1262
Covered Branches:	1262
Branch Coverage:	100.0%
Branches in Coverage %:	Yes
Coverage Strength:	1080.96 hits per line

---

💛 - Coveralls

[rubensworks]

Wait, but this converts a previously fully streaming pipeline into a buffered one.
Is it possible to make the fix without buffering?

[rubensworks]

Yes. In 150d073, imports are no longer collected into arrays. Each import's stream is still started as soon as its rdfs:seeAlso quad is seen. Once the importing stream has ended, the includer reads the import streams one after another with for await. A stream that isn't being read stops once its buffer (16 quads in object mode) is full, so imports that are waiting stay paused with bounded memory, and the import that is being read streams through.

The resulting order and blank node labels are the same as with buffering. All of Comunica's compiled engines are byte-identical to the buffered version and still identical across runs and installs. Compiling Comunica's query-sparql config takes about as long as on master (5.2 to 5.9 s for both). I also added a test that checks that the first quad of an import is emitted while that import is still being parsed; it times out against the buffered version.

Two things stay as they are, since the old code worked the same way:

• The quads of the importing file itself all come before any imported quad. This is what makes the order independent of timing.
• Pushing imported quads doesn't wait for the consumer of the includer, as the old code also pushed on data.

---

Generated by Claude Code