Skip to content

GH-3994: Improved TDB2 index access for single pattern queries - #3995

Open
Aklakan wants to merge 3 commits into
apache:mainfrom
Aklakan:20260613-index-for-basic-distinct
Open

Aklakan wants to merge 3 commits into
apache:mainfrom
Aklakan:20260613-index-for-basic-distinct

Conversation

@Aklakan

@Aklakan Aklakan commented Jun 16, 2026 •

Copy link
Copy Markdown
Contributor

GitHub issue resolved #3994

Pull request Description: Adds skip scan evaluation for single-pattern queries to TDB2. Added special code paths to OpExecutorTDB2 for OpDistinct and OpGroupBy.

Execution time to find all distinct predicates on Wikidata Truthy (8B triples) becomes between 1 and 100 seconds (warm-cold caches)
Also works for simple group by patterns such as
SELECT ?p (COUNT(DISTINCT ?g) AS ?c) { GRAPH ?g { ?s ?p ?o } } GROUP BY ?p

Need yet to add tests - such as create permutations of patterns and compare results with a reference engine (eg. ARQ).


  • Tests are included.
  • [ ] Documentation change and updates are provided for the Apache Jena website
  • Commits have been squashed to remove intermediate development commit messages.
  • Key commit messages start with the issue number (GH-xxxx)

By submitting this pull request, I acknowledge that I am making a contribution to the Apache Software Foundation under the terms and conditions of the Contributor's Agreement.


See the Apache Jena "Contributing" guide.

@Aklakan
Aklakan marked this pull request as draft June 16, 2026 10:27
@Aklakan
Aklakan force-pushed the 20260613-index-for-basic-distinct branch 8 times, most recently from 1f3a682 to adf134b Compare June 28, 2026 17:44
@Aklakan
Aklakan marked this pull request as ready for review June 28, 2026 17:45
@Aklakan

Aklakan commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor Author

The code is ready for review.

  • The most critical changes to existing code are those in BPTreeDistinctKeyPrefixIterator which are meant to restrict skip scans / index scans to custom sub-ranges.

    When no custom range is specified, then the range used to be bptRootNode.{minRecord, maxRecord}, which indicates that maxRecord is inclusive for page-level iteration. However, when doing record-level interation using records.getRecordBuffer().iterator(minRecord, maxRecord) then maxRecord is exclusive. So this is something @rvesse and/or @afs might want to look at whether this is correct and how it should be.
  • The class TestOpExecutorTDB2SkipScan generates SELECT DISTINCT ... and SELECT ... (COUNT(DISTINCT ?v) AS ?c) { ... } GROUP BY ... queries with different combinations and permutations.
  • There is the SystemTDB.symSkipScan symbol to disable the optimization.

public static <T> Iterator<T> dropWhile(Iterator<T> iter, Predicate<T> predicate) {
PeekIterator<T> iter2 = new PeekIterator<>(iter);
for(;;) {
T elt = iter2.peek();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Endless loop because iterator is never advanced.
This method was used during development before settling on
iter = records.getRecordBuffer().iterator(minRecord, maxRecordPlusOne); in BPTreeDistinctKeyPrefixIterator.

@rvesse rvesse left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Aklakan this looks like a really great potential improvement. I had always thought what I'd done previously for prefix scanning had potential to be applied to other kinds of queries but never any chance to explore that further so thanks for taking this on

I have various comments across the PR, these covers three main things (plus the odd typo):

  1. Whether this can/should handle other aggregates like MAX(DISTNCT ?var) since it would seem applying the distinct in the scan first wouldn't change their outputs either.
  2. Improvements to testing, particularly around regression and scale
  3. Leftover TODOs which may/may not remain relevant

@Aklakan
Aklakan force-pushed the 20260613-index-for-basic-distinct branch from bd29358 to e3ba4d2 Compare June 29, 2026 12:40
@Aklakan
Aklakan force-pushed the 20260613-index-for-basic-distinct branch 8 times, most recently from 189640a to ec0bfd1 Compare July 22, 2026 14:22
@Aklakan
Aklakan force-pushed the 20260613-index-for-basic-distinct branch 4 times, most recently from ea0eb86 to 974750f Compare July 28, 2026 10:17
@Aklakan

Aklakan commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor Author

I have completed my revision. The code is now more modular:

  • OpExecutorTDB2 may try to perform skip scan execution for OpDistinct and OpGroup. It is aimed to work for simple queries such as
    • SELECT (min(?o) AS ?min) (max(?o) AS ?max) { ?s ?p ?o } and
    • SELECT DISTINCT ?p { ?s ?p ?o }
  • SkipScanRewrite.isSkipScanCandidate restricts the op-structures eligible to skip scan execution is applied. The restriction could be relaxed in the future.
  • TransformDistinctPlacement implements a transformer that pushes DISTINCT operations down - ideally adjacent to a single the triple pattern / quad. This transform could eventually go to ARQ as a general tool.
  • OpExtSkipScan is a custom operator that effectively encapsulates (distinct (project (vars) (triple-or-quad)))
  • TestOpExecutorTDB2SkipScan generates data of configurable size and all permutations of single-{triple/quad} access patterns and compares the result set against a reference in-memory dataset.

@rvesse

rvesse commented Aug 3, 2026

Copy link
Copy Markdown
Member

I have completed my revision. The code is now more modular:

  • OpExecutorTDB2 may try to perform skip scan execution for OpDistinct and OpGroup. It is aimed to work for simple queries such as

    • SELECT (min(?o) AS ?min) (max(?o) AS ?max) { ?s ?p ?o } and
    • SELECT DISTINCT ?p { ?s ?p ?o }
  • SkipScanRewrite.isSkipScanCandidate restricts the op-structures eligible to skip scan execution is applied. The restriction could be relaxed in the future.

  • TransformDistinctPlacement implements a transformer that pushes DISTINCT operations down - ideally adjacent to a single the triple pattern / quad. This transform could eventually go to ARQ as a general tool.

  • OpExtSkipScan is a custom operator that effectively encapsulates (distinct (project (vars) (triple-or-quad)))

  • TestOpExecutorTDB2SkipScan generates data of configurable size and all permutations of single-{triple/quad} access patterns and compares the result set against a reference in-memory dataset.

Thanks @Aklakan, I will aim to review this again later in the week

@afs

afs commented Oct 4, 2026

Copy link
Copy Markdown
Member

@rvesse - how do the changes look to you?

@rvesse rvesse left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me now.

Test coverage is substantially improved which was one of my requests from my original review. Have checked out the branch and ran the new test suite locally.

Couple of minor comments

@Aklakan
Aklakan force-pushed the 20260613-index-for-basic-distinct branch from 767a4fd to 6e23db1 Compare October 7, 2026 20:11
Co-authored-by: Rob Vesse <rvesse@dotnetrdf.org>
@Aklakan
Aklakan force-pushed the 20260613-index-for-basic-distinct branch from 6e23db1 to 6b422ab Compare October 20, 2026 22:04
rvesse
rvesse previously approved these changes Oct 21, 2026
@Aklakan

Aklakan commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

I added the requested minor changes and had another look over the code and it still looked ok to me. So for now I'd say its done. In case a bug is revealed there is still SystemTDB.symSkipScan that can be set to false to disable this feature.

@afs

afs commented Oct 21, 2026

Copy link
Copy Markdown
Member

To get up-to-speed with this PR, I did a AI code review (Claude Code, model=Opus5, effort=xhigh).
This was a bit of an experiment. Just because it finds something does not necessarily mean there is an issue.

It came up with some findings. I got it to generate test cases then verified some of them (1, 4 and 6 - they have test SELECT queries) by taking the data and query then using tdb2.tdbquery --mem=DATA --query=QUERY using the PR code base and also Jena main. The ones I looked at do behave differently.

I think some of findings are the same underlying cause in appearing in different ways.

tdb2:skipScan works.

I'm not sure then best way to exchange the material. If there is a better way, please say so.

I've put the material on a branch in my afs/jena repo.
https://github.com/afs/jena/tree/pr3995-tdb2-single-pattern-queries

The code is the PR, rebased to Jena main as of today.

3 commits:

  1. The generated tests
  2. The report
  3. A fix needed for tdb2.tdbloader --mem=

Claude offered to "split into per-finding comments for posting inline on the PR (/code-review 3995 --comment does that)" if that would be helpful.

@Aklakan

Aklakan commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the Claude analysis. All wrong SPARQL answers were so far fixable. I am currently down to 6 failing test cases which I still need to look into - at least 2 seem to be invalid.

  • One conceptual issue is the min/max handling in BPTreeDistinctKeyPrefixIterator. Right now matching ranges uses an exclusive maximum - but perhaps it must be possible to match the maximum (0xffff...) value itself. Then the logic would have to be changed for the maximum to be inclusive, and an empty range would no longer be represented by [x, x[ but e.g. by [x, y] with y < x.

  • The other thing I noticed is that TupleLib reads records just based on NodeId.SIZE, but in the comments NodeId can have two encodings: "long" with 8+4 bytes and "short" with 8 bytes. But NodeId.SIZE is only 8. So now I wonder whether the BPTree can actually mix different lengths of NodeIds - if that was possible something might be more fundamentally broken.

Test05_EqualityLinkArrayOverflow

  • distinct_quadWithOneVarInEverySlot: SELECT DISTINCT ?s { ?s ?s ?s } Claude expects skip scan, but each row needs to be checked, so skip scan can't be used directly.
    It would be possible to rewrite the original query as below but I'd say this is out of scope for the PR.
SELECT ?s {
  { SELECT DISTINCT ?s { ?s [] [] } }
  FILTER EXISTS { ?s ?s ?s }
}

Test12_PrefixIteratorRange

  • First impression is that singletonPageShortcutRespectsLowerBound is invalid: The test case inserts records [00000-00005] into the BPT and then does a retrieval with the range [00001-00005] but a prefix length of 3. So effectively a query for the range [000_00, 000_FF]. It then fails because the retrieval yields 00000 which is less than the minimum of the request 00001. But for the given prefix length 00000 is correct.

I still have to look into Test10_RemapValueConditions, Test11_KeyPrefixLengthValidation.

@afs

afs commented Oct 22, 2026

Copy link
Copy Markdown
Member
  • The other thing I noticed is that TupleLib reads records just based on NodeId.SIZE, but in the comments NodeId can have two encodings: "long" with 8+4 bytes and "short" with 8 bytes. But NodeId.SIZE is only 8. So now I wonder whether the BPTree can actually mix different lengths of NodeIds - if that was possible something might be more fundamentally broken.

Only "short" is implemented.

While NodeId is designed for 8+4, it has never been used. It woudl need testing out before use in a release as the implicit assumpt of 8 may well have leaked into some code or other.

It was never intended for mixed use. One database - one size of NodeId.

@afs

afs commented Oct 22, 2026

Copy link
Copy Markdown
Member
  • distinct_quadWithOneVarInEverySlot: SELECT DISTINCT ?s { ?s ?s ?s } Claude expects skip scan, but each row needs to be checked, so skip scan can't be used directly.
    It would be possible to rewrite the original query as below but I'd say this is out of scope for the PR.

Good plan.

@rvesse
rvesse self-requested a review October 22, 2026 12:10
@rvesse
rvesse dismissed their stale review October 22, 2026 12:11

Other reviewers have discovered edge cases where the new behaviour breaks

@Aklakan

Aklakan commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor Author

I'm not sure then best way to exchange the material. If there is a better way, please say so.

Good question - (Heh, sounds like an AI response). So the nice thing is that Claude isolated all changes - even private access - using the Internals class.

I am nearing completion of my fixes, the question is how to add them. perhaps i add only the fixes to this PR, and you rebase the Claude tests on top of them? So ideally the tests would eventually use e.g. package-private access rather than reflection via Internals - that would require removing some 'private' declarations.
Alternatively, I could include the tests in this PR - whatever you prefer.

@afs

afs commented Oct 22, 2026

Copy link
Copy Markdown
Member

I am nearing completion of my fixes, the question is how to add them. perhaps i add only the fixes to this PR, and you rebase the Claude tests on top of them? So ideally the tests would eventually use e.g. package-private access rather than reflection via Internals - that would require removing some 'private' declarations.

Update the PR - pick any tests that are useful.

I can generate a new Claude review from scratch and it generate can tests based on the new state.

This is also an option

Claude offered to "split into per-finding comments for posting inline on the PR" if that would be helpful.

@Aklakan
Aklakan force-pushed the 20260613-index-for-basic-distinct branch from 73f4f08 to 2d940c5 Compare October 23, 2026 12:48
@Aklakan

Aklakan commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

I added the fixes and the test cases in a new commit and ran a review using Astra/Medium which found 4 more issues in my changes for Claude which are all fixed now and verified with additional tests. (I did not squash them yet so that the AI can track progress in the commits).
I think its ready for another round of review on your side.
There are some disabled tests which have a comment why they are disabled - eventually those need to be cleaned up.

The following points were raised as potential code smells, but the design choices were deliberately that way for the given reasons (in case Claude also marks them):

  • Executor stored in context: A workaround for OpExt.eval() not receiving the current executor. QC.execute() can create a different executor and break custom implementations. Retaining the exact instance is intentional; first-executor-wins remains a limitation.
  • Tuple/index coordinate types: EqualityLink and VarMap serve both coordinate systems. Separate types could prevent remapping mistakes, but add complexity. Clear naming, documentation, and tests are sufficient for now.
  • Equality checks through SPARQL expressions: Synthetic variables and filter/project iterators add overhead, but reuse established abort handling. Direct NodeId filtering is deferred unless performance warrants it.
  • Cache ownership: The cache is execution-scoped, not global. Shared child contexts merit awareness, but no invalid physical-index reuse was demonstrated; no cache redesign is proposed.

@afs

afs commented Nov 7, 2026

Copy link
Copy Markdown
Member

Review as downloads:

2026-11-07 Review, PR3995, rebased to current main # Review: GH-3994 — Improved TDB2 index access for single pattern queries

Commits reviewed: 6473e41c1c, f9eb5af200, f9a24c1b0e (47 files, +5049/−41)

Summary

Valuable and largely well-built. It builds clean, the full jena-tdb2 suite is green
(1976 tests, 2 skipped), and the speedup is real and large.

Two things need resolving before merge:

  1. A variable-name collision that silently drops rows (union graph + a user variable named ?G0).
  2. DISTINCT is applied per input binding rather than globally, which changes row counts.

Everything else below is "should fix" or cleanup.

Measured speedup

400k triples + 400k quads, 20k subjects × 20 predicates × 10 objects, best of 3 after warmup:

Query skipScan on skipScan off ratio
SELECT DISTINCT ?p { ?s ?p ?o } 2 ms 150 ms 75×
SELECT DISTINCT ?s { ?s ?p ?o } 21 ms 176 ms 8×
SELECT DISTINCT ?o { ?s ?p ?o } 1 ms 158 ms 158×
SELECT DISTINCT ?s ?p ?o { ?s ?p ?o } 53 ms 1150 ms 22×
SELECT DISTINCT ?g ?p { GRAPH ?g { ?s ?p ?o } } 5 ms 260 ms 52×
SELECT ?p (COUNT(DISTINCT ?o) AS ?c) { ?s ?p ?o } GROUP BY ?p 5 ms 116 ms 23×

Note the full-key case (DISTINCT ?s ?p ?o) also gains 22×, because it avoids the hash-set
QueryIterDistinct entirely.


Blocking

1. ?G0 collides with the synthetic union-graph variable — silently wrong answers

PatternQuery.java:131 allocates the graph slot variable for the union graph with:

VarAlloc varAlloc = new VarAlloc("G");
tuple = new Node[] { varAlloc.allocVar(), t.getSubject(), t.getPredicate(), t.getObject() };

new VarAlloc("G").allocVar() is Var.alloc("G0") — an ordinary variable name a user can
write. A fresh VarAlloc is created on every call, so it is always ?G0.

When the query really uses ?G0, TuplePatternSpec.create sees slot 0 and slot 1 holding the
same variable and builds an EqualityLink, forcing graph = subject:

SELECT DISTINCT ?G0 { GRAPH <urn:x-arq:UnionGraph> { ?G0 ?p ?o } }

  skipScan on  → 1 row   (:s1)              <-- only the row where graph == subject
  skipScan off → 3 rows  (:s1, :s2, :a)

No error is raised — rows are just missing. Confirmed on five variants:

Query on off
DISTINCT ?G0 { GRAPH <…UnionGraph> { ?G0 ?p ?o } } 1 3
DISTINCT ?G0 ?p { GRAPH <…UnionGraph> { ?G0 ?p ?o } } 1 5
DISTINCT ?o { GRAPH <…UnionGraph> { ?G0 ?p ?o } } 1 7
?G0 (COUNT(DISTINCT ?o)) … GROUP BY ?G0 1 group 4 groups
DISTINCT ?s { GRAPH <…UnionGraph> { ?s ?G0 ?o } } 0 4

?G1 and any other name are unaffected.

Fix: use a non-distinguished name, as the sibling code in
OpExecutorTDB2SkipScan.java:417 already does:

VarAlloc varAlloc = new VarAlloc(ARQConstants.allocVarAnonMarker + "G");

??G0 cannot be written in SPARQL, and ??G is unused elsewhere in ARQ (path flattening
uses ??P / ??Q).

The PR's own generated harness (TestOpExecutorTDB2SkipScan) only ever uses the variable
names ?g ?s ?p ?o, and the union graph appears in exactly one assertion in the whole test
set (TestAggregatorsSkipScan.java:88), which is why this wasn't caught.

2. DISTINCT is applied per input binding, not globally

OpExecutorTDB2SkipScan.java:125 wraps execution in QueryIter.flatMap(input, binding -> …),
and the DISTINCT runs inside the flatMap (line 163 for the failover path, line 364 for the
indirect-distinct path), with QueryIterCommonParent (line 171) re-attaching the parent
binding afterwards. When the input iterator carries duplicate bindings, the answer changes:

SELECT * { VALUES ?z { 1 1 } { SELECT DISTINCT ?o { ?s :p ?o } } }

  TDB2, skipScan on  : 4 rows
  TDB2, skipScan off : 2 rows
  ARQ in-memory      : 2 rows

Also reproduced with { { ?a :q ?b } UNION { ?a :q ?b } } on the left, and with
SELECT (COUNT(*) AS ?n) wrapped around either (4 vs 2).

In fairness to the PR, 4 looks like the spec-correct answer. The control without
DISTINCT gives 6 (= 3 sub-select rows × 2 VALUES rows) on all three engines, so the
sub-select's DISTINCT should leave 2 × 2 = 4. ARQ's 2 comes from linearising the join so
that OpDistinct ends up deduplicating across the parent's variables. So this is arguably a
long-standing ARQ deviation that the rewrite steps around.

It still needs a decision, because right now the answer depends on whether the rewrite fires:

SELECT (COUNT(*) AS ?n) { VALUES ?z { 1 1 } { SELECT DISTINCT ?o { ?s :p ?o } } }
  → on: 4, off: 2          (single triple — rewrite fires)

SELECT (COUNT(*) AS ?n) { VALUES ?z { 1 1 } { SELECT DISTINCT ?o { ?s :p ?o . ?s :p ?o2 } } }
  → on: 2, off: 2          (two triples — rewrite declines)

It would also fail the PR's own SkipScanTestLib.assertSkipScanAgrees contract ("the skip
scan answers the query exactly as the general machinery does") if a test exercised it.

Note LATERAL and OPTIONAL forms agree in all three (4/4/4) — it is specifically the
join/VALUES shape.


Should fix

3. resolveTable can hand a 4-wide table to a 3-tuple pattern

OpExecutorTDB2SkipScan.java:279:

if (tupleLength == 3) {
    GraphTDB graph = (GraphTDB)execCxt.getActiveGraph();
    return graph.getNodeTupleTable();
}

GraphTDB.getNodeTupleTable() delegates to DatasetGraphTDB.chooseNodeTupleTable(graphName),
which returns the quad table (4 columns) for any non-default graph name. With a named
GraphTDB as active graph, computeIndexMatch would do quad[tm.mapIdx(3)] on a length-3
array — ArrayIndexOutOfBoundsException, or silently the wrong graph if it didn't throw.

isForTDB only checks instanceof GraphTDB, which a named graph satisfies.

I could not reach this path: connectDataset(...).getNamedModel(...) yields
GraphViewSwitchable (not a GraphTDB), dynamic datasets (FROM / FROM NAMED) wrap graphs
in non-TDB views, and quadification removes OpBGP from under OpGraph. I instrumented
resolveTable to throw on a width mismatch and ran the whole jena-tdb2 suite clean, plus
probes with surviving OpGraph (paths, sub-selects inside GRAPH, SERVICE).

But the existing code treats this case as live — OpExecutorTDB2.executeBGP at lines 193-201
explicitly routes a BGP on a named GraphTDB to quad access, and
OpExecutorTDB2.execute(OpFilter) carries the comment "Still may be a TDB graph in a non-TDB
dataset (e.g. a named model)"
. Dropping that invariant silently is worth a guard.

Cheapest fix: have resolveTable return null (→ no candidate → normal path) unless
table.getTupleLen() == tupleLength.

4. New code adopts an API marked for removal, and cancellation is inconsistent

OpExecutorTDB2SkipScan.java:435 calls execCxt.checkCancelSignal(), which is
@Deprecated(forRemoval=true). The blanket @SuppressWarnings("removal") on exec
(line 372) hides the warning; -Xlint:removal shows it is the only cause. The deprecation
javadoc points at getCancelSignal() or at letting QueryIteratorBase handle it.

Related: that call is the only cancel check and it sits inside the residual-conditions
filter, so a scan with no residual conditions never reaches it. (Results stay cancellable
per emitted binding via QueryIteratorBase.nextBinding, so this is a consistency issue, not
a hang — the per-prefix page/subtree short-circuits bound the work inside one hasNext().)

5. isSkipScanCandidate misses the common FILTER … BIND … shape

SkipScanRewrite.java:79-85 peels OpFilter then OpExtend:

if (op instanceof OpFilter opF)  { op = opF.getSubOp(); }
if (op instanceof OpExtend opE)  { op = opE.getSubOp(); }

But ARQ produces extend(filter(bgp)) — extend above filter — for that query, so the gate
rejects it even though TransformDistinctPlacement + TransformSkipScan handle it fine:

SELECT DISTINCT ?s ?c { ?s ?p ?o FILTER(?s != :x) BIND(str(?s) AS ?c) }
  algebra: (distinct (project (?s ?c) (extend ((?c (str ?s)))
                                       (filter (!= ?s :x) (bgp (triple ?s ?p ?o))))))
  → rewrite declines

Replacing the two ifs with a loop over both makes it fire — verified, and the genuinely
non-pushable case (BIND(str(?s) AS ?c) BIND(str(?p) AS ?d) with ?p outside the
projection) is still correctly rejected downstream by canPush:

while (true) {
    if (op instanceof OpFilter opF) { op = opF.getSubOp(); continue; }
    if (op instanceof OpExtend opE) { op = opE.getSubOp(); continue; }
    break;
}

6. Passing an OpExecutor through the Context is fragile

OpExtSkipScan.java:42-49 stores the first OpExecutor it sees under a context symbol:

public static void setOpExecutorIfAbsent(Context cxt, OpExecutor opExecutor) {
    cxt.computeIfAbsent(symOpExecutor, x -> opExecutor);
}

Each OpExecutorTDB2 captures its own execCxt, and OpExecutorTDB2.java:78 says so
directly: "A new compile object is created for each op compilation. So the execCxt is changing
as we go through the query-compile-execute process."
The failover path then runs
opExecutor.executeOp(...) against a potentially stale active graph / execution context.

Failover is rare ("Should never be needed with default TDB2 configurations"), but
QC.execute(op, input, execCxt) using the execCxt actually passed to eval would remove
the mechanism entirely.


Cleanups

Dead production code

Record.padRight (new public API on a core DBOE class) has no production caller — the
only references are in Test06_RecordPadRightUpperBound, which exists to test it. Drop both,
or wire it in. Record.incrementKeyOrNull is used (OpExecutorTDB2SkipScan.java:429) and
fine.

Review scaffolding is shipping as production tests

  • Package org.apache.jena.tdb2.solver.skipscan_3995 embeds a PR number — and 3995 is the PR
    while the issue is 3994.
  • TS_SkipScan_3995 javadoc still reads "these tests fail against the PR as it stands — each
    failure is the finding"
    . They all pass now.
  • That same javadoc {@link}s Test15_UnusedCodeAndWeakAssertion, which does not exist
    anywhere in the tree — a dangling reference.
  • Test01…Test14 are split across skipscan and skipscan_3995 with no visible rule
    (05, 07, 09, 10, 13 in one; 01-04, 06, 08, 11, 12, 14 in the other).
  • Test_SPARQL_TDB — a long-standing core test class — now static-imports
    skipscan_3995.SkipScanTestLib.rows and .tdb2 and uses neither.
  • Neither suite class is referenced from any parent suite; the Test* names mean surefire
    picks the classes up directly anyway.

Suggest folding these into skipscan under descriptive names.

Two @Disabled tests contradict the shipped javadoc

BPlusTree.distinctByKeyPrefix's new javadoc says "Prefix lengths from zero through the full
key length are supported"
, but:

  • Test11_KeyPrefixLengthValidation.prefixLengthZeroIsRejected — @Disabled("A range of 0 should gracefully match all records.") — asserts length 0 throws.
  • TestBPTreeDistinctKeys.bptree_distinct_by_key_bad_01 — @Disabled("SELECT DISTINCT ?zz { ?s ?p ?o } can result in 0 length prefixes") — same assertion, disabled rather than updated.

Length 0 does work end-to-end (SELECT DISTINCT ?zz { ?s ?p ?o } returns the correct single
row), but only because every prefix comparison is vacuously equal. With a non-null max
bound and length 0, isPrefixInRequestRange returns false and the iterator yields nothing —
unreachable from the executor, which passes null there, but it contradicts the javadoc.
Pick one behaviour and have the tests assert it.

Leftover debug code

  • OpExecutorTDB2SkipScan.java:167-169 — // FIXME Ugly but just for testing plus two
    commented-out innerIter = … lines.
  • OpExecutorTDB2SkipScan.java:138 — // System.err.println("NO CACHE HIT: " + canonicalQuery);
  • PatternQuery.java:137 — trailing //, filter, extend, reapplyProject);

Unrelated diff noise

  • jena-geosparql/.../BasicTask.java — a javadoc unicode-hyphen fix, unrelated to this work.
  • jena-tdb2/pom.xml — trailing-whitespace edits mixed in with the real dependency change.

Both would be better as a separate commit.

Smaller points

  • commons-math4-legacy added as a test dependency with no direct use in jena-tdb2 test
    sources — presumably needed transitively at runtime for MEDIAN/MODE in
    TestAggregatorsSkipScan. A one-line comment would save the next person the hunt. (Version
    resolves fine from the parent's dependencyManagement.)
  • E_Equals for equality links (OpExecutorTDB2SkipScan.java:423) is value equality where
    term equality is meant. It happens to be safe — a repeated variable always spans at least
    one non-object slot, so both ends must be IRI or blank node — but E_SameTerm is clearer,
    cheaper, and doesn't rest on that argument.
  • SystemTDB now imports OpExecutorTDB2 purely for a javadoc {@link}, adding a
    sys → solver compile edge.
  • OpExecutorTDB2SkipScan exposes ~12 public static internals (computeIndexMatch,
    tryComputeIndexMap, remapProjection, remapEqualityLinks, remapValueConditions,
    resolveTable, planCandidates, pickBest, …) used only by tests. Package-private would
    keep the published API surface honest for an experimental feature.
  • PatternQuery.createOrNull$ — the trailing $ is unusual for this codebase.
  • IndexMatch.COMPARATOR wraps in Comparator.nullsLast(...), but pickBest never sees a
    null match.
  • On a cache miss, findBestCandidate and then tryExec each call resolveTable and
    TuplePatternSpec.create — the second pair is redundant work per input binding.
  • symSkipScan defaults on (isTrueOrUndef). Reasonable long-term, but given that finding 1
    produces silently wrong answers rather than an error, consider one release opt-in. The flag
    isn't documented anywhere either.

What I verified

  • mvn install on jena-dboe-base, jena-dboe-trans-data, jena-tdb2 — clean.
  • Full jena-tdb2 suite after clean: 1976 tests, 0 failures, 2 skipped (both the
    @Disabled ones above).
  • New suites specifically: TS_SkipScan_3995, TS_TDB_SkipScan, Test_SPARQL_TDB —
    1154 tests green.
  • A 44-query differential sweep (skipScan on vs off, order-insensitive) over a database with
    literals, language tags, xsd:integer/xsd:decimal pairs, blank nodes, a graph named the
    same as a subject, repeated variables in every position, quads, union graph, filters, BIND,
    all the suitable aggregators, VALUES, sub-selects, OPTIONAL, and no-match patterns —
    all 44 agreed. The ?G0 and duplicate-input-binding cases above were found by extending
    that sweep.
  • resolveTable instrumented to throw on a tuple-width mismatch; full module suite plus
    targeted OpGraph / path / dynamic-dataset probes — never tripped.
  • -Xlint:removal on the new package, to identify what the @SuppressWarnings is hiding.
  • Read through computeIndexMatch, TuplePatternSpec.create, tryComputeIndexMap,
    remapEqualityLinks, the finalProj sizing, the record min/max + incrementKeyOrNull
    bound construction, and the BPTreeDistinctKeyPrefixIterator range handling against
    RecordBufferIterator / BPTreeNode.iterator semantics. No array-overflow or
    off-by-one problems found; half-open bound handling is correct, including the all-ANY
    overflow-to-null (unbounded) case. Index-space vs tuple-space remapping is consistent
    throughout, despite the primary/secondary local names in remapEqualityLinks reading
    backwards.
  • SkipScanRewrite.tryRewriteAsSkipScan(OpGroup) aggregator analysis: the distinct-invariance
    argument holds for every aggregator in isSkipScanSuitable, and COUNT(*) /
    multi-variable / expression cases are correctly declined. (The convertToNonDistinct
    javadoc mentions SAMPLE as distinct-invariant while isSkipScanSuitable deliberately
    excludes non-distinct AggSample — just a doc inconsistency.)
  • TransformDistinctPlacement push rules for OpProject / OpFilter / OpExtend /
    nested OpDistinct, including the reapplyProject path and the unstable-expression
    rejection (rand(), uuid()), traced by hand on worked examples.

@afs

afs commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

For the VarAlloc:

Variables internally don't have to be valid syntax. A unique prefix of *G will do. See ARQConstants.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TDB2 Skip scan for single pattern queries.

3 participants