Repository navigation
Conversation
1f3a682 to
adf134b
Compare
|
The code is ready for review.
|
| public static <T> Iterator<T> dropWhile(Iterator<T> iter, Predicate<T> predicate) { | ||
| PeekIterator<T> iter2 = new PeekIterator<>(iter); | ||
| for(;;) { | ||
| T elt = iter2.peek(); |
There was a problem hiding this comment.
Endless loop because iterator is never advanced.
This method was used during development before settling on
iter = records.getRecordBuffer().iterator(minRecord, maxRecordPlusOne); in BPTreeDistinctKeyPrefixIterator.
rvesse
left a comment
There was a problem hiding this comment.
Hey @Aklakan this looks like a really great potential improvement. I had always thought what I'd done previously for prefix scanning had potential to be applied to other kinds of queries but never any chance to explore that further so thanks for taking this on
I have various comments across the PR, these covers three main things (plus the odd typo):
- Whether this can/should handle other aggregates like
MAX(DISTNCT ?var)since it would seem applying the distinct in the scan first wouldn't change their outputs either. - Improvements to testing, particularly around regression and scale
- Leftover TODOs which may/may not remain relevant
bd29358 to
e3ba4d2
Compare
189640a to
ec0bfd1
Compare
ea0eb86 to
974750f
Compare
|
I have completed my revision. The code is now more modular:
|
Thanks @Aklakan, I will aim to review this again later in the week |
|
@rvesse - how do the changes look to you? |
rvesse
left a comment
There was a problem hiding this comment.
This looks good to me now.
Test coverage is substantially improved which was one of my requests from my original review. Have checked out the branch and ran the new test suite locally.
Couple of minor comments
767a4fd to
6e23db1
Compare
Co-authored-by: Rob Vesse <rvesse@dotnetrdf.org>
6e23db1 to
6b422ab
Compare
|
I added the requested minor changes and had another look over the code and it still looked ok to me. So for now I'd say its done. In case a bug is revealed there is still |
|
To get up-to-speed with this PR, I did a AI code review (Claude Code, model=Opus5, effort=xhigh). It came up with some findings. I got it to generate test cases then verified some of them (1, 4 and 6 - they have test SELECT queries) by taking the data and query then using I think some of findings are the same underlying cause in appearing in different ways.
I'm not sure then best way to exchange the material. If there is a better way, please say so. I've put the material on a branch in my The code is the PR, rebased to Jena 3 commits:
Claude offered to "split into per-finding comments for posting inline on the PR (/code-review 3995 --comment does that)" if that would be helpful. |
|
Thanks for the Claude analysis. All wrong SPARQL answers were so far fixable. I am currently down to 6 failing test cases which I still need to look into - at least 2 seem to be invalid.
SELECT ?s {
{ SELECT DISTINCT ?s { ?s [] [] } }
FILTER EXISTS { ?s ?s ?s }
}
I still have to look into |
Only "short" is implemented. While NodeId is designed for 8+4, it has never been used. It woudl need testing out before use in a release as the implicit assumpt of 8 may well have leaked into some code or other. It was never intended for mixed use. One database - one size of NodeId. |
Good plan. |
Other reviewers have discovered edge cases where the new behaviour breaks
Good question - (Heh, sounds like an AI response). So the nice thing is that Claude isolated all changes - even private access - using the I am nearing completion of my fixes, the question is how to add them. perhaps i add only the fixes to this PR, and you rebase the Claude tests on top of them? So ideally the tests would eventually use e.g. package-private access rather than reflection via |
Update the PR - pick any tests that are useful. I can generate a new Claude review from scratch and it generate can tests based on the new state. This is also an option
|
73f4f08 to
2d940c5
Compare
|
I added the fixes and the test cases in a new commit and ran a review using Astra/Medium which found 4 more issues in my changes for Claude which are all fixed now and verified with additional tests. (I did not squash them yet so that the AI can track progress in the commits). The following points were raised as potential code smells, but the design choices were deliberately that way for the given reasons (in case Claude also marks them):
|
|
Review as downloads:
2026-11-07 Review, PR3995, rebased to current main# Review: GH-3994 — Improved TDB2 index access for single pattern queriesCommits reviewed: SummaryValuable and largely well-built. It builds clean, the full Two things need resolving before merge:
Everything else below is "should fix" or cleanup. Measured speedup400k triples + 400k quads, 20k subjects × 20 predicates × 10 objects, best of 3 after warmup:
Note the full-key case ( Blocking1.
|
| Query | on | off |
|---|---|---|
DISTINCT ?G0 { GRAPH <…UnionGraph> { ?G0 ?p ?o } } |
1 | 3 |
DISTINCT ?G0 ?p { GRAPH <…UnionGraph> { ?G0 ?p ?o } } |
1 | 5 |
DISTINCT ?o { GRAPH <…UnionGraph> { ?G0 ?p ?o } } |
1 | 7 |
?G0 (COUNT(DISTINCT ?o)) … GROUP BY ?G0 |
1 group | 4 groups |
DISTINCT ?s { GRAPH <…UnionGraph> { ?s ?G0 ?o } } |
0 | 4 |
?G1 and any other name are unaffected.
Fix: use a non-distinguished name, as the sibling code in
OpExecutorTDB2SkipScan.java:417 already does:
VarAlloc varAlloc = new VarAlloc(ARQConstants.allocVarAnonMarker + "G");??G0 cannot be written in SPARQL, and ??G is unused elsewhere in ARQ (path flattening
uses ??P / ??Q).
The PR's own generated harness (TestOpExecutorTDB2SkipScan) only ever uses the variable
names ?g ?s ?p ?o, and the union graph appears in exactly one assertion in the whole test
set (TestAggregatorsSkipScan.java:88), which is why this wasn't caught.
2. DISTINCT is applied per input binding, not globally
OpExecutorTDB2SkipScan.java:125 wraps execution in QueryIter.flatMap(input, binding -> …),
and the DISTINCT runs inside the flatMap (line 163 for the failover path, line 364 for the
indirect-distinct path), with QueryIterCommonParent (line 171) re-attaching the parent
binding afterwards. When the input iterator carries duplicate bindings, the answer changes:
SELECT * { VALUES ?z { 1 1 } { SELECT DISTINCT ?o { ?s :p ?o } } }
TDB2, skipScan on : 4 rows
TDB2, skipScan off : 2 rows
ARQ in-memory : 2 rows
Also reproduced with { { ?a :q ?b } UNION { ?a :q ?b } } on the left, and with
SELECT (COUNT(*) AS ?n) wrapped around either (4 vs 2).
In fairness to the PR, 4 looks like the spec-correct answer. The control without
DISTINCT gives 6 (= 3 sub-select rows × 2 VALUES rows) on all three engines, so the
sub-select's DISTINCT should leave 2 × 2 = 4. ARQ's 2 comes from linearising the join so
that OpDistinct ends up deduplicating across the parent's variables. So this is arguably a
long-standing ARQ deviation that the rewrite steps around.
It still needs a decision, because right now the answer depends on whether the rewrite fires:
SELECT (COUNT(*) AS ?n) { VALUES ?z { 1 1 } { SELECT DISTINCT ?o { ?s :p ?o } } }
→ on: 4, off: 2 (single triple — rewrite fires)
SELECT (COUNT(*) AS ?n) { VALUES ?z { 1 1 } { SELECT DISTINCT ?o { ?s :p ?o . ?s :p ?o2 } } }
→ on: 2, off: 2 (two triples — rewrite declines)
It would also fail the PR's own SkipScanTestLib.assertSkipScanAgrees contract ("the skip
scan answers the query exactly as the general machinery does") if a test exercised it.
Note LATERAL and OPTIONAL forms agree in all three (4/4/4) — it is specifically the
join/VALUES shape.
Should fix
3. resolveTable can hand a 4-wide table to a 3-tuple pattern
OpExecutorTDB2SkipScan.java:279:
if (tupleLength == 3) {
GraphTDB graph = (GraphTDB)execCxt.getActiveGraph();
return graph.getNodeTupleTable();
}GraphTDB.getNodeTupleTable() delegates to DatasetGraphTDB.chooseNodeTupleTable(graphName),
which returns the quad table (4 columns) for any non-default graph name. With a named
GraphTDB as active graph, computeIndexMatch would do quad[tm.mapIdx(3)] on a length-3
array — ArrayIndexOutOfBoundsException, or silently the wrong graph if it didn't throw.
isForTDB only checks instanceof GraphTDB, which a named graph satisfies.
I could not reach this path: connectDataset(...).getNamedModel(...) yields
GraphViewSwitchable (not a GraphTDB), dynamic datasets (FROM / FROM NAMED) wrap graphs
in non-TDB views, and quadification removes OpBGP from under OpGraph. I instrumented
resolveTable to throw on a width mismatch and ran the whole jena-tdb2 suite clean, plus
probes with surviving OpGraph (paths, sub-selects inside GRAPH, SERVICE).
But the existing code treats this case as live — OpExecutorTDB2.executeBGP at lines 193-201
explicitly routes a BGP on a named GraphTDB to quad access, and
OpExecutorTDB2.execute(OpFilter) carries the comment "Still may be a TDB graph in a non-TDB
dataset (e.g. a named model)". Dropping that invariant silently is worth a guard.
Cheapest fix: have resolveTable return null (→ no candidate → normal path) unless
table.getTupleLen() == tupleLength.
4. New code adopts an API marked for removal, and cancellation is inconsistent
OpExecutorTDB2SkipScan.java:435 calls execCxt.checkCancelSignal(), which is
@Deprecated(forRemoval=true). The blanket @SuppressWarnings("removal") on exec
(line 372) hides the warning; -Xlint:removal shows it is the only cause. The deprecation
javadoc points at getCancelSignal() or at letting QueryIteratorBase handle it.
Related: that call is the only cancel check and it sits inside the residual-conditions
filter, so a scan with no residual conditions never reaches it. (Results stay cancellable
per emitted binding via QueryIteratorBase.nextBinding, so this is a consistency issue, not
a hang — the per-prefix page/subtree short-circuits bound the work inside one hasNext().)
5. isSkipScanCandidate misses the common FILTER … BIND … shape
SkipScanRewrite.java:79-85 peels OpFilter then OpExtend:
if (op instanceof OpFilter opF) { op = opF.getSubOp(); }
if (op instanceof OpExtend opE) { op = opE.getSubOp(); }But ARQ produces extend(filter(bgp)) — extend above filter — for that query, so the gate
rejects it even though TransformDistinctPlacement + TransformSkipScan handle it fine:
SELECT DISTINCT ?s ?c { ?s ?p ?o FILTER(?s != :x) BIND(str(?s) AS ?c) }
algebra: (distinct (project (?s ?c) (extend ((?c (str ?s)))
(filter (!= ?s :x) (bgp (triple ?s ?p ?o))))))
→ rewrite declines
Replacing the two ifs with a loop over both makes it fire — verified, and the genuinely
non-pushable case (BIND(str(?s) AS ?c) BIND(str(?p) AS ?d) with ?p outside the
projection) is still correctly rejected downstream by canPush:
while (true) {
if (op instanceof OpFilter opF) { op = opF.getSubOp(); continue; }
if (op instanceof OpExtend opE) { op = opE.getSubOp(); continue; }
break;
}6. Passing an OpExecutor through the Context is fragile
OpExtSkipScan.java:42-49 stores the first OpExecutor it sees under a context symbol:
public static void setOpExecutorIfAbsent(Context cxt, OpExecutor opExecutor) {
cxt.computeIfAbsent(symOpExecutor, x -> opExecutor);
}Each OpExecutorTDB2 captures its own execCxt, and OpExecutorTDB2.java:78 says so
directly: "A new compile object is created for each op compilation. So the execCxt is changing
as we go through the query-compile-execute process." The failover path then runs
opExecutor.executeOp(...) against a potentially stale active graph / execution context.
Failover is rare ("Should never be needed with default TDB2 configurations"), but
QC.execute(op, input, execCxt) using the execCxt actually passed to eval would remove
the mechanism entirely.
Cleanups
Dead production code
Record.padRight (new public API on a core DBOE class) has no production caller — the
only references are in Test06_RecordPadRightUpperBound, which exists to test it. Drop both,
or wire it in. Record.incrementKeyOrNull is used (OpExecutorTDB2SkipScan.java:429) and
fine.
Review scaffolding is shipping as production tests
- Package
org.apache.jena.tdb2.solver.skipscan_3995embeds a PR number — and 3995 is the PR
while the issue is 3994. TS_SkipScan_3995javadoc still reads "these tests fail against the PR as it stands — each
failure is the finding". They all pass now.- That same javadoc
{@link}sTest15_UnusedCodeAndWeakAssertion, which does not exist
anywhere in the tree — a dangling reference. Test01…Test14are split acrossskipscanandskipscan_3995with no visible rule
(05, 07, 09, 10, 13 in one; 01-04, 06, 08, 11, 12, 14 in the other).Test_SPARQL_TDB— a long-standing core test class — now static-imports
skipscan_3995.SkipScanTestLib.rowsand.tdb2and uses neither.- Neither suite class is referenced from any parent suite; the
Test*names mean surefire
picks the classes up directly anyway.
Suggest folding these into skipscan under descriptive names.
Two @Disabled tests contradict the shipped javadoc
BPlusTree.distinctByKeyPrefix's new javadoc says "Prefix lengths from zero through the full
key length are supported", but:
Test11_KeyPrefixLengthValidation.prefixLengthZeroIsRejected—@Disabled("A range of 0 should gracefully match all records.")— asserts length 0 throws.TestBPTreeDistinctKeys.bptree_distinct_by_key_bad_01—@Disabled("SELECT DISTINCT ?zz { ?s ?p ?o } can result in 0 length prefixes")— same assertion, disabled rather than updated.
Length 0 does work end-to-end (SELECT DISTINCT ?zz { ?s ?p ?o } returns the correct single
row), but only because every prefix comparison is vacuously equal. With a non-null max
bound and length 0, isPrefixInRequestRange returns false and the iterator yields nothing —
unreachable from the executor, which passes null there, but it contradicts the javadoc.
Pick one behaviour and have the tests assert it.
Leftover debug code
OpExecutorTDB2SkipScan.java:167-169—// FIXME Ugly but just for testingplus two
commented-outinnerIter = …lines.OpExecutorTDB2SkipScan.java:138—// System.err.println("NO CACHE HIT: " + canonicalQuery);PatternQuery.java:137— trailing//, filter, extend, reapplyProject);
Unrelated diff noise
jena-geosparql/.../BasicTask.java— a javadoc unicode-hyphen fix, unrelated to this work.jena-tdb2/pom.xml— trailing-whitespace edits mixed in with the real dependency change.
Both would be better as a separate commit.
Smaller points
commons-math4-legacyadded as atestdependency with no direct use injena-tdb2test
sources — presumably needed transitively at runtime forMEDIAN/MODEin
TestAggregatorsSkipScan. A one-line comment would save the next person the hunt. (Version
resolves fine from the parent'sdependencyManagement.)E_Equalsfor equality links (OpExecutorTDB2SkipScan.java:423) is value equality where
term equality is meant. It happens to be safe — a repeated variable always spans at least
one non-object slot, so both ends must be IRI or blank node — butE_SameTermis clearer,
cheaper, and doesn't rest on that argument.SystemTDBnow importsOpExecutorTDB2purely for a javadoc{@link}, adding a
sys→solvercompile edge.OpExecutorTDB2SkipScanexposes ~12public staticinternals (computeIndexMatch,
tryComputeIndexMap,remapProjection,remapEqualityLinks,remapValueConditions,
resolveTable,planCandidates,pickBest, …) used only by tests. Package-private would
keep the published API surface honest for an experimental feature.PatternQuery.createOrNull$— the trailing$is unusual for this codebase.IndexMatch.COMPARATORwraps inComparator.nullsLast(...), butpickBestnever sees a
null match.- On a cache miss,
findBestCandidateand thentryExeceach callresolveTableand
TuplePatternSpec.create— the second pair is redundant work per input binding. symSkipScandefaults on (isTrueOrUndef). Reasonable long-term, but given that finding 1
produces silently wrong answers rather than an error, consider one release opt-in. The flag
isn't documented anywhere either.
What I verified
mvn installonjena-dboe-base,jena-dboe-trans-data,jena-tdb2— clean.- Full
jena-tdb2suite afterclean: 1976 tests, 0 failures, 2 skipped (both the
@Disabledones above). - New suites specifically:
TS_SkipScan_3995,TS_TDB_SkipScan,Test_SPARQL_TDB—
1154 tests green. - A 44-query differential sweep (skipScan on vs off, order-insensitive) over a database with
literals, language tags,xsd:integer/xsd:decimalpairs, blank nodes, a graph named the
same as a subject, repeated variables in every position, quads, union graph, filters,BIND,
all the suitable aggregators,VALUES, sub-selects,OPTIONAL, and no-match patterns —
all 44 agreed. The?G0and duplicate-input-binding cases above were found by extending
that sweep. resolveTableinstrumented to throw on a tuple-width mismatch; full module suite plus
targetedOpGraph/ path / dynamic-dataset probes — never tripped.-Xlint:removalon the new package, to identify what the@SuppressWarningsis hiding.- Read through
computeIndexMatch,TuplePatternSpec.create,tryComputeIndexMap,
remapEqualityLinks, thefinalProjsizing, the record min/max +incrementKeyOrNull
bound construction, and theBPTreeDistinctKeyPrefixIteratorrange handling against
RecordBufferIterator/BPTreeNode.iteratorsemantics. No array-overflow or
off-by-one problems found; half-open bound handling is correct, including the all-ANY
overflow-to-null(unbounded) case. Index-space vs tuple-space remapping is consistent
throughout, despite theprimary/secondarylocal names inremapEqualityLinksreading
backwards. SkipScanRewrite.tryRewriteAsSkipScan(OpGroup)aggregator analysis: the distinct-invariance
argument holds for every aggregator inisSkipScanSuitable, andCOUNT(*)/
multi-variable / expression cases are correctly declined. (TheconvertToNonDistinct
javadoc mentionsSAMPLEas distinct-invariant whileisSkipScanSuitabledeliberately
excludes non-distinctAggSample— just a doc inconsistency.)TransformDistinctPlacementpush rules forOpProject/OpFilter/OpExtend/
nestedOpDistinct, including thereapplyProjectpath and the unstable-expression
rejection (rand(),uuid()), traced by hand on worked examples.
|
For the Variables internally don't have to be valid syntax. A unique prefix of |
GitHub issue resolved #3994
Pull request Description: Adds skip scan evaluation for single-pattern queries to TDB2. Added special code paths to
OpExecutorTDB2forOpDistinctandOpGroupBy.Execution time to find all distinct predicates on Wikidata Truthy (8B triples) becomes between 1 and 100 seconds (warm-cold caches)
Also works for simple group by patterns such as
SELECT ?p (COUNT(DISTINCT ?g) AS ?c) { GRAPH ?g { ?s ?p ?o } } GROUP BY ?pNeed yet to add tests - such as create permutations of patterns and compare results with a reference engine (eg. ARQ).
[ ] Documentation change and updates are provided for the Apache Jena websiteBy submitting this pull request, I acknowledge that I am making a contribution to the Apache Software Foundation under the terms and conditions of the Contributor's Agreement.
See the Apache Jena "Contributing" guide.