[prrao87]

Why

The official LDBC IC14 text ("Trusted connection paths") takes about 2 minutes on SF1 (#123), which is way too slow, pointing at a suboptimal plan.

The benchmark suite doesn't run that text. graph-benchmark-ldbc-complex runs its own version of IC14: the same query, reworded in Cypher without reduce(). It unwinds each step of the path, counts the replies in each direction with size() of a pattern comprehension anchored on the two people, and adds them up with sum(). The benchmark's version takes 0.05 s and returns the same 58 rows as the official text.

The cause of the slowdown isn't reduce(): it is the pattern comprehension inside it, correlated with each path relationship r through

WHERE (a.ID = startNode(r).ID AND b.ID = endNode(r).ID)
   OR (a.ID = endNode(r).ID AND b.ID = startNode(r).ID)

Every branch reads both a and b, so no part of the condition can run before the expansion reaches b. The plan cross-joined the distinct path relationships with every Person, expanded three hops from all of them, and filtered afterwards. The same happens for any WHERE that accepts either direction of an edge, in MATCH, OPTIONAL MATCH, EXISTS or a comprehension.

What changed

When a WHERE is planned, each conjunct that is an OR of ANDs now also contributes one implied predicate per variable it reads: every branch with that variable's conjuncts removed, OR-ed back together. Dropping b above gives a.ID = startNode(r).ID OR a.ID = endNode(r).ID, which reads only a and outer values, so DataFusion's existing filter pushdown moves it below the expansion:

before: CrossJoin(r, every Person) -> expand 3 hops -> Filter(OR)
after:  NestedLoopJoin(r, Person ON a.ID = s OR a.ID = e) -> expand 3 hops from <= 2 people -> Filter(OR)

• The original predicate stays in the filter, so results can't change: a true branch makes each of its conjuncts true, hence the shortened branch too.
• Nothing is added when a branch would be left with no conjunct (a.x = 1 OR b.y = 2) or when nothing was dropped.
• Only comparisons and IN lists without subqueries are copied, so no pattern or comprehension is planned twice.
• The rewrite is not specific to IC14 or to id properties. The DataFusion internals page on optimizer rewrites documents it.

User impact: no API change. Queries with this kind of WHERE can get faster; others plan exactly as before.

What remains: the official text runs in 0.31 s, against 0.05 s for the benchmark's version. The remaining time is the nested-loop join comparing each relationship with every Person, since an OR of equalities isn't a hash join. Closing that gap would need a more targeted rewrite, so this PR stops here. The benchmark repository still runs its own version; switching it to the official text, and updating the benchmarks page, can follow a release with this change.

Refs #123

Validation

• Official IC14 text on SF1, memory mode, release build: about 2 min on main, 0.31 s on this branch. Its 58 rows (paths and weights) are identical to those of the benchmark's version.
• New Rust unit tests for the implied predicates, including cases that must imply nothing.
• New Python tests, with and without CSR, check results for an either-direction MATCH (null properties, a duplicate edge) and for the IC14 shape, and check that the implied predicate sits below the expansion.
• cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, make test (1,710 passed), make lint, the strict docs build and typos all pass locally. CI pending.

Benchmarks

Both suites, memory and lazy, run with the harness in CONTRIBUTING.md: release wheels from main and from this branch, alternated, two runs each. Every query's answer check passed. Times are the mean of the two runs per build.

Suite	Mode	Geomean branch / main	Gap between main runs
graph-benchmark-ldbc	memory	0.999	1.001
graph-benchmark-ldbc	lazy	0.988	0.971
graph-benchmark-ldbc-complex	memory	1.012	1.001
graph-benchmark-ldbc-complex	lazy	1.003	0.998

IC12 in memory mode came out 14% slower in both branch runs. Its plan is byte-identical on both builds (its only OR has single-condition branches, so the rewrite adds nothing), and a control rerun of IC12 alone, alternating builds three times, gave main 16.9 to 19.4 ms and the branch 17.8 to 19.7 ms. It is run-to-run noise.

The benchmarked IC14 is the benchmark's version, whose conditions have no OR, so the rewrite doesn't change it; the official text isn't in either suite yet.

graph-benchmark-ldbc (30 queries), memory: geomean 0.999, main run gap 1.001

Query	main (ms)	branch (ms)	Change
Q1	0.6	0.6	+1.0%
Q2	0.6	0.6	-0.4%
Q3	0.7	0.7	+0.7%
Q4	0.5	0.5	-0.4%
Q5	1.1	1.1	+1.0%
Q6	0.4	0.4	+0.0%
Q7	1.0	1.1	+2.7%
Q8	0.6	0.6	-0.5%
Q9	0.8	0.8	-0.8%
Q10	0.7	0.7	-0.6%
Q11	1.5	1.5	-0.5%
Q12	0.8	0.8	-0.6%
Q13	0.8	0.8	-1.0%
Q14	0.5	0.5	-0.5%
Q15	0.6	0.6	-0.4%
Q16	0.7	0.7	-0.5%
Q17	0.9	0.9	-1.3%
Q18	0.7	0.7	-0.6%
Q19	1.0	1.0	-0.5%
Q20	3.5	3.5	+0.8%
Q21	0.8	0.8	-0.5%
Q22	0.7	0.7	+0.1%
Q23	0.7	0.7	+0.8%
Q24	0.7	0.7	+0.4%
Q25	0.8	0.8	+0.0%
Q26	0.8	0.7	-1.4%
Q27	0.9	0.9	-0.7%
Q28	1.4	1.4	+1.2%
Q29	0.7	0.7	-1.0%
Q30	41.8	41.8	+0.1%

graph-benchmark-ldbc (30 queries), lazy: geomean 0.988, main run gap 0.971

Query	main (ms)	branch (ms)	Change
Q1	1.2	1.2	-1.3%
Q2	1.4	1.4	+0.1%
Q3	0.8	0.8	-2.1%
Q4	0.8	0.8	-1.3%
Q5	4.0	3.8	-5.1%
Q6	0.8	0.7	-1.7%
Q7	1.1	1.1	-2.7%
Q8	1.8	1.7	-2.6%
Q9	1.4	1.3	-1.5%
Q10	1.0	0.9	-1.8%
Q11	2.0	2.0	-0.9%
Q12	6.6	6.6	-0.7%
Q13	1.1	1.1	-0.1%
Q14	0.7	0.7	-0.1%
Q15	0.8	0.8	-1.0%
Q16	1.1	1.1	+0.9%
Q17	1.4	1.4	-1.3%
Q18	1.1	1.1	-0.6%
Q19	1.5	1.5	-1.1%
Q20	23.6	23.7	+0.3%
Q21	0.9	0.8	-2.3%
Q22	0.9	0.9	-1.9%
Q23	1.2	1.2	-0.9%
Q24	1.1	1.1	-1.3%
Q25	1.2	1.2	-1.4%
Q26	1.1	1.1	-1.1%
Q27	1.3	1.3	-1.5%
Q28	2.1	2.0	-1.0%
Q29	1.0	1.0	-1.2%
Q30	42.5	42.7	+0.5%

graph-benchmark-ldbc-complex (IC1 to IC14), memory: geomean 1.012, main run gap 1.001

Query	main (ms)	branch (ms)	Change
IC1	23.1	23.1	-0.0%
IC2	2.5	2.5	+0.3%
IC3	25.7	25.5	-1.0%
IC4	5.9	5.9	+0.0%
IC5	75.7	74.8	-1.1%
IC6	21.6	22.2	+2.5%
IC7	9.8	9.7	-1.1%
IC8	2.3	2.3	+0.2%
IC9	37.7	38.4	+1.8%
IC10	20.4	20.5	+0.1%
IC11	3.0	3.0	+0.1%
IC12	17.0	19.4	+14.3%
IC13	1.1	1.1	+1.0%
IC14	49.0	49.0	+0.0%

graph-benchmark-ldbc-complex (IC1 to IC14), lazy: geomean 1.003, main run gap 0.998

Query	main (ms)	branch (ms)	Change
IC1	32.4	32.2	-0.6%
IC2	25.4	26.0	+2.4%
IC3	165.5	164.6	-0.5%
IC4	12.9	12.9	-0.4%
IC5	112.5	110.7	-1.5%
IC6	23.7	23.7	-0.1%
IC7	14.6	14.6	+0.6%
IC8	3.3	3.3	-0.2%
IC9	308.8	300.5	-2.7%
IC10	29.3	29.3	-0.0%
IC11	5.4	5.4	+0.5%
IC12	21.8	22.8	+4.5%
IC13	1.4	1.5	+0.6%
IC14	50.2	51.0	+1.6%

Commits: lance-csr main ffc3b7e and this branch on top of it; graph-benchmark-ldbc 3afadb9; graph-benchmark-ldbc-complex fc7b44c. Machine: Apple M5, 24 GiB, macOS 26.6.2.

🤖 Generated with Claude Code