Rogue SPR: score leaf-pair candidates from parent split overlaps

.AGENTS/memory/Optimizations.md

@@ -1018,5 +1018,37 @@ pay; what pays is what the parent overlaps let a candidate *skip*.
   600k candidates).  Rejected: storing parent cell information and
   updating it by an O(1) delta (5.3 cycles/cell vs 8.5) - not worth a d^2
   double matrix once the fill is no longer dominant.
-- Next: depth-2 pair candidates (two leaves) by the same route; the
-  unbounded MCI could use the Tier 1 form if tests tolerate ulp changes.
+- Next: depth-2 pair candidates (two leaves) by the same route (done, see
+  below); the unbounded MCI could use the Tier 1 form if tests tolerate ulp
+  changes.
+
+## Rogue SPR: leaf-pair candidates from parent overlaps (2026-10-23)
+`SingleDrop` became `LeafDrop`, with `Prepare(i, j)` for depth-2 candidates.
+- Two removals are replayed on copies of the parent ledgers (leaf i once per
+  outer loop, then j per candidate), recording the vanishing split per tree
+  at each step.  Covers the cherry, sib(i) = par(j), mirror and root-child
+  cases without enumerating them; the one trap is that the root-child test
+  must be structural (`parent[parent[tip]] == parent[1]`), not
+  `row_of[parent] < 0`: after removing i, sib(i) can become the root's
+  child while still owning a row.
+- Overlap correction is popcount(row_has & col_has) on 2-bit masks;
+  `mutual_clustering_bounded_counts()` dispatches on `SplitCounts::pair` to
+  a template so the depth-1 kernel is unchanged.
+- Shared-split test: parent pairs whose polarized XOR is two leaves are
+  recorded as sorted pair keys; a pair also falls back if either leaf alone
+  makes a shared split.  Price merge chains: the first vanishing split's
+  merge target can itself vanish second, then it merges into the second's
+  target (prices only affect speed).
+- Fallback rate on 295 pairs (9-40 leaves, half near): 75% of incremental
+  pair candidates scored from overlaps; fallbacks are leaf 1 or a disabled
+  tree (14%), a single-leaf shared split (7%), a pair-shared split (1.6%),
+  reduction (1%).
+- Min of 3 interleaved runs vs main, depth 2: rand30 3.86 -> 1.65 s (-57%),
+  mid80 5.47 -> 3.10 s (-43%), near80 0.39 -> 0.21 s (-46%).  Depth 1
+  unchanged within noise (rand80 1.03-1.09 vs 1.07-1.11 s over 4 rounds).
+  SPRDist sums identical on all workloads.
+- Correctness: 295 tree pairs, 70,142 pair candidates, with ASSERTs on:
+  argmin sets of single and pair score vectors identical to
+  `incremental = FALSE`; max difference among near-best scores 7e-15.
+- Remaining at depth 2: candidates involving leaf 1 (~2/n) could be scored
+  by rerooting; ScoreWithout() for shared-split pairs.

.github/workflows/R-CMD-check.yml

@@ -338,7 +338,9 @@ jobs:
       !cancelled() && needs.sense-check.result == 'success' &&
       contains(fromJSON('["success", "skipped"]'), needs.codemeta.result) &&
       (github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch')
-    runs-on: ubuntu-latest
+    # Pinned: ubuntu-latest is now 26.04, on which setup-r apt-installs ~175
+    # packages and has timed out; RSPM below serves noble (24.04) binaries.
+    runs-on: ubuntu-24.04
     # A hung setup or a wedged benchmark otherwise runs to the 6-hour default,
     # holding a runner the whole time.
     timeout-minutes: 45
@@ -672,7 +674,7 @@ jobs:
     # and codemeta is skipped outside pull requests, so without this the
     # backstop never ran on push, schedule or workflow_dispatch.
     if: ${{ !cancelled() && needs.core.result == 'success' }}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04 # As benchmark
     timeout-minutes: 120  # revdepchecks are genuinely long
 
     name: revdepcheck (${{ matrix.config.deps }})

NEWS.md

@@ -35,6 +35,10 @@
 
 ## Performance
 
+- `CompareAll()` no longer takes time quadratic in the number of pairs on
+  older versions of R (e.g. 4.1), where each element access to a
+  `multiPhylo` object copied the whole list.
+
 - Trees with different leaf sets are pruned to their shared leaves in C++,
   without constructing pruned trees in R: `TreeDistance()` on 150 gene trees
   is c. 15× faster, with lower memory use (#123).
@@ -51,6 +55,10 @@
   the last few bits, which can break exact ties between candidates
   differently.
 
+- With `options(sprDepth = 2)`, `SPRDist(method = "Rogue")` likewise scores
+  most pairs of leaves from the parent trees' splits: 45–55% faster at 30–80
+  leaves.
+
 - Improve performance when comparing many similar trees with
   `ClusteringInfoDistance()`, `MatchingSplitDistance()`, `InfoRobinsonFoulds()`
   and `JaccardRobinsonFoulds()`.

R/tree_distance_utilities.R

@@ -535,28 +535,31 @@ Ntropy <- function(...) {
 CompareAll <- function(x, Func, FUN.VALUE = Func(x[[1]], x[[1]], ...),
                         ...) {
   nTree <- length(x)
+  labels <- names(x)
   countUp <- seq_len(nTree - 1)
-  i <- x[rep(countUp, rev(countUp))]
-  j <- x[unlist(sapply(countUp, function(n) n + seq_len(nTree - n)))]
-
+  i <- rep(countUp, rev(countUp))
+  j <- unlist(lapply(countUp, function(n) n + seq_len(nTree - n)))
+  # A plain list: on R 4.1, each `[[` on a multiPhylo copies the whole list
+  x <- lapply(seq_len(nTree), function(k) x[[k]])
+
   cluster <- getOption("TreeDist-cluster")
   ret <- if (is.null(cluster)) {
     cli_progress_bar("Comparing", total = length(i))
     vapply(seq_along(i), function(k) {
       cli_progress_update(1, .envir = parent.frame(2))
-      Func(i[[k]], j[[k]], ...)
+      Func(x[[i[k]]], x[[j[k]]], ...)
       }, FUN.VALUE)
   } else {
-    do.call("cbind", 
-            parLapply(cluster, seq_along(i), 
-                      function(k, Func) Func(i[[k]], j[[k]]),
+    do.call("cbind",
+            parLapply(cluster, seq_along(i),
+                      function(k, Func) Func(x[[i[k]]], x[[j[k]]]),
                       Func = Func))
   }
-  
+
   .WrapReturn <- function(dists) {
     structure(dists,
               Size = nTree,
-              Labels = names(x),
+              Labels = labels,
               Diag = FALSE,
               Upper = TRUE,
               class = "dist")

src/split_counts.h

@@ -4,13 +4,14 @@
 #include <cstdint>
 
 // Splits of a pair of trees, each obtained from a parent tree by removing
-// one leaf, described through the parent's splits.  The trees have `n_split`
-// splits each.  Child split i of the first tree restricts parent split
-// row[i]; it has row_in[i] leaves, and row_has[i] is 1 if the parent split
-// contained the removed leaf, else 0.  col, col_in and col_has describe the
-// second tree likewise.  overlap[r * stride + c] counts the leaves shared by
-// parent splits r and c, so the corresponding child splits share
-//   overlap[r * stride + c] - (row_has & col_has).
+// one leaf (or two, if `pair`), described through the parent's splits.  The
+// trees have `n_split` splits each.  Child split i of the first tree
+// restricts parent split row[i]; it has row_in[i] leaves, and bit k of
+// row_has[i] is set if the parent split contained removed leaf k.  col,
+// col_in and col_has describe the second tree likewise.
+// overlap[r * stride + c] counts the leaves shared by parent splits r and c,
+// so the corresponding child splits share
+//   overlap[r * stride + c] - popcount(row_has & col_has).
 struct SplitCounts {
   int n_tip; // Leaves in each child tree
   int n_split;
@@ -22,6 +23,7 @@ struct SplitCounts {
   const int *col;
   const int *col_in;
   const int16_t *col_has;
+  bool pair; // Two leaves removed; else row_has and col_has are 0 or 1
 };
 
 #endif