Skip to content

Fixes for MP/MR Datadeps parallel - #746

Merged
jpsamaroo merged 5 commits into
masterfrom
jps/hier-mp-mr-fixes
Sep 5, 2026
Merged

Fixes for MP/MR Datadeps parallel#746
jpsamaroo merged 5 commits into
masterfrom
jps/hier-mp-mr-fixes

Conversation

@jpsamaroo

@jpsamaroo jpsamaroo commented Sep 4, 2026

Copy link
Copy Markdown
Member

Written by Claude Opus

@github-actions

github-actions Bot commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Dagger benchmarks: dirty vs master

Median time

master dirty master / dirty
array/dagger/N=1024 (block 512)/add (X + X) 4.77 ± 2.7 ms 2.77 ± 2.6 ms 1.72 ± 1.9
array/dagger/N=1024 (block 512)/alloc (rand) 2.2 ± 0.078 ms 2.38 ± 0.6 ms 0.924 ± 0.23
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 2.54 ± 0.5 ms 2.84 ± 0.66 ms 0.897 ± 0.27
array/dagger/N=1024 (block 512)/map (sin.(X)) 6.21 ± 1.2 ms 5.68 ± 0.56 ms 1.09 ± 0.23
array/dagger/N=1024 (block 512)/norm 1.2 ± 0.55 ms 2.83 ± 2.5 ms 0.424 ± 0.43
array/dagger/N=1024 (block 512)/reduce (sum) 2.15 ± 0.15 ms 2.35 ± 1.7 ms 0.914 ± 0.66
array/dagger/N=1024 (block 512)/transpose (permutedims) 4.49 ± 1.4 ms 5.45 ± 1.4 ms 0.825 ± 0.34
array/dagger/N=256 (block 256)/add (X + X) 0.866 ± 0.15 ms 0.85 ± 0.57 ms 1.02 ± 0.7
array/dagger/N=256 (block 256)/alloc (rand) 0.678 ± 0.039 ms 0.693 ± 0.012 ms 0.979 ± 0.059
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.525 ± 0.22 ms 0.508 ± 0.19 ms 1.03 ± 0.58
array/dagger/N=256 (block 256)/map (sin.(X)) 0.982 ± 0.039 ms 0.952 ± 0.17 ms 1.03 ± 0.19
array/dagger/N=256 (block 256)/norm 0.494 ± 0.034 ms 0.453 ± 0.037 ms 1.09 ± 0.12
array/dagger/N=256 (block 256)/reduce (sum) 0.613 ± 0.0076 ms 0.53 ± 0.032 ms 1.16 ± 0.071
array/dagger/N=256 (block 256)/transpose (permutedims) 0.728 ± 0.18 ms 0.721 ± 0.041 ms 1.01 ± 0.25
linalg/dagger/N=1024 (block 512)/cholesky 27 ± 6.7 ms 21.6 ± 2.8 ms 1.25 ± 0.35
linalg/dagger/N=1024 (block 512)/lu 0.0466 ± 0.0082 s 0.0439 ± 0.0023 s 1.06 ± 0.19
linalg/dagger/N=1024 (block 512)/matmul (A*A) 0.0501 ± 0.01 s 0.048 ± 0.014 s 1.05 ± 0.38
linalg/dagger/N=1024 (block 512)/matvec (A*x) 3.06 ± 0.7 ms 3.06 ± 1.5 ms 1 ± 0.55
linalg/dagger/N=1024 (block 512)/qr 0.119 ± 0.0062 s 0.111 ± 0.011 s 1.08 ± 0.12
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.0535 ± 0.0088 s 0.0492 ± 0.0083 s 1.09 ± 0.26
linalg/dagger/N=1024 (block 512)/svd 0.0319 h 0.0361 h 0.885
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 0.0368 ± 0.0029 s 0.0405 ± 0.0057 s 0.909 ± 0.15
linalg/dagger/N=256 (block 256)/cholesky 3.68 ± 6.2 ms 3.26 ± 2.8 ms 1.13 ± 2.1
linalg/dagger/N=256 (block 256)/lu 3.36 ± 0.16 ms 4.06 ± 1.9 ms 0.829 ± 0.39
linalg/dagger/N=256 (block 256)/matmul (A*A) 2.46 ± 0.41 ms 2.5 ± 0.72 ms 0.983 ± 0.33
linalg/dagger/N=256 (block 256)/matvec (A*x) 1.38 ± 0.078 ms 2.56 ± 1.3 ms 0.54 ± 0.29
linalg/dagger/N=256 (block 256)/qr 4.64 ± 0.46 ms 4.17 ± 0.37 ms 1.11 ± 0.15
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 13 ± 5.7 ms 21.3 ± 8.5 ms 0.611 ± 0.36
linalg/dagger/N=256 (block 256)/svd 0.678 ± 0.078 s 0.615 ± 0.011 s 1.1 ± 0.13
linalg/dagger/N=256 (block 256)/syrk (A'*A) 3.33 ± 2.1 ms 3.94 ± 2.4 ms 0.845 ± 0.75
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 0.931 ± 0.039 s 0.905 ± 0.033 s 1.03 ± 0.057
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 0.491 ± 0.01 s 0.486 ± 0.021 s 1.01 ± 0.049
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.0533 ± 0.0055 s 0.063 ± 0.011 s 0.847 ± 0.17
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 0.923 ± 0.047 s 0.912 ± 0.021 s 1.01 ± 0.056
sparse/dagger/N=256 (block 16)/spgemm (S*S) 0.489 ± 0.023 s 0.467 ± 0.0095 s 1.05 ± 0.053
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.065 ± 0.016 s 0.0494 ± 0.0072 s 1.32 ± 0.38
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 7.11 ± 1.5 ms 6.79 ± 0.07 ms 1.05 ± 0.23
stencil/dagger/N=1024 (block 512)/assign (const) 1.32 ± 1.8 ms 1.06 ± 0.26 ms 1.25 ± 1.7
stencil/dagger/N=1024 (block 512)/multi-expr 4.94 ± 2.5 ms 3.98 ± 3.3 ms 1.24 ± 1.2
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 6.95 ± 1.3 ms 6.24 ± 1.5 ms 1.11 ± 0.34
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 6.05 ± 0.081 ms 6.09 ± 0.6 ms 0.993 ± 0.099
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 5.83 ± 0.16 ms 6.03 ± 0.14 ms 0.966 ± 0.035
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 6.96 ± 1.3 ms 5.63 ± 0.25 ms 1.24 ± 0.24
stencil/dagger/N=1024 (block 512)/update (+) 1.91 ± 0.5 ms 1.41 ± 0.93 ms 1.36 ± 0.97
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 1.76 ± 0.05 ms 1.87 ± 0.55 ms 0.942 ± 0.28
stencil/dagger/N=256 (block 256)/assign (const) 0.513 ± 0.016 ms 0.466 ± 0.053 ms 1.1 ± 0.13
stencil/dagger/N=256 (block 256)/multi-expr 2.99 ± 1.6 ms 2.42 ± 1.5 ms 1.24 ± 1
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.71 ± 0.28 ms 1.52 ± 0.072 ms 1.13 ± 0.19
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.32 ± 0.1 ms 2.41 ± 2.2 ms 0.548 ± 0.51
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.28 ± 0.092 ms 1.31 ± 0.33 ms 0.971 ± 0.25
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.47 ± 0.02 ms 1.55 ± 0.088 ms 0.945 ± 0.055
stencil/dagger/N=256 (block 256)/update (+) 0.667 ± 0.087 ms 1.71 ± 3 ms 0.39 ± 0.68
time_to_load 1.12 ± 0.0084 s 1.14 ± 0.012 s 0.986 ± 0.013

Allocations / memory

master dirty master / dirty
array/dagger/N=1024 (block 512)/add (X + X) 5.54 k allocs: 8.22 MB 5.53 k allocs: 8.22 MB 1
array/dagger/N=1024 (block 512)/alloc (rand) 2.74 k allocs: 8.09 MB 2.74 k allocs: 8.09 MB 1
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 2.4 k allocs: 8.08 MB 2.41 k allocs: 8.08 MB 1
array/dagger/N=1024 (block 512)/map (sin.(X)) 2.09 k allocs: 8.07 MB 2.1 k allocs: 8.07 MB 1
array/dagger/N=1024 (block 512)/norm 2.71 k allocs: 0.0905 MB 2.72 k allocs: 0.0907 MB 0.998
array/dagger/N=1024 (block 512)/reduce (sum) 4.16 k allocs: 0.142 MB 4.13 k allocs: 0.141 MB 1.01
array/dagger/N=1024 (block 512)/transpose (permutedims) 3.32 k allocs: 8.14 MB 3.34 k allocs: 8.14 MB 1
array/dagger/N=256 (block 256)/add (X + X) 1.73 k allocs: 0.575 MB 1.72 k allocs: 0.575 MB 1
array/dagger/N=256 (block 256)/alloc (rand) 0.754 k allocs: 0.526 MB 0.757 k allocs: 0.526 MB 1
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.651 k allocs: 0.522 MB 0.651 k allocs: 0.522 MB 1
array/dagger/N=256 (block 256)/map (sin.(X)) 0.574 k allocs: 0.52 MB 0.574 k allocs: 0.52 MB 1
array/dagger/N=256 (block 256)/norm 0.715 k allocs: 24.6 kB 0.715 k allocs: 24.6 kB 1
array/dagger/N=256 (block 256)/reduce (sum) 0.816 k allocs: 30.5 kB 0.815 k allocs: 30.5 kB 0.999
array/dagger/N=256 (block 256)/transpose (permutedims) 1.11 k allocs: 0.551 MB 1.11 k allocs: 0.551 MB 1
linalg/dagger/N=1024 (block 512)/cholesky 7.82 k allocs: 10.3 MB 7.82 k allocs: 10.3 MB 1
linalg/dagger/N=1024 (block 512)/lu 15.6 k allocs: 14.6 MB 15.6 k allocs: 14.6 MB 1
linalg/dagger/N=1024 (block 512)/matmul (A*A) 7.74 k allocs: 8.29 MB 7.75 k allocs: 8.29 MB 1
linalg/dagger/N=1024 (block 512)/matvec (A*x) 6.16 k allocs: 0.246 MB 6.05 k allocs: 0.242 MB 1.02
linalg/dagger/N=1024 (block 512)/qr 11.7 k allocs: 9.6 MB 11.5 k allocs: 9.59 MB 1
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 29.3 k allocs: 15.2 MB 30 k allocs: 15.2 MB 0.998
linalg/dagger/N=1024 (block 512)/svd 0.0498 M allocs: 0.198 GB 0.0503 M allocs: 0.197 GB 1
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 9.13 k allocs: 20.4 MB 9.11 k allocs: 20.4 MB 1
linalg/dagger/N=256 (block 256)/cholesky 3.06 k allocs: 0.634 MB 3.05 k allocs: 0.635 MB 1
linalg/dagger/N=256 (block 256)/lu 5.4 k allocs: 1.24 MB 5.42 k allocs: 1.24 MB 1
linalg/dagger/N=256 (block 256)/matmul (A*A) 1.94 k allocs: 0.582 MB 1.94 k allocs: 0.582 MB 1
linalg/dagger/N=256 (block 256)/matvec (A*x) 2.36 k allocs: 0.0995 MB 2.37 k allocs: 0.0999 MB 0.996
linalg/dagger/N=256 (block 256)/qr 3.45 k allocs: 0.779 MB 3.44 k allocs: 0.779 MB 1
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 11.9 k allocs: 1.53 MB 11.9 k allocs: 1.53 MB 1
linalg/dagger/N=256 (block 256)/svd 15.5 k allocs: 6.71 MB 15.5 k allocs: 6.71 MB 1
linalg/dagger/N=256 (block 256)/syrk (A'*A) 3.07 k allocs: 2.14 MB 3.06 k allocs: 2.14 MB 1
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 3.03 M allocs: 0.112 GB 3.09 M allocs: 0.114 GB 0.982
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 3.33 M allocs: 0.156 GB 2.91 M allocs: 0.142 GB 1.1
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.174 M allocs: 6.65 MB 0.211 M allocs: 7.92 MB 0.839
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 2.92 M allocs: 0.108 GB 3.03 M allocs: 0.112 GB 0.966
sparse/dagger/N=256 (block 16)/spgemm (S*S) 3.27 M allocs: 0.117 GB 2.92 M allocs: 0.106 GB 1.11
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.188 M allocs: 7.13 MB 0.193 M allocs: 7.3 MB 0.977
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 4.8 k allocs: 8.2 MB 4.79 k allocs: 8.2 MB 1
stencil/dagger/N=1024 (block 512)/assign (const) 2.18 k allocs: 0.0949 MB 2.17 k allocs: 0.0949 MB 0.999
stencil/dagger/N=1024 (block 512)/multi-expr 4.81 k allocs: 0.212 MB 4.8 k allocs: 0.211 MB 1
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 3.83 k allocs: 0.232 MB 3.8 k allocs: 0.232 MB 1
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 3.73 k allocs: 0.23 MB 3.72 k allocs: 0.23 MB 1
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 3.8 k allocs: 0.262 MB 3.8 k allocs: 0.262 MB 1
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 3.64 k allocs: 0.167 MB 3.65 k allocs: 0.168 MB 0.995
stencil/dagger/N=1024 (block 512)/update (+) 2.61 k allocs: 0.116 MB 2.62 k allocs: 0.117 MB 0.986
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 1.56 k allocs: 0.575 MB 1.56 k allocs: 0.575 MB 1
stencil/dagger/N=256 (block 256)/assign (const) 0.811 k allocs: 0.041 MB 0.8 k allocs: 0.0406 MB 1.01
stencil/dagger/N=256 (block 256)/multi-expr 1.73 k allocs: 0.0873 MB 1.73 k allocs: 0.0873 MB 1
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.27 k allocs: 0.0722 MB 1.26 k allocs: 0.0718 MB 1.01
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.21 k allocs: 0.0703 MB 1.21 k allocs: 0.07 MB 1
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.31 k allocs: 0.0811 MB 1.28 k allocs: 0.0801 MB 1.01
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.2 k allocs: 0.0625 MB 1.24 k allocs: 0.0638 MB 0.979
stencil/dagger/N=256 (block 256)/update (+) 0.924 k allocs: 0.0463 MB 0.902 k allocs: 0.0456 MB 1.02
time_to_load 0.147 k allocs: 10.8 kB 0.147 k allocs: 10.8 kB 1

Plots

No time regressions beyond 25.0% or allocation regressions beyond 25.0% (timing changes inside the reported ±spread don't count) 🎉

Within noise (7 metric(s) past threshold but inside the ±spread; not counted)
  • stencil/dagger/N=256 (block 256)/update (+) (time): 156.1%
  • array/dagger/N=1024 (block 512)/norm (time): 136.1%
  • linalg/dagger/N=256 (block 256)/matvec (A*x) (time): 85.1%
  • stencil/dagger/N=256 (block 256)/neighbors (Pad) (time): 82.6%
  • linalg/dagger/N=256 (block 256)/solve (A\b via lu) (time): 63.5%
  • stencil/dagger/N=1024 (block 512)/update (+) (time): -26.5%
  • array/dagger/N=1024 (block 512)/add (X + X) (time): -41.9%

Full results and plots (download the benchmark-results artifact).

jpsamaroo and others added 5 commits September 4, 2026 16:46
@jpsamaroo
datadeps/hierarchical: Rank by written args separately during partiti…
7b28e84 
…on_dag
@jpsamaroo @cursoragent
Sch/MPI: Cut fixed per-task overhead in placement and dispatch
8120495 
Three costs that each task paid regardless of how much work it did, and
which therefore grew to dominate as chunks got smaller with more workers
or ranks:

* `compatible_processors` scanned every processor in the cluster and
  constrained the scope against each. A scope built from `ExactScope`s
  already names its processors, so `scope_processors` reads them off
  directly and only checks that they still exist and are enabled. This is
  the shape Datadeps hands the scheduler for every pinned task.
* `estimate_task_costs!` walked a task's chunks to rank candidate
  processors even when there was only one candidate to rank. Datadeps
  pins each task to a single space, so this was pure overhead.
* MPI's uniform dispatch ran `promote_op` and `infer_effects` per task
  per rank to decide whether the result needs a status broadcast.
  Uncached, each dispatch allocated megabytes of inference state on
  every rank, and because ranks advance in lock-step the resulting GC
  pauses did not overlap with anything. Both queries now go through
  memos (`cached_return_type`, the new `cached_nothrow`).

Co-authored-by: Cursor <cursoragent@cursor.com>
@jpsamaroo @cursoragent
Datadeps: Decide slot pass-through by type, not by local inspection
1bb554d 
`slot_is_already_in_place` decides whether a chunk can serve as its own
slot instead of being copied. It required the chunk to be locally owned
and unwrappable here, because it answered the question by unwrapping the
value and asking whether `move_rewrap` would treat it as a leaf. That
excluded exactly the cases where a needless copy is most expensive:
under MPI every rank plans every task but owns only some of the data,
and under Distributed the chunk is often homed on another worker. In
both, the alternative is a `move_rewrap` that reproduces, message by
message, data already sitting where it is needed -- for a wrapper like
`HaloArray`, a header broadcast plus one transfer per child, per
argument, per region.

`slot_rewrap_is_identity` answers the same question from `chunktype`
alone: a rewrap is the identity unless the type contains a handle whose
resolution is the point of the rewrap (`Chunk`, `DTask`, `ChunkView`),
so wrappers over plain data now pass through. `chunktype` is uniform
across ranks, so the decision is too.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jpsamaroo @claude
Memory spaces: Fix operator precedence in will_alias short-circuits
9da4cd2 
`&&` binds tighter than `||`, so

    x isa NoAliasing || y isa NoAliasing && return false

parses as `x isa NoAliasing || (y isa NoAliasing && return false)`: when
`x` is the special ainfo the `||` short-circuits to `true` and the function
falls through without returning.

Harmless today. `NoAliasing` still answers correctly because its
`memory_spans` is empty, making the span loop below vacuous; and
`UnknownAliasing` is unreachable, since every site that would return it
calls `error_unknown_aliasing` first, which throws.

Worth fixing anyway, because the `UnknownAliasing` fallthrough is wrong in
a dangerous direction: it reaches the span comparison, where `may_alias`
rejects operands in different memory spaces, and answers `false`. "Cannot
determine aliasing" must be conservative -- an under-approximation here
drops a dependency and races. Datadeps interop work may well make this
branch reachable by returning `UnknownAliasing` for foreign data rather
than throwing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jpsamaroo
datadeps: Don't compute buffer aliasing on one rank only
fc4a5f6 
`gather_free_syncdeps!` decided whether it could inspect a free candidate's
backing buffer via `aliasing_available`, which under MPI was true only on the
rank owning the ref. But `aliasing` is *collective* there -- the owner computes
and broadcasts -- so the owning rank alone would enter a broadcast no other rank
was in. That consumed a tag on one rank, desynchronizing every subsequent
`to_tag`, and then deadlocked in the copy cycles at region end.

Symptom on this branch: 26 errors per rank in `test/mpi.jl` (Cholesky, LU, and
every `@stencil` case), all `ArgumentError: Non-uniform value` raised from
`take_ref_id!`. Bisected to 87273af -- not because that commit is wrong, but
because ranking by written args moves each task onto the rank owning the chunk
it writes, which is what first makes a buffer underlie only wrapper arguments
and so reach this branch at all. Every rank now takes the rank-uniform
`key_ainfo` fallback whenever execution is uniform, so the decision depends
solely on `uniform_execution()`.

That leaves `aliasing_available` with no remaining call site --
`uniform_execution(::MPIAcceleration)` is unconditionally `true`, so the MPIRef
method could never be reached past the short-circuit -- so it is removed along
with its generic fallback.

`test/mpi.jl` goes from 26 errors per rank to 0 (395/397/385/373 passing, no
failures); Distributed datadeps is unchanged at 1272 passing.

Also adds a "Partial slot currency" testset covering the case that motivated
this: writing through a `ChunkView` on a non-owner rank and then reading the
whole parent there, which must see the view's write and pull the rest from the
owner. Its expectation is snapshotted *before* the region -- `A` is a non-Chunk
argument, so rank 0 owns it and Datadeps writes back into it in place; deriving
the baseline afterwards folds the update in and then adds it again (and only
rank 0's copy is written back, so it would also disagree across ranks).
@jpsamaroo
jpsamaroo force-pushed the jps/hier-mp-mr-fixes branch from 8b99db0 to fc4a5f6 Compare September 4, 2026 23:46
@jpsamaroo
jpsamaroo merged commit 0bebbaa into master Sep 5, 2026
30 of 42 checks passed
@jpsamaroo
jpsamaroo deleted the jps/hier-mp-mr-fixes branch September 5, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

bug datadeps mpi performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpsamaroo