Skip to content

mpi: Fix halos of SubDomain Functions next to empty ranks - #3099

Open
EdCaunt wants to merge 6 commits into
mainfrom
fix-subdomain-function-halo
Open

EdCaunt wants to merge 6 commits into
mainfrom
fix-subdomain-function-halo

Conversation

@EdCaunt

@EdCaunt EdCaunt commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Fixes an issue which surfaced with Functions on SubDomains.

Issue: a Function on a SubDomain, run under MPI, gets garbage, NaN or stale values in its halo whenever a neighbouring rank holds none of the SubDomain. Two separate causes:

  1. Wrong neighbours at runtime. Codegen picks the SubDomain's neighbourhood, which says "no neighbour" (PROC_NULL) there. At apply, MPINeighborhood._arg_values replaced it with the root Grid's. Rank 0 then posted a receive from an empty rank that sent 0 floats, and copied an uninitialised posix_memalign buffer into its halo.
  2. Halo never zeroed. Decomposition.reshape gave the outhalo rows to the outermost rank even when that rank was empty. The default data_with_halo[:] = 0 sent those rows there, so rank 0's halo kept uninitialised memory, which the stencil read.

Comment thread devito/core/autotuning.py Outdated
if isinstance(p, MPINeighborhood):
at_args.update(
MPINeighborhood(p.neighborhood)._arg_values()
p._rebuild()._arg_values()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why rebuild?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

p._rebuild() without any agument is also supposed to be no-op so that's worrying

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It returns a copy, built from scratch iirc. That being said, the copy should be done explicitly rather than hidden behind a _rebuild. Tweaking.

@FabioLuporini FabioLuporini left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks sane, but I'd lie if I claimed I fully understood it

so corollary: there can be multiple neighbourhood objects in the gen code, in particular when Functions-on-subdomains are present?

@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 33.33333% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.47%. Comparing base (f7350b6) to head (0be3f47).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
tests/test_subdomains.py 16.66% 10 Missing ⚠️
devito/mpi/distributed.py 40.00% 6 Missing ⚠️
devito/types/grid.py 33.33% 2 Missing and 2 partials ⚠️
tests/test_pickle.py 0.00% 3 Missing ⚠️
devito/mpi/routines.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3099      +/-   ##
==========================================
- Coverage   84.51%   84.47%   -0.05%     
==========================================
  Files         260      260              
  Lines       58493    58533      +40     
  Branches     4962     4963       +1     
==========================================
+ Hits        49438    49443       +5     
- Misses       8226     8258      +32     
- Partials      829      832       +3     
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX 68.42% <38.09%> (-0.01%) ⬇️
pytest-gpu-gcc- 79.21% <33.33%> (-0.04%) ⬇️
pytest-gpu-icx- 79.14% <33.33%> (-0.02%) ⬇️
pytest-gpu-nvc-nvidiaX 69.12% <38.09%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@EdCaunt

EdCaunt commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

looks sane, but I'd lie if I claimed I fully understood it

Full disclaimer: this was Claude cooked whilst I was working on something else. The original bug arose when trying to implement the adjoint for CPML. I still need to give it a proper review myself.

so corollary: there can be multiple neighbourhood objects in the gen code, in particular when Functions-on-subdomains are present?

Yes, that's correct, since a function on a subdomain can have a different neighbourhood to the grid when the subdomain doesn't cross a rank edge (so you have an empty rank on the other side, for which there should be no halo exchange)

@mloubout mloubout added the MPI mpi-related label Nov 3, 2026
Comment thread devito/mpi/distributed.py
def _obj_neighborhood(self):
# The neighborhood is determined by the SubDomain thicknesses, hence the
# name. E.g., `nb_x0_7` for `{x: ('middle', 0, 7), y: y}`
name = ''.join(f'_{d.root.name}{d.ltkn.value}_{d.rtkn.value}'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why ltkn/rtkn.value instead of .name

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not mandatory to give subdomains unique names, although I agree that this solution is kinda inelegant

Comment thread devito/mpi/distributed.py Outdated

# Update `nb` based on the distributor of the (possibly overriding)
# Functions using it. E.g., an override may be defined on a new Grid
for p in inputs:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't look right to have to go through all of the op inputs withing an _arg_values

@EdCaunt
EdCaunt force-pushed the fix-subdomain-function-halo branch from af46c21 to 2ed4bc6 Compare November 5, 2026 15:27
Comment thread devito/mpi/distributed.py
__rkwargs__ = ('name',)

def __init__(self, neighborhood):
def __init__(self, neighborhood, name='nb'):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a better default name, since nb is used for e.g boundary size and such

@EdCaunt EdCaunt Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current default name is nb though, so this is no change really

Comment thread devito/mpi/distributed.py
# Use `nb` as provided by the (possibly overriding) Grid or SubDomain.
# E.g., an override may be defined on a new Grid
if self.name in kwargs.get('args', {}):
return {}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it be the usual

key = kwargs.get('args', {}).get(self.name, self)
return key._arg_defaults()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, since it should only fire here if the override/default has not already been processed by either Grid._arg_values or AbstractSubDomain._arg_values

Comment thread devito/types/grid.py Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

MPI mpi-related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants