Add optional force_nofill to skip needsfill - #2027
Open
sdbachman wants to merge 2 commits into
Open
Conversation
Incomplete compute maps (e.g. masked land points) set needsfill and can corrupt heap in the subset rearranger. Allow apps to opt out via PIOc_InitDecomp*_flags and Fortran PIO_initdecomp(force_nofill=...), preserving default behavior when the flag is omitted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Plain-language summary
Some applications may need to read/write only over a portion of the model grid, but PIO currently won't allow that because of the "needsfill" attribute. This PR creates a way to (optionally) bypass the path where needsfill is set so that PIO will allow reads/writes to only certain parts of the grid.
Summary
Add an opt-in way for applications to skip PIO’s decomposition coverage / fill-value path (needsfill) when initializing a decomposition.
New optional flag: force_nofill. When set, determine_fill() forces needsfill = false and returns immediately, so the subset rearranger does not build the holegrid / fill machinery for that decomposition.
Existing PIOc_InitDecomp / PIOc_InitDecomp_bc / Fortran PIO_initdecomp call sites are unchanged: omitting the flag preserves current behavior.
Problem
Some applications (e.g. coastal ocean models with land masking) define decompositions whose compute map does not cover every point of the global array. In that case, determine_fill() sets iodesc->needsfill = true, and the subset rearranger allocates and populates holegrid / fill structures.
With ROMS + MASKING on Derecho (parallel I/O via start/count PIO_initdecomp → PIOc_InitDecomp_bc, which always uses PIO_REARR_SUBSET), that path led to heap corruption / glibc malloc aborts during the first initdecomp, before any history I/O completed. Forcing needsfill = false for those decompositions avoided the crash and allowed the run to proceed.
Note: changing the rearranger at PIO_init (e.g. box vs subset) does not affect the block-cyclic (_bc) path, which hardcodes subset rearranging. A per-decomposition opt-out of needsfill is the targeted fix for this class of use case.
Changes
C API (pio.h, pioc.c)
• Add bool force_nofill to io_desc_t.
◦ Add:
◦ PIOc_InitDecomp_flags(..., int force_nofill)
◦ PIOc_InitDecomp_bc_flags(..., int force_nofill)
◦ Keep existing entry points as thin wrappers that pass force_nofill = 0:
◦ PIOc_InitDecomp(...) → PIOc_InitDecomp_flags(..., 0)
◦ PIOc_InitDecomp_bc(...) → PIOc_InitDecomp_bc_flags(..., 0)
• Broadcast force_nofill with the other initdecomp arguments on async IO systems.
Core logic (pio_rearrange.c)
In determine_fill():
if (iodesc->force_nofill) {
iodesc->needsfill = false;
return PIO_NOERR;
}
Fortran API (piolib_mod.F90)
◦ Add optional force_nofill to:
◦ PIO_initdecomp_bc (start/count path)
◦ PIO_initdecomp_dof_i4 / _i8 / internal
• If absent or .false., pass 0 to C (stock behavior).
• If .true., pass 1 and skip the needsfill path.
Example:
call PIO_initdecomp(iosystem, PIO_double, dims, start, count, iodesc, &
force_nofill=.true.)
Compatibility
When to use this
Use force_nofill=.true. / force_nofill != 0 when:
• The application guarantees that unwritten / non-owned points do not need PIO fill values (e.g. masked land points that the model never writes, or the file already has acceptable fill / the app does not rely on PIO hole filling), and
• The automatic needsfill / holegrid path is incorrect or harmful for that decomposition.
Do not use it if the application depends on PIO writing fill values for uncovered grid points.
Files touched