Dear SWIFT team,
I am an astrophysics student at Tübingen University, working on adding material strength and plasticity models for planetary embryo collisions. I was running the stock 'DemoImpact' example simulation under swift_mpi, and it consistently aborts partway through the impact with:
scheduler.h:scheduler_activate_pack():253: Missing link to pack task. MPI_ABORT was invoked on rank N in communicator MPI_COMM_WORLD with errorcode -1.
The serial swift binary, built from the same source with the same configure line and run with the same ICs and parameter file, completes the full simulation without error (14,072 steps, all 28 snapshots). My work environment is as follows:
Cluster: BinAC 2 (bwHPC), single node
CPU: 2× AMD EPYC 7543 (Zen 3), 64 physical cores / 128 logical per node
Compiler: GCC 11.5.0
MPI: Open MPI 4.1.8
HDF5: 1.12.1 (parallel)
GSL: 2.8
FFTW: 3.3.10 (MPI build)
METIS: 5.2.1 built from source, IDXTYPEWIDTH 64, METIS_PartGraphKway present
ParMETIS: not available
Here is how I configured:
./configure \ --enable-mpi \ --with-hydro=planetary \ --with-equation-of-state=planetary \ --with-metis=$HOME/local \ --with-hdf5=<...>/bin/h5pcc \ --with-gsl=<...> \ --with-fftw=<...> \ --disable-doxygen-doc \ CC=mpicc \ CFLAGS="-O3 -march=native" \ CPPFLAGS="-I$HOME/local/include" \ LDFLAGS="-L$HOME/local/lib -L$HOME/local/lib64 -L<...>/gsl/lib64" \ LIBS="-lGKlib -lm"
The configure summary confirms: MPI enabled: yes, HDF5 parallel: yes, METIS/ParMETIS: yes / no, Hydro scheme: planetary, Equation of state: planetary, Planetary fixed entropy: no.
The run command used: srun --mpi=pmix --cpu-bind=cores \ ./swift_mpi --hydro --self-gravity --threads=$SLURM_CPUS_PER_TASK \ demo_impact_n50.yml
Single node was maintained throughout. Additional commands used to run: OMPI_MCA_pml=ob1, OMPI_MCA_btl="self,vader".
I made some changes the yaml parameter file, notably:
Scheduler:
max_top_level_cells: 32 # stock is 64
links_per_tasks: 1500 # stock/default 25 overflows immediately under MPI
tasks_per_cell: 50 # not present in stock
Restarts:
enable: 1
subdir: restart
delta_hours: 1.0
DomainDecomposition:
repartition_type: none
Some of the results after simulations:
SWIFT , CFLAGS , max_top_level_cells , Ranks × threads , Outcome
v2026.04 , -03 -flto -ffast-math , 32 , 4 × 8 , fails step 8903, t≈3.186e+04
v2026.04 , -02 only , 32 , 4 × 8 , fails step 8830, t≈3.182e+04
v2026.04 , -02 only , 128 , 4 × 8 , fails step 1840, t≈7.96e+03
v2026.01 , -03 -flto -ffast-math , 32 , serial swift, 64 threads , completes, 28 snapshots
I observed two trends:
Fewer MPI ranks → failure occurs later. 8 ranks fails around t≈2.18e+04; 4 ranks around t≈3.18e+04; the 1-"rank" serial case completes.
More top-level cells → failure occurs much earlier. 128 cells fails at step 1840 versus 8830 at 32 cells, despite METIS partitioning cleanly at 128.
Things tried that did not prevent the failure:
links_per_tasks: 25 (default) → 200 → 800 → 1500
tasks_per_cell: default → 50, proved to be irrelevant
DomainDecomposition: repartition_type: none, again proved to be irrelevant
max_top_level_cells: 32 and 128
8 and 4 MPI ranks
Building with and without --enable-mpi-mesh-gravity (the runs are non-periodic)
Dropping -flto, -ffast-math, -funroll-loops via --disable-optimization
Possibly relevant observations - At max_top_level_cells: 32 (and 8 ranks especially), the following appeared:
check_complete: Region N is not present in partition partition_initial_partition: METIS initial partition failed, using a vectorised partition repart_edge_metis: Node N is not present after repartition repart_edge_metis: WARNING: repartition has failed, continuing with the current partition
Raising max_top_level_cells to 128 removed all of these — METIS partitioned cleanly — but the Missing link to pack task failure then occurred much earlier. So the failure does not appear to be a consequence of the degenerate partition, and increasing cell count makes it worse.
So my question is, is this a known limitation of the planetary scheme under MPI, or a genuine error in my build? Any suggestion for parameters that would let this example run under MPI would be very welcome — the serial binary works, but the intended use case (higher-resolution impact simulations for a thesis on material strength and plasticity models) will need MPI.
I will be happy to provide full logs, timesteps.txt, task graphs, or to run any diagnostic build that would help.
With warm regards,
Proshmit Dasgupta
M.Sc. Astro and Particle Physics
Dear SWIFT team,
I am an astrophysics student at Tübingen University, working on adding material strength and plasticity models for planetary embryo collisions. I was running the stock 'DemoImpact' example simulation under swift_mpi, and it consistently aborts partway through the impact with:
The serial swift binary, built from the same source with the same configure line and run with the same ICs and parameter file, completes the full simulation without error (14,072 steps, all 28 snapshots). My work environment is as follows:
Here is how I configured:
The configure summary confirms: MPI enabled: yes, HDF5 parallel: yes, METIS/ParMETIS: yes / no, Hydro scheme: planetary, Equation of state: planetary, Planetary fixed entropy: no.
The run command used:
srun --mpi=pmix --cpu-bind=cores \ ./swift_mpi --hydro --self-gravity --threads=$SLURM_CPUS_PER_TASK \ demo_impact_n50.ymlSingle node was maintained throughout. Additional commands used to run: OMPI_MCA_pml=ob1, OMPI_MCA_btl="self,vader".
I made some changes the yaml parameter file, notably:
Some of the results after simulations:
I observed two trends:
Fewer MPI ranks → failure occurs later. 8 ranks fails around t≈2.18e+04; 4 ranks around t≈3.18e+04; the 1-"rank" serial case completes.
More top-level cells → failure occurs much earlier. 128 cells fails at step 1840 versus 8830 at 32 cells, despite METIS partitioning cleanly at 128.
Things tried that did not prevent the failure:
links_per_tasks: 25 (default) → 200 → 800 → 1500
tasks_per_cell: default → 50, proved to be irrelevant
DomainDecomposition: repartition_type: none, again proved to be irrelevant
max_top_level_cells: 32 and 128
8 and 4 MPI ranks
Building with and without --enable-mpi-mesh-gravity (the runs are non-periodic)
Dropping -flto, -ffast-math, -funroll-loops via --disable-optimization
Possibly relevant observations - At max_top_level_cells: 32 (and 8 ranks especially), the following appeared:
check_complete: Region N is not present in partition partition_initial_partition: METIS initial partition failed, using a vectorised partition repart_edge_metis: Node N is not present after repartition repart_edge_metis: WARNING: repartition has failed, continuing with the current partitionRaising max_top_level_cells to 128 removed all of these — METIS partitioned cleanly — but the Missing link to pack task failure then occurred much earlier. So the failure does not appear to be a consequence of the degenerate partition, and increasing cell count makes it worse.
So my question is, is this a known limitation of the planetary scheme under MPI, or a genuine error in my build? Any suggestion for parameters that would let this example run under MPI would be very welcome — the serial binary works, but the intended use case (higher-resolution impact simulations for a thesis on material strength and plasticity models) will need MPI.
I will be happy to provide full logs, timesteps.txt, task graphs, or to run any diagnostic build that would help.
With warm regards,
Proshmit Dasgupta
M.Sc. Astro and Particle Physics