[cailmdaley]

ShapePipe now runs on every candide node I tested, with no node excludes. The tile store moves off one user's NFS directory onto each job's own /scratch. Catalogues are bit-identical to the NFS-store run.

One image-sim tile ran on each node, pinned there. SKiLLS 1z2z_grid_1 tile 233.293 through sp run, launched from a plain login shell. The reference is the same tile on n24 with the NFS store. The comparison covers all 74 columns of final_cat, 3120 rows.

node	CPU	was excluded for	state dir	final_cat vs reference
n09	Intel	no internet (reachable now)	/n23data1	bit-identical
n17	Intel	"mounts"	/n17data, its own disk	bit-identical
n23	Intel	Slurm job failures	/n23data1, its own disk	prepare and exposure stages pass; tile queued
n34	AMD	/tmp filled	/n23data1	bit-identical
n35	AMD	/tmp filled	/n23data1	bit-identical
n36	AMD	unstated (GPU node)	/n23data1	bit-identical

n17 and n23 ran the working-directory change, and n17 passed the prepare stage again on the final commit, which only scopes that change to candide. The other four ran the store change from just before it, which doesn't affect jobs on nodes that don't own their state dir.

The n17 exclude was hiding a path bug. /nXXdata works on every node. /automnt/nXXdata doesn't exist on the node that owns the disk. Snakemake resolves its working directory to the /automnt form and hands that to jobs in three places:

• the Slurm log path, which kills the job with exit 0:53
• sbatch -D, which drops the job into /tmp
• a .snakemake/tmp.* wait-for file, which makes the job wait 60 s and exit 0 having done nothing

The Snakefile now spells the working directory in the plain form before any of those paths exist, and the profile's relative slurm-logdir joins onto it. This patches os.getcwd and a private Snakemake attribute, so a Snakemake upgrade could break it. An admin symlink /automnt/nXXdata -> /nXXdata on each disk-owning node would make it unnecessary.

The tile store is per-job intermediate data, so it belongs on per-job disk. It holds one tile's vignets and WCS for one tile_shape job. tile_store_root: /scratch/%u/%j in the machines: table names Slurm's per-job directory on the node's own disk, which Slurm deletes when the job ends, including after a hard kill. Every job parses the Snakefile on its node, so the Snakefile binds that directory to /local/scratch. tile_local() is unchanged, so finished tiles aren't invalidated, and sp no longer rewrites the profile. Anyone can now run on candide without a writable shared store. Nibi is unaffected.

The --tmp=16000 floor is cleared on candide: it selects on Slurm's TmpDisk, which is 0 on n23 even though n23 has /scratch.

space	create and delete one file
job /scratch	12–70 µs
/n23data1	300–470 µs
/n17data	~58,000 µs, with my own probes contending

Launching from a shell with a node-local XDG_CACHE_HOME now works. The profile leaves source-cache out of shared-fs-usage and the Snakefile drops the variable, as sp_validation already does.

Before merging:

• /scratch per-job directories are confirmed on n09, n17, n23, n34 and n36. n08, n18, n24 and n25 haven't reported because they're busy. Probes on n03 and n30 were cancelled by root after 2 minutes with no output, cause unknown. A node without /scratch would fail at the store setup, loudly.
• The test tile is small. A real-data tile's store is ~5.6 GB, and the smallest /scratch I measured is ~890 GB, but I haven't run real data on this store.
• workflow/README.md records the rule: no candide node exclusions without a reproducible failure of a pinned job.
• sp_validation still has its own candide excludes and would take the same working-directory fix in a follow-up.

— Claude Opus on behalf of Cail

🤖 Generated with Claude Code

https://claude.ai/code/session_01JK7o9EnraydJLhkzaVpLsM