docs: add Ansys Mechanical (MAPDL) best-practices Readme

[nicolaven]

Summary

apps/AnsysMechanical/ shipped only an sbatch and the DynamoDB recorder — unlike CFX/Fluent/Abaqus there was no best-practices doc. This adds apps/AnsysMechanical/Readme.md, written from running the full official V26 Cluster benchmark set (all six models, DMP, Intel MPI + EFA) on AWS ParallelCluster 3.16 / Amazon Linux 2023.

What it covers

• AL2023 required libraries: MAPDL's ansys.e links a legacy X11/Motif stack absent from AL2023. Table of every missing library and where Ansys bundles it, a build recipe for libpng12 (not bundled, not in any repo, cannot be faked with a libpng16 symlink due to versioned symbols), and three dependency traps discovered the hard way.
• Scheduler execution: run as a normal user, never root (-ssh needs the inter-node keypair; Intel MPI refuses root); HOME must be set; module guard for non-login shells; dnf vs yum; -machines carries the core count (no -np in DMP).
• EFA: the settings block plus how to verify the efa libfabric provider is actually in use (I_MPI_DEBUG=5) — TCP fallback is silent.
• Memory-first performance guidance: in-core vs out-of-core dominates MAPDL runtime; size to the model's documented memory requirement first; adding a node raises the memory ceiling, not just cores; under-population with the Utils/flexible-cores pinning lists (and why lowering --ntasks-per-node alone is insufficient).
• Scratch/concurrency planning: out-of-core solves write ~0.5–0.8 TB scratch per job; unbounded concurrency on a shared filesystem can hang jobs and distorts benchmark timings; monitor metadata rate and bursts, not average throughput.
• Exit-code trap: fixed-iteration benchmark models exit non-zero after a successful run; success must be judged from RUN COMPLETED + Elapsed Time in the output (as Ansys's own runBench.py does). Automation gating on exit code silently drops valid results — including the recorder wiring in the current sbatch.

Testing

Every item was hit and verified while running the official V26 Cluster set (V26direct-4/5/6, V26iter-4/5/6) across hpc8a/hpc7a/hpc6a/hpc6id/c8i on ParallelCluster 3.16, AL2023, FSx for Lustre PERSISTENT_2.

Not included

Performance numbers — the Performance section is intentionally TBC; measured scaling/instance-comparison charts to follow separately.

[nicolaven]

Thanks for putting this together — the README fills a real gap, and changing the official fixed-iteration benchmark handling away from raw MAPDL exit-code gating is directionally correct. I validated the current tip (8a48db6) rather than relying only on static review.

bash -n, git diff --check, and all CodeQL checks pass. However, the new solve-verdict and NVMe stage-out paths have correctness/data-loss cases that make this revision unsafe to merge.

Blocking: solve verdict can invert success and failure

The current classifier is whitespace-sensitive and removes individual context lines rather than classifying complete MAPDL error blocks:

unexpected_errs=$(grep -A2 '^ \*\*\* ERROR' "$output_file" \
  | grep -v 'iterations exceeds' | grep -c '^ \*\*\* ERROR' || true)

I ran the exact PR logic against representative fixtures. Results:

normal success                     solver_rc=0   solve_ok=1 final_rc=0   (correct)
expected split iteration stop      solver_rc=1   solve_ok=0 final_rc=1   (wrong: valid benchmark rejected)
column-zero genuine error          solver_rc=2   solve_ok=1 final_rc=0   (wrong: real failure accepted)
truncated output but rc zero       solver_rc=0   solve_ok=0 final_rc=0   (wrong: unverified job reports success)
arbitrary nonzero plus markers     solver_rc=42  solve_ok=1 final_rc=0   (wrong: unrelated failure normalized)
zero elapsed accepted              solver_rc=0   solve_ok=1 final_rc=0   (wrong: invalid result accepted)

Two underlying issues:

1. ^ *** ERROR requires exactly one leading space, so a column-zero or differently-indented real error is invisible. For the normal two-line fixed-iteration block, removing the detail line still leaves and counts the header, rejecting the run.
2. The final expression forces any marker-verified nonzero exit to zero, but leaves an unverified rc=0 as zero:

[ "$solve_ok" -eq 1 ] && exit 0 || exit "$_rec_rc"

Please parse complete, indentation-insensitive MAPDL error blocks and whitelist only the exact known benign fixed-iteration/user-request signature. Normalize a nonzero status only when that exact signature, final RUN COMPLETED, and a positive elapsed value are all present. If output verification fails while MAPDL returned 0, synthesize a nonzero job status rather than reporting scheduler success.

Blocking: failed stage-out can delete the only solver log

The shared-filesystem copy result is discarded and local scratch is then deleted unconditionally:

cp -f "$output_file" "$shared_dir/" 2>/dev/null
srun ... rm -rf "$workdir"

If the shared filesystem is full, unavailable, or denies the write, this destroys the only MAPDL output. The job may still exit zero and even record a result from the already-parsed local log. Make successful durable stage-out a prerequisite for cleanup and success. On copy failure, retain /scratch, emit the node/path needed for recovery, and fail the job.

Recorder validity/replay

The zero-argument parser scans only output.log and *.out, while the launcher now writes output-${SLURM_JOB_ID}.log. A replay fixture against that filename emitted no time_to_solution_seconds. It also accepts and records explicit 0 and -5, and automatically derived timing is not gated on RUN COMPLETED/error validation.

Please:

• include the launcher's output-*.log in replay discovery,
• require finite timing > 0, including explicit values,
• require verified completion/no unexpected errors for timing derived automatically from solver output,
• keep explicit valid values authoritative and DynamoDB failure non-fatal (both work correctly today).

Documentation/operational corrections

These should be addressed in the same update:

• A temporary empty HOME=$(mktemp -d) cannot contain the ~/.ssh identity that -ssh requires. Recover the account's real home or fail with an actionable message.
• The libpng 1.2.59 recipe downloads and builds an EOL library without checksum/signature verification. Add integrity verification, prerequisites/permissions, and clearly state the EOL/security/support trade-off.
• The library paths are hard-coded for v261/polyflow26.1.0, while the text claims 2023+. Scope the tested recipe to 2026 R1 or provide verified per-release mappings.
• With I_MPI_OFI_PROVIDER=efa, EFA selection is fail-closed; the README's silent-TCP-fallback warning describes the default when the provider filter is absent. Please distinguish those cases.
• The quoted root-refusal diagnostic is associated with Open MPI, not Intel MPI. Keep the non-root recommendation, but ground it in least privilege and the configured SSH identity.
• Qualify absolute performance claims ("best", "fastest", "extra memory buys nothing") to the configurations actually tested until the TBC performance results are published.
• Document/require homogeneous divisible task placement or derive per-host counts from Slurm; README floor division and launcher ceiling division disagree for non-divisible allocations.

Given how subtle the verdict boundary is, please add fixture-driven shell tests for: normal completion, exact benign fixed-iteration termination, genuine errors with varied indentation, truncated output with rc=0, unrelated nonzero status, zero/negative elapsed time, recorder replay, and failed stage-out. These tests do not need Slurm or a licensed solver.

I’m holding approval/merge on the three behavioral blockers: verdict/status normalization, destructive stage-out, and invalid recorder timing/replay. Happy to re-review immediately after an update.

[nicolaven]

Thanks — this was a good catch, and validating against fixtures rather than reading the diff was the right call. All three blockers were real. Fixed in 4b0a662.

I reproduced your exact expression against the fixtures first, to be sure I was fixing the reported behaviour and not my reading of it. It confirms all six of your rows, plus two more I hadn't considered:

fixture	rc	old verdict	new verdict
normal completion	0	ok	ok
fixed-iteration stop	1	rejected (valid benchmark discarded)	accepted, rc→0
column-zero real error	2	accepted	failed, rc=2
deep-indent real error	2	accepted	failed, rc=2
truncated output	0	reported success	failed, rc=3
clean output, unrelated rc	42	masked to 0	failed, rc=42
zero elapsed	0	accepted	failed, rc=3
negative elapsed	0	reported success	failed, rc=3

The last two are the ones I'd missed: -5.000 fails the *[!0-9.]* character test, so solve_ok became 0, and then [ solve_ok -eq 1 ] && exit 0 || exit $_rec_rc exited with rc=0 anyway. Same shape of bug as the truncated case, from the other direction.

Verdict / status normalisation

Moved into lib/mapdl-verdict.sh, sourced by the sbatch, because you were right that it isn't testable inline.

It parses complete error blocks (marker matched anywhere on the line, block ends at a blank line, the next marker, or a box rule), and classifies each block whole rather than filtering lines — so removing a matching detail line can no longer leave its header behind to be counted. Only the exact iteration-limit/user-request signature is whitelisted. solve_ok requires RUN COMPLETED and elapsed > 0 and zero unexpected blocks. Status normalisation:

verified	solver rc	benign block	job exit
yes	0	—	0
yes	≠0	yes	0
yes	≠0	no	rc (unexplained, not masked)
no	≠0	—	rc
no	0	—	3 (synthesised)

One extra fail-open case surfaced while testing: an error block with an empty body printed a blank line, which grep -c . did not count, so an unclassifiable error was invisible. Blocks are now emitted with an ERR: prefix and counted on that.

Stage-out

mapdl_stage_out verifies the destination exists, is non-empty, and matches the source byte count; the same-path case (SCRATCH_MODE=shared) short-circuits instead of copying a file onto itself. Reclamation is gated on its return value. On failure the job retains /scratch, prints the node and absolute path, points at --run-dir for replay, and fails. Tested against a read-only destination, an empty source and a missing source (the read-only case skips when running as root, since permissions wouldn't apply).

Recorder

Discovery includes output-<jobid>.log; timing must be finite and > 0 whether explicit or derived; derived timing additionally requires RUN COMPLETED and no unexpected error blocks. Explicit valid values remain authoritative and DynamoDB failure remains non-fatal, as you noted both already worked. The recorder stays self-contained by design, so it carries a compact copy of the block classifier with a pointer to the authoritative one; both are covered by tests.

Two incidental fixes found by the new tests: --dry-run was writing its banner to stdout, so the output couldn't be piped to jq/json.load (banner moved to stderr, stdout is now pure JSON); and the put-item result was read via a separate $?, now tested directly.

Documentation and operational items

All seven addressed. Two worth calling out:

• libpng checksum. I didn't want to publish a digest I couldn't stand behind, so the recipe pins sha256:4bd4b5ce…1e9e, which I verified matches SourceForge's own published file metadata for libpng-1.2.59.tar.gz and two independent downloads. The GPG signature route is documented alongside it and recommended over the pinned hash, since it verifies authorship rather than matching a hash copied into a document. The EOL trade-off is stated explicitly, with scoping guidance (own prefix, LD_LIBRARY_PATH only, never ldconfig) and a container alternative for policies that forbid EOL libraries.
• Division disagreement. Rather than making the README match the launcher, both now derive per-host counts from SLURM_TASKS_PER_NODE, and the job aborts if the -machines total doesn't equal SLURM_NPROCS. Floor division drops ranks and ceiling division over-draws licence tokens; neither is a safe default. The README also documents the "require divisible placement and fail otherwise" alternative for anyone who prefers the simple form. Both expanders are tested against 64(x2), 43,42, 64(x2),32 and malformed input.

The remaining items: HOME is recovered via getent passwd with a temporary HOME only for single-node runs; library paths scoped to the tested 2026 R1 with a find recipe for other releases; EFA fail-closed vs. silent-fallback separated into the two distinct cases; root refusal grounded in the SSH identity and least privilege; absolute performance claims qualified to the configurations tested, pending the TBC section.

Tests

apps/AnsysMechanical/tests/run-tests.sh — 63 cases, no Slurm, no licensed solver, no credentials, no network:

• 28 verdict and stage-out, covering all eight fixtures across rc 0/1/2/42
• 17 recorder replay and timing validity
• 18 end-to-end runs of the assembled AnsysMechanical.sbatch with scontrol/srun/mpirun/module/curl/mapdl stubbed

That last suite also closes a gap I should have flagged myself in the previous revision: the launcher had only ever been bash -n'd, never executed. It now runs for real in CI-able form and asserts the job's exit status per fixture, that failed runs record nothing, that -machines matches the allocation, and that a missing verdict library aborts before any solver time is spent.

Performance section remains TBC at the data owner's request. Ready for re-review.