Conversation
…ad FFT cache Convention, AD and dead-code fixes found in a full-codebase audit. Every change is behaviour-preserving under the default `:dft` configuration; the breaking moves are confined to `:quad` and to hand-built configurations. phi_scale was only half-implemented. `synthesis` scaled its Fourier bins by `phi_inv_scale(cfg)` while `analysis` applied a fixed `cfg.cphi`, so under `:quad` the pair was not mutually inverse — `analysis(synthesis(alm))` came back as `alm/2π` exactly. Analysis now applies `cphi*nlon/phi_inv_scale(cfg)`, which is `cphi` under `:dft`, threaded through the batch, complex-packed, planned, distributed and adjoint paths. The direct evaluators (`synthesis_point`, `synthesis_axisym`, `SH_to_lat`, `SHqst_to_point`, `SH_to_grad_point`, `SHqst_to_lat` and the PencilArray locals) applied no φ factor at all and so disagreed with the grid they sample by 2π; they now carry the same factor. An unset `phi_scale` used to fall back to a grid-type guess, which made a regular grid from the exported keyword constructor disagree by 2π with the identical grid from `create_regular_config`; unset now means `:dft`, as every constructor emits. A Wigner-d rotation through a general β couples every |m'| <= l, so a layout with mmax < lmax silently dropped the components it could not hold — 14.8 % of the field energy at lmax=8/mmax=5, 24.0 % at mmax=3, with no error. Such a rotation now raises. The two non-mixing angles (β = 0, π) still work at any mmax, so pure Z-rotations expressed as ZYZ are unaffected. The turbo transforms walked a bare `0:mmax` instead of `0:mres:mmax`, making `analysis_turbo`/`synthesis_turbo` disagree with their core counterparts by O(1) on any `mres > 1` config, and nested `@threads :static` illegally inside an outer threaded region. Both now share the core's cached m ordering and threading predicate. Three rotation pullbacks (`SH_Yrotate`, `shtns_rotation_apply_cplx`, `shtns_rotation_apply_real`) read their primal input lazily, so an in-place call or a reused buffer silently corrupted the angle gradient; each now snapshots at primal time. The `analysis`/`synthesis`/sphtor rrules declared fewer keywords than their primals, which made ChainRules skip them entirely once a caller passed `use_rfft` or `fft_scratch`. `synthesis_qst`/`analysis_qst` had no rrule at all and crashed inside FFTW under Zygote. The `SHTNSKIT_CACHE_PENCILFFTS` knob and the four `*_fft_plan_cache*` controls forwarded to a cache in the parallel extension whose only reader had no call sites, storing NamedTuple placeholders the FFT wrappers ignored. The dead cache is deleted and the controls now address the one in `src/fftutils.jl` that the serial and distributed transforms actually use. `DistributedSpectralPlan2D` no longer attaches a finalizer, since `MPI_Comm_free` is collective and garbage collection is rank-local. Also: structural `SHTConfig` fields keep their derived state consistent or refuse assignment; the exported keyword constructor validates its invariants; `spatial_view` is exported as the missing bridge in the padding API; and the regular grids' `nlat >= 2*lmax + 1` exactness threshold is documented with measured numbers. Validation: 76,848 serial assertions; 1,714 parallel-grid assertions; MPI at 4 ranks (comprehensive, extended, audit-fixes, 2d-alignment) and 2 ranks (local correctness, comm cleanup, plan preflight, transpose operand preflight, transpose SHT, dealiased transpose, plan allocation). Julia 1.11.1, one thread, no CUDA or ROCm hardware present; Pkg.test/JET/Aqua and the GPU and parity runners were not re-run. Readiness evidence refreshed accordingly: the host-transfer allowlist regenerated (685 -> 676 entries) and the audited tree digest recomputed, with this run recorded separately from the historical readiness runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dbj4TtuW11gke9JwSNs7Nz
…same slack `test_qst_transforms.jl`'s `synthesis_qst_l` ceiling and `test_plan.jl`'s `cfg`-form `synthesis!` ceiling are calibrated for a single-threaded run. The shared m-loop orchestrators start `@threads` tasks whenever threads are available, and each threaded region costs a few hundred bytes to spawn — a constant independent of problem size, not the field-sized regression these budgets exist to catch. Both went red at `-t 4` (13392 > 13000 and 2464 > 128); the `test_plan.jl` one is pre-existing on main. Uses the same `_thread_alloc_slack()` helper the four other budget files already carry. The planned `synthesis!` / `synthesis_sphtor!` ceilings are left strict: those own their scratch and spawn no tasks, so they stay at 0 regardless of thread count. Serial suite: 76,848 pass at one thread and at four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dbj4TtuW11gke9JwSNs7Nz
`_evaluator_phi_scale` returns a `Float64`, so multiplying a `Float32` point or
latitude evaluation by it promoted every result to `Float64` — breaking the
element type these functions promise their caller, and the `Dual` types AD needs
to see through. `test/parity/runtests_cpu.jl` caught it: 64 failures on
`@test all(value -> value isa T, qst_host)` in the local-evaluation parity
suite, all element-type assertions, no numeric drift.
Adds `_evaluator_phi_scale(cfg, ::Type{T})`, which narrows the scale to the
evaluator's own real type, and uses it at every site: `SHqst_to_point`,
`SH_to_grad_point`, `SH_to_lat`, `SH_to_lat_cplx`, `SHqst_to_lat`,
`synthesis_point` and `synthesis_point_cplx`. The axisym pair and the PencilArray
locals already converted explicitly and are unchanged.
Pins the contract with a new testset that asserts the returned element type for
every evaluator across `Float32`/`Float64` and both `phi_scale` modes, so the
serial suite catches this without the parity runner.
test/parity/runtests_cpu.jl now exits 0 (was 2148 passed / 64 failed; stock
origin/main is 2212/2212, so this was a regression introduced by the evaluator
phi_scale change in d9950cf, not a pre-existing failure). Serial suite 76,884
pass at one thread and at four.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dbj4TtuW11gke9JwSNs7Nz
|
Two follow-up commits.
Fixed with Validation now
Readiness evidence refreshed for the new tree, with the CPU parity results On the SHTns 3.7 parity failure reported separatelyNot a defect in that check and nothing to remove. |
Full-codebase audit fixes, ported onto current
main. Every change isbehaviour-preserving under the default
:dftconfiguration; the breaking movesare confined to
:quadand to hand-built configurations.Correctness
phi_scalewas only half-implemented.synthesisscales its Fourier bins byphi_inv_scale(cfg)whileanalysisapplied a fixedcfg.cphi, so under:quadthe pair was not mutually inverse —analysis(cfg, synthesis(cfg, alm))came back as
alm / 2πexactly. Analysis now appliescphi · nlon / phi_inv_scale(cfg)(=cphiunder:dft), threaded through thebatch, complex-packed, planned, distributed and adjoint paths.
The direct evaluators applied no φ factor at all:
synthesis_point,synthesis_point_cplx,synthesis_axisym,synthesis_axisym_l,SH_to_lat,SH_to_lat_cplx,SHqst_to_point,SH_to_grad_point,SHqst_to_latand thePencilArray local evaluations each disagreed with the grid they claim to sample
by exactly 2π under
:quad. They now carry the same factorsynthesisdoes.An unset
phi_scale(:auto) fell back togrid_type == :gauss ? nlon : nlon/2π,so a regular grid built through the exported
SHTConfig(; …)constructordisagreed by 2π with the identical grid from
create_regular_config. Everyconstructor emits
:dft, so unset now means:dft.Order-mixing rotations silently truncated at
mmax < lmax. A Wigner-drotation through a general β couples
Y_l^mto every|m'| ≤ l; components pastmmaxhad nowhere to go and were dropped without an error — measured atlmax = 8, 14.8 % of the field energy atmmax = 5and 24.0 % atmmax = 3. Such a rotation now raises.β ≡ 0(diagonal) andβ ≡ π(anti-diagonal) are exempt, so pure Z-rotations still work at any
mmax.The turbo transforms ignored
mres. Both walked a bare0:mmax, soanalysis_turbopopulated — andsynthesis_turboconsumed — columns anmres > 1transform has no storage for; the disagreement with the core pair wasO(1), not roundoff. They also nested
@threads :static, illegal inside an outerthreaded region. Both now share the core's cached m ordering and threading
predicate.
Three rotation pullbacks read coefficients the primal had overwritten.
SH_Yrotate,shtns_rotation_apply_cplxandshtns_rotation_apply_realcaptured the primal input and read it lazily, so an in-place call
(
Rlm === Qlm) or any caller reusing the buffer corrupted the angle gradient.Each now snapshots at primal time (ChainRules and Zygote both).
rrules declared fewer keywords than their primals. Passinguse_rfftorfft_scratch— even at its default — made ChainRules skip theanalysis/synthesis/sphtor rules entirely and fall through to source tracing.synthesis_qst/analysis_qsthad no rule at all and crashed inside FFTW.Dead code and API
The
SHTNSKIT_CACHE_PENCILFFTSvariable and the four*_fft_plan_cache*controls forwarded to a cache in the parallel extension whose only reader,
_get_or_plan, had no call sites — the "plans" it stored wereNamedTupleplaceholders the FFT wrappers ignored, so every knob was a no-op. The dead cache
(and the unreachable helpers around it) is deleted; the controls now address the
cache in
src/fftutils.jlthat the serial and distributed transforms actuallyuse, without requiring the extension to be loaded.
DistributedSpectralPlan2Dno longer attaches a finalizer:closefreessub-communicators via
MPI_Comm_free, which is collective, while garbagecollection is rank-local and nondeterministic. Cleanup is explicit only.
spatial_view(cfg, A)is exported — the missing bridge in the padding API.allocate_padded_spatialreturnsnlat_padded ≥ nlatrows while every transformrequires exactly
nlat, so the padded buffer could not be passed toanalysisat all.
set_batch_size!is documented as advisory metadata (the batch entrypoints take the field count from
size(fields, 3)).Structural
SHTConfigfields keep their derived state consistent or refuseassignment —
cfg.lmax = 10used to leavesize(cfg.Nlm) == (7, 7)while thetransforms indexed it as
(lmax+1, mmax+1)under@inbounds. The exportedkeyword constructor now validates the invariants the
create_*_confighelpershave always enforced.
Documentation
The regular grids' exactness threshold is documented with measured numbers:
Fejér and Clenshaw–Curtis with
nlatnodes are exact through degreenlat - 1,and analysis integrates degree
2·lmax, so they neednlat ≥ 2·lmax + 1whereGauss–Legendre needs
nlat = lmax + 1. Below that nothing warns — 7.2e-2relative round-trip error at
lmax = 8, nlat = 10.Deliberately not ported
The audit branch also rewrote
src/plan.jlto route the planned transformsthrough the shared orchestrators (fixing a table-dispatch miss and a doubled
Legendre traversal). That is a performance refactor, and
mainhas sincere-engineered the same file for alias-safety and allocation-free non-canonical
operation. Rather than risk those properties, this PR ports only the
conformance test: planned vs
cfgagreement across the tables × rfft ×Robert-form matrix. It passes on
mainas-is, confirming there is no correctnessgap — only the unrealised speedup.
Validation
test_mpi_2d_alignment.jl(wired into CI)operand preflight, transpose SHT, dealiased transpose, plan allocation
Julia 1.11.1, one thread, no CUDA or ROCm hardware.
Pkg.test(), JET, Aqua andthe GPU/parity runners were not re-run. Readiness evidence was refreshed the
way
c421d3eddoes it: the host-transfer allowlist regenerated by its owngenerator (685 → 676 entries, reflecting the deleted dead code and the new
pullback snapshots), the audited tree digest recomputed, and this run recorded in
its own block so the historical readiness figures stay labelled as historical.
🤖 Generated with Claude Code