Restore SACC-based CosmoSIS inference with tomographic conversion

Dockerfile

@@ -124,8 +124,9 @@ RUN bash -c 'set -eo pipefail; \
     cmds=$(python -m cosmosis.configure); \
     eval "$cmds"; \
     export GSL_INC=/usr/include GSL_LIB=/usr/lib/x86_64-linux-gnu; \
+    export FFTW_INCLUDE_DIR=/usr/include FFTW_LIBRARY=/usr/lib/x86_64-linux-gnu; \
     make -C shear; \
-    test -f shear/cl_to_xi_nicaea/nicaea_interface.so'
+    python -c "import ctypes; ctypes.CDLL(\"shear/cl_to_xi_nicaea/nicaea_interface.so\")"'
 
 # Install sp_validation itself (editable) into the same venv; deps are already
 # satisfied by the sync above.

cosmo_inference/README.md

@@ -1,76 +1,88 @@
-# UNIONS Cosmological Inference Pipeline
-by Lisa Goh and Sacha Guerrini, CEA Paris-Saclay
+# SACC → CosmoSIS inference
 
-This folder contains the files neccessary to run the cosmological inference pipeline on the UNIONS galaxy catalogues. 
+The shared workflow converts an analysis SACC to CosmoSIS 2pt-FITS, renders a pipeline INI from explicit template/values/priors inputs, samples under MPI, and writes a weighted JSON summary.
+It never reads a catalogue or a blind record.
+A concealed SACC stays concealed; its `blind` stamp is recorded in the input provenance.
 
-### Requirements
-Everything the pipeline needs ships in the container: nothing to install, and no
-paths to edit before a run.
+```bash
+snakemake --profile workflow/profiles/default -j 32 \
+  -s workflow/inference/Snakefile inference_fiducial --configfile run.yaml
+```
 
-[CosmoSIS](https://cosmosis.readthedocs.io/en/latest/) comes in via the
-`workflow` extra, built with MPI support. The CosmoSIS Standard Library — the
-tree of modules the `.ini` pipelines name — is built into the image at
-`/opt/cosmosis-standard-library`, with `CSL_DIR` pointing there. CosmoSIS reads
-environment variables into an `.ini`'s `[DEFAULT]` section, so the templates'
-`COSMOSIS_DIR = %(CSL_DIR)s` resolves to it. Outside the container, export
-`CSL_DIR` at a build of your own and the same templates work unchanged.
+Run Snakemake on the host, from the real checkout path.
+Its jobs run in the CI dependency image and import the launched checkout.
+The standalone entry point needs only the config below; `workflow/Snakefile` includes the same rules for composition with measurement workflows.
+`inference_fiducial` gathers all named runs, not just preparation products.
+A single run can be requested as `<output_dir>/<run>/summary.json`.
 
-CSL is pinned to the **UNIONS-WL org fork**
-([UNIONS-WL/cosmosis-standard-library](https://github.com/UNIONS-WL/cosmosis-standard-library/))
-at `b7b1552a`, the fork's main branch: Sacha's four UNIONS commits are
-reapplied on current upstream, including the scipy `lpn` fix.
+```yaml
+inference:
+  output_dir: /path/to/run/inference
+  runs:
+    mock_00001:
+      sacc: /path/to/MOCK_00001_clean_tomo.sacc
+      covariance: /path/to/current_reference_tomo.sacc  # optional replacement
+      n_bins: 4
+      template: cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_ia.ini
+      values: /path/to/mock_values.ini
+      priors: /path/to/mock_priors.ini
+      threads: 32
+      sampler: nautilus
+      sampler_settings:
+        n_live: 300
+        n_eff: 1500
+        resume: T
+      scale_cuts:
+        XI_PLUS: [12.0, 83.0]
+        XI_MINUS: [12.0, 83.0]
+      overrides:
+        pipeline:
+          debug: F
+          timing: F
+```
 
-Launch sampling under MPI (`mpiexec -n N cosmosis --mpi ...`), not `--smp`:
-CosmoSIS's shared-memory pool is unmaintained and still crashes after sampling
-completes (`Pool` has no attribute `data`, `runtime/process_pool.py`) as of
-3.25.2.
+Each run produces `data.fits`, `input.json`, `pipeline.ini`, `chain.txt`, `cosmosis.log`, and `summary.json`.
+Keep values and priors outside the code: every sampled bin needs its own m and redshift-shift entries.
+The example sampler settings are economical mock-validation settings, not a convergence recommendation for a final analysis.
+The summary uses Nautilus log weights (or equal weights for unweighted chains) and reports means, standard deviations and central 68% intervals.
+MCMC burn-in must be handled by the sampler configuration before summarization.
 
-### To Run
-The inference pipeline is orchestrated through Snakemake. On the candide
-cluster, drive it with the committed profile — see
-[`workflow/README.md`](../workflow/README.md) for the one-time
-`uv tool install` setup and the full explanation. From the repository root:
+## Inputs and covariance
 
-```bash
-snakemake --profile workflow/profiles/candide \
-    -s workflow/Snakefile \
-    inference_fiducial --configfile <run config>
-```
+Conversion writes all ξ+ pairs, then all ξ− pairs, in numeric upper-triangle order, with 1-indexed BIN1/BIN2 and one NZ column per source bin.
+The covariance follows exactly the same SACC-index permutation, including cross-pair terms.
+Integration-grid and unrelated diagnostic points are excluded.
+A covariance reference contributes only its covariance, never its means or n(z).
+Reference rows are matched by statistic, tracer pair, grid and nominal angle (or measured angle if no nominal bin is stored).
+This allows a corrected data covariance to be used on a clean mock with identical nominal bins.
 
-Off-cluster, drop `--profile` and add `-j <jobs>` instead. Each job runs
-inside the sp_validation container automatically — no `apptainer shell` or
-`apptainer exec` needed by hand.
+The direct `sacc_to_twopoint_fits` converter retains the historical mixed ξ/Cℓ file layout and single-bin bytes.
+CSL's legacy reader cannot associate two separate covariances with such a mixed file, so inference preparation writes only the requested likelihood statistic.
+The default is `statistic: xi`.
+`statistic: cell` selects a CELL_EE-only likelihood product, but the SAMPLE representation does not propagate SACC bandpower windows: a harmonic inference run must validate its window treatment separately.
 
-This will automatically execute all steps:
-1. Calculate 2PCF ($\xi_{pm}$) via `cosmo_val.py`
-2. Compute covariance matrices using CosmoCov <!--- LG: now obsolete, making way for OnCovariance cauclation instead --->
-3. Prepare CosmoSIS data (FITS) via `cosmosis_fitting.py`
-4. Run CosmoSIS inference
+For the legacy single-bin PSF-leakage pipeline, add `rho_stats` and `tau_stats` sidecar FITS paths and select `cosmosis_pipeline_A_psf.ini`.
+Both sidecars are required; RHO_STATS carries variances absent from the SACC.
+The τ covariance comes from the SACC, including τ0–τ2 correlations.
+Tomographic PSF sidecars are not supported by that legacy model.
+The old ξ-file preparation CLI `scripts/cosmosis_fitting.py` is retired; its FITS builders survive only as a frozen byte-compatibility test oracle.
 
-For standalone FITS data preparation (real-space inputs plus optional pseudo-$C_\ell$ data), you can also use the Python script directly:
+## Runtime
 
-```bash
-python scripts/cosmosis_fitting.py \
-  --cosmosis-root "catalog_version_config" \
-  --data-dir "/path/to/output/chains" \
-  --nz-file "/path/to/nz_file.txt" \
-  --out-file "/path/to/output.fits" \
-  --xi "/path/to/xi_plus.fits" "/path/to/xi_minus.fits" \
-  --cov-xi "/path/to/covariance.txt" \
-  --use-rho-tau \
-  --rho-stats "/path/to/rho_stats.fits" \
-  --tau-stats "/path/to/tau_stats.fits" \
-  --cov-tau "/path/to/cov_tau.npy" \
-  --cl-file "/path/to/pseudo_cl.fits" \
-  --cov-cl "/path/to/pseudo_cl_cov.fits"
-```
-
-You can view all available options with:
-```bash
-python scripts/cosmosis_fitting.py --help
-``` 
+CosmoSIS and the UNIONS-WL CSL fork (`b7b1552a`) ship in the CI image at `/opt/cosmosis-standard-library`.
+The renderer resolves `CSL_DIR` explicitly, and the chain rule evaluates `python -m cosmosis.configure` before launch so compiled modules can find `libcosmosis.so`.
+Run sampling as `mpiexec -n N cosmosis --mpi`, never `--smp`.
+PolyChord requires its optional sampler package; Nautilus is available in the current image.
+Use resumable settings for long runs.
 
-Ensure the pseudo-$C_\ell$ spectra (`pseudo_cl_*.fits`) and their covariance (`pseudo_cl_cov_*.fits`) produced by `cosmo_val.py` exist for the requested catalog version (or mock seed) before running the standalone command.
+Each chain rule runs on one allocated node and disables Open MPI's attempt to rediscover Slurm, because the image has no `srun` binary.
+For manual single-node launches, use the same environment settings:
 
+```bash
+export APPTAINERENV_PRTE_MCA_ras='^slurm'
+export APPTAINERENV_PRTE_MCA_plm='^slurm'
+```
 
-This is the pipeline used to derive cosmological constraints with cosmic shear data from the UNIONS v1.4 catalogue.
+Set `inference.mpi_args: "--bind-to none"` when running several chains concurrently inside the allocation.
+Never use this local-launch setting to span unallocated nodes.
+The allocation, its CPU limit, and Snakemake's `--cores` remain the resource limits.

cosmo_inference/cosmosis_config/templates/cosmosis_pipeline_A_psf.ini

@@ -107,6 +107,8 @@ file = %(COSMOSIS_DIR)s/shear/xi_sys/tau_from_rho.py
 data_file=%(FITS_FILE)s
 
 [2pt_like]
+file = %(COSMOSIS_DIR)s/likelihood/2pt/2pt_like_xi_sys.py
+add_xi_sys = T
 data_file=%(FITS_FILE)s
 gaussian_covariance=F
 covmat_name=COVMAT