Refactor/adforce eval

README.md

@@ -80,9 +80,9 @@ Key scripts:
 python -m adbo.exp_1d --test True --exp_name test
 ```
 
-### `comp`
+### `adforce.eval`
 
-Validates the historical ADCIRC surge simulations (the SurgeNet training set, published on Hugging Face) against de-tided NOAA CO-OPS tide-gauge observations, scoring peak skill, time-series skill, and peak timing, with permutation/negative-control null tests and sensitivity checks. See [`comp/README.md`](https://github.com/sdat2/worstsurge/blob/main/comp/README.md) for the validation methodology, negative controls, and results.
+Validates the historical ADCIRC surge simulations (the SurgeNet training set, published on Hugging Face) against de-tided NOAA CO-OPS tide-gauge observations, scoring peak skill, time-series skill, and peak timing, with permutation/negative-control null tests and sensitivity checks. See [`adforce/eval/README.md`](https://github.com/sdat2/worstsurge/blob/main/adforce/eval/README.md) for the validation methodology, negative controls, and results. (Formerly the top-level `comp/` package; caches remain under `data/comp/`.)
 
 ### `worst`
 
@@ -104,8 +104,8 @@ A motivating toy example (`toy_example.py`): a single small Keras ReLU MLP is tr
 ```
 adbo/        Bayesian optimization loop (trieste)
 adforce/     ADCIRC wrapper and forcing utilities
+adforce/eval/  Historical surge validation against NOAA tide gauges (see adforce/eval/README.md)
 cle15/       CLE15 wind profile implementations (pure-Python & Numba)
-comp/        Historical surge validation against NOAA tide gauges (see comp/README.md)
 data/        Key input data (fort.22.nc, IBTrACS, ERA5, CMIP6, etc.)
 docs/        ReadTheDocs source
 img/         Key figures
@@ -152,7 +152,7 @@ dependency groups are available as extras:
 pip install -e .[bo]    # trieste/TensorFlow Bayesian optimization (adbo, worst)
 pip install -e .[cmip]  # intake/xESMF CMIP6 download + regridding (tcpips)
 pip install -e .[mpi]   # dask_mpi/mpi4py HPC parallelism (tcpips)
-pip install -e .[comp]  # utide/huggingface_hub tide-gauge validation (comp)
+pip install -e .[eval]  # utide/huggingface_hub tide-gauge validation (adforce.eval; [comp] is a deprecated alias)
 pip install -e .[all]   # everything
 ```
 

REPRODUCE.md

@@ -78,7 +78,7 @@ Bayesian optimization (`adbo.exp*`) → EVT with an upper bound (`worst`)**
 | 6. ADCIRC forcing | `python -m adforce.wrap` | ARCHER2 only (needs `padcirc`/`adcprep` binaries, see `adforce/config/files/`) | `wrap.slurm`, `n01_wrap.slurm` |
 | 7. Bayesian optimization | `python -m adbo.exp_1d/_2d/_3d ...` | ARCHER2 only (drives ADCIRC via `adforce`) | `[bo]` extra; `bo_*.slurm`, `exp.slurm`, per-city scripts, `n01_bo_first..fourth` |
 | 8. EVT with upper bound | `python -m worst.vary_samples_ns`, `worst.vary_noise`, `worst.vary_nonstationary`, `worst.sigma_robustness`, `worst.ns_evt_figs` | local (CPU; `[bo]` extra for TensorFlow fits) | synthetic experiments — no ADCIRC needed |
-| Validation (side chain) | `python -m comp.validate` / `comp.nulltest` / `comp.detide_sensitivity` / `comp.sensitivity` | local (needs internet + `[comp]` extra) | Hugging Face archived runs vs NOAA gauges |
+| Validation (side chain) | `python -m adforce.eval.validate` / `adforce.eval.nulltest` / `adforce.eval.detide_sensitivity` / `adforce.eval.sensitivity` | local (needs internet + `[eval]` extra) | Hugging Face archived runs vs NOAA gauges |
 | Observations (side chain, thesis) | `python -m tcpips.ibtracs` | local or ARCHER2 (`ibtracs.slurm`) | IBTrACS vs ERA5-derived PI/PS |
 
 Stages 6–7 require ARCHER2 (or another HPC machine with a ported
@@ -107,17 +107,17 @@ Some plotting entry points are driven from commented-out calls in the module
 | **Fig. 10** — AEP with known vs unknown upper bound, GEV resampling (Nr=600) (`worst:fig:evt`) | `python -m worst.vary_samples_ns` (hydra config `worst/config/`) | synthetic GEV draws (TensorFlow fit) | `img/worst/evt_fig_tens_*.pdf` | — (local) |
 | **Fig. 11** — effect of upper-bound uncertainty σ_ẑ* on return-value estimates (`worst:fig:vary_z_star_sigma`) | `python -m worst.vary_noise` | synthetic GEV draws; cache `data/worst/vary_z_star_*.nc` | `img/worst/vary_z_star_sigma.pdf` | — (local) |
 
-### Appendix A — historical surge validation (`comp`; see [`comp/README.md`](comp/README.md))
+### Appendix A — historical surge validation (`adforce.eval`; see [`adforce/eval/README.md`](adforce/eval/README.md))
 
 | Paper item | Command | Key inputs | Output | SLURM job |
 | --- | --- | --- | --- | --- |
-| **Table 3** — de-tiding-method sensitivity (`tab:detide-robustness`) | `python -m comp.detide_sensitivity` | de-tided gauge cache (`data/comp/ts_cache/`) | `<thesis>/paper/comp_detide_table.tex` (`\input` by the appendix) | — (local) |
-| **Table 4** — per-storm skill vs de-tided NOAA gauges (`tab:gauge-validation`) | `python -m comp.validate` | Hugging Face [`sdat2/surgenet-train`](https://huggingface.co/datasets/sdat2/surgenet-train); NOAA CO-OPS gauges (de-tided with `utide`) | `<thesis>/paper/comp_val_table.tex` | — (local) |
-| **Fig. 12** — peak-skill scatter (`fig:gauge-validation-scatter`) | `python -m comp.validate` (full 14-storm sweep) | as above | `img/comp/val_scatter.png` · `<thesis>/img/comp_val_scatter.pdf` | — (local) |
-| **Fig. 13** — example de-tided surge time series (`fig:gauge-validation-examples`) | `python -m comp.validate` (or `--examples-only` from cache) | as above (cache: `data/comp/ts_cache/`) | `img/comp/val_examples.png` · `<thesis>/img/comp_val_examples.pdf` | — (local) |
-| **Fig. 14** — negative-control null tests (`fig:gauge-validation-null`) | `python -m comp.nulltest` | `data/comp/out/val_summary.csv` (+ netCDFs for the lag null) | `<thesis>/img/comp_val_nulltests.pdf` · `img/comp/val_nulltests.png` | — (local) |
-| Summary metrics (support; pinned by `tests/test_comp.py`) | `python -m comp.validate` | as above | `data/comp/out/val_summary.csv` | — (local) |
-| Threshold/node-selection sensitivity (text numbers) | `python -m comp.sensitivity` | `val_summary.csv` | console report (headline r stability) | — (local) |
+| **Table 3** — de-tiding-method sensitivity (`tab:detide-robustness`) | `python -m adforce.eval.detide_sensitivity` | de-tided gauge cache (`data/comp/ts_cache/`) | `<thesis>/paper/comp_detide_table.tex` (`\input` by the appendix) | — (local) |
+| **Table 4** — per-storm skill vs de-tided NOAA gauges (`tab:gauge-validation`) | `python -m adforce.eval.validate` | Hugging Face [`sdat2/surgenet-train`](https://huggingface.co/datasets/sdat2/surgenet-train); NOAA CO-OPS gauges (de-tided with `utide`) | `<thesis>/paper/comp_val_table.tex` | — (local) |
+| **Fig. 12** — peak-skill scatter (`fig:gauge-validation-scatter`) | `python -m adforce.eval.validate` (full 14-storm sweep) | as above | `img/comp/val_scatter.png` · `<thesis>/img/comp_val_scatter.pdf` | — (local) |
+| **Fig. 13** — example de-tided surge time series (`fig:gauge-validation-examples`) | `python -m adforce.eval.validate` (or `validate.examples_only=true` from cache) | as above (cache: `data/comp/ts_cache/`) | `img/comp/val_examples.png` · `<thesis>/img/comp_val_examples.pdf` | — (local) |
+| **Fig. 14** — negative-control null tests (`fig:gauge-validation-null`) | `python -m adforce.eval.nulltest` | `data/comp/out/val_summary.csv` (+ netCDFs for the lag null) | `<thesis>/img/comp_val_nulltests.pdf` · `img/comp/val_nulltests.png` | — (local) |
+| Summary metrics (support; pinned by `tests/test_eval.py`) | `python -m adforce.eval.validate` | as above | `data/comp/out/val_summary.csv` | — (local) |
+| Threshold/node-selection sensitivity (text numbers) | `python -m adforce.eval.sensitivity` | `val_summary.csv` | console report (headline r stability) | — (local) |
 
 ### Appendix B — non-stationary EVT
 

comp/README.md → adforce/eval/README.md

@@ -1,4 +1,8 @@
-# comp — historical surge validation against tide gauges
+# adforce.eval — ADCIRC configuration evaluation against tide gauges
+
+(Formerly the top-level `comp/` package; on-disk caches stay under
+`data/comp/` because the raw CO-OPS responses are not reproducibly
+re-downloadable and the utide fits are expensive.)
 
 Validates the historical ADCIRC storm-surge simulations (the SurgeNet training set,
 228 IBTrACS North-Atlantic landfalling TCs on the EC95d mesh, published on Hugging
@@ -19,7 +23,7 @@ For each storm:
    NOAA CO-OPS water-level gauge in the storm's region box — NW Gulf (Texas → Florida
    panhandle; the New Orleans and Galveston study regions) for Gulf storms, or the
    Florida peninsula (Key West → Fernandina Beach; the Miami study region) for the
-   Atlantic/Florida storms (`FLORIDA_STORMS` in `comp/constants.py`);
+   Atlantic/Florida storms (`FLORIDA_STORMS` in [`constants.py`](constants.py));
 3. de-tide the gauge record with a robust [`utide`](https://github.com/wesleybowman/UTide)
    harmonic fit on the storm's calendar year (falls back to CO-OPS `predictions`);
 4. score skill on three axes:
@@ -35,10 +39,10 @@ For each storm:
 ## Run
 
 ```bash
-python -m comp.validate                       # full 19-storm sweep (Gulf + Florida)
-python -m comp.validate --storms "Ida 2021"   # one storm (skips example panels + table)
-python -m comp.validate --examples-only       # just the example figure, from cache (fast)
-python -m comp.validate --examples-only --refresh-cache   # recompute the cached series first
+python -m adforce.eval.validate                       # full 19-storm sweep (Gulf + Florida)
+python -m adforce.eval.validate 'storms=["Ida 2021"]'   # one storm (skips example panels + table)
+python -m adforce.eval.validate validate.examples_only=true   # example figure, from cache (fast)
+python -m adforce.eval.validate validate.examples_only=true validate.refresh=true   # recompute first
 ```
 
 A full sweep regenerates **everything the paper uses**, in one step, so the figures, the
@@ -54,8 +58,8 @@ table and the prose cannot drift apart:
 The slow step is the per-gauge `utide` de-tiding. A full sweep **caches** each storm's
 de-tided `(sim, obs)` series as Parquet under `data/comp/ts_cache/` (write-through), keyed by
 the node-selection + de-tiding parameters so the cache self-invalidates if any of those change.
-`--examples-only` then re-renders `comp_val_examples.pdf` from that cache in seconds (vs minutes)
-— use it to iterate on the figure's layout without re-detiding; `--refresh-cache` forces a recompute.
+`validate.examples_only=true` then re-renders `comp_val_examples.pdf` from that cache in seconds (vs minutes)
+— use it to iterate on the figure's layout without re-detiding; `validate.refresh=true` forces a recompute.
 
 The thesis tree is located by searching for `paper/appendix.tex`; override with the
 `WORSTSURGE_PAPER_ROOT` env var. Downloads/caches live under `data/comp/` (git-ignored).
@@ -65,26 +69,26 @@ The thesis tree is located by searching for `paper/appendix.tex`; override with
 `val_summary.csv` is enough to falsify the result without re-running ADCIRC:
 
 ```bash
-python -m comp.nulltest            # permutation, cross-storm, and temporal-lag nulls
-python -m comp.nulltest --no-lag   # peak-level nulls only (no netCDF)
-python -m comp.sensitivity         # threshold + node-selection robustness
-python -m comp.sensitivity --no-node
+python -m adforce.eval.nulltest            # permutation, cross-storm, and temporal-lag nulls
+python -m adforce.eval.nulltest nulltest.lag=false   # peak-level nulls only (no netCDF)
+python -m adforce.eval.sensitivity         # threshold + node-selection robustness
+python -m adforce.eval.sensitivity sensitivity.node=false
 ```
 
-`comp.nulltest` writes `img/comp_val_nulltests.pdf` and reports: a label-permutation null
+`adforce.eval.nulltest` writes `img/comp_val_nulltests.pdf` and reports: a label-permutation null
 (observed r=0.89 vs null max 0.31 over 5000 shuffles, p<1e-3); a within-storm permutation
 (observed spatial r=0.86 vs null max 0.61) showing real *spatial* skill; a cross-storm
 same-gauge null (collapses to r~0.1–0.2); and a temporal-lag curve (time-series r peaks
-sharply at lag 0 and decays to zero within two days). `comp.sensitivity` shows the headline
+sharply at lag 0 and decays to zero within two days). `adforce.eval.sensitivity` shows the headline
 is stable across the clean-filter cut-offs (r 0.81–0.89) and the node-selection knobs
 (r changes <0.001 with the wet-depth threshold).
 
-Unit + regression tests live in [`../tests/test_comp.py`](../tests/test_comp.py) (skill
+Unit + regression tests live in [`../../tests/test_eval.py`](../../tests/test_eval.py) (skill
 metrics, time-series alignment, the valid/clean gating, the LaTeX table, and pinned
 headline numbers + negative-control separations):
 
 ```bash
-python -m pytest tests/test_comp.py -o addopts=""
+python -m pytest tests/test_eval.py -o addopts=""
 ```
 
 ## Configuration
@@ -112,6 +116,57 @@ The slight low bias is consistent with the omitted wave setup/runup and medium m
 resolution (datum and node-sampling effects checked and found minor); over-predictions
 concentrate at shallow semi-enclosed bay/pass gauges during direct landfalls.
 
+## Comparing adforce configurations (general framework)
+
+Beyond the historical HF-archive validation above, the submodule compares
+**any** adforce configurations (resolution low/mid/high, tides on/off, SWAN)
+against gauges and against each other, driven entirely by YAML under
+[`config/`](config/):
+
+- **matrix** ([`config/matrix/`](config/matrix/)) — the comparison grid:
+  cartesian `axes` over wrap-config paths (`adcirc.resolution.value`,
+  `adcirc.tide.value`, ...) minus `exclude`, plus explicit `cells`, with a
+  designated `baseline` for model-vs-model.
+- **launch** (`python -m adforce.eval.launch`) — expands the matrix into
+  per-(cell, storm) runs under `<runs_root>/<study>/<cell>/<slug>/`, routes
+  every historical run through the training driver's per-storm input
+  generation (correct tidal windows — never the Katrina-pinned static
+  decks), skips completed runs, refuses FOREIGN directories, and records a
+  full-config sha256 per run in `eval_manifest.json`. `dry_run=true` is the
+  default: it prints the plan table and fires nothing. `controls=true` adds
+  the tide-only runs of the tide-surge-interaction triple.
+- **extract** (`python -m adforce.eval.extract`) — remote-side reducer:
+  `fort.63.nc` (5–8 GB) → `gauge_ts.parquet` per run (long format
+  `storm, sid, gauge, time, zeta`), same node selection as the validation.
+- **harvest** (`python -m adforce.eval.harvest`) — rsyncs the minimal
+  artifact set (`config.yaml`, `gauge_ts.parquet`, `fort.61.nc`,
+  `maxele.63.nc`, `slurm.out`, manifest) to the laptop.
+- **pairs** (`python -m adforce.eval.pairs`) — model-vs-model:
+  `resolution_bias_table` (reproduces `rerun/results/resolution_bias.csv`
+  exactly) and `tide_surge_interaction`
+  (`peak(both) − peak(storm) − peak(tide)`; reproduces the published
+  low-res rows of `rerun/results/tide_surge_interaction.csv` 683/683).
+  Both reproductions are pinned by tests.
+
+Example — low vs mid × tide on/off for two storms:
+
+```bash
+# remote (GCP spot VM): preview, then launch
+python -m adforce.eval.launch study=kat-ida matrix=res_x_tide \
+    'storms=["Katrina 2005","Ida 2021"]'                    # dry run (default)
+python -m adforce.eval.launch study=kat-ida matrix=res_x_tide \
+    'storms=["Katrina 2005","Ida 2021"]' dry_run=false controls=true
+# laptop:
+python -m adforce.eval.harvest remote=gcp-vm:/work/exp/eval study=kat-ida dry_run=false
+python -m adforce.eval.pairs action=interaction res=mid \
+    tide_series=... both_series=... out=data/comp/out/kat-ida_tsi.csv
+```
+
+Caveat (tide-on scoring): comparing a tide-on run against raw gauge water
+level needs the datum/steric offset handled (pre-storm-mean alignment) and
+skew surge as the headline metric; that obs-side path is staged follow-up
+work — do not publish tide-on skill numbers before it lands.
+
 ## Dependencies
 
 `huggingface_hub`, `utide`, `xarray`, `scipy`, `pandas`, `pyarrow`, `requests`, `matplotlib`,

comp/__init__.py → adforce/eval/__init__.py

@@ -5,7 +5,7 @@
 
 Entry point::
 
-    python -m comp.validate
+    python -m adforce.eval.validate
 """
 
 from .validate import run, validate_storm, metrics

adforce/eval/_cli.py

@@ -0,0 +1,31 @@
+"""Shared CLI helpers for the adforce.eval hydra entry points.
+
+The comp/-era argparse flags were replaced by hydra overrides when the module
+moved to adforce/eval/. Stale invocations (old docs, shell history, thesis
+Makefiles) should fail loudly with the exact translation rather than be
+half-parsed by hydra -- so each entry point calls :func:`reject_legacy_flags`
+with its flag map before handing argv to hydra. Only known legacy flags are
+rejected; hydra's own ``--help``/``--cfg``/etc. pass through untouched.
+"""
+
+import sys
+from typing import Dict
+
+
+def reject_legacy_flags(mapping: Dict[str, str], module: str) -> None:
+    """Exit with the hydra translation if an old argparse flag is on argv.
+
+    Args:
+        mapping: legacy flag -> replacement hydra override (shown verbatim).
+        module: dotted module path for the usage line, e.g. "adforce.eval.validate".
+    """
+    hits = [a for a in sys.argv[1:] if a.split("=")[0] in mapping]
+    if not hits:
+        return
+    lines = [
+        f"{module}: argparse flags were replaced by hydra overrides "
+        "when comp/ moved to adforce/eval/. Translation:"
+    ]
+    lines += [f"  {old:<18} ->  {new}" for old, new in mapping.items()]
+    lines.append(f"e.g.  python -m {module} " + mapping[hits[0].split("=")[0]])
+    raise SystemExit("\n".join(lines))