diff --git a/README.md b/README.md index 0fd6aa9c..14125225 100755 --- a/README.md +++ b/README.md @@ -80,9 +80,9 @@ Key scripts: python -m adbo.exp_1d --test True --exp_name test ``` -### `comp` +### `adforce.eval` -Validates the historical ADCIRC surge simulations (the SurgeNet training set, published on Hugging Face) against de-tided NOAA CO-OPS tide-gauge observations, scoring peak skill, time-series skill, and peak timing, with permutation/negative-control null tests and sensitivity checks. See [`comp/README.md`](https://github.com/sdat2/worstsurge/blob/main/comp/README.md) for the validation methodology, negative controls, and results. +Validates the historical ADCIRC surge simulations (the SurgeNet training set, published on Hugging Face) against de-tided NOAA CO-OPS tide-gauge observations, scoring peak skill, time-series skill, and peak timing, with permutation/negative-control null tests and sensitivity checks. See [`adforce/eval/README.md`](https://github.com/sdat2/worstsurge/blob/main/adforce/eval/README.md) for the validation methodology, negative controls, and results. (Formerly the top-level `comp/` package; caches remain under `data/comp/`.) ### `worst` @@ -104,8 +104,8 @@ A motivating toy example (`toy_example.py`): a single small Keras ReLU MLP is tr ``` adbo/ Bayesian optimization loop (trieste) adforce/ ADCIRC wrapper and forcing utilities +adforce/eval/ Historical surge validation against NOAA tide gauges (see adforce/eval/README.md) cle15/ CLE15 wind profile implementations (pure-Python & Numba) -comp/ Historical surge validation against NOAA tide gauges (see comp/README.md) data/ Key input data (fort.22.nc, IBTrACS, ERA5, CMIP6, etc.) docs/ ReadTheDocs source img/ Key figures @@ -152,7 +152,7 @@ dependency groups are available as extras: pip install -e .[bo] # trieste/TensorFlow Bayesian optimization (adbo, worst) pip install -e .[cmip] # intake/xESMF CMIP6 download + regridding (tcpips) pip install -e .[mpi] # dask_mpi/mpi4py HPC parallelism (tcpips) -pip install -e .[comp] # utide/huggingface_hub tide-gauge validation (comp) +pip install -e .[eval] # utide/huggingface_hub tide-gauge validation (adforce.eval; [comp] is a deprecated alias) pip install -e .[all] # everything ``` diff --git a/REPRODUCE.md b/REPRODUCE.md index 47e1b96a..548a1328 100644 --- a/REPRODUCE.md +++ b/REPRODUCE.md @@ -78,7 +78,7 @@ Bayesian optimization (`adbo.exp*`) → EVT with an upper bound (`worst`)** | 6. ADCIRC forcing | `python -m adforce.wrap` | ARCHER2 only (needs `padcirc`/`adcprep` binaries, see `adforce/config/files/`) | `wrap.slurm`, `n01_wrap.slurm` | | 7. Bayesian optimization | `python -m adbo.exp_1d/_2d/_3d ...` | ARCHER2 only (drives ADCIRC via `adforce`) | `[bo]` extra; `bo_*.slurm`, `exp.slurm`, per-city scripts, `n01_bo_first..fourth` | | 8. EVT with upper bound | `python -m worst.vary_samples_ns`, `worst.vary_noise`, `worst.vary_nonstationary`, `worst.sigma_robustness`, `worst.ns_evt_figs` | local (CPU; `[bo]` extra for TensorFlow fits) | synthetic experiments — no ADCIRC needed | -| Validation (side chain) | `python -m comp.validate` / `comp.nulltest` / `comp.detide_sensitivity` / `comp.sensitivity` | local (needs internet + `[comp]` extra) | Hugging Face archived runs vs NOAA gauges | +| Validation (side chain) | `python -m adforce.eval.validate` / `adforce.eval.nulltest` / `adforce.eval.detide_sensitivity` / `adforce.eval.sensitivity` | local (needs internet + `[eval]` extra) | Hugging Face archived runs vs NOAA gauges | | Observations (side chain, thesis) | `python -m tcpips.ibtracs` | local or ARCHER2 (`ibtracs.slurm`) | IBTrACS vs ERA5-derived PI/PS | Stages 6–7 require ARCHER2 (or another HPC machine with a ported @@ -107,17 +107,17 @@ Some plotting entry points are driven from commented-out calls in the module | **Fig. 10** — AEP with known vs unknown upper bound, GEV resampling (Nr=600) (`worst:fig:evt`) | `python -m worst.vary_samples_ns` (hydra config `worst/config/`) | synthetic GEV draws (TensorFlow fit) | `img/worst/evt_fig_tens_*.pdf` | — (local) | | **Fig. 11** — effect of upper-bound uncertainty σ_ẑ* on return-value estimates (`worst:fig:vary_z_star_sigma`) | `python -m worst.vary_noise` | synthetic GEV draws; cache `data/worst/vary_z_star_*.nc` | `img/worst/vary_z_star_sigma.pdf` | — (local) | -### Appendix A — historical surge validation (`comp`; see [`comp/README.md`](comp/README.md)) +### Appendix A — historical surge validation (`adforce.eval`; see [`adforce/eval/README.md`](adforce/eval/README.md)) | Paper item | Command | Key inputs | Output | SLURM job | | --- | --- | --- | --- | --- | -| **Table 3** — de-tiding-method sensitivity (`tab:detide-robustness`) | `python -m comp.detide_sensitivity` | de-tided gauge cache (`data/comp/ts_cache/`) | `/paper/comp_detide_table.tex` (`\input` by the appendix) | — (local) | -| **Table 4** — per-storm skill vs de-tided NOAA gauges (`tab:gauge-validation`) | `python -m comp.validate` | Hugging Face [`sdat2/surgenet-train`](https://huggingface.co/datasets/sdat2/surgenet-train); NOAA CO-OPS gauges (de-tided with `utide`) | `/paper/comp_val_table.tex` | — (local) | -| **Fig. 12** — peak-skill scatter (`fig:gauge-validation-scatter`) | `python -m comp.validate` (full 14-storm sweep) | as above | `img/comp/val_scatter.png` · `/img/comp_val_scatter.pdf` | — (local) | -| **Fig. 13** — example de-tided surge time series (`fig:gauge-validation-examples`) | `python -m comp.validate` (or `--examples-only` from cache) | as above (cache: `data/comp/ts_cache/`) | `img/comp/val_examples.png` · `/img/comp_val_examples.pdf` | — (local) | -| **Fig. 14** — negative-control null tests (`fig:gauge-validation-null`) | `python -m comp.nulltest` | `data/comp/out/val_summary.csv` (+ netCDFs for the lag null) | `/img/comp_val_nulltests.pdf` · `img/comp/val_nulltests.png` | — (local) | -| Summary metrics (support; pinned by `tests/test_comp.py`) | `python -m comp.validate` | as above | `data/comp/out/val_summary.csv` | — (local) | -| Threshold/node-selection sensitivity (text numbers) | `python -m comp.sensitivity` | `val_summary.csv` | console report (headline r stability) | — (local) | +| **Table 3** — de-tiding-method sensitivity (`tab:detide-robustness`) | `python -m adforce.eval.detide_sensitivity` | de-tided gauge cache (`data/comp/ts_cache/`) | `/paper/comp_detide_table.tex` (`\input` by the appendix) | — (local) | +| **Table 4** — per-storm skill vs de-tided NOAA gauges (`tab:gauge-validation`) | `python -m adforce.eval.validate` | Hugging Face [`sdat2/surgenet-train`](https://huggingface.co/datasets/sdat2/surgenet-train); NOAA CO-OPS gauges (de-tided with `utide`) | `/paper/comp_val_table.tex` | — (local) | +| **Fig. 12** — peak-skill scatter (`fig:gauge-validation-scatter`) | `python -m adforce.eval.validate` (full 14-storm sweep) | as above | `img/comp/val_scatter.png` · `/img/comp_val_scatter.pdf` | — (local) | +| **Fig. 13** — example de-tided surge time series (`fig:gauge-validation-examples`) | `python -m adforce.eval.validate` (or `validate.examples_only=true` from cache) | as above (cache: `data/comp/ts_cache/`) | `img/comp/val_examples.png` · `/img/comp_val_examples.pdf` | — (local) | +| **Fig. 14** — negative-control null tests (`fig:gauge-validation-null`) | `python -m adforce.eval.nulltest` | `data/comp/out/val_summary.csv` (+ netCDFs for the lag null) | `/img/comp_val_nulltests.pdf` · `img/comp/val_nulltests.png` | — (local) | +| Summary metrics (support; pinned by `tests/test_eval.py`) | `python -m adforce.eval.validate` | as above | `data/comp/out/val_summary.csv` | — (local) | +| Threshold/node-selection sensitivity (text numbers) | `python -m adforce.eval.sensitivity` | `val_summary.csv` | console report (headline r stability) | — (local) | ### Appendix B — non-stationary EVT diff --git a/comp/README.md b/adforce/eval/README.md similarity index 53% rename from comp/README.md rename to adforce/eval/README.md index 7ae2f62c..8affc5af 100644 --- a/comp/README.md +++ b/adforce/eval/README.md @@ -1,4 +1,8 @@ -# comp — historical surge validation against tide gauges +# adforce.eval — ADCIRC configuration evaluation against tide gauges + +(Formerly the top-level `comp/` package; on-disk caches stay under +`data/comp/` because the raw CO-OPS responses are not reproducibly +re-downloadable and the utide fits are expensive.) Validates the historical ADCIRC storm-surge simulations (the SurgeNet training set, 228 IBTrACS North-Atlantic landfalling TCs on the EC95d mesh, published on Hugging @@ -19,7 +23,7 @@ For each storm: NOAA CO-OPS water-level gauge in the storm's region box — NW Gulf (Texas → Florida panhandle; the New Orleans and Galveston study regions) for Gulf storms, or the Florida peninsula (Key West → Fernandina Beach; the Miami study region) for the - Atlantic/Florida storms (`FLORIDA_STORMS` in `comp/constants.py`); + Atlantic/Florida storms (`FLORIDA_STORMS` in [`constants.py`](constants.py)); 3. de-tide the gauge record with a robust [`utide`](https://github.com/wesleybowman/UTide) harmonic fit on the storm's calendar year (falls back to CO-OPS `predictions`); 4. score skill on three axes: @@ -35,10 +39,10 @@ For each storm: ## Run ```bash -python -m comp.validate # full 19-storm sweep (Gulf + Florida) -python -m comp.validate --storms "Ida 2021" # one storm (skips example panels + table) -python -m comp.validate --examples-only # just the example figure, from cache (fast) -python -m comp.validate --examples-only --refresh-cache # recompute the cached series first +python -m adforce.eval.validate # full 19-storm sweep (Gulf + Florida) +python -m adforce.eval.validate 'storms=["Ida 2021"]' # one storm (skips example panels + table) +python -m adforce.eval.validate validate.examples_only=true # example figure, from cache (fast) +python -m adforce.eval.validate validate.examples_only=true validate.refresh=true # recompute first ``` A full sweep regenerates **everything the paper uses**, in one step, so the figures, the @@ -54,8 +58,8 @@ table and the prose cannot drift apart: The slow step is the per-gauge `utide` de-tiding. A full sweep **caches** each storm's de-tided `(sim, obs)` series as Parquet under `data/comp/ts_cache/` (write-through), keyed by the node-selection + de-tiding parameters so the cache self-invalidates if any of those change. -`--examples-only` then re-renders `comp_val_examples.pdf` from that cache in seconds (vs minutes) -— use it to iterate on the figure's layout without re-detiding; `--refresh-cache` forces a recompute. +`validate.examples_only=true` then re-renders `comp_val_examples.pdf` from that cache in seconds (vs minutes) +— use it to iterate on the figure's layout without re-detiding; `validate.refresh=true` forces a recompute. The thesis tree is located by searching for `paper/appendix.tex`; override with the `WORSTSURGE_PAPER_ROOT` env var. Downloads/caches live under `data/comp/` (git-ignored). @@ -65,26 +69,26 @@ The thesis tree is located by searching for `paper/appendix.tex`; override with `val_summary.csv` is enough to falsify the result without re-running ADCIRC: ```bash -python -m comp.nulltest # permutation, cross-storm, and temporal-lag nulls -python -m comp.nulltest --no-lag # peak-level nulls only (no netCDF) -python -m comp.sensitivity # threshold + node-selection robustness -python -m comp.sensitivity --no-node +python -m adforce.eval.nulltest # permutation, cross-storm, and temporal-lag nulls +python -m adforce.eval.nulltest nulltest.lag=false # peak-level nulls only (no netCDF) +python -m adforce.eval.sensitivity # threshold + node-selection robustness +python -m adforce.eval.sensitivity sensitivity.node=false ``` -`comp.nulltest` writes `img/comp_val_nulltests.pdf` and reports: a label-permutation null +`adforce.eval.nulltest` writes `img/comp_val_nulltests.pdf` and reports: a label-permutation null (observed r=0.89 vs null max 0.31 over 5000 shuffles, p<1e-3); a within-storm permutation (observed spatial r=0.86 vs null max 0.61) showing real *spatial* skill; a cross-storm same-gauge null (collapses to r~0.1–0.2); and a temporal-lag curve (time-series r peaks -sharply at lag 0 and decays to zero within two days). `comp.sensitivity` shows the headline +sharply at lag 0 and decays to zero within two days). `adforce.eval.sensitivity` shows the headline is stable across the clean-filter cut-offs (r 0.81–0.89) and the node-selection knobs (r changes <0.001 with the wet-depth threshold). -Unit + regression tests live in [`../tests/test_comp.py`](../tests/test_comp.py) (skill +Unit + regression tests live in [`../../tests/test_eval.py`](../../tests/test_eval.py) (skill metrics, time-series alignment, the valid/clean gating, the LaTeX table, and pinned headline numbers + negative-control separations): ```bash -python -m pytest tests/test_comp.py -o addopts="" +python -m pytest tests/test_eval.py -o addopts="" ``` ## Configuration @@ -112,6 +116,57 @@ The slight low bias is consistent with the omitted wave setup/runup and medium m resolution (datum and node-sampling effects checked and found minor); over-predictions concentrate at shallow semi-enclosed bay/pass gauges during direct landfalls. +## Comparing adforce configurations (general framework) + +Beyond the historical HF-archive validation above, the submodule compares +**any** adforce configurations (resolution low/mid/high, tides on/off, SWAN) +against gauges and against each other, driven entirely by YAML under +[`config/`](config/): + +- **matrix** ([`config/matrix/`](config/matrix/)) — the comparison grid: + cartesian `axes` over wrap-config paths (`adcirc.resolution.value`, + `adcirc.tide.value`, ...) minus `exclude`, plus explicit `cells`, with a + designated `baseline` for model-vs-model. +- **launch** (`python -m adforce.eval.launch`) — expands the matrix into + per-(cell, storm) runs under `////`, routes + every historical run through the training driver's per-storm input + generation (correct tidal windows — never the Katrina-pinned static + decks), skips completed runs, refuses FOREIGN directories, and records a + full-config sha256 per run in `eval_manifest.json`. `dry_run=true` is the + default: it prints the plan table and fires nothing. `controls=true` adds + the tide-only runs of the tide-surge-interaction triple. +- **extract** (`python -m adforce.eval.extract`) — remote-side reducer: + `fort.63.nc` (5–8 GB) → `gauge_ts.parquet` per run (long format + `storm, sid, gauge, time, zeta`), same node selection as the validation. +- **harvest** (`python -m adforce.eval.harvest`) — rsyncs the minimal + artifact set (`config.yaml`, `gauge_ts.parquet`, `fort.61.nc`, + `maxele.63.nc`, `slurm.out`, manifest) to the laptop. +- **pairs** (`python -m adforce.eval.pairs`) — model-vs-model: + `resolution_bias_table` (reproduces `rerun/results/resolution_bias.csv` + exactly) and `tide_surge_interaction` + (`peak(both) − peak(storm) − peak(tide)`; reproduces the published + low-res rows of `rerun/results/tide_surge_interaction.csv` 683/683). + Both reproductions are pinned by tests. + +Example — low vs mid × tide on/off for two storms: + +```bash +# remote (GCP spot VM): preview, then launch +python -m adforce.eval.launch study=kat-ida matrix=res_x_tide \ + 'storms=["Katrina 2005","Ida 2021"]' # dry run (default) +python -m adforce.eval.launch study=kat-ida matrix=res_x_tide \ + 'storms=["Katrina 2005","Ida 2021"]' dry_run=false controls=true +# laptop: +python -m adforce.eval.harvest remote=gcp-vm:/work/exp/eval study=kat-ida dry_run=false +python -m adforce.eval.pairs action=interaction res=mid \ + tide_series=... both_series=... out=data/comp/out/kat-ida_tsi.csv +``` + +Caveat (tide-on scoring): comparing a tide-on run against raw gauge water +level needs the datum/steric offset handled (pre-storm-mean alignment) and +skew surge as the headline metric; that obs-side path is staged follow-up +work — do not publish tide-on skill numbers before it lands. + ## Dependencies `huggingface_hub`, `utide`, `xarray`, `scipy`, `pandas`, `pyarrow`, `requests`, `matplotlib`, diff --git a/comp/__init__.py b/adforce/eval/__init__.py similarity index 90% rename from comp/__init__.py rename to adforce/eval/__init__.py index a0c5cfae..f5f44034 100644 --- a/comp/__init__.py +++ b/adforce/eval/__init__.py @@ -5,7 +5,7 @@ Entry point:: - python -m comp.validate + python -m adforce.eval.validate """ from .validate import run, validate_storm, metrics diff --git a/adforce/eval/_cli.py b/adforce/eval/_cli.py new file mode 100644 index 00000000..e375a93c --- /dev/null +++ b/adforce/eval/_cli.py @@ -0,0 +1,31 @@ +"""Shared CLI helpers for the adforce.eval hydra entry points. + +The comp/-era argparse flags were replaced by hydra overrides when the module +moved to adforce/eval/. Stale invocations (old docs, shell history, thesis +Makefiles) should fail loudly with the exact translation rather than be +half-parsed by hydra -- so each entry point calls :func:`reject_legacy_flags` +with its flag map before handing argv to hydra. Only known legacy flags are +rejected; hydra's own ``--help``/``--cfg``/etc. pass through untouched. +""" + +import sys +from typing import Dict + + +def reject_legacy_flags(mapping: Dict[str, str], module: str) -> None: + """Exit with the hydra translation if an old argparse flag is on argv. + + Args: + mapping: legacy flag -> replacement hydra override (shown verbatim). + module: dotted module path for the usage line, e.g. "adforce.eval.validate". + """ + hits = [a for a in sys.argv[1:] if a.split("=")[0] in mapping] + if not hits: + return + lines = [ + f"{module}: argparse flags were replaced by hydra overrides " + "when comp/ moved to adforce/eval/. Translation:" + ] + lines += [f" {old:<18} -> {new}" for old, new in mapping.items()] + lines.append(f"e.g. python -m {module} " + mapping[hits[0].split("=")[0]]) + raise SystemExit("\n".join(lines)) diff --git a/comp/annual_max.py b/adforce/eval/annual_max.py similarity index 87% rename from comp/annual_max.py rename to adforce/eval/annual_max.py index 09aaf527..c04796d4 100644 --- a/comp/annual_max.py +++ b/adforce/eval/annual_max.py @@ -4,20 +4,20 @@ (one calendar-year request at a time -- the CO-OPS ``hourly_height`` product accepts up to a year per request), de-tides each calendar year separately with a robust ``utide`` harmonic fit (mean + trend included, following -:func:`comp.coops.observed_residual`), and records the ANNUAL MAXIMUM of the +:func:`adforce.eval.coops.observed_residual`), and records the ANNUAL MAXIMUM of the surge residual for every year with adequate data. Missing-data policy: a year is skipped unless (a) it has at least ``UTIDE_MIN_SAMPLES`` hourly samples (stable harmonic fit, same threshold as -``comp.coops``) and (b) at least ``MIN_YEAR_COVERAGE`` (80%) of the year's +``adforce.eval.coops``) and (b) at least ``MIN_YEAR_COVERAGE`` (80%) of the year's hours are present, since with large gaps the true annual maximum may fall in a gap and the recorded maximum would be biased low. Even in accepted years the maximum can be truncated if the gauge failed *during* the peak (e.g. Grand -Isle in Ida 2021, see ``comp.constants.KNOWN_FAILED``); such years are flagged +Isle in Ida 2021, see ``adforce.eval.constants.KNOWN_FAILED``); such years are flagged with ``max_at_gap_edge`` rather than dropped. Everything is cached under ``data/comp/``: the raw CO-OPS responses land in -``COOPS_CACHE`` (via :mod:`comp.coops`), the per-year de-tided residuals and +``COOPS_CACHE`` (via :mod:`adforce.eval.coops`), the per-year de-tided residuals and the annual-maxima table are Parquet files under ``ANNUAL_MAX_CACHE``, keyed by the de-tiding parameters so the cache self-invalidates if those change. Reruns are therefore free. @@ -26,20 +26,22 @@ insensitive to the datum because the harmonic fit absorbs the mean (and a linear trend, which also removes most local sea-level rise within a year). -Run:: +Run (hydra overrides; config root adforce/eval/config/annual_max_config.yaml):: - python -m comp.annual_max --station 8761724 # Grand Isle, LA - python -m comp.annual_max --station 8735180 --start 1980 # Dauphin Island, AL + python -m adforce.eval.annual_max station=8761724 # Grand Isle, LA + python -m adforce.eval.annual_max station=8735180 start=1980 # Dauphin Island, AL """ from __future__ import annotations -import argparse import json import os import time from typing import Optional +import hydra +from omegaconf import DictConfig + import numpy as np import pandas as pd @@ -68,7 +70,7 @@ def _hours_in_year(year: int) -> int: def station_meta(station: str) -> Optional[dict]: """Name/lat/lon for a CO-OPS station from the cached station list. - Reuses the same ``stations.json`` cache as :func:`comp.coops.gulf_gauges` + Reuses the same ``stations.json`` cache as :func:`adforce.eval.coops.gulf_gauges` (all water-level stations, no bounding box), fetching it once if absent. Args: @@ -96,7 +98,7 @@ def station_meta(station: str) -> Optional[dict]: # --------------------------------------------------------------------------- # -# Fetching. comp.coops._coops caches every response, including transport +# Fetching. adforce.eval.coops._coops caches every response, including transport # failures (as zero-byte files). A genuine "no data" year is a NON-empty API # error message and is accepted at once; a zero-byte cache entry is deleted and # the year re-requested a few times, so a one-off network blip cannot @@ -127,7 +129,7 @@ def fetch_year_wl( ) -> pd.Series: """Hourly water level (MSL, GMT) for one calendar year, cached + polite. - Delegates to :func:`comp.coops.fetch_year` (verified ``hourly_height``, + Delegates to :func:`adforce.eval.coops.fetch_year` (verified ``hourly_height``, falling back to preliminary ``water_level``); on an empty result caused by a cached transport failure the bad cache entry is cleared and the request retried with a growing sleep. Tests monkeypatch this function. @@ -164,7 +166,7 @@ def fetch_year_wl( def detide_year(wl: pd.Series, lat: float, method: str = "robust") -> pd.Series: """De-tide one calendar year of hourly water levels with ``utide``. - Mirrors :func:`comp.coops.observed_residual` (harmonic fit with mean and + Mirrors :func:`adforce.eval.coops.observed_residual` (harmonic fit with mean and linear trend over the full year, residual = observed - reconstruction), with one deliberate difference: the DatetimeIndex is passed to ``utide`` directly instead of via ``matplotlib.dates.date2num``. With matplotlib's @@ -260,6 +262,7 @@ def year_residual( Returns: Optional[pd.Series]: Residual series, or None if the year is skipped. """ + C.ensure_dirs() # cache dirs are created lazily, not at import fp = _resid_path(station, year, method) if not refresh and os.path.exists(fp): df = pd.read_parquet(fp) @@ -305,6 +308,7 @@ def annual_maxima( ``station, name, year, ann_max_m, t_max, n_obs, coverage, max_at_gap_edge``. """ + C.ensure_dirs() # cache dirs are created lazily, not at import tp = _table_path(station, start, end, method) if not refresh and os.path.exists(tp): df = pd.read_parquet(tp) @@ -351,31 +355,27 @@ def annual_maxima( return df -def main() -> None: - ap = argparse.ArgumentParser(description=__doc__) - ap.add_argument( - "--station", - default="8761724", - help="CO-OPS station id (default 8761724, Grand Isle LA)", - ) - ap.add_argument("--start", type=int, default=C.AM_START_YEAR) - ap.add_argument("--end", type=int, default=C.AM_END_YEAR) - ap.add_argument( - "--method", - default="robust", - choices=["robust", "ols"], - help="utide.solve method (default robust)", - ) - ap.add_argument( - "--refresh", action="store_true", help="recompute instead of reading the caches" +_LEGACY_FLAGS = { + "--station": "station=8761724", + "--start": "start=1980", + "--end": "end=2025", + "--method": "method=robust", + "--refresh": "refresh=true", +} + + +@hydra.main(version_base=None, config_path="config", config_name="annual_max_config") +def main(cfg: DictConfig) -> None: + if cfg.method not in ("robust", "ols"): + raise SystemExit(f"method must be robust|ols, got {cfg.method!r}") + df = annual_maxima( + str(cfg.station), cfg.start, cfg.end, method=cfg.method, refresh=cfg.refresh ) - a = ap.parse_args() - df = annual_maxima(a.station, a.start, a.end, method=a.method, refresh=a.refresh) if df.empty: print("no usable years") return print( - f"\n{len(df)} usable years of {a.end - a.start + 1}; " + f"\n{len(df)} usable years of {cfg.end - cfg.start + 1}; " f"max residual {df.ann_max_m.max():.2f} m in " f"{int(df.loc[df.ann_max_m.idxmax(), 'year'])}; " f"{int(df.max_at_gap_edge.sum())} year(s) flagged max-near-gap" @@ -383,4 +383,7 @@ def main() -> None: if __name__ == "__main__": + from ._cli import reject_legacy_flags + + reject_legacy_flags(_LEGACY_FLAGS, "adforce.eval.annual_max") main() diff --git a/adforce/eval/cells.py b/adforce/eval/cells.py new file mode 100644 index 00000000..8b71e52e --- /dev/null +++ b/adforce/eval/cells.py @@ -0,0 +1,170 @@ +"""Configuration-cell identity for adforce run comparison. + +A *cell* is one point in the comparison matrix (resolution x tide x swan x +forcing x physics tag). Its canonical string (:func:`cell_id`) names run +directories, cache files, and model-vs-model join keys; provenance is always +established from the ``/config.yaml`` that ``adforce.config.save_config`` +writes (never from a directory name alone). + +Storm identity is kept separate from the cell: the same cell is run once per +storm, in ``////``. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Optional + +# Wrap-config paths for the physical axes (the one place the wrap tree's +# {value, ...} nesting appears on the eval side). +AXIS_PATHS = { + "resolution": "adcirc.resolution.value", + "tide": "adcirc.tide.value", + "swan": "adcirc.swan.value", +} + + +@dataclass(frozen=True, order=True) +class ConfigCell: + """One comparison-matrix cell (storm-independent).""" + + resolution: str = "mid" # low | mid | high + tide: bool = False # tidal forcing in fort.15 + swan: bool = False # SWAN wave coupling (padcswan) + forcing: str = "storm" # storm | tide | both (tide = wind-off control run) + physics_tag: str = "default" # opaque label for future physics variants + mannings_n: Optional[float] = None # fort.13 override -- VERIFIED NO-OP (NWP=0), blocked at plan time + friction_cf: Optional[float] = None # fort.15 hybrid-friction CF override (None = generated 0.0025) + + +def _onoff(b: bool) -> str: + return "on" if b else "off" + + +def cell_id(c: ConfigCell) -> str: + """Deterministic cell name, e.g. ``res-mid_tide-off_swan-off_f-storm``. + + The ``physics_tag`` is appended only when non-default, so today's standard + cells stay short and stable. + """ + parts = [ + f"res-{c.resolution}", + f"tide-{_onoff(c.tide)}", + f"swan-{_onoff(c.swan)}", + f"f-{c.forcing}", + ] + if c.mannings_n is not None: + parts.append(f"n{c.mannings_n:g}") + if c.friction_cf is not None: + parts.append(f"cf{c.friction_cf:g}") + if c.physics_tag != "default": + parts.append(f"p-{c.physics_tag}") + return "_".join(parts) + + +def dir_to_storm(d: str) -> Optional[str]: + """``'22_MICHAEL_2018'`` -> ``'Michael 2018'`` (val_summary storm naming). + + Ported from ``rerun/adcirc/score_resolution.py`` so legacy GCP/ARCHER2 + sweep directories (``__``) can be ingested read-only. + Returns None when the basename does not match the convention. + """ + m = re.match(r"\d+_(.+)_(\d{4})$", d) + if m is None: + return None + name = m.group(1).replace("_", " ").title() + return f"{name} {m.group(2)}" + + +def storm_to_slug(storm: str, fname: str) -> str: + """``('Katrina 2005', '152_KATRINA_2005.nc')`` -> ``'152_KATRINA_2005'``. + + The slug matches both the HF archive filename stem and the run-dir + basename produced by the training driver, so one name keys both sources. + """ + return fname[:-3] if fname.endswith(".nc") else fname + + +def expand_matrix(matrix_cfg) -> list: + """Expand the matrix-YAML grammar into ``[(cell_name, overrides), ...]``. + + Grammar (see ``adforce/eval/config/matrix/``): ``axes`` maps WRAP config + paths (``adcirc.resolution.value`` etc.) to value lists whose cartesian + product, minus partial-match ``exclude`` dicts, forms the grid; + ``cells``/``include`` append fully explicit ``{name, overrides}`` entries. + Cell names come from ``name_keys`` aliases in axes order (bools render + ``on``/``off``), e.g. ``res-low_tide-off``. + + Raises: + ValueError: On duplicate cell names or a ``baseline`` naming no cell. + """ + from itertools import product + + axes = dict(matrix_cfg.get("axes") or {}) + name_keys = dict(matrix_cfg.get("name_keys") or {}) + excludes = [dict(e) for e in (matrix_cfg.get("exclude") or [])] + + def render(path, val) -> str: + alias = name_keys.get(path, path.split(".")[-1]) + v = ("on" if val else "off") if isinstance(val, bool) else str(val) + return f"{alias}-{v}" + + cells = [] + if axes: + paths = list(axes) + for combo in product(*(list(axes[p]) for p in paths)): + ov = dict(zip(paths, combo)) + if any(all(ov.get(k) == v for k, v in ex.items()) for ex in excludes): + continue + cells.append(("_".join(render(p, ov[p]) for p in paths), ov)) + for group in ("cells", "include"): + for item in matrix_cfg.get(group) or []: + cells.append((str(item["name"]), dict(item["overrides"]))) + + names = [n for n, _ in cells] + dups = sorted({n for n in names if names.count(n) > 1}) + if dups: + raise ValueError(f"duplicate cell names in matrix: {dups}") + baseline = matrix_cfg.get("baseline") + if baseline and baseline not in names: + raise ValueError(f"baseline {baseline!r} is not one of the cells {names}") + return cells + + +def provenance_match(run_cfg, cell: ConfigCell) -> bool: + """Does a run's dumped ``config.yaml`` match this cell's physical axes? + + Args: + run_cfg: OmegaConf ``DictConfig`` loaded from ``/config.yaml`` + (via :func:`adforce.config.load_config`). + cell (ConfigCell): The cell the directory is expected to hold. + + Returns: + bool: True when resolution/tide/swan all agree, and -- for cells that + carry non-default eval-side axes (mannings_n set, or a tide-only + control) -- when the run's ``eval_axes`` block matches too. Legacy + run dirs predate ``eval_axes``, so those checks only fire for + non-default cells. Missing wrap keys count as a mismatch (FOREIGN) + -- a run without provenance is never trusted. + """ + try: + adcirc = run_cfg.adcirc + if not ( + str(adcirc.resolution.value) == cell.resolution + and bool(adcirc.tide.value) == cell.tide + and bool(adcirc.swan.value) == cell.swan + ): + return False + axes = run_cfg.get("eval_axes") or {} + for field in ("mannings_n", "friction_cf"): + want = getattr(cell, field) + if want is not None: + got = axes.get(field) + if got is None or abs(float(got) - want) > 1e-9: + return False + if cell.forcing == "tide" and axes.get("forcing") != "tide": + return False + return True + except Exception: + return False diff --git a/adforce/eval/config/annual_max_config.yaml b/adforce/eval/config/annual_max_config.yaml new file mode 100644 index 00000000..00c78232 --- /dev/null +++ b/adforce/eval/config/annual_max_config.yaml @@ -0,0 +1,12 @@ +# Root config for python -m adforce.eval.annual_max (obs-side annual-maximum +# de-tided residual pipeline; feeds the gauge-EVT fits in worst/gauge_fit*.py). +hydra: + output_subdir: null + run: + dir: . + +station: "8761724" # CO-OPS station id (default: Grand Isle, LA) +start: 1980 # first year; = C.AM_START_YEAR (drift-tested) +end: 2025 # last year; = C.AM_END_YEAR (drift-tested) +method: robust # utide.solve method: robust | ols +refresh: false # recompute instead of reading the Parquet caches diff --git a/adforce/eval/config/eval_config.yaml b/adforce/eval/config/eval_config.yaml new file mode 100644 index 00000000..5aee5726 --- /dev/null +++ b/adforce/eval/config/eval_config.yaml @@ -0,0 +1,36 @@ +# Root config for the adforce.eval analysis entry points +# (validate, nulltest, sensitivity, detide_sensitivity). +# +# House style: flat keys + comments (infrastructure/analysis config, like +# adforce/config/slurm|files); the {value, description, units} nesting is +# reserved for physical parameters in the wrap tree. +defaults: + - scoring: default + - _self_ + +# hydra hygiene: every output goes to an absolute path from +# adforce.eval.constants; keep the CWD and write no .hydra/ run dirs. +hydra: + output_subdir: null + run: + dir: . + +storms: null # subset of storm display names, e.g. 'storms=["Ida 2021","Katrina 2005"]' (null = all) + +validate: + refresh: false # force-recompute the cached time series (was --refresh-cache) + examples_only: false # only the example-panel figure, from cache (was --examples-only) + failures: false # per-city worst-case panels from the cached sweep (was --failures) + n_failures: 6 # panels per city for validate.failures=true (was --n-failures) + +nulltest: + lag: true # temporal-lag null (netCDF-heavy); --no-lag -> nulltest.lag=false + +sensitivity: + node: true # netCDF node-selection sweep; --no-node -> sensitivity.node=false + +detide_sensitivity: + methods: null # subset of method ids, e.g. '["godin_lowpass","noaa_predictions"]' (null = all) + limit: null # only the first N clean pairs (quick look; suppresses the table) + skew: false # instantaneous-residual vs skew-surge peak metric (was --skew) + table_only: false # regenerate the appendix table from the cached CSV (was --table-only) diff --git a/adforce/eval/config/extract_config.yaml b/adforce/eval/config/extract_config.yaml new file mode 100644 index 00000000..13026ab3 --- /dev/null +++ b/adforce/eval/config/extract_config.yaml @@ -0,0 +1,15 @@ +# Root config for python -m adforce.eval.extract (run dir -> gauge_ts.parquet +# reducer; runs remote where the fort.63.nc files are). +defaults: + - scoring: default + - _self_ + +hydra: + output_subdir: null + run: + dir: . + +runs_root: ??? # parent of the per-storm run directories (required) +gauges_csv: null # sid,name,lat,lon CSV; null -> CO-OPS panel over both boxes (needs cache/net) +out: null # optional combined parquet (legacy data/comp/{lowres,midres} convention) +per_run: true # write /gauge_ts.parquet beside each fort.63.nc (harvest convention) diff --git a/adforce/eval/config/harvest_config.yaml b/adforce/eval/config/harvest_config.yaml new file mode 100644 index 00000000..e7f5b428 --- /dev/null +++ b/adforce/eval/config/harvest_config.yaml @@ -0,0 +1,10 @@ +# Root config for python -m adforce.eval.harvest (remote -> laptop rsync). +hydra: + output_subdir: null + run: + dir: . + +remote: ??? # host:/path/to/exp/eval (parent of study dirs), e.g. gcp-vm:/work/exp/eval +study: ??? # study subtree to pull +dest: null # local mirror root (null -> data/comp/runs/) +dry_run: true # print the rsync command without running it diff --git a/adforce/eval/config/launch_config.yaml b/adforce/eval/config/launch_config.yaml new file mode 100644 index 00000000..cd34087d --- /dev/null +++ b/adforce/eval/config/launch_config.yaml @@ -0,0 +1,26 @@ +# Root config for python -m adforce.eval.launch (sweep orchestrator). +defaults: + - matrix: res_x_tide + - _self_ + +hydra: + output_subdir: null + run: + dir: . + +study: default # namespaces /// and the manifest +runs_root: ${oc.env:WORSTSURGE_RUNS_ROOT,runs/eval} # relative -> under the repo root +storms: null # subset of storm display names (null = all 19) +controls: false # auto-add tide-only control cells (interaction triples) +spinup_days: auto # auto: 6.0 for tidal cells, else 0.0 (hist_sweep SPINUP) +recommended_dt: 5.0 # ADCIRC timestep [s] passed to input generation +dry_run: true # DEFAULT: print the plan table, launch nothing +skip_completed: true # resume semantics: skip runs with a success receipt +overwrite: false # refuse FOREIGN run dirs (config.yaml mismatch) unless true +extract_after_run: true # reduce each run to gauge_ts.parquet right after it finishes +strip_after_extract: # then delete the bulky outputs (hist_sweep disk lesson) + - fort.63.nc + - fort.64.nc + - fort.73.nc + - fort.74.nc + - "PE*" diff --git a/adforce/eval/config/matrix/archive_default.yaml b/adforce/eval/config/matrix/archive_default.yaml new file mode 100644 index 00000000..8c7e0900 --- /dev/null +++ b/adforce/eval/config/matrix/archive_default.yaml @@ -0,0 +1,14 @@ +# The published HF EC95d no-tide archive as a degenerate one-cell matrix: +# scoring with this matrix reproduces the paper's validation numbers inside +# the general framework. +axes: {} +name_keys: {} +exclude: [] +include: [] +cells: + - name: ec95d-archive + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: false + adcirc.swan.value: false +baseline: ec95d-archive diff --git a/adforce/eval/config/matrix/friction_storm.yaml b/adforce/eval/config/matrix/friction_storm.yaml new file mode 100644 index 00000000..c15788d3 --- /dev/null +++ b/adforce/eval/config/matrix/friction_storm.yaml @@ -0,0 +1,25 @@ +# Surge sensitivity to bottom friction (adforce/eval/tidal_diagnosis.md +# take-2 verdict, step 2): CF=0.005 fixes the semidiurnal tide, but the +# published surge skill (bias -0.22 m, r 0.871) was earned at CF=0.0025 and +# is already biased LOW -- more friction may damp surge peaks further. +# Storm-only runs (the published configuration: no tides, no spinup) at the +# control and the tide-side sweet spot, scored against the cached de-tided +# gauge residuals. +axes: {} +name_keys: {} +exclude: [] +include: [] +cells: + - name: storm-cf0.0025 # control (= the published configuration, regenerated) + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: false + forcing: storm + friction_cf: 0.0025 + - name: storm-cf0.005 # the tide-side sweet spot + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: false + forcing: storm + friction_cf: 0.005 +baseline: storm-cf0.0025 diff --git a/adforce/eval/config/matrix/friction_tide.yaml b/adforce/eval/config/matrix/friction_tide.yaml new file mode 100644 index 00000000..91b7bfc7 --- /dev/null +++ b/adforce/eval/config/matrix/friction_tide.yaml @@ -0,0 +1,34 @@ +# Bottom-friction sensitivity on tide-only runs (mid resolution), take 2. +# The Manning's-n sweep was a verified no-op: the decks carry NWP=0, so +# fort.13 is never read and friction is fort.15's NOLIBF=2 hybrid line +# `CF HBREAK FTHETA FGAMMA = 0.0025 1 10 0.333333` -- a uniform quadratic +# coefficient with Manning-like scaling only below 1 m depth. friction_cf +# rewrites that CF (control regenerated identically for apples-to-apples). +# Hypothesis (adforce/eval/tidal_diagnosis.md): the near-resonant GoM +# diurnal band (O1 +34%, K1 +18%, uniform phase lead) is under-damped at +# CF=0.0025; doubling/tripling CF should pull O1/K1 toward 1 and shrink the +# lead without destroying M2. +axes: {} +name_keys: {} +exclude: [] +include: [] +cells: + - name: tide-cf0.0025 # control (= generated deck default) + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: true + forcing: tide + friction_cf: 0.0025 + - name: tide-cf0.005 + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: true + forcing: tide + friction_cf: 0.005 + - name: tide-cf0.0075 + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: true + forcing: tide + friction_cf: 0.0075 +baseline: tide-cf0.0025 diff --git a/adforce/eval/config/matrix/mannings_tide.yaml b/adforce/eval/config/matrix/mannings_tide.yaml new file mode 100644 index 00000000..b3bd8bca --- /dev/null +++ b/adforce/eval/config/matrix/mannings_tide.yaml @@ -0,0 +1,35 @@ +# Manning's-n friction sensitivity on tide-only runs (mid resolution) -- +# the experiment recommended by adforce/eval/tidal_diagnosis.md: the diurnal +# band (O1 +34%, K1 +18%) is over-amplified with a uniform phase lead, +# pointing at the under-damped basin response under the shipped uniform +# n = 0.022. Success = O1/K1 amp ratios -> 1 and the phase lead shrinking +# without breaking M2 (score each cell with adforce.eval.tidecheck + +# tideconst). +# +# forcing/mannings_n are eval-side pseudo-axes (never touch the wrap tree); +# n0.022 regenerates the control with identical machinery so the comparison +# is apples-to-apples. +axes: {} +name_keys: {} +exclude: [] +include: [] +cells: + - name: tide-n0.022 # control (= shipped fort.13 default) + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: true + forcing: tide + mannings_n: 0.022 + - name: tide-n0.028 + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: true + forcing: tide + mannings_n: 0.028 + - name: tide-n0.035 + overrides: + adcirc.resolution.value: mid + adcirc.tide.value: true + forcing: tide + mannings_n: 0.035 +baseline: tide-n0.022 diff --git a/adforce/eval/config/matrix/res_x_tide.yaml b/adforce/eval/config/matrix/res_x_tide.yaml new file mode 100644 index 00000000..3fc5bcd2 --- /dev/null +++ b/adforce/eval/config/matrix/res_x_tide.yaml @@ -0,0 +1,13 @@ +# low/mid resolution x tide on/off comparison matrix. +# Axis keys are WRAP config paths (hence the .value nesting -- the wrap +# tree's convention; eval's own keys stay flat). +axes: + adcirc.resolution.value: [low, mid] + adcirc.tide.value: [false, true] +name_keys: + adcirc.resolution.value: res + adcirc.tide.value: tide +exclude: [] # partial-match dicts over axis paths, e.g. {adcirc.resolution.value: low} +include: [] # extra fully-explicit {name, overrides} cells appended to the grid +cells: [] # alternative: fully explicit named cells instead of axes +baseline: res-mid_tide-off # model-vs-model reference cell diff --git a/adforce/eval/config/pairs_config.yaml b/adforce/eval/config/pairs_config.yaml new file mode 100644 index 00000000..e8014c1f --- /dev/null +++ b/adforce/eval/config/pairs_config.yaml @@ -0,0 +1,20 @@ +# Root config for python -m adforce.eval.pairs (model-vs-model comparison). +hydra: + output_subdir: null + run: + dir: . + +action: resolution_bias # resolution_bias | interaction + +# resolution_bias inputs +series: ??? # candidate-cell gauge-series parquet (long format) +val_summary: null # baseline val_summary.csv (null -> data/comp/out/val_summary.csv) +out: null # CSV to write +min_obs_peak: 1.0 # pooled-ratio report threshold on observed peak [m] + +# interaction inputs +res: low # resolution label for the output rows +storm_series: null # storm-only gauge-series parquet (null -> archive fallback only) +tide_series: null # tide-only gauge-series parquet +both_series: null # storm+tide gauge-series parquet +use_archive: true # fall back to HF-archive val_summary peaks for storm-only diff --git a/adforce/eval/config/scoring/default.yaml b/adforce/eval/config/scoring/default.yaml new file mode 100644 index 00000000..ed7a986b --- /dev/null +++ b/adforce/eval/config/scoring/default.yaml @@ -0,0 +1,18 @@ +# Node-selection, clean-pair, and uncertainty gates for the gauge validation. +# +# Mirrors adforce/eval/constants.py EXACTLY: constants.py remains the single +# source of truth consumed by library code (and by the ts-cache tag, whose +# byte-stability keeps the utide-expensive caches valid), while this group +# makes the values visible/overridable from the CLI for future sweep studies. +# tests/test_eval.py::test_scoring_config_matches_constants pins the equality +# so the two cannot drift (the worstsurge audit flagged constants drift as a +# live failure mode). +max_node_deg: 0.12 # = C.MAX_NODE_DEG +wet_min_m: 0.3 # = C.WET_MIN_M +knn: 60 # = C.KNN +max_timing_hr: 6.0 # = C.MAX_TIMING_HR +min_obs_peak_m: 0.4 # = C.MIN_OBS_PEAK_M +ts_min_overlap: 24 # = C.TS_MIN_OVERLAP +n_bootstrap: 2000 # = C.N_BOOTSTRAP +bootstrap_seed: 0 # = C.BOOTSTRAP_SEED +utide_min_samples: 2000 # = C.UTIDE_MIN_SAMPLES diff --git a/adforce/eval/config/source/hf_archive.yaml b/adforce/eval/config/source/hf_archive.yaml new file mode 100644 index 00000000..2b9c50da --- /dev/null +++ b/adforce/eval/config/source/hf_archive.yaml @@ -0,0 +1,5 @@ +# The published SurgeNet HF archive (sdat2/surgenet-train): element-centroid +# dual-graph fields, ssh = WD + DEM, tides excluded (pure surge). +# Consumed by adforce.eval.sources.HFArchiveSource. +kind: hf_archive +repo: sdat2/surgenet-train # = constants.HF_REPO (drift would break download_storm) diff --git a/adforce/eval/config/source/run_dirs.yaml b/adforce/eval/config/source/run_dirs.yaml new file mode 100644 index 00000000..e6f85746 --- /dev/null +++ b/adforce/eval/config/source/run_dirs.yaml @@ -0,0 +1,6 @@ +# Completed adforce run directories as the simulation source. +# Consumed by adforce.eval.sources.RunDirSource. +kind: run_dirs +runs_root: ??? # parent of the per-storm run directories (required) +prefer: [gauge_ts, fort61, fort63] # artifact preference; gauges fall through to the next +verify_config: true # provenance check vs /config.yaml (see eval.status) diff --git a/adforce/eval/config/tidecheck_config.yaml b/adforce/eval/config/tidecheck_config.yaml new file mode 100644 index 00000000..7fbd5641 --- /dev/null +++ b/adforce/eval/config/tidecheck_config.yaml @@ -0,0 +1,10 @@ +# Root config for python -m adforce.eval.tidecheck (tide-only run validation +# vs CO-OPS astronomical predictions -- the gate before tide-on surge skill). +hydra: + output_subdir: null + run: + dir: . + +series: ??? # gauge-series parquet path or glob (tide-only runs) +label: tide # names the default output data/comp/out/tidecheck_