PSF star selection vetoes the UNIONS manual mask

astra.yaml

@@ -342,34 +342,52 @@ analyses:
               corrupted, so acting on them is an analysis decision.
               Rasterising would bake one mask version into every shape.
       psf_star_mask_veto:
-        label: PSF-star candidates rejected on instrument flags only
+        label: PSF-star candidates rejected on instrument flags and the manual mask
         rationale: >-
-          Of the masks, star selection cuts only on IMAFLAGS_ISO. mask_query sits
-          between SExtractor and setools in the exposure chain; when
-          MASK_PATHS names maps it writes MASK_EXT (0 clean, nonzero flagged,
-          off-coverage clean) onto each CCD's catalogue. MASK_PATHS ships
-          commented out, so the module passes catalogues through without
-          MASK_EXT. The intended map is the UNIONS star-body product (bit 2);
-          halo bits 0 and 1 are left out because halos say nothing about
-          whether a star is a good PSF sample.
+          Star selection cuts on IMAFLAGS_ISO and on MASK_EXT from the UNIONS
+          manual mask (bit 8: large galaxies, globular clusters and other
+          extended objects). mask_query sits between SExtractor and setools in
+          the exposure chain and writes MASK_EXT (0 clean, nonzero flagged,
+          off-coverage clean) onto each CCD's catalogue. Outlier rejection
+          alone copes with crowded star fields: in run 1, M13, M92 and the
+          Draco and Ursa Minor dwarf spheroidals are 3-15x overdense in PSF
+          stars with no oversized candidates, and normal-size stars fit the
+          model as well as the surrounding field does. It fails where compact
+          extended sources outnumber stars on a CCD. Inside 0.15 deg of M33,
+          24% of selected PSF stars are more than 1.2x the PSF size, and
+          normal-size stars come out 2.4% smaller than the PSF model, so the
+          contaminants inflate the model itself. That region carries most of
+          the first-bin rho-statistics variance and flips the sign of the
+          fitted leakage coefficients. The manual mask removes 1,024 of the
+          1,025 oversized candidates within 1 deg of M33 and every radius
+          where the model is biased. It costs 0.5% of PSF stars survey-wide
+          (CosmoStat/shapepipe#993). Halo bits 0 and 1 stay out because a
+          star inside another star's halo is not thereby a bad PSF sample.
           Values:
           MASK:star_selection.IMAFLAGS_ISO = "== 0";
           MASK:preselect.IMAFLAGS_ISO = "== 0";
           MASK:flag.IMAFLAGS_ISO = "== 0";
-          MASK:star_selection.MASK_EXT = absent;
-          MASK:preselect.MASK_EXT = absent;
-          MASK:flag.MASK_EXT = absent.
-        default: instrument_flags_only
+          MASK:star_selection.MASK_EXT = "== 0";
+          MASK:preselect.MASK_EXT = "== 0";
+          MASK:flag.MASK_EXT = "== 0";
+          config_exp_psfex.ini#MASK_QUERY_RUNNER.MASK_PATHS = $SP_INPUT_MASKS/mask_ugriz_nside131072_n8.hsp;
+          config_exp_mccd.ini#MASK_QUERY_RUNNER.MASK_PATHS = $SP_INPUT_MASKS/mask_ugriz_nside131072_n8.hsp.
+        default: manual_mask_veto
         options:
+          manual_mask_veto:
+            label: IMAFLAGS_ISO == 0 and MASK_EXT == 0 on the manual mask (bit 8)
           instrument_flags_only:
             label: IMAFLAGS_ISO == 0; no sky map queried
+            description: >-
+              Leaves outlier rejection to handle every sky-fixed contaminant.
+              Run 1 ran this way; M33's HII regions and clusters entered the
+              PSF model.
           star_body_veto:
-            label: Also reject candidates on the star-body map (MASK_EXT == 0)
+            label: Also reject candidates on the star-body map (bit 2)
             description: >-
-              Set MASK_PATHS to the star-body map and add MASK_EXT == 0 beside
-              each IMAFLAGS_ISO cut in star_selection.setools (one line per
-              mask block). Querying without the cut records MASK_EXT and
-              changes no star.
+              Add the star-body map to MASK_PATHS. It flags 12.3% of run-1 PSF
+              stars, because it masks star bodies, the objects star selection
+              is meant to keep.
           star_body_and_halo_veto:
             label: Also reject candidates inside star halos
             excluded: true

docs/source/pipeline_tutorial.md

@@ -214,7 +214,7 @@ consumed by *querying them at object positions*, never by rasterizing them onto
 pixels. Two modules do the querying, from the same shared lookup
 (`shapepipe.utilities.mask_query`): `mask_query` runs between `sextractor` and
 `setools` on the single-exposure single-CCD catalogues and writes one integer
-`MASK_EXT` column (0 = clean), recording the star-body map (bit 2) against every
+`MASK_EXT` column (0 = clean), recording the manual mask (bit 8) against every
 detection; `make_cat` writes one
 `MASK_<band>` column per band onto the final tile catalogue, carrying the map
 value verbatim so downstream selections choose their own cuts. Map paths and
@@ -228,16 +228,16 @@ these are the only masks that reject anything inside the pipeline. The
 sits, not that its pixels are broken, so what to do about one is an analysis
 decision and is made downstream.
 
-**Nothing in the pipeline cuts on the queried columns**, and on exposures the
-query ships off entirely: `MASK_PATHS` is commented out, which makes
-`mask_query` a strict no-op that passes the catalogue through with no
-`MASK_EXT` column (the module stays in the chain, so enabling it is
-uncommenting one line). `star_selection.setools` rejects on `IMAFLAGS_ISO == 0`
-and nothing else, deliberately starting from outlier rejection alone, and the
-final catalogue's `MASK_<band>` columns are written unfiltered. `MASK_EXT` is
-the configurable pickup if outlier rejection proves insufficient: add
-`MASK_EXT == 0` beside each `IMAFLAGS_ISO == 0`, one line per mask block, as
-that file's header documents.
+**One queried mask cuts inside the pipeline: the manual mask, on PSF stars.**
+`star_selection.setools` rejects candidates with `IMAFLAGS_ISO != 0` or
+`MASK_EXT != 0`. Outlier rejection handles crowded star fields on its own, but
+where compact extended sources outnumber stars on a CCD (the HII regions and
+clusters of M33) they pass selection and inflate the PSF model, so the manual
+mask, which covers those objects, keeps them out of the PSF fit
+([#993](https://github.com/CosmoStat/shapepipe/issues/993)). The final
+catalogue's `MASK_<band>` columns are written unfiltered. Without `MASK_PATHS`,
+`mask_query` is a strict no-op that writes no `MASK_EXT` column, so dropping
+the map means dropping the `MASK_EXT` lines in `star_selection.setools` too.
 
 No internet access is needed at any point, and there is no reference star
 catalogue to download.

src/shapepipe/modules/mask_query_package/__init__.py

@@ -24,8 +24,9 @@
 **Healsparse masks** are sky-fixed LOCATION flags — a star halo, a manual
 region, a band with no data. They say where an object is, not that its pixels
 are broken, so what to do about one is an analysis decision. They are queried
-into columns and every rejection happens downstream: ``MASK_EXT`` here,
-``MASK_<band>`` in ``make_cat``. Nothing in the pipeline cuts on either.
+into columns: ``MASK_EXT`` here, ``MASK_<band>`` in ``make_cat``. The one
+in-pipeline cut on them is PSF-star selection, which rejects candidates with
+``MASK_EXT != 0``; the ``MASK_<band>`` columns are never cut on.
 
 Description
 ===========
@@ -48,24 +49,16 @@
 ``< > <= >= == !=`` — no bitwise operators — so any bit selection has to happen
 here, leaving the config a plain test for zero.
 
-Off by default, and a no-op when off
-====================================
+A no-op without maps
+====================
 
-``MASK_PATHS`` ships COMMENTED OUT. With no maps configured the module is a
-strict no-op: the catalogue is copied through unchanged, with no ``MASK_EXT``
-column at all — the same gating ``make_cat`` gives ``MASK_EXT_PATHS``. It stays
-in the ``MODULE`` chain either way, so enabling the query is uncommenting one
-line and never editing the chain.
-
-Even with maps configured, nothing cuts on the result. The PSF star selection
-deliberately starts from outlier rejection alone, rejecting only on the
-instrument flags, and ``MASK_EXT`` is the configurable pickup if that proves
-insufficient. Writing the column without cutting on it is what lets the effect
-of a mask on the star sample be *measured* before it is imposed.
-
-That pickup is one line: add ``MASK_EXT == 0`` beside each
-``IMAFLAGS_ISO == 0`` in ``star_selection.setools``. The escape hatch is
-deliberate, and that file's header says so.
+With no ``MASK_PATHS`` the module is a strict no-op: the catalogue is copied
+through unchanged, with no ``MASK_EXT`` column at all — the same gating
+``make_cat`` gives ``MASK_EXT_PATHS``. It stays in the ``MODULE`` chain either
+way. The shipped data configs name the UNIONS manual mask (bit 8), and
+``star_selection.setools`` rejects ``MASK_EXT != 0`` in its ``preselect``,
+``flag`` and ``star_selection`` blocks; a config without the map must drop those
+lines too, because ``setools`` cannot cut on a missing column.
 
 The lookup itself lives in :mod:`shapepipe.utilities.mask_query`, shared with
 ``make_cat``'s per-band ``MASK_<band>`` columns, so the healsparse primitive is
@@ -74,12 +67,14 @@
 What gets queried is deliberately narrow
 ========================================
 
-``MASK_PATHS`` is a *list of maps to record against each PSF-star candidate*,
-not a list of every mask that exists. The committed configs name exactly one
-map — the UNIONS star-body product (bit 2). Halo bits 0 and 1 are excluded on
-purpose: halos flag objects for the final catalogue, they say nothing about
-whether a star is a good PSF sample (mask-force telecon, 2026-07-21). MaxiMask
-is not queried here either.
+``MASK_PATHS`` is a *list of maps that veto PSF-star candidates*, not a list
+of every mask that exists. The committed configs name exactly one map — the
+UNIONS manual mask (bit 8), which covers large galaxies, globular clusters and
+other extended objects. Halo bits 0 and 1 are excluded on purpose: halos flag
+objects for the final catalogue, they say nothing about whether a star is a
+good PSF sample (mask-force telecon, 2026-07-21). The star-body map (bit 2) is
+excluded because it flags the stars themselves. MaxiMask is not queried here
+either.
 
 Widening it costs a config edit and no code — add a path. That is why the
 contract is a path list rather than a bit mask: the UNIONS products are one

src/shapepipe/modules/mask_query_runner.py

@@ -30,9 +30,8 @@ def mask_query_runner(
 
     @sc [decision:masking.psf_star_mask_veto,label:scope] mask-query-carries-not-cuts
     Without MASK_PATHS the catalogue passes through with no MASK_EXT column;
-    with it, MASK_EXT is carried for measurement and nothing here removes a
-    star. The PSF-star veto is a setools edit (``MASK_EXT == 0`` beside
-    IMAFLAGS_ISO), not a change here.
+    with it, MASK_EXT is written and nothing here removes a star. The PSF-star
+    veto is the ``MASK_EXT == 0`` cut in star_selection.setools.
 
     """
     sexcat_path = input_file_list[0]

tests/module/test_mask_query.py

@@ -317,7 +317,7 @@ def test_empty_coverage_raises_clearly(tmp_path):
 def test_mask_query_no_maps_is_noop(tmp_path):
     """No MASK_PATHS means a strict pass-through, not a zero column.
 
-    The shipped configs comment MASK_PATHS out, so this is the DEFAULT path.
+    The image-sims configs carry no MASK_PATHS, so this is their path.
     mask_query stays in the MODULE chain either way — setools reads its output
     — so the no-op has to publish a file, and that file must be identical to
     its input with no MASK_EXT column at all (the gating make_cat gives

tests/module/test_setools_star_selection.py

@@ -37,6 +37,7 @@ def test_star_selection_preselects_on_angular_fwhm(tmp_path):
             ("MAG_AUTO", "f8"),
             ("FLAGS", "i4"),
             ("IMAFLAGS_ISO", "i4"),
+            ("MASK_EXT", "i4"),
             ("X_IMAGE", "f8"),
             ("Y_IMAGE", "f8"),
         ],
@@ -66,3 +67,60 @@ def test_star_selection_preselects_on_angular_fwhm(tmp_path):
     ).round(6) or abs(
         float(stats["Mean star fwhm selected (arcsec)"]) - 0.75
     ) < 0.02
+
+
+def _run(catalogue, tmp_path):
+    tools = SETools(catalogue, str(tmp_path), "-test", str(CONFIG), cat_file=False)
+    try:
+        tools.process(_NullLogger())
+    finally:
+        tools._config_file.close()
+    return tools.mask["star_selection"]
+
+
+def _crowded_ccd(mask_ext):
+    """30 stars at 0.75" and 60 compact extended sources at 1.05".
+
+    The extended sources outnumber the stars, as M33's HII regions and clusters
+    do on its CCDs, and sit inside the preselection's FWHM range.
+    """
+    rng = np.random.default_rng(2)
+    n_star, n_ext = 30, 60
+    fwhm_arcsec = np.r_[np.full(n_star, 0.75), np.full(n_ext, 1.05)]
+    pixel_scale = rng.uniform(0.185, 0.188, fwhm_arcsec.size)
+    fwhm_image = fwhm_arcsec / pixel_scale + rng.uniform(-0.05, 0.05, fwhm_arcsec.size)
+    catalogue = np.zeros(
+        fwhm_image.size,
+        dtype=[
+            ("FWHM_IMAGE", "f8"),
+            ("FWHM_WORLD", "f8"),
+            ("MAG_AUTO", "f8"),
+            ("FLAGS", "i4"),
+            ("IMAFLAGS_ISO", "i4"),
+            ("MASK_EXT", "i4"),
+            ("X_IMAGE", "f8"),
+            ("Y_IMAGE", "f8"),
+        ],
+    )
+    catalogue["FWHM_IMAGE"] = fwhm_image
+    catalogue["FWHM_WORLD"] = fwhm_image * pixel_scale / 3600.0
+    catalogue["MAG_AUTO"] = rng.uniform(19.0, 20.5, fwhm_image.size)
+    catalogue["X_IMAGE"] = rng.uniform(0, 2048, fwhm_image.size)
+    catalogue["Y_IMAGE"] = rng.uniform(0, 4612, fwhm_image.size)
+    catalogue["MASK_EXT"][n_star:] = mask_ext
+    return catalogue, np.arange(fwhm_image.size) < n_star
+
+
+def test_extended_sources_capture_the_locus_without_the_manual_mask(tmp_path):
+    """Outlier rejection alone selects the extended sources, not the stars."""
+    catalogue, is_star = _crowded_ccd(mask_ext=0)
+    selected = _run(catalogue, tmp_path)
+    assert selected[~is_star].all()
+    assert not selected[is_star].any()
+
+
+def test_manual_mask_restores_the_stellar_locus(tmp_path):
+    """MASK_EXT != 0 keeps extended sources out of the locus and the sample."""
+    catalogue, is_star = _crowded_ccd(mask_ext=1)
+    selected = _run(catalogue, tmp_path)
+    np.testing.assert_array_equal(selected, is_star)

tests/unit/test_decisions.py

@@ -630,11 +630,18 @@ def test_repository_decisions_have_sites_and_all_values_match():
         ),
         (
             "workflow/config/cfis/star_selection.setools",
-            "[MASK:star_selection]\n",
-            "[MASK:star_selection]\nMASK_EXT == 0\n",
+            "IMAFLAGS_ISO == 0\nMASK_EXT == 0\n\n[MASK:fwhm_mag_cut]",
+            "IMAFLAGS_ISO == 0\n\n[MASK:fwhm_mag_cut]",
             "masking.psf_star_mask_veto",
             "MASK_EXT",
         ),
+        (
+            "workflow/config/cfis/config_exp_mccd.ini",
+            "\nMASK_PATHS = ",
+            "\n; MASK_PATHS = ",
+            "masking.psf_star_mask_veto",
+            "MASK_PATHS",
+        ),
         (
             "workflow/config/cfis/default_tile.sex",
             "SATUR_KEY",

tests/unit/test_parse_guards.py

@@ -89,3 +89,41 @@ def test_root_guard_runs_on_the_raw_flags():
     compute phase, and the prepare parse should already refuse."""
     assert _called('refuse_one_root_cleaners(PRODUCTS_DIR, RUN_DIR, '
                    'flag(config.get("clean")),')
+
+
+# --- masks: every chain that queries them needs inputs.masks -------------------
+
+@pytest.mark.parametrize(
+    "input_type, psf_model, n_users",
+    [("data", "psfex", 2), ("data", "mccd", 2),
+     ("image_sims", "psfex", 1), ("image_sims", "mccd", 1),
+     ("image_sims", "fake", 0)],
+)
+def test_mask_users(input_type, psf_model, n_users):
+    assert len(_lift("mask_users")(input_type, psf_model)) == n_users
+
+
+def test_sims_with_stars_without_masks_are_refused():
+    users = _lift("mask_users")("image_sims", "psfex")
+    with pytest.raises(WorkflowError) as exc:
+        _lift("refuse_missing_masks")(users, None, "here")
+    assert "config_exp_psfex.ini" in str(exc.value)
+
+
+def test_fake_psf_sims_need_no_masks():
+    users = _lift("mask_users")("image_sims", "fake")
+    _lift("refuse_missing_masks")(users, None, "here")
+
+
+def test_sims_share_the_masked_star_selection():
+    """The reason sims with stars need masks: they run the data star selection."""
+    sims = REPO_ROOT / "workflow/config/cfis_image_sims"
+    data = REPO_ROOT / "workflow/config/cfis"
+    for name in ("star_selection.setools", "config_exp_psfex.ini", "config_exp_mccd.ini"):
+        assert (sims / name).resolve() == (data / name).resolve(), name
+    assert "MASK_EXT == 0" in (data / "star_selection.setools").read_text()
+
+
+def test_mask_guard_runs_on_the_parsed_run():
+    assert _called("MASK_USERS = mask_users(INPUT_TYPE, PSF_MODEL)")
+    assert _called("refuse_missing_masks(")

workflow/README.md

@@ -308,12 +308,12 @@ profiles/nibi/config.yaml  SLURM executor; apptainer SDM; per-user jobs cap; kee
   healsparse maps and QUERIED once per object: the `mask_query` module writes a
   `MASK_EXT` column onto each CCD's detection catalogue (inside `exp_psf`), and
   `make_cat` writes one `MASK_<band>` column per band onto the final catalogue
-  (inside `tile_make_cat`). Neither is cut on in the pipeline: instrument flags
-  mark corrupted measurements and are the only masks that reject anything here,
-  while the healsparse masks are location flags and every decision about them is
-  downstream. On exposures the query ships off — `MASK_PATHS` is commented out,
-  making `mask_query` a no-op pass-through — so turning it on is a config edit,
-  not a chain edit. Map paths are config, not
+  (inside `tile_make_cat`). Instrument flags mark corrupted measurements and
+  reject objects here; the healsparse masks are location flags, and decisions
+  about them are downstream, with one exception: PSF-star selection rejects
+  candidates on the UNIONS manual mask (bit 8, `MASK_EXT == 0`), because
+  extended objects such as M33 otherwise enter the PSF model itself. The final
+  catalogue's `MASK_<band>` columns are never cut on. Map paths are config, not
   code, so regenerated products cost a config edit. Nothing is fetched from a
   catalogue server, staged, or rasterized, which is why the old
   `star_catalogue` / `exp_star_cat` / `exp_mask` rules and their cache root are
@@ -340,9 +340,10 @@ profiles/nibi/config.yaml  SLURM executor; apptainer SDM; per-user jobs cap; kee
     --fragment "$products/exp/${exp:0:2}/$exp/maps/maps-$exp.hsp" \
     --manifest "$products/exp/${exp:0:2}/$exp/manifests/exp_maps.json"
   ```
-- **External masks are wired, on the tile side only (data runs).** `inputs.masks`
-  is a third input root beside tiles and exposures, set per machine in the
-  `machines:` table, exported as `$SP_INPUT_MASKS` and
+- **External masks are wired.** `inputs.masks` is a third input root beside
+  tiles and exposures, set per machine in the `machines:` table, required for
+  data runs and for any run that selects PSF stars (`psf_model` psfex or mccd,
+  simulations included), exported as `$SP_INPUT_MASKS` and
   pointing at the UNIONS DR6 ugriz bit ladder: one boolean healsparse map per
   bit, nside 131072, `True` = masked. `config_tile_Mc.ini` names all 11 of them
   in `MASK_EXT_PATHS` under a `<flag value>_<name>` label, so `make_cat`