diff --git a/astra.yaml b/astra.yaml index e336ef639..43a004589 100644 --- a/astra.yaml +++ b/astra.yaml @@ -342,34 +342,52 @@ analyses: corrupted, so acting on them is an analysis decision. Rasterising would bake one mask version into every shape. psf_star_mask_veto: - label: PSF-star candidates rejected on instrument flags only + label: PSF-star candidates rejected on instrument flags and the manual mask rationale: >- - Of the masks, star selection cuts only on IMAFLAGS_ISO. mask_query sits - between SExtractor and setools in the exposure chain; when - MASK_PATHS names maps it writes MASK_EXT (0 clean, nonzero flagged, - off-coverage clean) onto each CCD's catalogue. MASK_PATHS ships - commented out, so the module passes catalogues through without - MASK_EXT. The intended map is the UNIONS star-body product (bit 2); - halo bits 0 and 1 are left out because halos say nothing about - whether a star is a good PSF sample. + Star selection cuts on IMAFLAGS_ISO and on MASK_EXT from the UNIONS + manual mask (bit 8: large galaxies, globular clusters and other + extended objects). mask_query sits between SExtractor and setools in + the exposure chain and writes MASK_EXT (0 clean, nonzero flagged, + off-coverage clean) onto each CCD's catalogue. Outlier rejection + alone copes with crowded star fields: in run 1, M13, M92 and the + Draco and Ursa Minor dwarf spheroidals are 3-15x overdense in PSF + stars with no oversized candidates, and normal-size stars fit the + model as well as the surrounding field does. It fails where compact + extended sources outnumber stars on a CCD. Inside 0.15 deg of M33, + 24% of selected PSF stars are more than 1.2x the PSF size, and + normal-size stars come out 2.4% smaller than the PSF model, so the + contaminants inflate the model itself. That region carries most of + the first-bin rho-statistics variance and flips the sign of the + fitted leakage coefficients. The manual mask removes 1,024 of the + 1,025 oversized candidates within 1 deg of M33 and every radius + where the model is biased. It costs 0.5% of PSF stars survey-wide + (CosmoStat/shapepipe#993). Halo bits 0 and 1 stay out because a + star inside another star's halo is not thereby a bad PSF sample. Values: MASK:star_selection.IMAFLAGS_ISO = "== 0"; MASK:preselect.IMAFLAGS_ISO = "== 0"; MASK:flag.IMAFLAGS_ISO = "== 0"; - MASK:star_selection.MASK_EXT = absent; - MASK:preselect.MASK_EXT = absent; - MASK:flag.MASK_EXT = absent. - default: instrument_flags_only + MASK:star_selection.MASK_EXT = "== 0"; + MASK:preselect.MASK_EXT = "== 0"; + MASK:flag.MASK_EXT = "== 0"; + config_exp_psfex.ini#MASK_QUERY_RUNNER.MASK_PATHS = $SP_INPUT_MASKS/mask_ugriz_nside131072_n8.hsp; + config_exp_mccd.ini#MASK_QUERY_RUNNER.MASK_PATHS = $SP_INPUT_MASKS/mask_ugriz_nside131072_n8.hsp. + default: manual_mask_veto options: + manual_mask_veto: + label: IMAFLAGS_ISO == 0 and MASK_EXT == 0 on the manual mask (bit 8) instrument_flags_only: label: IMAFLAGS_ISO == 0; no sky map queried + description: >- + Leaves outlier rejection to handle every sky-fixed contaminant. + Run 1 ran this way; M33's HII regions and clusters entered the + PSF model. star_body_veto: - label: Also reject candidates on the star-body map (MASK_EXT == 0) + label: Also reject candidates on the star-body map (bit 2) description: >- - Set MASK_PATHS to the star-body map and add MASK_EXT == 0 beside - each IMAFLAGS_ISO cut in star_selection.setools (one line per - mask block). Querying without the cut records MASK_EXT and - changes no star. + Add the star-body map to MASK_PATHS. It flags 12.3% of run-1 PSF + stars, because it masks star bodies, the objects star selection + is meant to keep. star_body_and_halo_veto: label: Also reject candidates inside star halos excluded: true diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index eb5586dc3..1fb9efcec 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -214,7 +214,7 @@ consumed by *querying them at object positions*, never by rasterizing them onto pixels. Two modules do the querying, from the same shared lookup (`shapepipe.utilities.mask_query`): `mask_query` runs between `sextractor` and `setools` on the single-exposure single-CCD catalogues and writes one integer -`MASK_EXT` column (0 = clean), recording the star-body map (bit 2) against every +`MASK_EXT` column (0 = clean), recording the manual mask (bit 8) against every detection; `make_cat` writes one `MASK_` column per band onto the final tile catalogue, carrying the map value verbatim so downstream selections choose their own cuts. Map paths and @@ -228,16 +228,16 @@ these are the only masks that reject anything inside the pipeline. The sits, not that its pixels are broken, so what to do about one is an analysis decision and is made downstream. -**Nothing in the pipeline cuts on the queried columns**, and on exposures the -query ships off entirely: `MASK_PATHS` is commented out, which makes -`mask_query` a strict no-op that passes the catalogue through with no -`MASK_EXT` column (the module stays in the chain, so enabling it is -uncommenting one line). `star_selection.setools` rejects on `IMAFLAGS_ISO == 0` -and nothing else, deliberately starting from outlier rejection alone, and the -final catalogue's `MASK_` columns are written unfiltered. `MASK_EXT` is -the configurable pickup if outlier rejection proves insufficient: add -`MASK_EXT == 0` beside each `IMAFLAGS_ISO == 0`, one line per mask block, as -that file's header documents. +**One queried mask cuts inside the pipeline: the manual mask, on PSF stars.** +`star_selection.setools` rejects candidates with `IMAFLAGS_ISO != 0` or +`MASK_EXT != 0`. Outlier rejection handles crowded star fields on its own, but +where compact extended sources outnumber stars on a CCD (the HII regions and +clusters of M33) they pass selection and inflate the PSF model, so the manual +mask, which covers those objects, keeps them out of the PSF fit +([#993](https://github.com/CosmoStat/shapepipe/issues/993)). The final +catalogue's `MASK_` columns are written unfiltered. Without `MASK_PATHS`, +`mask_query` is a strict no-op that writes no `MASK_EXT` column, so dropping +the map means dropping the `MASK_EXT` lines in `star_selection.setools` too. No internet access is needed at any point, and there is no reference star catalogue to download. diff --git a/src/shapepipe/modules/mask_query_package/__init__.py b/src/shapepipe/modules/mask_query_package/__init__.py index 30d2a53dd..928896aca 100644 --- a/src/shapepipe/modules/mask_query_package/__init__.py +++ b/src/shapepipe/modules/mask_query_package/__init__.py @@ -24,8 +24,9 @@ **Healsparse masks** are sky-fixed LOCATION flags — a star halo, a manual region, a band with no data. They say where an object is, not that its pixels are broken, so what to do about one is an analysis decision. They are queried -into columns and every rejection happens downstream: ``MASK_EXT`` here, -``MASK_`` in ``make_cat``. Nothing in the pipeline cuts on either. +into columns: ``MASK_EXT`` here, ``MASK_`` in ``make_cat``. The one +in-pipeline cut on them is PSF-star selection, which rejects candidates with +``MASK_EXT != 0``; the ``MASK_`` columns are never cut on. Description =========== @@ -48,24 +49,16 @@ ``< > <= >= == !=`` — no bitwise operators — so any bit selection has to happen here, leaving the config a plain test for zero. -Off by default, and a no-op when off -==================================== +A no-op without maps +==================== -``MASK_PATHS`` ships COMMENTED OUT. With no maps configured the module is a -strict no-op: the catalogue is copied through unchanged, with no ``MASK_EXT`` -column at all — the same gating ``make_cat`` gives ``MASK_EXT_PATHS``. It stays -in the ``MODULE`` chain either way, so enabling the query is uncommenting one -line and never editing the chain. - -Even with maps configured, nothing cuts on the result. The PSF star selection -deliberately starts from outlier rejection alone, rejecting only on the -instrument flags, and ``MASK_EXT`` is the configurable pickup if that proves -insufficient. Writing the column without cutting on it is what lets the effect -of a mask on the star sample be *measured* before it is imposed. - -That pickup is one line: add ``MASK_EXT == 0`` beside each -``IMAFLAGS_ISO == 0`` in ``star_selection.setools``. The escape hatch is -deliberate, and that file's header says so. +With no ``MASK_PATHS`` the module is a strict no-op: the catalogue is copied +through unchanged, with no ``MASK_EXT`` column at all — the same gating +``make_cat`` gives ``MASK_EXT_PATHS``. It stays in the ``MODULE`` chain either +way. The shipped data configs name the UNIONS manual mask (bit 8), and +``star_selection.setools`` rejects ``MASK_EXT != 0`` in its ``preselect``, +``flag`` and ``star_selection`` blocks; a config without the map must drop those +lines too, because ``setools`` cannot cut on a missing column. The lookup itself lives in :mod:`shapepipe.utilities.mask_query`, shared with ``make_cat``'s per-band ``MASK_`` columns, so the healsparse primitive is @@ -74,12 +67,14 @@ What gets queried is deliberately narrow ======================================== -``MASK_PATHS`` is a *list of maps to record against each PSF-star candidate*, -not a list of every mask that exists. The committed configs name exactly one -map — the UNIONS star-body product (bit 2). Halo bits 0 and 1 are excluded on -purpose: halos flag objects for the final catalogue, they say nothing about -whether a star is a good PSF sample (mask-force telecon, 2026-07-21). MaxiMask -is not queried here either. +``MASK_PATHS`` is a *list of maps that veto PSF-star candidates*, not a list +of every mask that exists. The committed configs name exactly one map — the +UNIONS manual mask (bit 8), which covers large galaxies, globular clusters and +other extended objects. Halo bits 0 and 1 are excluded on purpose: halos flag +objects for the final catalogue, they say nothing about whether a star is a +good PSF sample (mask-force telecon, 2026-07-21). The star-body map (bit 2) is +excluded because it flags the stars themselves. MaxiMask is not queried here +either. Widening it costs a config edit and no code — add a path. That is why the contract is a path list rather than a bit mask: the UNIONS products are one diff --git a/src/shapepipe/modules/mask_query_runner.py b/src/shapepipe/modules/mask_query_runner.py index 5f2290393..1a74f8f1c 100644 --- a/src/shapepipe/modules/mask_query_runner.py +++ b/src/shapepipe/modules/mask_query_runner.py @@ -30,9 +30,8 @@ def mask_query_runner( @sc [decision:masking.psf_star_mask_veto,label:scope] mask-query-carries-not-cuts Without MASK_PATHS the catalogue passes through with no MASK_EXT column; - with it, MASK_EXT is carried for measurement and nothing here removes a - star. The PSF-star veto is a setools edit (``MASK_EXT == 0`` beside - IMAFLAGS_ISO), not a change here. + with it, MASK_EXT is written and nothing here removes a star. The PSF-star + veto is the ``MASK_EXT == 0`` cut in star_selection.setools. """ sexcat_path = input_file_list[0] diff --git a/tests/module/test_mask_query.py b/tests/module/test_mask_query.py index f52d2e669..a13f8467d 100644 --- a/tests/module/test_mask_query.py +++ b/tests/module/test_mask_query.py @@ -317,7 +317,7 @@ def test_empty_coverage_raises_clearly(tmp_path): def test_mask_query_no_maps_is_noop(tmp_path): """No MASK_PATHS means a strict pass-through, not a zero column. - The shipped configs comment MASK_PATHS out, so this is the DEFAULT path. + The image-sims configs carry no MASK_PATHS, so this is their path. mask_query stays in the MODULE chain either way — setools reads its output — so the no-op has to publish a file, and that file must be identical to its input with no MASK_EXT column at all (the gating make_cat gives diff --git a/tests/module/test_setools_star_selection.py b/tests/module/test_setools_star_selection.py index ad2b64fe7..561589d68 100644 --- a/tests/module/test_setools_star_selection.py +++ b/tests/module/test_setools_star_selection.py @@ -37,6 +37,7 @@ def test_star_selection_preselects_on_angular_fwhm(tmp_path): ("MAG_AUTO", "f8"), ("FLAGS", "i4"), ("IMAFLAGS_ISO", "i4"), + ("MASK_EXT", "i4"), ("X_IMAGE", "f8"), ("Y_IMAGE", "f8"), ], @@ -66,3 +67,60 @@ def test_star_selection_preselects_on_angular_fwhm(tmp_path): ).round(6) or abs( float(stats["Mean star fwhm selected (arcsec)"]) - 0.75 ) < 0.02 + + +def _run(catalogue, tmp_path): + tools = SETools(catalogue, str(tmp_path), "-test", str(CONFIG), cat_file=False) + try: + tools.process(_NullLogger()) + finally: + tools._config_file.close() + return tools.mask["star_selection"] + + +def _crowded_ccd(mask_ext): + """30 stars at 0.75" and 60 compact extended sources at 1.05". + + The extended sources outnumber the stars, as M33's HII regions and clusters + do on its CCDs, and sit inside the preselection's FWHM range. + """ + rng = np.random.default_rng(2) + n_star, n_ext = 30, 60 + fwhm_arcsec = np.r_[np.full(n_star, 0.75), np.full(n_ext, 1.05)] + pixel_scale = rng.uniform(0.185, 0.188, fwhm_arcsec.size) + fwhm_image = fwhm_arcsec / pixel_scale + rng.uniform(-0.05, 0.05, fwhm_arcsec.size) + catalogue = np.zeros( + fwhm_image.size, + dtype=[ + ("FWHM_IMAGE", "f8"), + ("FWHM_WORLD", "f8"), + ("MAG_AUTO", "f8"), + ("FLAGS", "i4"), + ("IMAFLAGS_ISO", "i4"), + ("MASK_EXT", "i4"), + ("X_IMAGE", "f8"), + ("Y_IMAGE", "f8"), + ], + ) + catalogue["FWHM_IMAGE"] = fwhm_image + catalogue["FWHM_WORLD"] = fwhm_image * pixel_scale / 3600.0 + catalogue["MAG_AUTO"] = rng.uniform(19.0, 20.5, fwhm_image.size) + catalogue["X_IMAGE"] = rng.uniform(0, 2048, fwhm_image.size) + catalogue["Y_IMAGE"] = rng.uniform(0, 4612, fwhm_image.size) + catalogue["MASK_EXT"][n_star:] = mask_ext + return catalogue, np.arange(fwhm_image.size) < n_star + + +def test_extended_sources_capture_the_locus_without_the_manual_mask(tmp_path): + """Outlier rejection alone selects the extended sources, not the stars.""" + catalogue, is_star = _crowded_ccd(mask_ext=0) + selected = _run(catalogue, tmp_path) + assert selected[~is_star].all() + assert not selected[is_star].any() + + +def test_manual_mask_restores_the_stellar_locus(tmp_path): + """MASK_EXT != 0 keeps extended sources out of the locus and the sample.""" + catalogue, is_star = _crowded_ccd(mask_ext=1) + selected = _run(catalogue, tmp_path) + np.testing.assert_array_equal(selected, is_star) diff --git a/tests/unit/test_decisions.py b/tests/unit/test_decisions.py index 8c0f39766..0aa79a2b3 100644 --- a/tests/unit/test_decisions.py +++ b/tests/unit/test_decisions.py @@ -630,11 +630,18 @@ def test_repository_decisions_have_sites_and_all_values_match(): ), ( "workflow/config/cfis/star_selection.setools", - "[MASK:star_selection]\n", - "[MASK:star_selection]\nMASK_EXT == 0\n", + "IMAFLAGS_ISO == 0\nMASK_EXT == 0\n\n[MASK:fwhm_mag_cut]", + "IMAFLAGS_ISO == 0\n\n[MASK:fwhm_mag_cut]", "masking.psf_star_mask_veto", "MASK_EXT", ), + ( + "workflow/config/cfis/config_exp_mccd.ini", + "\nMASK_PATHS = ", + "\n; MASK_PATHS = ", + "masking.psf_star_mask_veto", + "MASK_PATHS", + ), ( "workflow/config/cfis/default_tile.sex", "SATUR_KEY", diff --git a/tests/unit/test_parse_guards.py b/tests/unit/test_parse_guards.py index ba7397a01..d1650cd10 100644 --- a/tests/unit/test_parse_guards.py +++ b/tests/unit/test_parse_guards.py @@ -89,3 +89,41 @@ def test_root_guard_runs_on_the_raw_flags(): compute phase, and the prepare parse should already refuse.""" assert _called('refuse_one_root_cleaners(PRODUCTS_DIR, RUN_DIR, ' 'flag(config.get("clean")),') + + +# --- masks: every chain that queries them needs inputs.masks ------------------- + +@pytest.mark.parametrize( + "input_type, psf_model, n_users", + [("data", "psfex", 2), ("data", "mccd", 2), + ("image_sims", "psfex", 1), ("image_sims", "mccd", 1), + ("image_sims", "fake", 0)], +) +def test_mask_users(input_type, psf_model, n_users): + assert len(_lift("mask_users")(input_type, psf_model)) == n_users + + +def test_sims_with_stars_without_masks_are_refused(): + users = _lift("mask_users")("image_sims", "psfex") + with pytest.raises(WorkflowError) as exc: + _lift("refuse_missing_masks")(users, None, "here") + assert "config_exp_psfex.ini" in str(exc.value) + + +def test_fake_psf_sims_need_no_masks(): + users = _lift("mask_users")("image_sims", "fake") + _lift("refuse_missing_masks")(users, None, "here") + + +def test_sims_share_the_masked_star_selection(): + """The reason sims with stars need masks: they run the data star selection.""" + sims = REPO_ROOT / "workflow/config/cfis_image_sims" + data = REPO_ROOT / "workflow/config/cfis" + for name in ("star_selection.setools", "config_exp_psfex.ini", "config_exp_mccd.ini"): + assert (sims / name).resolve() == (data / name).resolve(), name + assert "MASK_EXT == 0" in (data / "star_selection.setools").read_text() + + +def test_mask_guard_runs_on_the_parsed_run(): + assert _called("MASK_USERS = mask_users(INPUT_TYPE, PSF_MODEL)") + assert _called("refuse_missing_masks(") diff --git a/workflow/README.md b/workflow/README.md index 897e2a204..18c908494 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -308,12 +308,12 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee healsparse maps and QUERIED once per object: the `mask_query` module writes a `MASK_EXT` column onto each CCD's detection catalogue (inside `exp_psf`), and `make_cat` writes one `MASK_` column per band onto the final catalogue - (inside `tile_make_cat`). Neither is cut on in the pipeline: instrument flags - mark corrupted measurements and are the only masks that reject anything here, - while the healsparse masks are location flags and every decision about them is - downstream. On exposures the query ships off — `MASK_PATHS` is commented out, - making `mask_query` a no-op pass-through — so turning it on is a config edit, - not a chain edit. Map paths are config, not + (inside `tile_make_cat`). Instrument flags mark corrupted measurements and + reject objects here; the healsparse masks are location flags, and decisions + about them are downstream, with one exception: PSF-star selection rejects + candidates on the UNIONS manual mask (bit 8, `MASK_EXT == 0`), because + extended objects such as M33 otherwise enter the PSF model itself. The final + catalogue's `MASK_` columns are never cut on. Map paths are config, not code, so regenerated products cost a config edit. Nothing is fetched from a catalogue server, staged, or rasterized, which is why the old `star_catalogue` / `exp_star_cat` / `exp_mask` rules and their cache root are @@ -340,9 +340,10 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee --fragment "$products/exp/${exp:0:2}/$exp/maps/maps-$exp.hsp" \ --manifest "$products/exp/${exp:0:2}/$exp/manifests/exp_maps.json" ``` -- **External masks are wired, on the tile side only (data runs).** `inputs.masks` - is a third input root beside tiles and exposures, set per machine in the - `machines:` table, exported as `$SP_INPUT_MASKS` and +- **External masks are wired.** `inputs.masks` is a third input root beside + tiles and exposures, set per machine in the `machines:` table, required for + data runs and for any run that selects PSF stars (`psf_model` psfex or mccd, + simulations included), exported as `$SP_INPUT_MASKS` and pointing at the UNIONS DR6 ugriz bit ladder: one boolean healsparse map per bit, nside 131072, `True` = masked. `config_tile_Mc.ini` names all 11 of them in `MASK_EXT_PATHS` under a `_` label, so `make_cat` diff --git a/workflow/Snakefile b/workflow/Snakefile index bb7c32933..14505890e 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -83,15 +83,6 @@ if _unresolved: f"$variable for machine={MACHINE!r}, " f"input_type={INPUT_TYPE!r}: {', '.join(_unresolved)}. Set them in " f"your run config (SP_RUN_CONFIG).") -# The external masks are a data-only input: config_tile_Mc.ini's -# MASK_EXT_PATHS reads $SP_INPUT_MASKS, and the sims Mc.ini carries no mask -# block. Checked here rather than in run_config.REQUIRED because it depends on -# input_type. -if INPUT_TYPE == "data" and not config.get("inputs", {}).get("masks"): - raise WorkflowError( - f"inputs.masks is unset for machine={MACHINE!r}, input_type='data': " - "the tile chain queries the UNIONS sky-fixed masks (MASK_EXT_PATHS in " - "config/cfis/config_tile_Mc.ini). Set it in the run config.") # The image every job runs in: this user's sandbox, else their cached SIF, # else the run config's `container:`. Checked at parse time, since an empty @@ -128,6 +119,36 @@ if PSF_MODEL == "fake" and not PSF_DICT: raise WorkflowError("psf_model=fake needs `psf_dict:` (the simulation's " "pickled PSF dictionary) in the run config.") +def mask_users(input_type, psf_model): + """Name the chains that query the UNIONS sky-fixed masks ($SP_INPUT_MASKS). + + Data tiles query them per band (MASK_EXT_PATHS in config_tile_Mc.ini; the + sims Mc.ini has no mask block). Every exposure chain that selects PSF stars + vetoes candidates on the manual mask (MASK_PATHS in config_exp_psfex.ini and + config_exp_mccd.ini, which the simulations share through symlinks, together + with star_selection.setools). Only psf_model=fake runs neither. + """ + users = [] + if input_type == "data": + users.append("the tile chain (MASK_EXT_PATHS in config_tile_Mc.ini)") + if psf_model != "fake": + users.append(f"PSF-star selection (MASK_PATHS in config_exp_{psf_model}.ini)") + return users + + +def refuse_missing_masks(users, masks, where): + """Refuse a run whose chains query the masks when inputs.masks is unset.""" + if users and not masks: + raise WorkflowError( + f"inputs.masks is unset for {where}: {' and '.join(users)} " + "query the UNIONS sky-fixed masks. Set it in the run config.") + + +MASK_USERS = mask_users(INPUT_TYPE, PSF_MODEL) +refuse_missing_masks( + MASK_USERS, config.get("inputs", {}).get("masks"), + f"machine={MACHINE!r}, input_type={INPUT_TYPE!r}, psf_model={PSF_MODEL!r}") + # How images reach the run (exported as SP_RETRIEVE for the Git/Gie configs): # `symlink` to a local mirror, or `vos` download. Sims are always local. RETRIEVE_MODES = {"symlink", "vos"} @@ -1073,7 +1094,7 @@ def unit_pre(stage, unit, *, exp_name=None, forest=None, env=None, ] if PSF_DICT: lines.append(f"export PSF_DICT='{PSF_DICT}'") - if INPUT_TYPE == "data": + if MASK_USERS: lines.append(f"export SP_INPUT_MASKS='{INPUTS['masks']}'") if forest: lines.append(f"export SP_EXP='{forest}'") diff --git a/workflow/config.yaml b/workflow/config.yaml index d2f0c37bb..ac693e318 100644 --- a/workflow/config.yaml +++ b/workflow/config.yaml @@ -61,11 +61,13 @@ input_types: # - tile_list: text file with tile IDs # - retrieve: method to get input data, allowed are symlink, vos # - inputs.tiles, .exposures: path for input tile and exposure -# - inputs.masks (data only): the UNIONS sky-fixed masks, one boolean -# healsparse map per bit of the band-combined ugriz mask (nside 131072, -# True = masked), queried per object by make_cat -- never rasterized. -# Consumed by MASK_EXT_PATHS in config/cfis/config_tile_Mc.ini, which -# carries the bit table. Required for input_type: data. +# - inputs.masks: the UNIONS sky-fixed masks, one boolean healsparse map per +# bit of the band-combined ugriz mask (nside 131072, True = masked), queried +# per object -- never rasterized. Data tiles consume them through +# MASK_EXT_PATHS in config/cfis/config_tile_Mc.ini, which carries the bit +# table; PSF-star selection vetoes the manual mask (bit 8) through MASK_PATHS +# in the exposure configs, which the simulations share. Required unless +# input_type: image_sims with psf_model: fake (mask_users in the Snakefile). # - inputs.catalogues: UNIONS per-tile catalogues, a local directory # (symlinked in) or a vos: URL (tile_detection: unions_catalogue) # - psf_dict: simulation PSF stamps (psf_model: fake) @@ -109,6 +111,7 @@ machines: # picks the grid by setting `run:` alone. tiles: /n09data/hervas/skills_out/$run/images/SP_tiles exposures: /n09data/hervas/skills_out/$run/images/SP_exp + masks: $base_dir/masks/dr6-2026-08 # Dictionary of PSF model stamps, which psf_model: fake reads in place # of an exposure PSF fit. psf_dict: /home/hervas/fhervas/workdir_skills/input/psf_files/Full_psf_dict.pickle diff --git a/workflow/config/cfis/config_exp_mccd.ini b/workflow/config/cfis/config_exp_mccd.ini index cdfdd9de9..dcf4fde93 100644 --- a/workflow/config/cfis/config_exp_mccd.ini +++ b/workflow/config/cfis/config_exp_mccd.ini @@ -133,6 +133,7 @@ SUFFIX = sexcat MAKE_POST_PROCESS = FALSE +# @sc [decision:masking.psf_star_mask_veto] [MASK_QUERY_RUNNER] INPUT_DIR = $SP_RUN/output/run_sp_exp_SxSePsf/sextractor_runner/output @@ -143,28 +144,21 @@ FILE_PATTERN = sexcat NUMBERING_SCHEME = -0000000-0 -# The PSF-star diet, and it is deliberately NARROW: instrument flags (read by -# SExtractor as IMAFLAGS_ISO) plus the healsparse star-body map (UNIONS bit 2), -# and nothing else. Halo bits 0 and 1 are excluded on purpose — halos flag -# objects for the final catalogue, they do not reject PSF stars — and -# MaxiMask is not in the diet either. Widen it by -# adding paths here; every map that is True (boolean) or nonzero (integer) at a -# detection sets MASK_EXT. Comma-separated. +# The PSF-star sky-mask veto: the UNIONS manual mask (bit 8), which covers +# large galaxies, globular clusters and other extended objects. Every map that +# is True (boolean) or nonzero (integer) at a detection sets MASK_EXT, and +# star_selection.setools rejects candidates with MASK_EXT != 0. Halo bits 0 +# and 1 stay out because halos say nothing about whether a star is a good PSF +# sample; the star-body map (bit 2) stays out because it flags 12% of PSF +# stars. Comma-separated; widening the veto is adding a path. # -# UNSET BY DEFAULT, and the module is a strict no-op without it: the catalogue -# is passed through with no MASK_EXT column, the same gating make_cat gives -# MASK_EXT_PATHS. mask_query stays in the MODULE chain either way, so turning -# the query on is uncommenting one line and never editing the chain. -# -# Even with it set, NOTHING CUTS ON IT: the star selection ships permissive -# (instrument flags only) and the column is carried for transparency and -# measurement — see star_selection.setools' header for the one-line change -# that would impose it. -; MASK_PATHS = $SP_CONFIG/mask_ugriz_nside131072_n4.hsp +# Commenting it out makes the module a strict no-op (no MASK_EXT column), so +# star_selection.setools' MASK_EXT cuts must go with it. +MASK_PATHS = $SP_INPUT_MASKS/mask_ugriz_nside131072_n8.hsp # Optional: restrict integer maps to these bits (value & MASK_BITS). Absent, # any nonzero value flags. Boolean maps — the UNIONS per-bit products, one map -# per bit — ignore it, which is why the diet above is a path list and not a +# per bit — ignore it, which is why the veto above is a path list and not a # bit mask. ; MASK_BITS = 4 diff --git a/workflow/config/cfis/config_exp_psfex.ini b/workflow/config/cfis/config_exp_psfex.ini index 05382bb16..0201cf7b2 100644 --- a/workflow/config/cfis/config_exp_psfex.ini +++ b/workflow/config/cfis/config_exp_psfex.ini @@ -150,28 +150,21 @@ FILE_PATTERN = sexcat NUMBERING_SCHEME = -0000000-0 -# The PSF-star diet, and it is deliberately NARROW: instrument flags (read by -# SExtractor as IMAFLAGS_ISO) plus the healsparse star-body map (UNIONS bit 2), -# and nothing else. Halo bits 0 and 1 are excluded on purpose — halos flag -# objects for the final catalogue, they do not reject PSF stars (mask-force -# telecon, 2026-07-21) — and MaxiMask is not in the diet either. Widen it by -# adding paths here; every map that is True (boolean) or nonzero (integer) at a -# detection sets MASK_EXT. Comma-separated. +# The PSF-star sky-mask veto: the UNIONS manual mask (bit 8), which covers +# large galaxies, globular clusters and other extended objects. Every map that +# is True (boolean) or nonzero (integer) at a detection sets MASK_EXT, and +# star_selection.setools rejects candidates with MASK_EXT != 0. Halo bits 0 +# and 1 stay out because halos say nothing about whether a star is a good PSF +# sample; the star-body map (bit 2) stays out because it flags 12% of PSF +# stars. Comma-separated; widening the veto is adding a path. # -# UNSET BY DEFAULT, and the module is a strict no-op without it: the catalogue -# is passed through with no MASK_EXT column, the same gating make_cat gives -# MASK_EXT_PATHS. mask_query stays in the MODULE chain either way, so turning -# the query on is uncommenting one line and never editing the chain. -# -# Even with it set, NOTHING CUTS ON IT: the star selection ships permissive -# (instrument flags only) and the column is carried for transparency and -# measurement — see star_selection.setools' header for the one-line change -# that would impose it. -; MASK_PATHS = $SP_CONFIG/mask_ugriz_nside131072_n4.hsp +# Commenting it out makes the module a strict no-op (no MASK_EXT column), so +# star_selection.setools' MASK_EXT cuts must go with it. +MASK_PATHS = $SP_INPUT_MASKS/mask_ugriz_nside131072_n8.hsp # Optional: restrict integer maps to these bits (value & MASK_BITS). Absent, # any nonzero value flags. Boolean maps — the UNIONS per-bit products, one map -# per bit — ignore it, which is why the diet above is a path list and not a +# per bit — ignore it, which is why the veto above is a path list and not a # bit mask. ; MASK_BITS = 4 diff --git a/workflow/config/cfis/config_tile_Mc.ini b/workflow/config/cfis/config_tile_Mc.ini index 18429216c..a1a0d2518 100644 --- a/workflow/config/cfis/config_tile_Mc.ini +++ b/workflow/config/cfis/config_tile_Mc.ini @@ -108,9 +108,8 @@ N_EPOCH_SLOTS = 12 # coverage reads boolean False, indistinguishable from clean. ShapePipe cuts # on none of these columns; which to cut on is decided in sp_validation. # -# PSF-star selection deliberately does NOT consume these: MASK_PATHS in -# config_exp_psfex.ini stays commented out, keeping the star diet narrow -# (instrument flags only). See that file's header. +# PSF-star selection queries the manual mask (bit 8) separately, through +# mask_query's MASK_PATHS in config_exp_psfex.ini and config_exp_mccd.ini. # @sc [decision:masking.sky_mask_application] MASK_EXT_PATHS = 1_Faint_star_halos:$SP_INPUT_MASKS/mask_ugriz_nside131072_n1.hsp, diff --git a/workflow/config/cfis/star_selection.setools b/workflow/config/cfis/star_selection.setools index 930600c37..7974400fb 100644 --- a/workflow/config/cfis/star_selection.setools +++ b/workflow/config/cfis/star_selection.setools @@ -1,15 +1,15 @@ ## SETools configuration file for star/galaxy separation based on size/mag properties ## -## The mask cut is IMAFLAGS_ISO == 0: instrument flags identify corrupted -## pixels, while sky-fixed external masks identify locations rather than -## unusable measurements. PSF outlier rejection handles location-dependent -## contamination without an external-mask veto. +## PSF-star candidates are rejected on instrument flags (IMAFLAGS_ISO, corrupted +## pixels) and on the UNIONS manual mask (MASK_EXT, written by mask_query from +## its MASK_PATHS). Outlier rejection copes with crowded star fields, but not +## where compact extended sources outnumber stars on a CCD (M33's HII regions +## and clusters): they shift the stellar locus and enter the PSF model. The +## manual mask covers those objects. ## -## To veto external masks, enable the desired maps in mask_query's MASK_PATHS -## and add MASK_EXT == 0 beside each IMAFLAGS_ISO line below. MASK_PATHS ships -## commented out, so mask_query is a pass-through and writes no MASK_EXT. -## SETools has no bitwise operators; select mask bits in MASK_PATHS. - +## MASK_EXT exists only when mask_query's MASK_PATHS is set; dropping the map +## means dropping the MASK_EXT lines below. SETools has no bitwise operators; +## select mask bits in MASK_PATHS. # @sc [decision:masking.psf_star_mask_veto,decision:star_selection_psf.star_selection_box] [MASK:preselect] @@ -22,6 +22,7 @@ FWHM_WORLD < 1.5 / 3600 FLAGS == 0 IMAFLAGS_ISO == 0 +MASK_EXT == 0 NO_SAVE # @sc [decision:masking.psf_star_mask_veto] @@ -31,6 +32,7 @@ NO_SAVE FLAGS == 0 IMAFLAGS_ISO == 0 +MASK_EXT == 0 NO_SAVE @@ -46,6 +48,7 @@ FWHM_IMAGE >= mode(FWHM_IMAGE{preselect}) - 3 * locus_width(FWHM_IMAGE{preselect FLAGS == 0 # @sc [decision:masking.pixel_mask_source] IMAFLAGS_ISO == 0 +MASK_EXT == 0 [MASK:fwhm_mag_cut] FWHM_IMAGE > 0