Skip to content

cosmo_val: pair-count shape noise in the OneCovariance covariance - #432

Draft
cailmdaley wants to merge 13 commits into
developfrom
cosmo-val-paircount-noise
Draft

cailmdaley wants to merge 13 commits into
developfrom
cosmo-val-paircount-noise

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Measured pair-count shape noise now covers every tomographic bin pair as well as the whole sample.
On the blinded v2 tomographic data, pair-count noise doesn't rescue the joint ξ−B null test: PTE 0.0045, against 0.0046 with analytic noise. Analytic/measured noise medians are 0.973–1.006 at 12–83′.
This remains a draft pending validation on GLASS mocks with a known answer (#443).

OneCovariance's uniform-area estimate misses mask holes and spatial weight variation.
The workflow measures W = Σwᵢwⱼ and Q = Σ(wᵢwⱼ)² from positions and weights, then supplies effective counts through its tabulated-pair input.
For tomography it also reads the bin-ID field, excludes nonpositive IDs, and counts every auto and cross pair separately on each covariance grid.

  • covariance.shape_noise: pair_counts is the cosmo_val default for both whole-sample and tomographic covariances; analytic restores the uniform-area estimate.
  • Tomographic counts use shear.tomo_bin_col, with files in upper-triangle bin order; the covariance retains the binned sample's cov_th.tomo area and mask.
  • The count cache contains no shapes, signals or blind records.
    The mixed, sample-variance, non-Gaussian and SSC terms are unchanged.

The normalization differs for autos and crosses.
OneCovariance takes per-component dispersions sᵢ.
Auto-bin tables contain 2W²/Q; disjoint cross-bin tables contain W²/Q, because only one Kronecker contraction survives.
Both give the independent-shape variance 2sᵢ²sⱼ²Q/W².
Doubling the cross count would halve its noise incorrectly.
The derivation and cache conventions are in docs/pair-count-noise.md.

The tests include explicit per-bin-pair weighted answers, shape-free HDF5 reads, DAG dependencies and bin-pair file order, analytic opt-out, and footprint inputs.
An opt-in probe executes the real OneCovariance table reader and real-space noise kernel with unequal bin dispersions; its Fourier cosmological terms are stubbed to zero.
It reproduces all auto/cross answers and vanishing cross-pair and ξ+/ξ− pure-noise covariance against candide OneCovariance 70265be3.
It does not validate the full cosmological integration.

Existing non-tomographic full-v2 evidence

The analytic/measured shape-noise variance ratio is 0.809–0.856 (median 0.839) over 12–83′ on the full v2 area.
The broad deficit persists at full area; the pilot's extra large-scale decline reflects its boundaries.

Analytic / measured-pair shape-noise variance vs θ

With fixed full-v2 B vectors at 12–83′ and only the noise diagonal substituted:

Statistic dof PTE, analytic PTE, measured pairs
COSEBIs B, 20 modes 20 0.438 0.643
Pure-B ξ₊ 7 0.347 0.480
Pure-B ξ₋ 7 0.072 0.129
Pure-B combined 14 0.157 0.294

B-mode PTEs, analytic vs measured-pair noise

These are controlled noise-diagonal replacements, not fresh full-area OneCovariance runs.
They do not validate mask-dependent mixed or sample-variance terms, and they do not establish the tomographic correction.

Closes #431.

— Codex on behalf of Cail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L5YaFeu6oHriuZyFwNAbPd
@cailmdaley
cailmdaley marked this pull request as ready for review October 28, 2026 18:48
cailmdaley added a commit that referenced this pull request Oct 28, 2026
Resolve covariance.smk against #432: keep #436's per-component pass-through
(sigma_e_convention from #432 is superseded); adapt #432's pair-noise test
and doc to the single convention.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley and others added 5 commits October 30, 2026 01:00
# Conflicts:
#	workflow/rules/covariance.smk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For w_i=c, Q=c²W exactly, so one geometry traversal supplies both moments. Real-data impact: nonuniform-weight catalogues are unchanged; uniform-weight catalogues avoid a redundant traversal. All 14 pair-count tests pass inside the image on the allocation.
The previous commit also carried the catalogue-metadata invalidation helper and
the painted-mock README section, which belong to their own PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley
cailmdaley marked this pull request as draft October 30, 2026 23:02
cailmdaley pushed a commit that referenced this pull request Oct 30, 2026
…entry

A tomographic covariance reads cov_th.tomo (per-bin n_e, sigma_e) and the
entry's multi-bin n(z). A non-tomographic covariance of an entry that
declares shear.tomo_bin_col reads the whole-sample cov_th and the
bin-summed n(z) written by nz_bin_sum. Without this, such an entry's
non-tomographic covariance was silently built on the 4-bin n(z).

From run/tomo1030 (80d2bc2, 5e5f313), without #432's pair-count inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
covariance.shape_noise governs the non-tomographic covariance. A config
that sets pair_counts explicitly, as the cosmo_val template does, now
builds its tomographic covariances analytic instead of refusing them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
cailmdaley added a commit that referenced this pull request Oct 31, 2026
…stic per bin pair (#447)

* Order NaMaster covariance by SACC point labels

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Write multiple bin pairs for SACC statistics

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Seal tomographic pseudo-Cl pairs in one SACC part

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Derive tomographic COSEBIs and pure E/B covariance

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Write tomographic rho and tau SACC parts

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Wire tomographic statistics into workflow assembly

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Restore pair-native SACC writer documentation

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Test OneCovariance ordering through SACC assembly

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Test tomographic jackknife and assembled covariance order

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Make pseudo-Cl measurements private and retain only sealed spectra

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Derive tomographic covariance dimensions from block FITS

Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>

* Reuse tomographic ρ only when its cache records the current measurement

The PSF-only ρ shared across source bins was reused whenever its FITS and
covariance existed, bypassing the measurement metadata check, so a cache
from other means settings was republished. Drop the patched-correlation
copy fed to estimate_multi_cov: TreeCorr builds jackknife realisations from
the per-patch results, which the means pass leaves intact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9FFN33ALe6iizjiJMa

* Allow BLAS round-off in the derived-covariance workflow test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9FFN33ALe6iizjiJMa

* Tomographic and whole-sample OneCovariance inputs from one catalogue entry

A tomographic covariance reads cov_th.tomo (per-bin n_e, sigma_e) and the
entry's multi-bin n(z). A non-tomographic covariance of an entry that
declares shear.tomo_bin_col reads the whole-sample cov_th and the
bin-summed n(z) written by nz_bin_sum. Without this, such an entry's
non-tomographic covariance was silently built on the 4-bin n(z).

From run/tomo1030 (80d2bc2, 5e5f313), without #432's pair-count inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT

* Refuse a tomographic covariance built from whole-sample cov_th

For an entry that declares shear.tomo_bin_col, a _tomo covariance without
cov_th.tomo got single whole-sample n_e and sigma_e standing in for every
bin. It now raises at DAG build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT

* Refuse a bin sum of unit-normalised per-bin n(z)

The whole-sample n(z) is the sum of the bins' columns, so each column
must carry its bin's weight fraction. A file whose every column
integrates to one would weight the bins equally; nz_bin_sum refuses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT

* Read n_e as a float in the sigma_e convention test

get_onecov_cat_params returns per-bin comma-separated strings for both
sigma_e and n_e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT

---------

Co-authored-by: GPT-6.1 Terra <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cail Daley <cdaley@l1.nibi.sharcnet>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cosmo_val: pair-count shape noise in the OneCovariance covariance

1 participant