Repository navigation
cosmo_val: pair-count shape noise in the OneCovariance covariance - #432
Draft
cailmdaley wants to merge 13 commits into
Draft
cailmdaley wants to merge 13 commits into
cailmdaley wants to merge 13 commits into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5YaFeu6oHriuZyFwNAbPd
cailmdaley
marked this pull request as ready for review
October 28, 2026 18:48
# Conflicts: # workflow/rules/covariance.smk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For w_i=c, Q=c²W exactly, so one geometry traversal supplies both moments. Real-data impact: nonuniform-weight catalogues are unchanged; uniform-weight catalogues avoid a redundant traversal. All 14 pair-count tests pass inside the image on the allocation.
The previous commit also carried the catalogue-metadata invalidation helper and the painted-mock README section, which belong to their own PRs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley
marked this pull request as draft
October 30, 2026 23:02
cailmdaley
pushed a commit
that referenced
this pull request
Oct 30, 2026
…entry A tomographic covariance reads cov_th.tomo (per-bin n_e, sigma_e) and the entry's multi-bin n(z). A non-tomographic covariance of an entry that declares shear.tomo_bin_col reads the whole-sample cov_th and the bin-summed n(z) written by nz_bin_sum. Without this, such an entry's non-tomographic covariance was silently built on the 4-bin n(z). From run/tomo1030 (80d2bc2, 5e5f313), without #432's pair-count inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
covariance.shape_noise governs the non-tomographic covariance. A config that sets pair_counts explicitly, as the cosmo_val template does, now builds its tomographic covariances analytic instead of refusing them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
cailmdaley
added a commit
that referenced
this pull request
Oct 31, 2026
…stic per bin pair (#447) * Order NaMaster covariance by SACC point labels Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Write multiple bin pairs for SACC statistics Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Seal tomographic pseudo-Cl pairs in one SACC part Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Derive tomographic COSEBIs and pure E/B covariance Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Write tomographic rho and tau SACC parts Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Wire tomographic statistics into workflow assembly Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Restore pair-native SACC writer documentation Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Test OneCovariance ordering through SACC assembly Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Test tomographic jackknife and assembled covariance order Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Make pseudo-Cl measurements private and retain only sealed spectra Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Derive tomographic covariance dimensions from block FITS Co-Authored-By: GPT-6.1 Terra <noreply@openai.com> * Reuse tomographic ρ only when its cache records the current measurement The PSF-only ρ shared across source bins was reused whenever its FITS and covariance existed, bypassing the measurement metadata check, so a cache from other means settings was republished. Drop the patched-correlation copy fed to estimate_multi_cov: TreeCorr builds jackknife realisations from the per-patch results, which the means pass leaves intact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9FFN33ALe6iizjiJMa * Allow BLAS round-off in the derived-covariance workflow test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9FFN33ALe6iizjiJMa * Tomographic and whole-sample OneCovariance inputs from one catalogue entry A tomographic covariance reads cov_th.tomo (per-bin n_e, sigma_e) and the entry's multi-bin n(z). A non-tomographic covariance of an entry that declares shear.tomo_bin_col reads the whole-sample cov_th and the bin-summed n(z) written by nz_bin_sum. Without this, such an entry's non-tomographic covariance was silently built on the 4-bin n(z). From run/tomo1030 (80d2bc2, 5e5f313), without #432's pair-count inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT * Refuse a tomographic covariance built from whole-sample cov_th For an entry that declares shear.tomo_bin_col, a _tomo covariance without cov_th.tomo got single whole-sample n_e and sigma_e standing in for every bin. It now raises at DAG build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT * Refuse a bin sum of unit-normalised per-bin n(z) The whole-sample n(z) is the sum of the bins' columns, so each column must carry its bin's weight fraction. A file whose every column integrates to one would weight the bins equally; nz_bin_sum refuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT * Read n_e as a float in the sigma_e convention test get_onecov_cat_params returns per-bin comma-separated strings for both sigma_e and n_e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT --------- Co-authored-by: GPT-6.1 Terra <noreply@openai.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Cail Daley <cdaley@l1.nibi.sharcnet>
…t-noise # Conflicts: # workflow/rules/covariance.smk
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured pair-count shape noise now covers every tomographic bin pair as well as the whole sample.
On the blinded v2 tomographic data, pair-count noise doesn't rescue the joint ξ−B null test: PTE 0.0045, against 0.0046 with analytic noise. Analytic/measured noise medians are 0.973–1.006 at 12–83′.
This remains a draft pending validation on GLASS mocks with a known answer (#443).
OneCovariance's uniform-area estimate misses mask holes and spatial weight variation.
The workflow measures W = Σwᵢwⱼ and Q = Σ(wᵢwⱼ)² from positions and weights, then supplies effective counts through its tabulated-pair input.
For tomography it also reads the bin-ID field, excludes nonpositive IDs, and counts every auto and cross pair separately on each covariance grid.
covariance.shape_noise: pair_countsis the cosmo_val default for both whole-sample and tomographic covariances;analyticrestores the uniform-area estimate.shear.tomo_bin_col, with files in upper-triangle bin order; the covariance retains the binned sample'scov_th.tomoarea and mask.The mixed, sample-variance, non-Gaussian and SSC terms are unchanged.
The normalization differs for autos and crosses.
OneCovariance takes per-component dispersions sᵢ.
Auto-bin tables contain 2W²/Q; disjoint cross-bin tables contain W²/Q, because only one Kronecker contraction survives.
Both give the independent-shape variance 2sᵢ²sⱼ²Q/W².
Doubling the cross count would halve its noise incorrectly.
The derivation and cache conventions are in docs/pair-count-noise.md.
The tests include explicit per-bin-pair weighted answers, shape-free HDF5 reads, DAG dependencies and bin-pair file order, analytic opt-out, and footprint inputs.
An opt-in probe executes the real OneCovariance table reader and real-space noise kernel with unequal bin dispersions; its Fourier cosmological terms are stubbed to zero.
It reproduces all auto/cross answers and vanishing cross-pair and ξ+/ξ− pure-noise covariance against candide OneCovariance
70265be3.It does not validate the full cosmological integration.
Existing non-tomographic full-v2 evidence
The analytic/measured shape-noise variance ratio is 0.809–0.856 (median 0.839) over 12–83′ on the full v2 area.
The broad deficit persists at full area; the pilot's extra large-scale decline reflects its boundaries.
With fixed full-v2 B vectors at 12–83′ and only the noise diagonal substituted:
These are controlled noise-diagonal replacements, not fresh full-area OneCovariance runs.
They do not validate mask-dependent mixed or sample-variance terms, and they do not establish the tomographic correction.
Closes #431.
— Codex on behalf of Cail