diff --git a/GOVERNANCE.md b/GOVERNANCE.md index e2f54ad..aa7310a 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -57,6 +57,16 @@ for name, mod in (('views_frames', c), ('views_frames_summarize', s), ('views_fr to any of them, so reading `CONFORMANCE_FLOOR` tells a consumer exactly which contract version its CI asserts. Additive surface (a new law or method) is MINOR and does **not** bump the floor. +- **The floor is not a safe dependency pin (#237).** It versions the conformance + suite, not the behaviour of the operations. Outputs can change above it in a MINOR + (the tower estimators did three times, in 1.2.0, 1.3.0 and 1.9.0), and a behaviour + can arrive in a MAJOR, as donor fill did in 3.0.0. A consumer that relies on a + behaviour should floor its dependency at the release that introduced it, for example + `views-frames>=3.0.0` for reconciliation that places every country's mass. A loose + floor such as `>=1.10.2` lets a lockfile or a resolver keep an old release while + every check stays green: views-datafactory's lock sat at 1.0.0 from June to August + 2026, and on 2026-11-02 the `chrome_box` forecast lane still resolved 1.11.0 a day + after 3.0.0 shipped. ## Versioning (SemVer for a contract) diff --git a/notebooks/03_reconciliation.ipynb b/notebooks/03_reconciliation.ipynb index 45c6ee2..0278bc2 100644 --- a/notebooks/03_reconciliation.ipynb +++ b/notebooks/03_reconciliation.ipynb @@ -96,8 +96,10 @@ "The numpy core: `reconcile_proportional(grid, country)` rescales `grid` `(S, cells)` so\n", "each draw sums to `country` `(S,)`, **top-down by forecast proportion**. Three guarantees,\n", "shown on a tiny explicit example: per-draw conservation, **zeros stay zero**, and the\n", - "**all-zero-draw edge** (a draw where every cell is 0 has no proportions to scale — it stays\n", - "all-zero rather than dividing by zero)." + "**all-zero-draw edge** (a draw where every cell is 0 has no proportions to scale, so this\n", + "function leaves it all-zero rather than dividing by zero). That edge belongs to this low-level\n", + "function only: since 3.0.0, `ReconciliationModule` fills such a draw first when its country\n", + "value is positive (§4b, ADR-030), so the reconciled grid always sums to the country." ] }, { @@ -206,6 +208,48 @@ " print(\"fails loud on a wrong-level input :\", str(e)[:58], \"...\")" ] }, + { + "cell_type": "markdown", + "id": "4e1105a9", + "metadata": {}, + "source": [ + "### 4b · An empty grid with a positive country: donor fill (3.0.0)\n", + "\n", + "Scaling is multiplication, so `reconcile_proportional` cannot place a positive country value\n", + "into a draw whose grid is all zeros. Until 3.0.0 those fatalities were silently lost\n", + "(2.9–7.3% of the mass on the first production run, views-frames#283). The module now fills\n", + "such a draw first: it copies the grid pattern of a random **donor** draw from the same\n", + "`(time, country)`, then scales it as usual. A group with no donor at all raises `ValueError`.\n", + "`reconcile_result` reports how many draws were filled." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "43738044", + "metadata": {}, + "outputs": [], + "source": [ + "from views_frames_reconcile import METHOD_DONOR_FILL\n", + "from views_frames_reconcile.frames import prediction_frame_from_arrays\n", + "\n", + "# one country, 3 cells, 4 draws; draw 1 has an empty grid but a country value of 7\n", + "grid = np.array([[2.0, 0.0, 1.0, 0.0],\n", + " [1.0, 0.0, 0.0, 3.0],\n", + " [0.0, 0.0, 0.0, 0.0]], dtype=np.float32) # (cells, draws)\n", + "pgm_tiny = prediction_frame_from_arrays(\n", + " np.full(3, 500), np.array([11, 12, 13]), grid, level=SpatialLevel.PGM)\n", + "cm_tiny = prediction_frame_from_arrays(\n", + " np.array([500]), np.array([1]), np.array([[3.0, 7.0, 5.0, 6.0]]), level=SpatialLevel.CM)\n", + "keys = np.array([[500, 11], [500, 12], [500, 13]]); vals = np.array([1, 1, 1])\n", + "\n", + "res = ReconciliationModule(keys, vals).reconcile_result(cm_tiny, pgm_tiny)\n", + "print(\"method :\", res.method, res.method == METHOD_DONOR_FILL)\n", + "print(\"draws filled :\", res.filled_draws)\n", + "print(\"every draw sums to its country :\", np.asarray(res.frame.values).sum(axis=0))\n", + "print(\"filled draw 1 copies a donor's zero cells:\", np.asarray(res.frame.values)[:, 1])" + ] + }, { "cell_type": "markdown", "id": "3eea4c0d", @@ -475,7 +519,7 @@ "| You saw | The takeaway |\n", "|---|---|\n", "| `reconcile_proportional` (§2) | per-draw top-down; conserves, preserves zeros, handles the all-zero edge |\n", - "| `ReconciliationModule` (§4) | injected mapping (ADR-014), de-mutated, fail-loud |\n", + "| `ReconciliationModule` (§4) | injected mapping (ADR-014), de-mutated, fail-loud; fills empty-grid draws from a donor (§4b) |\n", "| `assert_reconcile_contract` (§5) | published contract; the mapping is load-bearing |\n", "| `ES(aggregate) ≠ Σ ES(cell)` (§6) | joint sampling; tail risk is subadditive |\n", "| bit-identity to the oracle (§7) | proves *faithful relocation*, **not** method quality |\n", diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index acf22a2..f53f0e8 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -486,7 +486,7 @@ Cross-refs: C-47 (eval provenance kept out of the generic header — the precede | ID | C-99 | | Tier | 1 | | Resolved | 2026-11-02 | -| Resolution | Released in **3.0.0** (ADR-030): an empty-grid draw borrows the pattern of a random donor draw from the same `(time, country)`, and a group with no donor raises. Verified on the real archive (0 unplaced mass) and on the published wheel. | +| Resolution | Released in **3.0.0** (ADR-030): an empty-grid draw borrows the pattern of a random donor draw from the same `(time, country)`, and a group with no donor raises. On the real archive the mass is no longer lost (0 unplaced). That figure is true by construction once a donor fills a draw, so it shows the mass is placed, not that it lands in the right cells. | | Source | views-models' first production run (`chrome_crate` → `chrome_box`, 2026-10-30), reported as #283; downstream investigation views-pipeline-core#579 | | Trigger | Any reconciliation where, for some `(time, country, draw)`, the country value is positive and every grid cell is zero. Common, not exotic: 94–96% of grid cells are zero in a conflict forecast. | | Location | `src/views_frames_reconcile/proportional.py:81-98` (the all-zero guard returns zeros); `src/views_frames_reconcile/conformance.py` (law 4 required it); `docs/CICs/Reconcile.md` §3/§6 ("an edge, not a failure"). | @@ -509,6 +509,12 @@ conformance law (`TestEachLawHasTeeth`). Mutation testing killed 23 of 23 mutati path (#289). On the real 2026-10-30 archive the 3.0.0 reconciler left **0** unplaced mass on all three targets, filling exactly the 546,813 draws measured (#292). **Resolved 2026-11-02:** 3.0.0 is on PyPI (#294), installed from there into a clean venv, and passes `assert_reconcile_contract`. +**What this does not show.** "0 unplaced" holds by construction, so it says nothing about +whether a donor's pattern puts the mass in the right cells; testing that needs the grid +before reconciliation, which the 2026-10-30 archive did not keep (ADR-030, Open Questions). +**Adoption is separate.** The defect is fixed in this package. The production lane that +exposed it still resolved views-frames 1.11.0 on 2026-11-02, because views-datafactory pinned +`views-frames<2` (views-datafactory#527); views-models#663 then raises the lane's floors. --- diff --git a/tests/test_reconciliation_scale.py b/tests/test_reconciliation_scale.py index 992c65c..6bffa29 100644 --- a/tests/test_reconciliation_scale.py +++ b/tests/test_reconciliation_scale.py @@ -60,15 +60,18 @@ def test_many_groups_correct_and_fast(self): # Loose smoke: a reintroduced O(groups x N) grouping would balloon this. assert elapsed < 10.0, f"reconcile unexpectedly slow ({elapsed:.1f}s)" - # Correctness across many groups: conservation holds per draw, except - # all-zero-input draws (which stay zero) — checked on a spread of countries. + # Correctness across many groups: every draw sums to its country (ADR-030), + # including draws whose grid was all zeros, which donor fill places. Checked + # on a spread of countries plus every country that has such a draw, so the + # empty-grid case is always exercised rather than assumed. cmv = {int(u): cm.values[i] for i, u in enumerate(cm.index.unit)} - for c in (1, 137, 1000, 2000): + empty = { + int(c) + for c in np.unique(gcountry) + if (pgm.values[gcountry == c] == 0).all(axis=0).any() + } + assert empty, "the fixture must contain an empty-grid draw to check" + for c in sorted({1, 137, 1000, 2000} | empty): sel = np.nonzero(gcountry == c)[0] - inp_allzero = (pgm.values[sel] == 0).all(axis=0) grid_sum = out.values[sel].sum(axis=0) - active = ~inp_allzero - np.testing.assert_allclose( - grid_sum[active], cmv[c][active], rtol=1e-3, atol=1e-2 - ) - assert (grid_sum[inp_allzero] == 0).all() + np.testing.assert_allclose(grid_sum, cmv[c], rtol=1e-3, atol=1e-2)