Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions GOVERNANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,16 @@ for name, mod in (('views_frames', c), ('views_frames_summarize', s), ('views_fr
to any of them, so reading `CONFORMANCE_FLOOR` tells a consumer exactly which contract
version its CI asserts. Additive surface (a new law or method) is MINOR and does
**not** bump the floor.
- **The floor is not a safe dependency pin (#237).** It versions the conformance
suite, not the behaviour of the operations. Outputs can change above it in a MINOR
(the tower estimators did three times, in 1.2.0, 1.3.0 and 1.9.0), and a behaviour
can arrive in a MAJOR, as donor fill did in 3.0.0. A consumer that relies on a
behaviour should floor its dependency at the release that introduced it, for example
`views-frames>=3.0.0` for reconciliation that places every country's mass. A loose
floor such as `>=1.10.2` lets a lockfile or a resolver keep an old release while
every check stays green: views-datafactory's lock sat at 1.0.0 from June to August
2026, and on 2026-11-02 the `chrome_box` forecast lane still resolved 1.11.0 a day
after 3.0.0 shipped.

## Versioning (SemVer for a contract)

Expand Down
50 changes: 47 additions & 3 deletions notebooks/03_reconciliation.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,10 @@
"The numpy core: `reconcile_proportional(grid, country)` rescales `grid` `(S, cells)` so\n",
"each draw sums to `country` `(S,)`, **top-down by forecast proportion**. Three guarantees,\n",
"shown on a tiny explicit example: per-draw conservation, **zeros stay zero**, and the\n",
"**all-zero-draw edge** (a draw where every cell is 0 has no proportions to scale — it stays\n",
"all-zero rather than dividing by zero)."
"**all-zero-draw edge** (a draw where every cell is 0 has no proportions to scale, so this\n",
"function leaves it all-zero rather than dividing by zero). That edge belongs to this low-level\n",
"function only: since 3.0.0, `ReconciliationModule` fills such a draw first when its country\n",
"value is positive (§4b, ADR-030), so the reconciled grid always sums to the country."
]
},
{
Expand Down Expand Up @@ -206,6 +208,48 @@
" print(\"fails loud on a wrong-level input :\", str(e)[:58], \"...\")"
]
},
{
"cell_type": "markdown",
"id": "4e1105a9",
"metadata": {},
"source": [
"### 4b · An empty grid with a positive country: donor fill (3.0.0)\n",
"\n",
"Scaling is multiplication, so `reconcile_proportional` cannot place a positive country value\n",
"into a draw whose grid is all zeros. Until 3.0.0 those fatalities were silently lost\n",
"(2.9–7.3% of the mass on the first production run, views-frames#283). The module now fills\n",
"such a draw first: it copies the grid pattern of a random **donor** draw from the same\n",
"`(time, country)`, then scales it as usual. A group with no donor at all raises `ValueError`.\n",
"`reconcile_result` reports how many draws were filled."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43738044",
"metadata": {},
"outputs": [],
"source": [
"from views_frames_reconcile import METHOD_DONOR_FILL\n",
"from views_frames_reconcile.frames import prediction_frame_from_arrays\n",
"\n",
"# one country, 3 cells, 4 draws; draw 1 has an empty grid but a country value of 7\n",
"grid = np.array([[2.0, 0.0, 1.0, 0.0],\n",
" [1.0, 0.0, 0.0, 3.0],\n",
" [0.0, 0.0, 0.0, 0.0]], dtype=np.float32) # (cells, draws)\n",
"pgm_tiny = prediction_frame_from_arrays(\n",
" np.full(3, 500), np.array([11, 12, 13]), grid, level=SpatialLevel.PGM)\n",
"cm_tiny = prediction_frame_from_arrays(\n",
" np.array([500]), np.array([1]), np.array([[3.0, 7.0, 5.0, 6.0]]), level=SpatialLevel.CM)\n",
"keys = np.array([[500, 11], [500, 12], [500, 13]]); vals = np.array([1, 1, 1])\n",
"\n",
"res = ReconciliationModule(keys, vals).reconcile_result(cm_tiny, pgm_tiny)\n",
"print(\"method :\", res.method, res.method == METHOD_DONOR_FILL)\n",
"print(\"draws filled :\", res.filled_draws)\n",
"print(\"every draw sums to its country :\", np.asarray(res.frame.values).sum(axis=0))\n",
"print(\"filled draw 1 copies a donor's zero cells:\", np.asarray(res.frame.values)[:, 1])"
]
},
{
"cell_type": "markdown",
"id": "3eea4c0d",
Expand Down Expand Up @@ -475,7 +519,7 @@
"| You saw | The takeaway |\n",
"|---|---|\n",
"| `reconcile_proportional` (§2) | per-draw top-down; conserves, preserves zeros, handles the all-zero edge |\n",
"| `ReconciliationModule` (§4) | injected mapping (ADR-014), de-mutated, fail-loud |\n",
"| `ReconciliationModule` (§4) | injected mapping (ADR-014), de-mutated, fail-loud; fills empty-grid draws from a donor (§4b) |\n",
"| `assert_reconcile_contract` (§5) | published contract; the mapping is load-bearing |\n",
"| `ES(aggregate) ≠ Σ ES(cell)` (§6) | joint sampling; tail risk is subadditive |\n",
"| bit-identity to the oracle (§7) | proves *faithful relocation*, **not** method quality |\n",
Expand Down
8 changes: 7 additions & 1 deletion reports/technical_risk_register.md
Original file line number Diff line number Diff line change
Expand Up @@ -486,7 +486,7 @@ Cross-refs: C-47 (eval provenance kept out of the generic header — the precede
| ID | C-99 |
| Tier | 1 |
| Resolved | 2026-11-02 |
| Resolution | Released in **3.0.0** (ADR-030): an empty-grid draw borrows the pattern of a random donor draw from the same `(time, country)`, and a group with no donor raises. Verified on the real archive (0 unplaced mass) and on the published wheel. |
| Resolution | Released in **3.0.0** (ADR-030): an empty-grid draw borrows the pattern of a random donor draw from the same `(time, country)`, and a group with no donor raises. On the real archive the mass is no longer lost (0 unplaced). That figure is true by construction once a donor fills a draw, so it shows the mass is placed, not that it lands in the right cells. |
| Source | views-models' first production run (`chrome_crate` → `chrome_box`, 2026-10-30), reported as #283; downstream investigation views-pipeline-core#579 |
| Trigger | Any reconciliation where, for some `(time, country, draw)`, the country value is positive and every grid cell is zero. Common, not exotic: 94–96% of grid cells are zero in a conflict forecast. |
| Location | `src/views_frames_reconcile/proportional.py:81-98` (the all-zero guard returns zeros); `src/views_frames_reconcile/conformance.py` (law 4 required it); `docs/CICs/Reconcile.md` §3/§6 ("an edge, not a failure"). |
Expand All @@ -509,6 +509,12 @@ conformance law (`TestEachLawHasTeeth`). Mutation testing killed 23 of 23 mutati
path (#289). On the real 2026-10-30 archive the 3.0.0 reconciler left **0** unplaced mass on all
three targets, filling exactly the 546,813 draws measured (#292). **Resolved 2026-11-02:** 3.0.0 is on
PyPI (#294), installed from there into a clean venv, and passes `assert_reconcile_contract`.
**What this does not show.** "0 unplaced" holds by construction, so it says nothing about
whether a donor's pattern puts the mass in the right cells; testing that needs the grid
before reconciliation, which the 2026-10-30 archive did not keep (ADR-030, Open Questions).
**Adoption is separate.** The defect is fixed in this package. The production lane that
exposed it still resolved views-frames 1.11.0 on 2026-11-02, because views-datafactory pinned
`views-frames<2` (views-datafactory#527); views-models#663 then raises the lane's floors.

---

Expand Down
21 changes: 12 additions & 9 deletions tests/test_reconciliation_scale.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,18 @@ def test_many_groups_correct_and_fast(self):
# Loose smoke: a reintroduced O(groups x N) grouping would balloon this.
assert elapsed < 10.0, f"reconcile unexpectedly slow ({elapsed:.1f}s)"

# Correctness across many groups: conservation holds per draw, except
# all-zero-input draws (which stay zero) — checked on a spread of countries.
# Correctness across many groups: every draw sums to its country (ADR-030),
# including draws whose grid was all zeros, which donor fill places. Checked
# on a spread of countries plus every country that has such a draw, so the
# empty-grid case is always exercised rather than assumed.
cmv = {int(u): cm.values[i] for i, u in enumerate(cm.index.unit)}
for c in (1, 137, 1000, 2000):
empty = {
int(c)
for c in np.unique(gcountry)
if (pgm.values[gcountry == c] == 0).all(axis=0).any()
}
assert empty, "the fixture must contain an empty-grid draw to check"
for c in sorted({1, 137, 1000, 2000} | empty):
sel = np.nonzero(gcountry == c)[0]
inp_allzero = (pgm.values[sel] == 0).all(axis=0)
grid_sum = out.values[sel].sum(axis=0)
active = ~inp_allzero
np.testing.assert_allclose(
grid_sum[active], cmv[c][active], rtol=1e-3, atol=1e-2
)
assert (grid_sum[inp_allzero] == 0).all()
np.testing.assert_allclose(grid_sum, cmv[c], rtol=1e-3, atol=1e-2)
Loading