Wildfire fraction data still non-canonical after the #891/v1.1 fix, inflating impact via values >1

[simonameiler]

Describe the bug

Hazard.check_matrices() (added in #893 to fix #891) calls sum_duplicates()
on both intensity and fraction. For the wildfire v1/v1.1 API datasets,
fraction is stored with the same duplicate (event, centroid) positions as
intensity (both are artefacts of the same underlying data generation), and
every stored fraction entry is exactly 1.0. Summing N duplicate positions
therefore produces fraction = N — a value of 2, 3, or 4 for a field that is
supposed to be bounded in [0, 1]. Nothing in ImpactCalc checks or clamps
this, so fraction is multiplied directly into the impact
(fraction * mdr * value), inflating the result by that same factor for
every affected cell.

#893's own description says it "does not fix the problem of unintentionally
summed values — it just ensures this problem is visible when looking at the
matrix data." That's accurate for intensity (a threshold-based impact
function mostly doesn't care whether a duplicated reading is summed first),
but it means the same duplication silently produces out-of-range fraction
values with no visibility and no impact-calculation safeguard.

I checked v1.1 (the API's patch for #891) too: it deduplicates intensity
upstream, but not fraction — the duplicate positions and all-1.0
values are identical between v1 and v1.1. So v1.1 fixes half of #891
and leaves the other half in place.

On a 72-country sweep of the wildfire dataset using real population exposure
(GPW v4.11 via LitPop), comparing the real impact total against the same
total with fraction clamped to 1, this mechanism alone inflates TOTAL_abs
by a median 6-7%, up to ~20% for COD specifically — depends on how exposure
value is distributed across the affected centroids, so it's a
population-weighted number, not reproducible from the minimal example below.
The minimal, unweighted version (one unit of exposure at every centroid, see
"To Reproduce") shows the same mechanism at 8.8% for COD.

To Reproduce

Using the same country as #891 (COD), which has 104,708 duplicate
(event, centroid) positions:

import numpy as np
from climada.util.api_client import Client

client = Client()
haz = client.get_hazard("wildfire", properties={"country_iso3alpha": "COD"}, version="v1")

print(f"fraction.has_canonical_format: {haz.fraction.has_canonical_format}")
print(f"fraction.nnz (raw): {haz.fraction.nnz}")
print(f"fraction raw data unique values: {np.unique(haz.fraction.data)}")

# .max() (like check_matrices()/sum_duplicates()) canonicalises as a side effect
print(f"fraction.max(): {haz.fraction.max()}")
print(f"fraction.nnz after summing: {haz.fraction.nnz}")
print(f"fraction entries now > 1: {int((haz.fraction.data > 1.0).sum())}")

Output:

fraction.has_canonical_format: False
fraction.nnz (raw): 1296686
fraction raw data unique values: [1.]
fraction.max(): 4.0
fraction.nnz after summing: 1191978
fraction entries now > 1: 99939

v1.1 for the same country:

haz_v11 = client.get_hazard("wildfire", properties={"country_iso3alpha": "COD"}, version="v1.1")
print(f"v1.1 intensity.nnz: {haz_v11.intensity.nnz}")   # 1191978 -- deduplicated
print(f"v1.1 fraction.nnz: {haz_v11.fraction.nnz}")     # 1296686 -- NOT deduplicated, same as v1

And the downstream effect on ImpactCalc, with a minimal synthetic exposure
(one unit of value at every centroid, so no external exposure data is
needed) and a step impact function at 300 K:

import geopandas as gpd
from climada.entity import Exposures, ImpactFunc, ImpactFuncSet
from climada.engine import ImpactCalc

def unit_exposure(haz):
    lat, lon = haz.centroids.lat, haz.centroids.lon
    exp = Exposures(gpd.GeoDataFrame(
        {"value": np.ones(len(lat))},
        geometry=gpd.points_from_xy(lon, lat), crs="EPSG:4326"))
    exp.gdf[f"impf_{haz.haz_type}"] = 1
    exp.assign_centroids(haz, threshold=100)
    return exp

impfset = ImpactFuncSet([ImpactFunc.from_step_impf((0, 300, 1000), haz_type=haz.haz_type)])

haz_real = client.get_hazard("wildfire", properties={"country_iso3alpha": "COD"}, version="v1")
total_real = float(np.sum(ImpactCalc(unit_exposure(haz_real), impfset, haz_real)
                          .impact(assign_centroids=False).at_event))

haz_clamped = client.get_hazard("wildfire", properties={"country_iso3alpha": "COD"}, version="v1")
exp_clamped = unit_exposure(haz_clamped)
impcalc = ImpactCalc(exp_clamped, impfset, haz_clamped)  # this call runs check_matrices()
haz_clamped.fraction.data = np.minimum(haz_clamped.fraction.data, 1.0)  # clamp AFTER summing
total_clamped = float(np.sum(impcalc.impact(assign_centroids=False).at_event))

print(f"TOTAL_abs, fraction as served: {total_real}")      # 1296686.0
print(f"TOTAL_abs, fraction clamped to <=1: {total_clamped}")  # 1191978.0
print(f"inflation ratio: {total_real / total_clamped:.4f}")    # 1.0878

With uniform unit exposure this reduces to a clean identity: the unclamped
total equals fraction.nnz before summing (1,296,686) and the clamped total
equals the number of unique (event, centroid) positions after summing
(1,191,978) — an 8.8% inflation from this mechanism alone, even before
weighting by real, unevenly-distributed exposure value.

Expected behavior

Either Hazard.check_matrices()/prune_csr_matrix() should deduplicate
fraction the same way it does for datasets like this one (e.g. via max
rather than sum, since fraction is a bounded quantity, not an
accumulator), or ImpactCalc/Hazard should validate that fraction stays
within [0, 1] and raise or warn otherwise. At minimum, the v1.1 wildfire
regeneration for #891 should have deduplicated fraction alongside
intensity, since it's the same underlying defect.

Climada Version: 6.1.1.dev0 (also present conceptually in any version
with #893's check_matrices() fix and not yet checked against versions
before it)

System Information:

• macOS 26.5.2
• Python 3.11.14

Additional context

Found while trying to reproduce a 2023 published study's wildfire exposure
numbers on current CLIMADA. It explains roughly half (median 50%, r = 0.865
across 72 countries) of a systematic gap between the 2023 numbers and today's
recompute — not a version regression in the reproduction sense, but a
genuine defect in how the fixed intensity-duplication issue's sibling field
is handled. Related: #891, #893.