Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 51 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
"TwoStageDiD" (previously "TwoStageDiD (Gardner)"); the
BusinessReport/DiagnosticReport Bacon caveats name classes likewise.

### Changed
- **`n_bootstrap` is validated and the `inference` selector fails closed**
(v4 program 2(d) PR-B; ledger rows [M-081] + [M-096] → `done`).
(a) A shared `validate_n_bootstrap` (promoted verbatim from
ChangesInChanges' local validator: non-negative integer, numpy integers
accepted, bool/None/float/negative rejected) now runs at `__init__` for
CallawaySantAnna, SunAbraham, EfficientDiD, ImputationDiD, TwoStageDiD,
WooldridgeDiD, ContinuousDiD, StaggeredTripleDifference and the DiD
family (DifferenceInDifferences; MultiPeriodDiD/TwoWayFixedEffects
inherit) - values that previously meant silent-bootstrap-off (negatives)
or latent breakage (floats/bools/None) now raise; `0` stays legal and
still means bootstrap off on every `> 0`-gated analytical lane. No
numeric defaults changed. (b) `inference=` accepts exactly
`{"analytical", "wild_bootstrap"}` (string-typed) at construction and
transactional `set_params` - unknown or non-string values raise instead
of silently running analytical inference. (c) At fit,
`DifferenceInDifferences` with `inference="wild_bootstrap"` and no
`cluster=` now raises `ValueError` where it previously fell back to
analytical SILENTLY - the pinned test
`test_did_wild_bootstrap_requires_cluster` flipped BY DESIGN to assert
the raise. (d) DiD/TWFE with `wild_bootstrap` and `n_bootstrap < 2`
raise at fit, closing the `n_bootstrap ∈ {0, 1}` states that ran WCR
with too few draws and returned a wild-labeled all-NaN inference tuple
with no warning. TWFE's unit auto-cluster still satisfies the cluster
prerequisite; the survey and Conley rejections keep precedence;
MultiPeriodDiD's warn-and-analytical-fallback is unchanged (and
n_bootstrap-independent). Emitted-guidance surfaces
(`docs/troubleshooting.rst`, `docs/choosing_estimator.rst`, `README.md`,
the bundled `llms*.txt` guides) now state the `cluster=` prerequisite
where they recommend wild bootstrap for DiD.

### Added
- **HeterogeneousAdoptionDiD post-fit `aggregate()` + panel-shape mode
inference, and the per-level bootstrap-gate convergence** (v4 program 2(b)
Expand Down Expand Up @@ -891,14 +922,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
lands while `__version__` is still 3.8.x) - the `M-091`/`M-092` pattern.
`M-096` covers the `inference="wild_bootstrap"` selector contract, which
governs wild CLUSTER bootstrap offered as an alternative to analytical SEs -
a roster of `DifferenceInDifferences` + `TwoWayFixedEffects`. Two things the
audit established. (i) The selector does **not** fail closed today:
`inference=` is stored with no valid-value check anywhere, and DiD routes
`wild_bootstrap` without `cluster=` silently to analytical (a currently
*pinned* behavior), so a typo or a missing prerequisite quietly changes which
SE/p-value/CI procedure runs. Phase 2 must validate the accepted value set on
`__init__` and transactional `set_params` and handle the missing-cluster case
explicitly. (ii) The apparent `inference=` vs `n_bootstrap>0` split is not
a SUPPORT roster of `DifferenceInDifferences` + `TwoWayFixedEffects` (the
selector *param* is additionally exposed by `MultiPeriodDiD`, which has no
wild path and falls back - support ≠ exposure). Two things the
audit established. (i) The selector did **not** fail closed at audit time:
`inference=` was stored with no valid-value check anywhere, and DiD routed
`wild_bootstrap` without `cluster=` silently to analytical (a then-*pinned*
behavior), so a typo or a missing prerequisite quietly changed which
SE/p-value/CI procedure ran. SHIPPED later in this release (the 2(d) PR-B
`### Changed` entry above): the accepted value set is validated on
`__init__` and transactional `set_params`, and the missing-cluster case
raises at fit. (ii) The apparent `inference=` vs `n_bootstrap>0` split is not
drift - estimators whose bootstrap *is* their inference method run materially
different procedures (CallawaySantAnna an influence-function multiplier
bootstrap; SunAbraham a unit-pairs bootstrap, Rao-Wu rescaled on survey
Expand Down Expand Up @@ -1339,6 +1373,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
default-flip and warning-retirement sweeps). Repo-internal documentation +
test only - **no public API or numerical behavior change.**

### Fixed
- **Stale wild-bootstrap metadata on refits**: `DifferenceInDifferences`
and `TwoWayFixedEffects` never cleared their per-fit bootstrap state, so
a wild-bootstrap fit followed by `set_params(inference="analytical")`
and a refit reported `inference_method="wild_bootstrap"` plus stale
`n_bootstrap`/`n_clusters`/`p_val_type` on an analytically-inferred
result. Both estimators now reset the state at the top of `fit()`;
refit transitions label inference from the current fit only.

## [3.8.0] - 2026-07-18

### Added
Expand Down
2 changes: 1 addition & 1 deletion DEFERRED.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ For survey-specific limitations (`NotImplementedError` paths), see the
| `SpilloverDiD` estimator-level end-to-end vcov reconstruction tests (`bread @ meat @ bread` against `res.vcov`): requires exposing the estimator's internal `X_2_kept` design arrays; the surface is currently pinned from different angles (uniform-weight bit-identity, drift goldens, manual lincom reconstruction at rtol=1e-6). | `spillover.py`, `tests/test_spillover.py` | follow-up | Low |
| `SpilloverDiD` Wave E.3 `finite_mask + design-subset` hygiene not yet adopted for TwoStageDiD's analogous pattern (`two_stage.py:567-601`) — separate parity follow-up noted in the `docs/api/spillover.rst` Restrictions block. | `two_stage.py` | Wave E.3 | Low |
| `HeterogeneousAdoptionDiD` `covariates=` (Theorem 6 multivariate-covariate extension) not implemented — `fit(covariates=...)` raises `NotImplementedError` via the shipped future-work trap (locked by the `test_had.py` / `test_methodology_had.py` L73 tests); the deferred work is the Theorem 6 extension itself. | `had.py` | Phase 2a | Low |
| MultiPeriodDiD wild bootstrap not supported (falls back to analytical) — user-facing edge-case limitation. | `estimators.py:1647` | — | Low |
| MultiPeriodDiD wild bootstrap not supported (falls back to analytical, n_bootstrap-independent) — user-facing edge-case limitation; the 4.0 removal replaces the fallback with a raise (v4-design §4.1). | `estimators.py:1574` | — | Low |
| `predict()` raises `NotImplementedError` — rarely needed; user-facing limitation. | `estimators.py:890-911` | — | Low |

## Version-gated (v4)
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ For rigorous DiD analysis, follow these 8 steps. Skipping diagnostic steps produ
2. **State identification assumptions** - which parallel trends variant (unconditional, conditional, PT-GT-Nev, PT-GT-NYT), no-anticipation, overlap.
3. **Test parallel trends** - simple 2x2: `check_parallel_trends()`, `equivalence_test_trends()`; staggered: inspect CS event-study pre-period coefficients (generic PT tests are invalid for staggered designs). Insignificant pre-trends do NOT prove PT holds.
4. **Choose estimator** - staggered adoption -> CS/SA/BJS (NOT plain TWFE); few treated units -> SDiD; factor confounding -> TROP; simple 2x2 -> DiD. Run `BaconDecomposition` to diagnose TWFE bias.
5. **Estimate** - `estimator.fit(data, ...)`. Always print the cluster count first and choose inference method based on the result (cluster-robust if >= 50 clusters, wild bootstrap if fewer).
5. **Estimate** - `estimator.fit(data, ...)`. Always print the cluster count first and choose inference method based on the result (cluster-robust if >= 50 clusters, wild bootstrap if fewer - for DifferenceInDifferences pass `cluster=`; TwoWayFixedEffects auto-clusters at unit level).
6. **Sensitivity analysis** - `compute_honest_did(results)` for bounds under PT violations (MultiPeriodDiD, CS, or dCDH natively; a StackedDiD `results.aggregate('event_study')` container also admits - needs `kappa_pre >= 2`), `run_all_placebo_tests()` for 2x2 falsification, specification comparisons for staggered designs.
7. **Heterogeneity** - CS: `results.aggregate('group')`/`'event_study'` (post-fit, no refit); SA: `results.event_study_effects` / `to_dataframe(level='cohort')`; Stacked: `results.aggregate('event_study')`/`'simple'` post-fit views (surface always computed since 3.9); EDiD: `results.aggregate(...)` post-fit from retained EIFs (3.9); ImputationDiD/TwoStageDiD: `results.aggregate(...)` post-fit from panel-backed kits (3.9); ContinuousDiD: `results.aggregate('dose'/'simple'/'event_study')` post-fit (3.9; dose/simple are views, event_study recomputes); subgroup re-estimation.
8. **Robustness** - compare 2-3 estimators (CS vs SA vs BJS), report with and without covariates (shows whether conditioning drives identification), present pre-trends and sensitivity bounds.
Expand Down
1 change: 1 addition & 0 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m

| Issue | Location | Origin | Effort | Priority |
|-------|----------|--------|--------|----------|
| Type-blind `n_bootstrap` acceptance in already-validated estimators - HAD bool (`isinstance(..., int)` passes `True`, runs as 1 replicate), dCDH bool+float (its bare `< 0` check passes both `True` and `2.5`), TROP float (`2.5` passes the `>= 2` floor), SyntheticDiD float under all three variance methods + bool/negative under jackknife (its floor check is skipped there) - align these local checks with the `utils.validate_n_bootstrap` type guard (M-081 kept them out of the sweep: it scoped to previously-UNvalidated estimators only) | `diff_diff/had.py`, `diff_diff/chaisemartin_dhaultfoeuille.py`, `diff_diff/trop.py`, `diff_diff/synthetic_did.py` | 2(d) PR-B | Quick | Low |
| M-020-era CS fit-time `aggregate=` teachings persist in troubleshooting.rst (:215/:241/:244) and choosing_estimator.rst (:243) - CS examples still fit with the deprecated kwarg; migrate to post-fit `results.aggregate('event_study')` (the two HAD examples in the same file were migrated with M-027) | `docs/troubleshooting.rst` | 2(b) PR-4 | Quick | Low |
| Evaluate adding the `BaseEstimator` param surface (get_params/set_params) to the exported classes that never had it - `PowerAnalysis`, `LinearRegression`, `BusinessReport`, `DiagnosticReport`, `TWFEWeightsResult` (a NEW public surface, deliberately out of the 2(c)-i pure-refactor scope; `LinearRegression` is the one `fit`-bearing class excluded from the contract suite's roster-completeness test). | `diff_diff/linalg.py`, `diff_diff/power.py` | mixin PR | Mid | Low |
| Tighten the mypy suppressions that back the enforced-zero posture: burn down `prep_dgp`'s per-module `[index]` override (needs a None-vs-array restructure that preserves the seeded RNG stream), and evaluate re-enabling the globally disabled codes (`arg-type`, `return-value`, `var-annotated`, `assignment`) one at a time — `assignment` alone hid several real annotation drifts found during the 2026-07 triage. | `pyproject.toml` `[tool.mypy]`, `diff_diff/prep_dgp.py` | lint-CI | Mid | Low |
Expand Down
10 changes: 2 additions & 8 deletions diff_diff/changes_in_changes.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
safe_inference_batch,
validate_binary,
validate_covariate_names,
validate_n_bootstrap,
)

# Default quantile grid: qte's ``probs = seq(0.05, 0.95, 0.05)`` (19 points), pinned to
Expand Down Expand Up @@ -1132,13 +1133,6 @@ def _validate_quantiles(quantiles: Optional[Any]) -> None:
raise ValueError(f"quantiles must be finite and strictly inside (0, 1), got '{quantiles}'")


def _validate_n_bootstrap(n_bootstrap: Any) -> None:
if isinstance(n_bootstrap, bool) or not isinstance(n_bootstrap, (int, np.integer)):
raise ValueError(f"n_bootstrap must be a non-negative integer, got '{n_bootstrap}'")
if n_bootstrap < 0:
raise ValueError(f"n_bootstrap must be a non-negative integer, got '{n_bootstrap}'")


def _validate_alpha(alpha: Any) -> None:
if not isinstance(alpha, (int, float, np.floating)) or isinstance(alpha, bool):
raise ValueError(f"alpha must be a float strictly between 0 and 1, got '{alpha}'")
Expand All @@ -1161,7 +1155,7 @@ def _validate_seed(seed: Any) -> None:
def _validate_all_params(params: Dict[str, Any]) -> None:
"""Validate the full hyperparameter dict (used by __init__, set_params, and fit)."""
_validate_quantiles(params["quantiles"])
_validate_n_bootstrap(params["n_bootstrap"])
validate_n_bootstrap(params["n_bootstrap"])
_validate_alpha(params["alpha"])
_validate_panel(params["panel"])
_validate_seed(params["seed"])
Expand Down
3 changes: 2 additions & 1 deletion diff_diff/continuous_did.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
build_unit_first_row_index,
compute_survey_vcov,
)
from diff_diff.utils import safe_inference
from diff_diff.utils import safe_inference, validate_n_bootstrap

if TYPE_CHECKING:
from diff_diff.survey import ResolvedSurveyDesign, SurveyDesign
Expand Down Expand Up @@ -286,6 +286,7 @@ def __init__(
self.anticipation = anticipation
self.base_period = base_period
self.alpha = alpha
validate_n_bootstrap(n_bootstrap)
self.n_bootstrap = n_bootstrap
self.bootstrap_weights = bootstrap_weights
self.seed = seed
Expand Down
3 changes: 2 additions & 1 deletion diff_diff/efficient_did.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@
compute_omega_star_nocov,
enumerate_valid_triples,
)
from diff_diff.utils import safe_inference
from diff_diff.utils import safe_inference, validate_n_bootstrap

# Re-export for convenience
__all__ = ["EfficientDiD", "EfficientDiDResults", "EDiDBootstrapResults"]
Expand Down Expand Up @@ -359,6 +359,7 @@ def __init__(
self.cluster = cluster
self.vcov_type = vcov_type
self.control_group = control_group
validate_n_bootstrap(n_bootstrap)
self.n_bootstrap = n_bootstrap
self.bootstrap_weights = bootstrap_weights
self.seed = seed
Expand Down
49 changes: 48 additions & 1 deletion diff_diff/estimators.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,17 @@
validate_covariate_names,
validate_design_term_names,
validate_df_convention,
validate_n_bootstrap,
wild_bootstrap_se,
)

# Accepted values for the `inference` selector (M-096). Exposed by exactly
# DifferenceInDifferences, MultiPeriodDiD and TwoWayFixedEffects (the two
# subclasses inherit this __init__); the fail-closed check lives in
# DifferenceInDifferences.__init__ and set_params inherits it via the
# BaseEstimator probe re-init.
_INFERENCE_METHODS = ("analytical", "wild_bootstrap")


class DifferenceInDifferences(BaseEstimator):
"""
Expand Down Expand Up @@ -108,9 +116,16 @@ class DifferenceInDifferences(BaseEstimator):
inference : str, default="analytical"
Inference method: "analytical" for standard asymptotic inference,
or "wild_bootstrap" for wild cluster bootstrap (recommended when
number of clusters is small, <50).
number of clusters is small, <50). Exactly these two (string)
values are accepted; anything else raises ``ValueError`` at
construction. ``"wild_bootstrap"`` requires ``cluster=`` — a fit
without it raises ``ValueError`` (since 3.9; previously it fell
back to analytical inference silently).
n_bootstrap : int, default=999
Number of bootstrap replications when inference="wild_bootstrap".
Must be a non-negative integer; ``>= 2`` is required when
``inference="wild_bootstrap"`` (0 or 1 replications cannot produce
bootstrap inference — the fit raises ``ValueError``).
bootstrap_weights : str, default="rademacher"
Type of bootstrap weights: "rademacher" (standard), "webb"
(recommended for <10 clusters), or "mammen" (skewness correction).
Expand Down Expand Up @@ -233,6 +248,13 @@ def __init__(
from diff_diff.linalg import resolve_vcov_type

validate_df_convention(df_convention)
validate_n_bootstrap(n_bootstrap)
# Fail-closed inference selector (M-096): an unrecognized or
# non-string value must never silently route to analytical. The
# isinstance guard matters — bare tuple membership admits a
# one-element ndarray via elementwise __eq__.
if not isinstance(inference, str) or inference not in _INFERENCE_METHODS:
raise ValueError(f"inference must be one of {_INFERENCE_METHODS}, got {inference!r}")

# `robust` is deprecated (rows M-045..M-047; removed in 4.0). None is
# the not-supplied sentinel: default constructions and get_params
Expand Down Expand Up @@ -380,6 +402,11 @@ def fit(
)
# Body-local name; the public parameter is post (M-030).
time = post
# Per-fit bootstrap state: cleared up front so the result builder
# labels inference from THIS fit only. Without the reset, a wild fit
# followed by set_params(inference="analytical") + refit reported
# stale inference_method="wild_bootstrap" + bootstrap metadata.
self._bootstrap_results = None
# Parse formula if provided
if formula is not None:
outcome, treatment, time, covariates = self._parse_formula(formula, data)
Expand Down Expand Up @@ -513,6 +540,26 @@ def fit(
cluster=self.cluster,
)

# Fail-closed wild-bootstrap coherence (M-096). Placed AFTER the
# survey and Conley front doors so their NotImplementedError
# rejections keep precedence (raising "pass cluster=" on a
# wild+Conley fit would be contradictory guidance — Conley rejects
# the combination regardless of cluster).
if self.inference == "wild_bootstrap":
if self.cluster is None:
raise ValueError(
"inference='wild_bootstrap' requires cluster=. The wild cluster "
"bootstrap resamples at the cluster level; pass cluster= or use "
"inference='analytical'."
)
if self.n_bootstrap < 2:
raise ValueError(
f"inference='wild_bootstrap' requires n_bootstrap >= 2 "
f"(got {self.n_bootstrap}). At least 2 replications are needed "
f"for bootstrap inference; use inference='analytical' for "
f"analytical SEs."
)

if absorb:
# FWL theorem: demean ALL regressors alongside outcome.
# Regressors collinear with absorbed FE (e.g., treatment after
Expand Down
4 changes: 2 additions & 2 deletions diff_diff/guides/llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ DifferenceInDifferences(
vcov_type: str | None = None, # Variance family: "hc1" (default), "classical", "hc2", "hc2_bm", "conley"
cluster: str | None = None, # Column for cluster-robust SEs
alpha: float = 0.05, # Significance level
inference: str = "analytical", # "analytical" or "wild_bootstrap"
n_bootstrap: int = 999, # Bootstrap replications (if inference="wild_bootstrap")
inference: str = "analytical", # "analytical" or "wild_bootstrap" (wild_bootstrap requires cluster=)
n_bootstrap: int = 999, # Bootstrap replications; >= 2 under inference="wild_bootstrap"
bootstrap_weights: str = "rademacher", # "rademacher", "webb", or "mammen"
seed: int | None = None, # Random seed
rank_deficient_action: str = "warn", # "warn", "error", or "silent"
Expand Down
Loading
Loading