Repository navigation
ML4H 2026 rebuttal: reviewer-requested controls (visit-end override, random-code control) - #287
Merged
Merged
Conversation
The concept head is supervised only at visit ends, yet the override was applied at every position with the running label (Appendix E names this as a possible reason the true value does not help). The new option applies truth/flip/random and the calibrated modes only at each visit's last position (chunk.visit_end, the flag the concept loss pools on) and leaves every other position as under none. Each mode gains a visit_end_positions block: top-1 and loss over only the visit-end positions with an observed label inside the band and a next-token target, n_positions and n_subjects, and on the left-hand mode the paired subject-clustered bootstrap differences (truth minus none, truth minus flip, flip minus none) in top-1 and loss. The default (all) is unchanged byte for byte; --hazard-heads combines with it (the landmark rows are not visit ends, documented). WP1 of the rebuttal plan gains the VM command. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A reviewer asks for a random-code control to show the concept ranking
helps. Table 9's numbers came from an uncommitted scratch harness
(cohort_worsen.py); only its stdout logs were banked. This adds the
cohort driver to the package, `python -m
odyssey.inference.concept_edit_attribution`, reproducing that log
format so scripts/parse_edit_attribution_logs.py reads both, and gives
it the control arm.
- `--code-selection {attributed,random}` (default attributed) and
`--random-seed`: the random arm draws min(top_k, pool) codes per
subject uniformly from the same candidate pool the occlusion loop
ranks, seeded per subject from (seed, subject_id), then builds the
identical worsen edits (worsen_edits_for_codes, split out of
worsen_edits_from_attribution with no behaviour change).
- `--candidate-pool {mappable,all}`: the default pool is the distinct
window codes that resolve to a panel signal with a worsen edit, the
only codes either arm can act on; `all` ranks every window code and
drops unmappable picks the way the attributed arm always did.
- `--index-frac` on _index_times_by_subject for the banked headers'
index_frac field.
- Parser tags runs with `selection` (attributed when absent) and
`random_seed`; make_edit_attribution_table gains `--with-random`
(Codes column, random row under its attributed row), default output
byte-identical on the banked JSON; the CI table labels random blocks.
- Tests: pool filtering, seeded draw, same-count/same-subject
guarantee, parser round trip, table flag. docs/ml4h2026_rebuttal_plan
WP9 carries the VM command lines for both arms on MIMIC-IV and eICU.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…o rebuttal/reviewer-controls
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #287 +/- ##
==========================================
+ Coverage 90.98% 91.24% +0.26%
==========================================
Files 71 71
Lines 11263 11577 +314
==========================================
+ Hits 10248 10564 +316
+ Misses 1015 1013 -2
🚀 New features to boost your workflow:
|
codecov/patch was at 79% against the 91% project target: the new concept_edit_attribution CLI (_main) and the run-dir driver's visit-end branch had no tests. Adds a CLI test with load_run, the concept pins and the shard loader stubbed, the cohort_worsen edge paths (unknown concept, max_subjects, empty pool, an unedited random draw under pool=all, index_frac in the header), and an evaluate_interventions test with a two-admission fixture that attaches the visit-end blocks, plus the override-positions guard and the empty-table case. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges the two branches that answer Reviewer 3P7y's requests, plus the plan's WP9 results.
rebuttal/visit-end-override(b57deb4):--override-positions visit_endon the label-override test. The override is applied only at the last position of each visit (where the concept loss is pooled); every other position is left as undernone. Each result carries avisit_end_positionsblock (top-1 and loss at the eligible visit-end positions, n_positions, n_subjects, paired subject-clustered bootstrap truth-minus-none / truth-minus-flip / flip-minus-none). Tests intests/odyssey/inference/test_interventions_visit_end.py.rebuttal/random-code-control(5bd3096):python -m odyssey.inference.concept_edit_attributioncohort CLI with--code-selection {attributed,random},--random-seed,--candidate-pool {mappable,all},--top-k,--index-frac;parse_edit_attribution_logs.pyreadsselection/random_seed;make_edit_attribution_table.py --with-randomrenders the random-code rows (Table 9 unchanged without the flag).docs/ml4h2026_rebuttal_plan.md: WP9 description (from the random-code branch) and a WP9 results section.Results (banked under
research_journal/figure_data, not committed)Test plan
ruff format --check,ruff check: cleanmypyon the touched modules: cleanpyteston interventions, edit-attribution, counterfactual and the two script tests: 111 passed, 4 skipped🤖 Generated with Claude Code