What was measured
An adversarial review of #200 built a 46-mutant set against tools/ip65_hw_checks.py — larger than the 14 in the in-tree tools/mutate_ip65_hw_checks.py. 29 were caught; 17 survived.
The decisive part is what came next: applying each of the 17 survivors and re-running the "the C64 did nothing" adversarial capture shows that not one of them changes a single verdict. They are defence-in-depth branches that are subsumed at the call site — an input that would exercise them is already rejected by an earlier or a neighbouring check.
So this is not "the checker is wrong". It is "17 branches exist that nothing distinguishes", which is a maintenance hazard rather than a live defect: a future caller that reaches one of them directly gets behaviour no test pins.
Why it is not fixed in #200
The in-tree harness catches 13/13 of its own detectable set and closed four genuine coverage gaps when it was written (each one a red case that passed whether or not the checker worked). Extending it to 46 mutants and then either pinning or deleting the 17 unreachable branches is a separate, larger piece of work, and #200 is already a first-hardware-result PR carrying its own evidence burden.
Suggested shape
For each survivor, one of:
- pin it — add a red case that reaches the branch directly, if a caller could plausibly reach it;
- delete it — if it is genuinely unreachable given how the rig calls the function, say so and remove it, the way
check_mac_on_wire's redundancy is now documented rather than silently carried;
- record it as known-equivalent —
tools/mutate_ip65_hw_checks.py already has that category and reports it rather than suppressing it (check_http_response's length test is the existing example).
The third option is the one to reach for last: a growing known-equivalent list is how a mutation score stops meaning anything.
Related
What was measured
An adversarial review of #200 built a 46-mutant set against
tools/ip65_hw_checks.py— larger than the 14 in the in-treetools/mutate_ip65_hw_checks.py. 29 were caught; 17 survived.The decisive part is what came next: applying each of the 17 survivors and re-running the "the C64 did nothing" adversarial capture shows that not one of them changes a single verdict. They are defence-in-depth branches that are subsumed at the call site — an input that would exercise them is already rejected by an earlier or a neighbouring check.
So this is not "the checker is wrong". It is "17 branches exist that nothing distinguishes", which is a maintenance hazard rather than a live defect: a future caller that reaches one of them directly gets behaviour no test pins.
Why it is not fixed in #200
The in-tree harness catches 13/13 of its own detectable set and closed four genuine coverage gaps when it was written (each one a red case that passed whether or not the checker worked). Extending it to 46 mutants and then either pinning or deleting the 17 unreachable branches is a separate, larger piece of work, and #200 is already a first-hardware-result PR carrying its own evidence burden.
Suggested shape
For each survivor, one of:
check_mac_on_wire's redundancy is now documented rather than silently carried;tools/mutate_ip65_hw_checks.pyalready has that category and reports it rather than suppressing it (check_http_response's length test is the existing example).The third option is the one to reach for last: a growing known-equivalent list is how a mutation score stops meaning anything.
Related