feat(doctor): count the update backups nobody was ever told about (#681) - #682
Merged
Conversation
`keel update` copies every `keel*.db` before it installs and NEVER deletes one. That is correct -- `update.py` names them as the data-recovery path, and an updater that pruned its own rollback would be an updater you cannot roll back from. What was missing is that nothing ever told the operator they were there. The live deployment, measured 2026-09-01: 88 files, 7.0 GB in ~/keel plus 408 MB in ~/keel/backups, database backups for 24 releases going back to 0.4.0. Three databases at 130-180 MB, one set per release, roughly one release a week -- so it grows ~470 MB per update and nothing bounds it. A FAILED update wrote a set too until #676 reordered downloads first, so retries used to compound it. 343 GB free today, so this is not urgent. It is INVISIBLE, which is the actual problem: the first signal would be a disk filling during an update, which is the one moment a rollback path matters most. SURFACE, NEVER DELETE, and the fix line is where that is enforced. `doctor` gains `backups.footprint` -- WARN past three per database, OK below, because a handful of recent backups is the design working and warning about three would train the finding to be ignored by the time it matters. The fix text says to prune BY HAND and says keel will not do it, and a test asserts the string offers no `keel backups prune`, no `--prune` and no `rm`. A second test scans `keel/` for any line that both mentions `bak-before` and calls `unlink`/`rmtree`/`os.remove`, so a future change to the rollback guarantee has to delete that test to make it. THE COUNT IS THE ACTIONABLE NUMBER, not the bytes. "keel.db: 23 going back to 0.4.0" says what to do; "7 GB" says only that something is large. A hand-named backup (`keel.db.bak-before-recordflow-...`, which exists in the live folder) is deliberately never sorted as the oldest RELEASE -- reporting the oldest as "recordflow" is both wrong and unactionable, since the operator cannot decide whether to keep a release they cannot name. `keel update`'s plan also names what is already kept, at the one moment the operator is thinking about backups anyway: immediately before another set is written. 9 mutants, 9 killed, and two of them changed the code rather than the tests. Replacing the launch-folder read with an empty footprint passed everything -- the seeded deployment has no backups, so both paths reported `ok` and the finding was never shown reading anywhere real. And the `try/except OSError` around the glob turned out to be UNREACHABLE: `Path.glob` returns an empty iterator for a missing directory and for one at mode 000 alike, so deleting the handler changed nothing. It is gone, with the correction recorded where it was -- what genuinely races is `stat` on a file that vanished between the glob and the read, and that one is guarded where it can actually happen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #681.
keel updatecopies everykeel*.dbbefore it installs and never deletes one. That is correct —update.pynames them as the data-recovery path, and an updater that pruned its own rollback would be an updater you cannot roll back from. What was missing is that nothing ever told the operator they were there.The live deployment, measured 2026-09-01
Three databases at 130–180 MB, one set per release, roughly a release a week — so it grows ~470 MB per update and nothing bounds it.
dfsays 343 GB free, so this is not urgent. It is invisible, which is the actual problem: the first signal would be a disk filling during an update, the one moment a rollback path matters most.Surface, never delete
doctorgainsbackups.footprint— WARN past three per database, OK below. A handful of recent backups is the design working, and warning about three would train the finding to be ignored by the time it matters.Two tests carry the "never delete" half:
keel backups prune, no--prune, norm;keel/for any line that both mentionsbak-beforeand callsunlink/rmtree/os.remove. A future change to the rollback guarantee has to delete that test to make it.The count is the actionable number, not the bytes. "keel.db: 23 going back to 0.4.0" says what to do; "7 GB" says only that something is large. A hand-named backup (
keel.db.bak-before-recordflow-…, which exists in the live folder) is never sorted as the oldest release — reporting "recordflow" as the oldest is both wrong and unactionable, since you can't decide whether to keep a release you can't name.keel update's plan also names what is already kept, immediately before another set is written.Verification
9 mutants, 9 killed — and two changed the code, not the tests:
okand the finding was never shown reading anywhere real. There's now a test that seeds four and monkeypatchesupdate._launch_dir.try/except OSErroraround the glob was unreachable.Path.globreturns an empty iterator for a missing directory and for one at mode 000 alike, so deleting the handler changed nothing — which is how it was found. It's gone, with the correction recorded in its place: what genuinely races isstaton a file that vanished between the glob and the read, and that one is guarded where it can actually happen.Full suite 5177 passed / 3 skipped; ruff and mypy clean.