Skip to content

feat(doctor): count the update backups nobody was ever told about (#681) - #682

Merged
eaitbrahim merged 1 commit into
mainfrom
feat-681-backup-footprint
Sep 1, 2026
Merged

feat(doctor): count the update backups nobody was ever told about (#681)#682
eaitbrahim merged 1 commit into
mainfrom
feat-681-backup-footprint

Conversation

@eaitbrahim

Copy link
Copy Markdown
Contributor

Closes #681.

keel update copies every keel*.db before it installs and never deletes one. That is correct — update.py names them as the data-recovery path, and an updater that pruned its own rollback would be an updater you cannot roll back from. What was missing is that nothing ever told the operator they were there.

The live deployment, measured 2026-09-01

88 files — 7.0 GB in ~/keel, plus 408 MB in ~/keel/backups
database backups for 24 releases, going back to 0.4.0

Three databases at 130–180 MB, one set per release, roughly a release a week — so it grows ~470 MB per update and nothing bounds it. df says 343 GB free, so this is not urgent. It is invisible, which is the actual problem: the first signal would be a disk filling during an update, the one moment a rollback path matters most.

Surface, never delete

doctor gains backups.footprint — WARN past three per database, OK below. A handful of recent backups is the design working, and warning about three would train the finding to be ignored by the time it matters.

Two tests carry the "never delete" half:

  • the fix text must say prune by hand and that keel will not do it — asserted to contain no keel backups prune, no --prune, no rm;
  • a scan of keel/ for any line that both mentions bak-before and calls unlink/rmtree/os.remove. A future change to the rollback guarantee has to delete that test to make it.

The count is the actionable number, not the bytes. "keel.db: 23 going back to 0.4.0" says what to do; "7 GB" says only that something is large. A hand-named backup (keel.db.bak-before-recordflow-…, which exists in the live folder) is never sorted as the oldest release — reporting "recordflow" as the oldest is both wrong and unactionable, since you can't decide whether to keep a release you can't name.

keel update's plan also names what is already kept, immediately before another set is written.

Verification

9 mutants, 9 killed — and two changed the code, not the tests:

  • Replacing the launch-folder read with an empty footprint passed everything. The seeded deployment has no backups, so both paths reported ok and the finding was never shown reading anywhere real. There's now a test that seeds four and monkeypatches update._launch_dir.
  • The try/except OSError around the glob was unreachable. Path.glob returns an empty iterator for a missing directory and for one at mode 000 alike, so deleting the handler changed nothing — which is how it was found. It's gone, with the correction recorded in its place: what genuinely races is stat on a file that vanished between the glob and the read, and that one is guarded where it can actually happen.

Full suite 5177 passed / 3 skipped; ruff and mypy clean.

`keel update` copies every `keel*.db` before it installs and NEVER deletes one.
That is correct -- `update.py` names them as the data-recovery path, and an
updater that pruned its own rollback would be an updater you cannot roll back
from. What was missing is that nothing ever told the operator they were there.

The live deployment, measured 2026-09-01: 88 files, 7.0 GB in ~/keel plus 408 MB
in ~/keel/backups, database backups for 24 releases going back to 0.4.0. Three
databases at 130-180 MB, one set per release, roughly one release a week -- so it
grows ~470 MB per update and nothing bounds it. A FAILED update wrote a set too
until #676 reordered downloads first, so retries used to compound it.

343 GB free today, so this is not urgent. It is INVISIBLE, which is the actual
problem: the first signal would be a disk filling during an update, which is the
one moment a rollback path matters most.

SURFACE, NEVER DELETE, and the fix line is where that is enforced. `doctor` gains
`backups.footprint` -- WARN past three per database, OK below, because a handful
of recent backups is the design working and warning about three would train the
finding to be ignored by the time it matters. The fix text says to prune BY HAND
and says keel will not do it, and a test asserts the string offers no `keel
backups prune`, no `--prune` and no `rm`. A second test scans `keel/` for any
line that both mentions `bak-before` and calls `unlink`/`rmtree`/`os.remove`, so
a future change to the rollback guarantee has to delete that test to make it.

THE COUNT IS THE ACTIONABLE NUMBER, not the bytes. "keel.db: 23 going back to
0.4.0" says what to do; "7 GB" says only that something is large. A hand-named
backup (`keel.db.bak-before-recordflow-...`, which exists in the live folder) is
deliberately never sorted as the oldest RELEASE -- reporting the oldest as
"recordflow" is both wrong and unactionable, since the operator cannot decide
whether to keep a release they cannot name.

`keel update`'s plan also names what is already kept, at the one moment the
operator is thinking about backups anyway: immediately before another set is
written.

9 mutants, 9 killed, and two of them changed the code rather than the tests.
Replacing the launch-folder read with an empty footprint passed everything --
the seeded deployment has no backups, so both paths reported `ok` and the finding
was never shown reading anywhere real. And the `try/except OSError` around the
glob turned out to be UNREACHABLE: `Path.glob` returns an empty iterator for a
missing directory and for one at mode 000 alike, so deleting the handler changed
nothing. It is gone, with the correction recorded where it was -- what genuinely
races is `stat` on a file that vanished between the glob and the read, and that
one is guarded where it can actually happen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL
@eaitbrahim eaitbrahim self-assigned this Sep 1, 2026
@eaitbrahim
eaitbrahim merged commit 65a4192 into main Sep 1, 2026
4 checks passed
@eaitbrahim
eaitbrahim deleted the feat-681-backup-footprint branch September 1, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nothing surfaces the update backups: 88 files and 7.0 GB in the live launch folder, unreported

1 participant