fix(update): retry a transport stall, and stop paying 466 MB before the step that fails (#675, #676)

[eaitbrahim]

Closes #675 and #676. Both came out of the failed keel update to 0.13.2 this morning.

downloading keel_core-0.13.2-py3-none-any.whl
update did not complete: could not download …: The read operation timed out

The asset was healthy the whole time — HTTP 200, 51,701 bytes, 0.82s a minute later, download counter still 0. A transient stall cost the entire update.

#675 — retries

_download_file made one attempt, and _HTTP_TIMEOUT_SEC = 15 is a per-socket-read timeout, not a deadline for the transfer — so a CDN going quiet for fifteen seconds fails a 51 KB download. Five wheels, five independent chances to trip. _http_get had the same shape and aborts the update before it even starts.

The trap this avoids. urllib.error.HTTPError is a subclass of OSError, so the obvious except OSError: retry retries a 404 as eagerly as a dropped connection — three times wrong instead of once — and retries a 403, which on the unauthenticated releases endpoint is the rate limit, spending the budget that just ran out. _is_retryable judges an HTTPError by its status and treats everything else as transport.

429 is deliberately not retryable, even though it's transient. It means the 60-per-hour budget is spent, and asking again is the one response that can't help; _http_get already turns it into a message naming the manual procedure, which needs no API call at all.

Timeout 15 → 30: with retries in hand, 30-and-retry recovers from more than 60-without while giving up on a genuinely dead host sooner. Backoff is a tuple rather than a formula so the added latency of a doomed update is legible at a glance — 4 seconds, once.

#676 — order

Backups ran before any download, on the reasoning that "if anything after this point half-happens, the databases' pre-update state exists on disk."

That guarantee is real and unchanged here: a download cannot make anything half-happen to a database, because _download_file writes only into Release/, and the two steps that do touch one — install and migrate — still run after every backup exists. The existing pin (_FakeOps.install asserting backups are already on disk and open cleanly as SQLite snapshots) passes untouched.

What the old order did was spend the expensive work before the failure-prone one: ~466 MB of sqlite3.backup() across three databases, thrown away by a stall on the first wheel. Backups are timestamped and deliberately never deleted, so the retry left a second full set behind rather than reusing the first — ~/keel holds 7.0 GB of them going back to 0.9.1.

Plan and gate wording updated to match: back up first → then back up.

Verification

12 mutants, 12 killed, and the last three were the lesson:

• _with_retries being correct proves nothing about _download_file using it. Every retry test called the helper directly, so removing the call from the production downloader left them all green. Same for _http_get. Both now have tests that drive the real function through a monkeypatched urlopen.
• backups=() on a failed download passed against a glob("*.bak-before-*") implementation until a test seeded an earlier run's backup — which is the normal state of a real launch folder, not an edge case. Reporting someone else's backup as this run's would send an operator to restore from the wrong file.

One more worth recording: a mutation whose anchor appeared three times never applied and read as a survivor until the count was checked.

Full suite 5124 passed / 3 skipped; ruff and mypy clean.

Note

This does not fix the update you ran — re-run keel update whenever; the release is healthy. This makes the next stall a non-event.