Repository navigation
fix(cascade): quiesce drains fully, the reconcile pass commits once - #487
Merged
Merged
Conversation
Follow-ups from the post-merge review of #481. POST /cascade/quiesce called sync_once once, whose drain stops at the worker's bound (max_passes x batch size = 5000 rows). The ledger the one-time reconcile queues after an upgrade can exceed that, so quiesce froze a partial index while reporting quiesced=true. drain_fully now lives on the orchestrator and the CLI's sync / rebuild and the API route share it. force_enqueue_all committed once per ledger row (~0.9 ms each), holding the server's startup for the whole pass on a large root; it now issues every row in one session with one commit (0.32 ms/row measured on 2000 rows), handing out LSNs in ledger order. CHANGELOG and the cascade runbook name the two roots the automatic pass does not cover: a server started with EVEROS_DISABLE_CASCADE=1 (run cascade sync --force-all once) and a multi-project root where reflection ran before 1.4.2 (cascade rebuild once; the pass does not clear the deprecated_by stamps one project's reflection left on another's rows). import-linter's forbidden list drops the deleted backends package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Kendrick-Song
approved these changes
Oct 12, 2026
A real-host run of POST /cascade/quiesce on a 12000-row ledger left 50 rows behind with the previous commit: inside a server the background worker claims batches too, so a pass of ours can find nothing pending while the worker still holds the last batch; quiesce then cancels the worker and that batch stays orphaned in `processing` until the next start. The race predates #481; the ledger an upgrade re-queues made it routine. drain_fully now stops on the ledger, not on its own pass: it loops while the queue summary reports pending rows (which count `processing`), waits one poll interval when a pass finds nothing, and gives up after 120 s without progress so a hung handler cannot hold the caller forever. Same root, pre-fix binary: 11350 rows left; this commit: 0, 12000 done. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 12, 2026
Closed
gloryfromca
added a commit
that referenced
this pull request
Oct 14, 2026
Pre-release so this cycle's index rework — per-project primary keys (#481), the quiesce drain (#487), the capped LanceDB caches and the query embedding budget (#496) — is exercised against a real PyPI resolution before 1.4.2 goes out. The release pipeline's own smoke installs the built wheel with `--no-deps`, so it cannot catch a resolution break on its own. `docs/openapi.json` moves with the version: `app.openapi()["info"] ["version"]` reads `everos.__version__` (installed package metadata), and `make lint` fails on a stale dump. Testers must pass `--pre`; pip skips pre-releases by default and would silently install 1.4.1 instead. Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups from the post-merge adversarial review of #481 (two independent reviewers; neither found a blocker, both found these).
POST /cascade/quiescecalledsync_onceonce, whose drain stops at the worker's bound (max_passes× batch size = 5000 rows). The ledger the one-time reconcile queues after an upgrade can exceed that, so quiesce froze a partial index while reportingquiesced=true.drain_fullynow lives on the orchestrator; the CLI'ssync/rebuildand the API route share it. A real-host run then showed a second way quiesce froze a partial index: the server's own worker claims batches too, so a pass can find nothing pending while the worker still holds the last batch, and the stop that follows orphans it inprocessing(50 of 12000 rows). The drain now stops on the ledger (pendingcountsprocessing), waits a poll interval when a pass finds nothing, and gives up after 120 s without progress.force_enqueue_allcommitted once per ledger row (~0.9 ms each), holding the server's startup for the whole pass on a large root. It now issues every row in one session with one commit: 0.32 ms/row on 2000 rows (0.64 s vs 1.8 s).EVEROS_DISABLE_CASCADE=1(runcascade sync --force-allonce) and a multi-project root where reflection ran before 1.4.2 (cascade rebuildonce; the pass does not clear thedeprecated_bystamps one project's reflection left on another project's rows).backendspackage deleted in fix(index): key rows on (app_id, project_id, id) #481.Area
Verification
Checklist
main..envfiles, dependency folders, or generated output.Notes for Reviewers
### Upgradegroup of that release.force_enqueue_allstill issues one statement per row inside the single transaction; a chunked multi-rowINSERT ... ON CONFLICTis the next step if a 100k-file root makes startup wait (noted in code).🤖 Generated with Claude Code