Skip to content

fix(cascade): quiesce drains fully, the reconcile pass commits once - #487

Merged
gloryfromca merged 2 commits into
mainfrom
fix/cascade-reconcile-followups
Oct 12, 2026
Merged

gloryfromca merged 2 commits into
mainfrom
fix/cascade-reconcile-followups

Conversation

@gloryfromca

@gloryfromca gloryfromca commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Follow-ups from the post-merge adversarial review of #481 (two independent reviewers; neither found a blocker, both found these).

  • POST /cascade/quiesce called sync_once once, whose drain stops at the worker's bound (max_passes × batch size = 5000 rows). The ledger the one-time reconcile queues after an upgrade can exceed that, so quiesce froze a partial index while reporting quiesced=true. drain_fully now lives on the orchestrator; the CLI's sync / rebuild and the API route share it. A real-host run then showed a second way quiesce froze a partial index: the server's own worker claims batches too, so a pass can find nothing pending while the worker still holds the last batch, and the stop that follows orphans it in processing (50 of 12000 rows). The drain now stops on the ledger (pending counts processing), waits a poll interval when a pass finds nothing, and gives up after 120 s without progress.
  • force_enqueue_all committed once per ledger row (~0.9 ms each), holding the server's startup for the whole pass on a large root. It now issues every row in one session with one commit: 0.32 ms/row on 2000 rows (0.64 s vs 1.8 s).
  • CHANGELOG and the cascade runbook name the two roots the automatic pass does not cover: a server started with EVEROS_DISABLE_CASCADE=1 (run cascade sync --force-all once) and a multi-project root where reflection ran before 1.4.2 (cascade rebuild once; the pass does not clear the deprecated_by stamps one project's reflection left on another project's rows).
  • import-linter's forbidden list drops the backends package deleted in fix(index): key rows on (app_id, project_id, id) #481.

Area

  • Architecture method
  • Benchmark
  • Use case
  • Documentation
  • Developer experience
  • CI, build, or release

Verification

make lint                       # ruff + import-linter (4 contracts kept) + repo gates + openapi drift
uv run pytest tests/unit        # 2667 passed, 4 skipped
uv run pytest tests/integration # 186 passed, 7 skipped
Mutation check: quiesce back to one sync_once -> test_drains_past_the_worker_bound red (1 != 5003);
  drain_fully back to the pass-driven loop -> its three orchestrator tests red (bound / worker race / idle cap).
Real host, POST /cascade/quiesce on a 12000-row ledger (acceptance/quiesce_smoke.py, counting-stub embedding):
  pre-fix binary 1805535: pending_before 11913 -> pending_after 11350; this branch: 11955 -> 0, ledger 12000 done.
Real host, upgrade-recovery group (v1.4.1 roots upgraded in place, embedding through a counting stub):
  T10.1 first start recovers the overwritten rows, T11.1 interrupted --force-all converges,
  T14.1 rebuild, T29.1 failed-row requeue -- all five PASS in 231 s (T18.1-R2 included).

Checklist

  • I kept the change scoped to the relevant area.
  • I am opening this from a separate branch, not pushing directly to main.
  • I updated docs, examples, or setup notes when behavior changed.
  • I added or updated tests when the change affects behavior.
  • I did not commit secrets, .env files, dependency folders, or generated output.
  • Active relative links in Markdown files resolve.

Notes for Reviewers

  • Meant to land before 1.4.2 is cut; the two doc sentences become the ### Upgrade group of that release.
  • force_enqueue_all still issues one statement per row inside the single transaction; a chunked multi-row INSERT ... ON CONFLICT is the next step if a 100k-file root makes startup wait (noted in code).

🤖 Generated with Claude Code

Follow-ups from the post-merge review of #481.

POST /cascade/quiesce called sync_once once, whose drain stops at the
worker's bound (max_passes x batch size = 5000 rows). The ledger the
one-time reconcile queues after an upgrade can exceed that, so quiesce
froze a partial index while reporting quiesced=true. drain_fully now
lives on the orchestrator and the CLI's sync / rebuild and the API route
share it.

force_enqueue_all committed once per ledger row (~0.9 ms each), holding
the server's startup for the whole pass on a large root; it now issues
every row in one session with one commit (0.32 ms/row measured on 2000
rows), handing out LSNs in ledger order.

CHANGELOG and the cascade runbook name the two roots the automatic pass
does not cover: a server started with EVEROS_DISABLE_CASCADE=1 (run
cascade sync --force-all once) and a multi-project root where reflection
ran before 1.4.2 (cascade rebuild once; the pass does not clear the
deprecated_by stamps one project's reflection left on another's rows).

import-linter's forbidden list drops the deleted backends package.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A real-host run of POST /cascade/quiesce on a 12000-row ledger left 50
rows behind with the previous commit: inside a server the background
worker claims batches too, so a pass of ours can find nothing pending
while the worker still holds the last batch; quiesce then cancels the
worker and that batch stays orphaned in `processing` until the next
start. The race predates #481; the ledger an upgrade re-queues made it
routine.

drain_fully now stops on the ledger, not on its own pass: it loops while
the queue summary reports pending rows (which count `processing`), waits
one poll interval when a pass finds nothing, and gives up after 120 s
without progress so a hung handler cannot hold the caller forever.

Same root, pre-fix binary: 11350 rows left; this commit: 0, 12000 done.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gloryfromca
gloryfromca merged commit be8ffcd into main Oct 12, 2026
11 checks passed
@gloryfromca
gloryfromca deleted the fix/cascade-reconcile-followups branch October 12, 2026 09:12
gloryfromca added a commit that referenced this pull request Oct 14, 2026
Pre-release so this cycle's index rework — per-project primary keys
(#481), the quiesce drain (#487), the capped LanceDB caches and the
query embedding budget (#496) — is exercised against a real PyPI
resolution before 1.4.2 goes out. The release pipeline's own smoke
installs the built wheel with `--no-deps`, so it cannot catch a
resolution break on its own.

`docs/openapi.json` moves with the version: `app.openapi()["info"]
["version"]` reads `everos.__version__` (installed package metadata),
and `make lint` fails on a stale dump.

Testers must pass `--pre`; pip skips pre-releases by default and would
silently install 1.4.1 instead.

Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants