Actions queue saturation: 120 open PRs + self-amplifying scheduler block all org merges (pg-erd-cloud: 0 merges since 2026-08-20)

[seonghobae]

Symptom

ContextualWisdomLab/pg-erd-cloud has merged nothing to main since 2026-08-20 (~50 PRs piled up). Every new PR fails the required opencode-review check. Same pattern is expected across other org repos that use the central required workflows.

Root cause (investigated 2026-09-01)

1. Required check opencode-review (.github/workflows/opencode-review.yml) dispatches repository_dispatch{merge-scheduler} then polls pulls/<n>/reviews for an opencode-agent[bot] head-SHA verdict for 90 min (for attempt in $(seq 1 180); sleep 30, timeout-minutes: 100), else fails closed.
2. The verdict never arrives in time because the org Actions queue is saturated: ~148 queued runs vs ~6 in_progress, and growing tick over tick.
3. Queue load, last ~24h (800-run sample):
   • pr-review-merge-scheduler.yml — 166 runs/day (self-triggers on push + every pull_request_target + every pull_request_review + workflow_run completion + */15 & */30 crons + repository_dispatch)
   • noema-review.yml — 87/day
   • strix.yml 39, plus osv/security/secret/scorecard/sbom/semgrep/python-security/codeql/close-empty at ~30/day each
   • repair-pr827-coderabbit-comments.yml — 52 runs / 3 days, all skipped (PR fix(coverage): replace stale LLVM runtime-boundary repair #827 is closed; on: pull_request has no branch filter so it fires on every .github PR event)
4. The per-PR scan load is multiplied by ~120 open PRs in this repo (98 @seonghobae, 11 @cursor, 10 dependabot), oldest feat(coverage): add bounded PyO3 peer-evidence gate #789 from 2026-08-05. Many are month-old automation self-repair PRs that never merged; each new push re-runs ~10 scans. Self-amplifying.
5. ~190 of 337 registered workflows are one-shot apply-pr*/finalize-pr*/one-shot-*/repair-pr*/export-pr* for long-closed PRs (608, 676, 759, 767, 782, 785, 787, 797, 952…). Most are dormant (no runs in 3 days) — orphaned registrations whose files are gone from main — so they are cleanup debt, not active load, except repair-pr827 above.

Actions taken this pass (safe, reversible)

• Disabled repair-pr827-coderabbit-comments.yml (gh workflow disable 334949216) — closed PR, fired+skipped on every repo PR event.

Decisions needed from a maintainer (not safe to do autonomously)

• Triage the ~120 open PRs. Most are @seonghobae automation-repair PRs; closing/rebasing the superseded ones removes the bulk of the per-PR scan load. An autonomous agent should not mass-close a human's PRs.
• Whether to reduce pr-review-merge-scheduler.yml cron cadence (*/15 org-sweep → */30 or hourly) during the incident.
• Whether to raise/relax the 90-min opencode-review poll ceiling or make it retry-on-rerun instead of fail-closed, so transient queue backlog doesn't block every repo.
• Delete the ~189 dormant one-shot workflow registrations.

Workaround for downstream repos meanwhile

Re-run the failed opencode-review check after the queue drains; PRs with all other required checks green will pass once opencode-agent[bot] posts a current-head verdict.

Filed by the pg-erd-cloud autonomous review/merge loop.

[seonghobae]

Additional finding: opencode-review-dispatch.yml also fails independently of the queue (stale head_sha TOCTOU)

Recent opencode-review-dispatch runs that do execute still end in failure at the validate-pr-metadata job:

##[error]repository_dispatch metadata does not match the live pull request: head_sha.
supplied_head=feat/.../5e34896…   live_head=feat/.../5ff9317…

The merge-scheduler dispatch captures head_sha at dispatch time; the validate job then sits in the backlog long enough for the PR head to advance, so the exact-match check rejects it → no verdict → downstream opencode-review fails closed. The queue delay causes the mismatch, so the busiest PRs are the most likely to never get a verdict.

Suggested fix: when supplied_head_sha != live_head_sha but the target/PR is authorized, the base matches, and the PR is open, proceed with the re-fetched live head_sha (a review gate should always evaluate the current head) instead of exit 1. Keep the exact-match rejection for base repo / PR number / actor / allowlist.

[seonghobae]

Landed the head_sha fix as #1533 (merged to main, admin). validate-pr-metadata no longer fails closed when the PR head advances between merge-scheduler dispatch and the validate job — it now proceeds on the re-fetched live head (downstream jobs already re-check with STALE_HEAD guards). This removes root cause #2. Root cause #1 (Actions queue depth ~193, driven by ~120 open PRs here × per-PR scans + pr-review-merge-scheduler.yml at 166 runs/day) still needs a maintainer decision on triaging the PR pile.

[seonghobae]

Fresh queue reproduction — stale/closed runs are not being drained (2026-09-01)

A complete three-page read now returns 293 queued runs (API total: 296). This is not only transient capacity pressure:

• Required PR Review Merge Scheduler: 52 queued runs.
• CodeQL PR: 21; Python Security: 21; Secret Scan: 18; SAST Semgrep: 17; SBOM Generation: 17.
• The oldest queued run is 32214648177, created 2026-08-19T04:08:23Z — nearly two weeks old.
• Closed/predecessor work is still present, including a queued Required OpenCode run for closed fix(ci): correct required-workflow-bootstrap job-scope check boundary #1528 and multiple predecessor-head review runs. Existing close/supersession cancellation therefore does not cover this queued backlog.

Repair acceptance at the central owner boundary:

1. Before dispatch/enqueue, deduplicate on (workflow, target_repository, PR, exact current head) and refuse a second pending/queued equivalent.
2. Sweep queued runs whose PR is closed or whose requested head is no longer the live PR head; cancel only after re-fetching the encoded repository/PR/head identity.
3. Keep at most one durable scheduler wake-up per target/head; a cancelled or superseded model scan must be re-enqueued fairly without multiplying scheduler runs.
4. Add a fixture containing an open-current run, a predecessor-head run, a closed-PR run, and two duplicate scheduler runs; RED must prove the current implementation retains/multiplies the latter three, GREEN must preserve only the open-current run and one scheduler wake-up.
5. Report terminal cancellation/dedup evidence separately from review verdict evidence; do not convert cancellation or queue cleanup into a passing semantic review.

Protected main is 1186a9f4e5eda7683b23ae63d2c806831743432a; current queue depth increased despite #1533, so its head-advance repair did not address this stale-run lifecycle defect.

[seonghobae]

2026-09-01 control-plane follow-up executed from fresh main evidence:

• Closed chore(ci): keep PR #827 repair workflow (still reopenable); fix exact-head-path-policy CI gate #1488 as superseded: its code delta duplicated fix(ci): bound required-workflow-bootstrap awk extraction to its own job #1506's more complete awk/pipefail repair, while its remaining unique rationale preserved the PR fix(coverage): replace stale LLVM runtime-boundary repair #827 one-shot writer workflow.
• Opened chore(ci): retire completed PR 827 source-fix workflow #1539 from protected main@1186a9f4e5eda7683b23ae63d2c806831743432a, exact head 1548ebff5a883c53fddd388b29f4296d6d1fefc9, deleting .github/workflows/repair-pr827-coderabbit-comments.yml. Fresh fix(coverage): replace stale LLVM runtime-boundary repair #827 state is closed/unmerged; the workflow subscribed to all synchronize/reopened/ready_for_review PR events and filtered fix(coverage): replace stale LLVM runtime-boundary repair #827 only at job time, so unrelated PR activity still allocated skipped runs. chore(ci): retire completed PR 827 source-fix workflow #1539 changes no reviewer, credential, scheduler, provider, or branch-protection semantics.
• The associated repair driver/self-test remain for a separate bounded hollow-path cleanup after its fixture contract is rewritten; this keeps the queue-reduction change minimal and avoids breaking 100%-coverage policy in the same patch.
• Fresh main inspection also confirms the central scheduler still has both */30 and */15 schedules plus push/review/workflow_run/repository_dispatch wakeups. That broader amplification remains open in this incident; it should be reduced only with the pinned org-queue-sweep contract/docs/tests changed together rather than by an unreviewed cron-only edit.
• fix(ci): repair main's stale review-dispatch blob pin and security-boundary test #1536 is the current small main-red repair; its two Devin threads are informational, have been resolved, and its exact-head security workflows are presently queued rather than treated as passing evidence.

This issue should remain open until #1539 is integrated and the scheduler-trigger/queue-capacity causal chain is reduced with exact-head hosted evidence. No bypass, force push, synthetic status, or predecessor evidence transfer was used.

[seonghobae]

2026-09-01 live loop update:

• chore(ci): retire completed PR 827 source-fix workflow #1539 now removes .github/workflows/repair-pr827-coderabbit-comments.yml, the completed PR fix(coverage): replace stale LLVM runtime-boundary repair #827 source-fix workflow that subscribed to every synchronize/reopened/ready_for_review event and filtered fix(coverage): replace stale LLVM runtime-boundary repair #827 only at job time. Exact head 1548ebff5a883c53fddd388b29f4296d6d1fefc9; Devin reports no issues. Hosted security runs are queued, so no merge evidence is being fabricated.
• chore(ci): keep PR #827 repair workflow (still reopenable); fix exact-head-path-policy CI gate #1488 was closed unmerged as duplicate/conflicting lineage: its CI-gate fix overlaps fix(ci): bound required-workflow-bootstrap awk extraction to its own job #1506 while its unique 'keep fix(coverage): replace stale LLVM runtime-boundary repair #827 workflow' decision contradicts the purpose-complete source-fix retirement contract and this queue incident.
• fix(ci): repair main's stale review-dispatch blob pin and security-boundary test #1536 is the immediate protected-main red-suite repair. Exact head 5cba0ccf57b043ba39f46912bf532cd945ea916d is mechanically mergeable and both Devin threads are resolved; current security workflows are queued, so merge waits for exact-head evidence while other work continues.
• fix(noema): don't treat a deleted file's expected head-content 404 as a review-blocking error #1524 is the canonical Noema deletion-context repair needed so dead-code/workflow deletion PRs are reviewed from base-side content rather than falsely blocked by expected HEAD 404s. A Ready-for-review mutation was attempted, but the GitHub connector itself failed on an invalid GraphQL field (Repository.fullDatabaseId); that external tooling defect is not being converted into a repository-policy bypass. An exact-head OpenCode re-review was requested.
• Exact-head OpenCode re-reviews were also requested on fix(ci): repair main's stale review-dispatch blob pin and security-boundary test #1536 and chore(ci): retire completed PR 827 source-fix workflow #1539.

Remaining causal queue work after these land: measure the contribution of the dual */15 + */30 scheduler heartbeats and event-trigger amplification, then change cadence only with the corresponding workflow-contract/doctoring updates and without weakening fail-closed review/merge evidence.

[seonghobae]

#1533 (head-advance TOCTOU fix) was reverted in #1540. Automated review (Codex P1×3 / P2, Devin) showed it was incomplete:

1. Left tests/test_opencode_agent_contract.py:2663 red (still asserts the removed exact-head check).
2. Draft-review path: active_draft_review_request() uses the exact-head artifact as sole authorization; adopting the live head without revalidating the draft marker for the new head can publish a review for an unrequested draft commit.
3. run-name still ends @A after adopting head B; active_review_run_refs() + dispatch_opencode_review() then classify the run stale and force-cancel it before it produces a verdict — defeating the fix.
4. Empty/malformed pr_head_sha now validated (warning skipped when empty).

Handing the head-advance fix to whoever owns the scheduler run-identity / queued-run lifecycle work (the #1531 dedup/sweep contract, #1532, #1539): the adopted live head must be reflected in run-ownership identity, SUPPLIED_HEAD_SHA must stay a required 40-char SHA, draft PRs must revalidate the request marker, and the contract test moves with it. The .Jules/.jules and pr827-disable items from earlier ticks stand; I'm stepping back from .github to avoid colliding with the in-flight control-plane work and will focus on downstream product repos.

[seonghobae]

Fresh closed-head production RED — 2026-09-01

A five-page queued-run enumeration observed 433 queued runs (the point-in-time total was 429 immediately before this write, confirming the queue is moving rather than terminal-clean).

The lifecycle defect reproduced on a PR closed during this maintenance pass:

• fix(ci): repair main's stale review-dispatch blob pin and security-boundary test #1536 closed at 2026-09-01T02:06:31Z, exact head 5cba0ccf57b043ba39f46912bf532cd945ea916d.
• After closure, three new runs were created at 2026-09-01T02:06:33Z and remained queued:
  • Required Noema: 33461294496
  • Strix: 33461294521
  • Required OpenCode: 33461294504

The same enumeration still contained closed #1540 and reverted #1533 review runs, while the oldest queued run remained 32214648177, created 2026-08-19T04:08:23Z.

Current dominant backlog: merge scheduler 81; OSV/Python/Secret/CodeQL 30 each; Scorecard/SBOM 29 each; Security/SAST/Close Empty 28 each. This is a concrete RED for admission-time suppression plus durable closed/stale cancellation; merely adding another sweep job is insufficient if closing a PR itself enqueues three more runs.

[seonghobae]

Fresh 2026-09-01 revalidation after protected main moved to 7b1a028e704a98ae8a807bb827f44aeaee0399af: #1540 reverted #1533, so the earlier incident comment claiming the dispatch head_sha relaxation is landed is no longer current. Do not build queue recovery on that reverted behavior. The safer current owner for the review-lifecycle part is #1508 at exact head c86911a6301ef25737ae220c67b60c136381caee: it removes Noema's workflow_run trigger entirely, keeps pull-request/repository-dispatch PR identity in one concurrency group, cancels in-progress Noema only for synchronize/closed, revalidates the exact expected head before inference and again before publication, and adds exact-head/paginated stale-worker cleanup for OpenCode/autofix plus obsolete Strix-run cleanup. This directly addresses both the observed same-head Noema cancellations and unnecessary workflow-run wakeups without relaxing evidence. #1508 is currently mergeable but required exact-head evidence is still non-passing: its protected-main Noema check was cancelled under the very current-main concurrency policy it replaces, and there is no qualifying current-head approval yet. I requested fresh OpenCode and Noema reviews on c86911a...; no bypass or predecessor evidence is being used. #1539 remains the focused PR827 workflow retirement. The incident remains open until these control-plane fixes land through normal governance and a fresh queue snapshot demonstrates stale/duplicate backlog falling rather than merely shifting.