Skip to content

feat: add read-only Actions queue health evidence - #1150

Merged
80 commits merged into
mainfrom
codex/pr1142-current-main-successor
Sep 17, 2026
Merged

80 commits merged into
mainfrom
codex/pr1142-current-main-successor

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Canonical read-only Actions queue-health owner lane and current-main successor for #1142. It classifies Actions admission states without cancelling/rerunning observed workflows, mutating leaf branches, synthesizing success, weakening required checks, or merging repositories under observation.

Current protected-base reconciliation — 2026-09-17 KST

Protected .github/main is now exact 8c77327a52883cbb7a9a698a67c20b3c2bec6700, the signed merge commit that brought canonical #2234 CodeQL endpoint/parser/wake-verdict repair into main. Prior queue-health owner exact was 42bb922f03bf75aed1bc1931d9fbaf04a5433e20.

Fresh comparison showed protected main had advanced by 26 commits while none of the queue-health owner paths had conflicting content. I therefore reconciled this lane by ordinary non-force two-parent ancestry, not by source copy or destructive rebase. Current exact is 6f9f2b4d9d1223355a417ba48d8a11500f267986 with parents 42bb922f... (owner) and 8c77327... (protected main). Its tree is protected-main tree plus the prior queue-health owner blobs. Comparison to both parents is ahead-only / behind 0. Effective delta against current main is 12 paths; two historical owner paths (docs/doctoring/actions-queue-saturation-hourly-sweep.md and tests/test_actions_queue_saturation_scheduler_cadence.py) are no longer effective diffs because identical content is already present on current main.

No old checks or reviews transfer. Fresh exact-head generation on 6f9f2b4... is nonterminal: Security Scan 35214786186 queued, Python Security 35214786233 queued, SAST Semgrep 35214786215 queued, and CodeQL PR 35214786187 pending.

Owner-side incident contract

Independent ContextualWisdomLab repositories reproduce distinct pre-source-execution states:

  • a required workflow can terminate startup_failure before any job exists;
  • a workflow can materialize jobs that remain pre-checkout with no runner assignment/checkout/steps;
  • a job can be cancelled after materialization but before runner assignment.

The collector keeps those states distinct and fail closed. Ordinary head-bound evidence uses bounded supported workflow-run queries and local classification. pull_request_target cancellation candidates are accepted only after linked current-open-PR number/head identity is proven. PR identity is re-read after terminal/job evidence; transient incomplete reads receive bounded retry while persistent incompleteness or identity drift remains a hard evidence failure.

Downstream ordering

#2201 remains a dependent queue-health enrollment/integration lane and still carries historical terminal CodeQL RED provenance from inherited pre-#2234 source. Because #2234 is now canonical protected-main ancestry, the correct sequence is: #1150 6f9f2b4... terminal exact-head acceptance → normal/non-force #2201 reconciliation/restack → fresh #2201 evidence. Do not blind-rerun the old #2201 head or copy #2234 source into it.

RED / GREEN closure criterion

RED is either an exact-current required workflow terminating before jobs exist or an exact-current materialized required job with no runner assignment, checkout identity, or steps. GREEN requires unchanged exact owner/leaf heads to obtain actual runner assignment, checkout identity and terminal required conclusions under the current ruleset while the collector continues to distinguish pre-job and pre-checkout cases fail closed.

Keep Draft until this unchanged exact has terminal applicable protected checks, all valid findings are resolved, and live review policy is satisfied. No self-approval, administrator bypass, force update, destructive rebase, no-op freshness commit, blind rerun, runner-selector churn, cancellation campaign, source copy, gate weakening, predecessor-evidence transfer, or synthetic success.

@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1ae1f3de-5a46-45ff-b947-18ee3b0e41fb

📝 Walkthrough

Walkthrough

시간별 GitHub Actions 큐 상태 수집 기능을 추가했습니다. 허용 저장소의 실행과 작업을 읽기 전용으로 조회하고, 실행 상태와 queue-age SLO를 분류합니다. 결과를 JSON·HTML 아티팩트로 저장하며, 워크플로 계약과 처리 경로를 테스트합니다.

Changes

Actions Queue Health

Layer / File(s) Summary
입력 계약 및 GitHub API 수집 기반
scripts/ci/actions_queue_health.py, tests/test_actions_queue_health.py
저장소, 타임스탬프, API 응답을 검증합니다. gh api를 읽기 전용으로 호출하고 PR, 작업, workflow run 데이터를 정규화합니다.
허용 저장소 스냅샷 수집
config/actions_queue_health_repositories.json, scripts/ci/actions_queue_health.py, CHANGELOG.md, tests/test_actions_queue_health.py
4개 허용 저장소에서 PR과 queued/in-progress 실행을 수집합니다. 현재 head 실행의 작업을 조회하고, 불완전한 PR 식별 응답은 한 번 재시도합니다.
실행 분류 및 보고서 출력
scripts/ci/actions_queue_health.py, tests/test_actions_queue_health.py
실행을 current-head, obsolete, unlinked로 분류합니다. runner 할당, queue-age SLO, 중복 lane, 차단 사유를 JSON·HTML 보고서에 기록합니다.
워크플로 연결 및 운영 계약 검증
.github/workflows/actions-queue-health.yml, scripts/ci/actions_queue_health.py, tests/test_actions_queue_health.py, tests/test_actions_queue_health_contract.py, docs/doctoring/actions-queue-health.md, CHANGELOG.md, scripts/ci/organization_commercial_readiness_loop.py
시간별 읽기 전용 워크플로가 토큰, allowlist, 스크립트, JSON·HTML 아티팩트를 연결합니다. CLI 오류와 SLO 경고, 워크플로 권한, 고정 액션 버전, 운영 제한을 검증하고 문서화합니다. GitHubClient.__init__ 설명과 모듈 진입점 동작도 갱신합니다.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟡 Moderate · up to 47bb2

The new scheduled read-only queue-health collector can remain stuck on an unresponsive GitHub API call and occupy the workflow for up to its platform limit, delaying later runs; bounded API and job timeouts should be added before merge, with several smaller validation and robustness follow-ups remaining.

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler as GitHub Actions scheduler
  participant Workflow as actions-queue-health.yml
  participant Script as actions_queue_health.py
  participant GitHub as GitHub API
  participant Artifact as Actions artifacts

  Scheduler->>Workflow: 매시간 collect 작업 실행
  Workflow->>Script: allowlist와 토큰으로 수집 시작
  Script->>GitHub: 저장소·PR·실행·작업 읽기 요청
  GitHub-->>Script: 큐 및 runner 상태 반환
  Script-->>Workflow: JSON·HTML 보고서 생성
  Workflow->>Artifact: 실행별 보고서 업로드
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 41.38% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 읽기 전용 GitHub Actions 큐 상태 증거를 추가하는 주요 변경 사항을 정확하고 간결하게 설명합니다.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/pr1142-current-main-successor

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Current-main successor for #1142: head 36006cda56f2400f09d30f467a308e74cb42f289 is based on main bbedc1a51ec1a2421f129955c629b3cd0507a4ec. Fresh local evidence: queue-health tests 40 passed; interrogate 100%; compileall, actionlint, and diff-check pass. Project #1 item added; protected current-head checks and review-thread requirements remain authoritative.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Current-head review request for PR #1150:

  • Exact head: 36006cda56f2400f09d30f467a308e74cb42f289.
  • Adds a read-only, explicitly allowlisted Actions queue-health report for .github, TEPP, naruon, and contextual-orchestrator, with exact-head classification, bounded pagination, runner assignment/SLO evidence, deterministic JSON, and accessible HTML.
  • Cross-repository reads use the configured read credential and the workflow has no cancellation, dispatch, merge, or write permissions.
  • Local evidence: queue-health and commercial-readiness tests 54 passed; actionlint .github/workflows/actions-queue-health.yml, Ruff, compileall, and diff-check passed.

Please provide a fresh independent review for this exact head. Protected current-head checks and qualifying approval remain required before merge.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Current-head verification for c3697b63 against live main@c47bee59: Actions queue-health focused tests passed (40 passed); compileall, CLI help, actionlint, and diff checks passed. The read-only credential/config boundary remains intact; review this exact head only.

@seonghobae

Copy link
Copy Markdown
Contributor Author

@opencode-agent Review exact current head c3697b63 against main@c47bee59. Inspect read-only Actions queue-health evidence, cross-repository credential selection, transient PR identity retries, bounded pagination, and hosted checks. Do not reuse predecessor evidence.

@seonghobae
seonghobae force-pushed the codex/pr1142-current-main-successor branch from c3697b6 to f92f08e Compare August 20, 2026 04:09
@seonghobae

Copy link
Copy Markdown
Contributor Author

@opencode-agent Review exact current head f92f08e against main@2cce96f8. Rebased the read-only Actions queue-health evidence onto current main; allowlisted repositories, bounded pagination/identity retries, named read credentials, and no write/dispatch/merge permissions are preserved. Verified: 76 queue-health/commercial-readiness tests, actionlint, compileall, interrogate, ruff, and git diff --check passed.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head ecosystem review requested. Verify the current source and checks only; preserve protected gates and read-only credential boundaries.

@opencode-agent review

@seonghobae
seonghobae enabled auto-merge (squash) August 20, 2026 05:34

Copy link
Copy Markdown
Contributor Author

@opencode-agent Review the current exact head of the read-only Actions queue-health successor. Verify complete bounded pagination, exact pull-request/head identity, transient incomplete-response retry, runner-assignment and queue-age classification, deterministic JSON/accessible HTML, and the absence of cancellation, dispatch, branch, merge, or credential-expansion authority. Reacquire exact-head quality, security, CodeQL, SAST, dependency/SBOM, and formal review evidence. Do not alter the branch.

@opencode-agent
opencode-agent Bot disabled auto-merge August 20, 2026 06:58
@seonghobae

seonghobae commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Current exact head 7bbd13393b7d7bfc7f76b544d4e33fc572ccb471 is based on protected main@6479989bbff475404cc2cccc468d5fb1d6c632e5; all predecessor checks and review evidence are historical and non-passing.

The read-only queue-health collector bounds Actions workflow-run responses to 50 records per page, rejects path-traversal repository segments, retries one transient incomplete PR identity response after a bounded delay, records repository-scoped collection failures as explicit incomplete evidence, rejects duplicate repositories, applies a 30-second API subprocess timeout, and has a 30-minute workflow ceiling. It never cancels runs, changes branches, dispatches workflows, or mutates merge state.

Systematic RED → GREEN:

  • RED on predecessor 94e41c6fc3b05cd342fa912586cabebcb9fe5d9e: required interrogate failed at 99.9%, identifying the undocumented FakeClient.__init__.
  • Minimal GREEN: adopted the same compatible constructor docstring already present in the canonical Strix successor instead of creating a competing implementation.
  • local focused queue-health suites: 51 passed
  • local full central suite: 1285 passed, 16 subtests passed
  • local docstring gate: 100.0%
  • local queue collector coverage: 367 statements / 164 branches / 100% across 49 tests
  • Python compile and git diff --check: passed

Exact-head hosted evidence:

  • Organization Commercial Readiness Loop Quality CI run 32378389730, job 96455250457: log proves checkout of 7bbd13393b7d7bfc7f76b544d4e33fc572ccb471; 34 passed; completed success.
  • Strix Changed Path Quality CI run 32378389741, job 96455249166: log proves checkout of 7bbd13393b7d7bfc7f76b544d4e33fc572ccb471; 1285 passed, 16 subtests passed; completed success.

The eight security/SBOM/CodeQL runs and a qualifying exact-head formal Reviews API verdict remain non-passing prerequisites.

coderabbitai[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor Author

Exact current head evidence for a6ac2620207d419a988a9cfabe8903feb378b542:

  • full suite: 1285 passed, 16 subtests passed
  • production coverage: 8205 statements / 3210 branches, 100%
  • interrogate scripts/ci --fail-under=100: passed
  • Python compileall, git diff --check, and actionlint .github/workflows/actions-queue-health.yml: passed
  • fixed verified review findings: reject ./.. repository segments; typed pull-request identity retry with bounded delay; isolate per-repository API/pagination failures and publish explicit JSON/HTML incomplete-evidence entries

Please review and evaluate Checks against this exact SHA only. No self-approval or predecessor-head evidence transfer.

Copy link
Copy Markdown
Contributor Author

Final exact current head is now 94e41c6f7d6db6f9d1d8c0d4a4d1f2b1eebfbe4c after the doctoring APA 7 references were added (docs-only follow-up to the verified code commit). Prior exact-head evidence remains valid for the code at a6ac2620207d419a988a9cfabe8903feb378b542; please bind any independent review and required Checks to 94e41c6f7d6db6f9d1d8c0d4a4d1f2b1eebfbe4c.

Copy link
Copy Markdown
Contributor Author

Correction to my immediately preceding comment: the exact final HEAD is 94e41c6fc3b05cd342fa912586cabebcb9fe5d9e. The prior SHA text in that comment was incorrect; disregard it. The docs-only follow-up is on top of the verified code commit a6ac2620207d419a988a9cfabe8903feb378b542. Please bind review and Checks only to 94e41c6fc3b05cd342fa912586cabebcb9fe5d9e.

Copy link
Copy Markdown
Contributor Author

@opencode-agent Please perform the independent review on exact current head 94e41c6fc3b05cd342fa912586cabebcb9fe5d9e only. The prior CodeRabbit findings were addressed in the code commit below it; the latest commit only adds APA 7 doctoring references. Bind the verdict to current Checks and do not transfer predecessor-head evidence.

Copy link
Copy Markdown
Contributor Author

#2213 current exact is now ecbdc3507148e2ab922deba8432db87c129cf433 (ahead 3 / behind 0 from this owner head; same two effective paths). Focused CodeRabbit re-review withdrew its earlier aggregate-count concern after verifying call order: _CORE_BUILD_REPORT counts the row as core terminal before the wrapper relabels it, so terminal_job_count remains 1. ecbdc350... adds that invariant as an explicit regression assertion only; production repair remains b016348d....

Re-review found no different causal defect: failure diagnostics survive collection, only zero-step/no-runner failed jobs become terminal_pre_execution_failure, runner-backed/stepped failures remain ordinary terminal, and startup/cancelled predicates remain unchanged. Source inspection only; hosted exact-head workflows are still nonterminal, so no executable GREEN or merge readiness is claimed.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP Ready-state canary on unchanged exact head: ContextualWisdomLab/TEPP#523@c3767beedc76329983c939b6b35b2f48b29c3a80 is not Draft, yet Rust Foundation run 34998902305 still has all four materialized jobs queued pre-checkout with runner_id=0, empty runner name and steps=[]: coverage 104482071205, format/lint/test/rustdoc 104482071339, Live PostgreSQL 104482071379, repository/Python contracts 104482071447. Documentation run 34998902174 is also queued. This separates the current queue condition from TEPP's Draft guard; no leaf rerun, no-op commit, selector change or cancellation was performed. Please treat it as another read-only owner-side admission sample, not as incident resolution or GREEN.

Copy link
Copy Markdown
Contributor Author

Fresh queue-health canaries extend the same pre-checkout class without touching leaf workflows:

  • ContextualWisdomLab/TEPP#523@c3767beedc76329983c939b6b35b2f48b29c3a80 is Ready for Review, not Draft. Rust Foundation run 34998902305 has four ubuntu-latest jobs (104482071205, 104482071339, 104482071379, 104482071447) queued since 2026-09-15T17:03:57Z; every job still reports runner_id=0, empty runner/group names and steps=[]. This separates the observed admission state from TEPP's Draft guard.
  • ContextualWisdomLab/TEPP#527@9d0c6efc2b03d470447bae37a5bbb5fd980bfa3c has current Rust Foundation 35013310119, Security 35013310049, Semgrep 35013310083 and CodeQL 35013310107 all queued.
  • The current main-target mhtml successor is now ContextualWisdomLab/mhtml-etl-gateway#97@cd0ab1a6e9a59b673c2b3d186f75a58ca412e209. Repository Quality run 35014858000 materialized Python 3.11–3.14 jobs (104535611214, 104535611311, 104535611263, 104535611018); all four are queued with runner_id=0, empty runner/group names and steps=[]. Security 35014858030, Semgrep 35014858002, CodeQL 35014858041 are also queued.

#2212@22d6a9679f015cd2571ae859d1925d92c3e0d676 therefore remains repository-correct enrollment work even though its prose names the older #95 canary. Its Security and Semgrep runs are terminal success; CodeQL remains queued. No leaf rerun, no-op commit, runner-selector change, cancellation or synthetic status was performed.

Copy link
Copy Markdown
Contributor Author

Fresh child-owner settlement specimen on #2213 exact ecbdc3507148e2ab922deba8432db87c129cf433: SAST Semgrep run 34979970295 is now terminal SUCCESS on the unchanged exact head, while CodeQL PR 34979970129 and Security Scan 34979970240 remain queued/nonterminal. This is additional same-head selective-admission evidence, not overall GREEN and not evidence for leaf repositories. Preserve the pending jobs; no blind rerun/no-op commit is needed. #2213 remains the bounded terminal-pre-execution-failure classification repair child of this exact #1150 head.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP canary after successor consolidation: ContextualWisdomLab/TEPP#538 is now Ready, targets protected main, and exact head is e2b0007ab7e3a8ebe9334f4dd3cb5157c104754d. Rust Foundation run 35023809995 has four materialized ubuntu-latest jobs — production coverage 104565827308, format/lint/test/rustdoc/dependency 104565827587, repository/Python contracts 104565827602, Live PostgreSQL 104565827641 — all still pre-checkout with runner_id=0, empty runner/group names and steps=[]. Documentation Quality 35023809973 is also queued. This is not explained by the leaf Draft guard: #538 was explicitly marked Ready before these runs were created, targets main, and GitHub currently reports it mergeable. No rerun/no-op/selector/cancel action was taken. Treat this as another current exact-head pre-checkout admission sample for the read-only queue-health owner; leaf acceptance remains pending actual assignment/execution.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP queue-health canary on the current main-target successor: ContextualWisdomLab/TEPP#538@df389349ce4c9d750d6aed6cbad2d353c06797d1 is Ready for Review, targets protected main, is currently mergeable, and its Rust Foundation run 35028802024 has four materialized ubuntu-latest jobs still pre-checkout: format/lint/test/rustdoc/dependency 104582168871, Live PostgreSQL 104582168954, repository/Python contracts 104582168964, production line/branch coverage 104582169198. Every job currently reports runner_id=0, empty runner/group names and steps=[]. Documentation Quality 35028802027, Security Scan 35028801919, Semgrep 35028802060, and CodeQL 35028802062 are also queued on the same exact head.

This supersedes the older #538 e2b0007... canary as the live leaf sample. No rerun, no-op commit, runner-selector change, cancellation, synthetic status, or gate weakening was performed. Treat this as current read-only pre-checkout admission evidence, not incident resolution or TEPP GREEN.

Copy link
Copy Markdown
Contributor Author

Fresh read-only canary from ContextualWisdomLab/TEPP#538 after its block-comment RED was causally repaired and the exact head returned to Ready. Current leaf head is 680790b90baede67be26b93e4a3b6d3e31d8ed1b targeting protected main@a243f18da4a4ca8a8d068c39922537f1f8ed6ad0. Ready-state Rust Foundation run 35042536446 has materialized all four jobs but they remain pre-checkout queued with no executed steps: repository/Python coverage 104625278363, Live PostgreSQL 104625278488, format/lint/test/rustdoc/dependency policy 104625278503, production line/branch coverage 104625278535. Documentation Quality 35042536444, Security Scan 35042498958, Semgrep 35042498938, and CodeQL PR 35042498973 are likewise nonterminal on the same exact head. No leaf rerun, no-op push, cancellation, runner-selector change, bypass, or synthetic success was used. Treat this only as a current-head admission specimen; GREEN still requires actual runner assignment/checkout and terminal required conclusions on unchanged head.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP queue canary after causal coverage repair: ContextualWisdomLab/TEPP#538 exact 758e06d75d7c3ff71bc1cf5c91153e29eb6adf7d is Ready/main-target. Rust Foundation run 35044250882 materialized all four ubuntu-latest jobs but all remain pre-checkout queued with runner_id=null, runner_name=null, steps=[]: repository/Python 104630548338, format/test/rustdoc/dependency 104630548526, production line/branch coverage 104630548560, Live PostgreSQL 104630548595. Leaf rerun/no-op push/runner-selector changes were not used. This supersedes the prior TEPP #538 canary head for queue-health evidence.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP canary for the queue-health owner path: ContextualWisdomLab/TEPP#538 is Ready/main-target at exact head 0a6947b11f2dba9e161f2abe8bdec0f652d4b47a. Rust Foundation run 35045658457 has four ubuntu-latest jobs materialized but still pre-checkout queued with runner_id=0, empty runner/group names, and steps=[]: Live PostgreSQL 104634796152, Production line/branch coverage 104634796279, Repository contracts/Python branch coverage 104634796299, Format/lint/test/rustdoc/dependency 104634796389. The head advanced ordinary-forward from 758e06d... by one test-only coverage-edge commit; no leaf rerun/no-op push/runner-selector change/cancel is being used. Treat this as a read-only current-head specimen, not GREEN evidence.

Copy link
Copy Markdown
Contributor Author

A benign reproducer of this owner's third incident class, plus a correction to my own disclosure above.

The PR body enumerates "a job can be cancelled after materialization but before runner assignment." ContextualWisdomLab/semantic-data-portal just produced one, and its cause is entirely ordinary.

35043486426 on ContextualWisdomLab/semantic-data-portal#102, head eac988c, event=pull_request: created_at=2026-09-16T01:17:24Z, conclusion=cancelled, updated_at=2026-09-16T03:53:23Z. It never received a runner — 2h 36m materialized-pre-runner, then cancelled. The cause is .github/workflows/fuzz.yml in that repository:

concurrency:
  group: fuzz-${{ github.ref }}
  cancel-in-progress: true

I pushed the next documentation commit to the same ref at 03:53:23Z and that cancelled the waiting run. No cancellation campaign, no runner-selector change, no attempt to influence a conclusion — just an author committing again while a run sat in the queue.

Why this matters to the collector. A cancelled-pre-runner job is indistinguishable at read time from the incident class unless the cause is checked. Any repository with cancel-in-progress: true and an active author will generate these continuously, and they are not admission failures. If the collector treats cancelled-after-materialization as RED without discriminating, this class will be its noisiest one. The discriminator is available in the run record — a cancellation whose updated_at coincides with a newer run on the same concurrency group is an ordinary supersede.

Correction to my earlier comments. I wrote that no cancellation was used. That was true of the runs I reported, which all completed and were left alone until they scheduled themselves. It is not true of the repository as a whole: this run was cancelled, by me, in the way described. I would rather state that than leave the earlier wording implying otherwise.

It also censors the dwell figures I gave you. Those nine waits are only the runs that got a runner before my next push. Longer waits are likelier to be cut short by a subsequent commit, so the sample is biased toward shorter values, and the 3h16m–6h01m interval should be read as a floor on the spread rather than its true extent. This strengthens rather than weakens the earlier point: a dwell threshold is not a sound discriminator, and here even the observed distribution is not trustworthy as a distribution.

If the collector computes any duration statistic from terminal runs, it will need to account for supersede-cancelled runs or it will systematically under-measure the wait.


Generated by Claude Code

seonghobae commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Fresh TEPP downstream specimen after Evidence #527 advanced ordinary-forward to exact 5c109c201b2f552e7ba573de9fe1512983e4dc6a: Rust Foundation run 35058423733 is queued before runner admission after the final rustfmt-compatible regression update. The immediately preceding exact head 31edb0d... had all four jobs materialized with runner_id=0, empty runner/group and steps=[] (production coverage 104672871535, format/lint/test/rustdoc/dependency 104672871688, repository/Python coverage 104672871697, Live PostgreSQL 104672871783). Treat that previous-head tuple as queue diagnostics only; no leaf rerun, no-op push, runner-selector change, cancellation, or synthetic success is being used. #527 remains Draft and is not being promoted ahead of foundation successor #538.

@seonghobae seonghobae mentioned this pull request Sep 16, 2026
2 tasks

Copy link
Copy Markdown
Contributor Author

TEPP canary update: the previously pre-checkout Ready-head Rust Foundation specimen has now recovered on unchanged ContextualWisdomLab/TEPP#538 exact 0a6947b11f2dba9e161f2abe8bdec0f652d4b47a. Run 35045658457 is terminal SUCCESS; all four jobs obtained runner/checkout/steps and completed successfully: Live PostgreSQL 104634796152, production authored line/branch coverage 104634796279, repository/Python branch coverage 104634796299, and format/lint/test/rustdoc/dependency policy 104634796389. This is a useful GREEN transition for the same exact leaf head, not synthetic success and not evidence that the organization-wide queue incident is closed. On the same TEPP head, Security Scan 35045658427 and CodeQL PR 35045658451 are still queued/nonterminal, so leaf promotion remains blocked.

Copy link
Copy Markdown
Contributor Author

TEPP queue canary refinement on unchanged ContextualWisdomLab/TEPP#538@0a6947b11f2dba9e161f2abe8bdec0f652d4b47a: the top-level Security and CodeQL workflows are no longer uniformly pre-admission. Security Scan 35045658427 admitted Detect changed scope (104634796717) and completed it SUCCESS, then spawned four new jobs that are still pre-checkout queued with runner_id=0, empty runner names, and steps=[]: osv-scan 104707865051, scorecard 104707865097, dependency-review 104707865105, trivy-fs 104707865120. CodeQL PR 35045658451 likewise admitted Detect CodeQL languages (104634795923) SUCCESS, then left both compatibility analyses pre-checkout queued: Python 104703325392, Actions 104703325427. This narrows the specimen from whole-workflow admission failure to downstream fan-out job admission delay on the same exact head. Rust Foundation remains terminal SUCCESS. No leaf rerun, no-op push, cancel, or runner-selector change was used; this is queue-health evidence only, not incident closure or TEPP promotion evidence.

Copy link
Copy Markdown
Contributor Author

TEPP canary refinement on unchanged ContextualWisdomLab/TEPP#538@0a6947b11f2dba9e161f2abe8bdec0f652d4b47a: CodeQL fan-out has now admitted both compatibility jobs. Python 104703325392 and Actions 104703325427 each obtained a hosted runner and completed their verdict-read step, then intentionally concluded FAILURE because VERDICT_STATE=pending after a successful exact-head dispatch request; the log explicitly says the dispatch workflow will rerun the exact failed shard after publishing a terminal verdict. The follow-on Dispatch current-head CodeQL scan job 104768105534 is currently queued pre-checkout. This supersedes the prior description of those two compatibility jobs as pre-checkout queued: the current bottleneck has moved one stage downstream to dispatch admission/terminal verdict publication. Security Scan 35045658427 is unchanged: changed-scope classification succeeded, while osv-scan 104707865051, scorecard 104707865097, dependency-review 104707865105, and trivy-fs 104707865120 remain queued. No leaf rerun, no-op push, cancellation, runner-selector change, or synthetic success was used.

Copy link
Copy Markdown
Contributor Author

TEPP queue-health refinement on unchanged ContextualWisdomLab/TEPP#538@0a6947b11f2dba9e161f2abe8bdec0f652d4b47a: Security Scan fan-out is no longer a queue specimen. Run 35045658427 admitted all four downstream jobs; osv-scan 104707865051, scorecard 104707865097, and trivy-fs 104707865120 are terminal SUCCESS. dependency-review 104707865105 also obtained a hosted runner, verified the exact checkout, and then failed semantically/fail-closed because the authenticated public-repository dependency-graph compare returned HTTP 403 (curl_exit=0); the pinned Dependency Review action was therefore skipped. That failure belongs to canonical Dependency Review owner #810, not queue-health. CodeQL remains the live admission specimen: Dispatch current-head CodeQL scan 104768105534 is still queued pre-checkout after the two compatibility shards admitted and intentionally failed on pending verdict. No leaf rerun/no-op push/cancel/runner-selector change was used.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP canary update for queue-health classification: ContextualWisdomLab/TEPP#538 exact head 0a6947b11f2dba9e161f2abe8bdec0f652d4b47a no longer has the CodeQL dispatch job in pre-checkout queue. Dispatch current-head CodeQL scan job 104768105534 was admitted at 2026-09-16T17:56:18Z and completed SUCCESS at 17:56:27Z on the unchanged exact head. The earlier Python/Actions compatibility shards had already failed while the dispatch verdict was pending; after the dispatch succeeded, TEPP re-ran failed jobs through the supported Actions rerun operation rather than using a no-op push or gate bypass. Treat the old dispatch specimen as recovered evidence, not a current queue-health blocker. This does not close #1150's broader owner lane or transfer GREEN to other repositories.

Copy link
Copy Markdown
Contributor Author

Follow-up after the supported failed-job rerun on the same TEPP #538 exact head: CodeQL run 35045658451, attempt 2 now has fresh compatibility jobs 104915624181 (python) and 104915624335 (actions) both queued with runner_id=0 and steps=[]; language detection remains terminal SUCCESS. So the old dispatch pre-checkout specimen is recovered, but the rerun exposed a new current pre-checkout admission specimen at the compatibility-shard layer. Keep those identities separate in queue-health evidence; no additional rerun/no-op push/cancellation is being used while this attempt is queued.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP consumer evidence changes the CodeQL specimen classification for PR #538 exact 0a6947b11f2dba9e161f2abe8bdec0f652d4b47a.

CodeQL attempt 2 compatibility jobs are no longer pre-runner queue specimens: Python job 104915624181 and Actions job 104915624335 both obtained hosted runners and then terminally failed in Read current-head CodeQL dispatch verdict / Release runner or enforce current-head CodeQL verdict. Only after those failures, a new Dispatch current-head CodeQL scan job 104992789328 materialized and is currently queued pre-checkout with runner_id=0 and no steps.

So the current queue-health specimen is the dispatch job only. The two compatibility failures are a separate orchestration/order evidence path and should not be counted as runner-admission failures. No leaf rerun/no-op push/gate weakening was performed. TEPP #538 remains fail-closed pending a terminal current-head dispatch verdict.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP canary adds a distinct partial matrix-admission specimen on one unchanged exact head. ContextualWisdomLab/TEPP#538@0a6947b11f2dba9e161f2abe8bdec0f652d4b47a is Ready/open/mergeable. CodeQL PR run 35045658451 attempt 3 has language detection 105046923125 terminal SUCCESS and Python compatibility 105046943410 assigned to GitHub-hosted runner 1002000343, then terminal FAILURE while reading the current-head dispatch verdict. In the same run and exact head, Actions compatibility 105046922540 is still queued with runner_id=0, empty runner identity and steps=[].

This is not a Draft-guard specimen and not a source-test failure classification for the queued Actions shard: the sibling Python shard was admitted and executed while Actions remained pre-checkout. Please preserve the matrix/job identity distinction in queue-health evidence rather than collapsing the whole workflow into one queue state. No rerun, no-op commit, runner-selector change, cancellation, or synthesized status was used.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP #538 exact-head queue evidence (2026-09-17): CodeQL PR run 35045658451 on head 0a6947b11f2dba9e161f2abe8bdec0f652d4b47a has advanced since the prior leaf report. Detect CodeQL languages job 105046923125 is terminal SUCCESS. Compatibility jobs are now both terminal FAILURE: actions 105046922540 completed steps Set up jobRead current-head CodeQL dispatch verdict SUCCESS → Release runner or enforce current-head CodeQL verdict FAILURE; python 105046943410 failed its verdict/enforcement path. A newer Dispatch current-head CodeQL scan job 105120402219 is currently QUEUED with no steps materialized in the jobs payload. This supersedes the prior classification that the actions compatibility job itself was still pre-checkout queued. No leaf rerun, no-op commit, runner-selector mutation, or gate weakening was performed.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP leaf evidence on unchanged foundation #538 exact 0a6947b11f2dba9e161f2abe8bdec0f652d4b47a, CodeQL PR run 35045658451 attempt 3: language detection 105046923125 succeeded; Python compatibility 105046943410 ran on a hosted runner and failed at Read current-head CodeQL dispatch verdict; Actions compatibility 105046922540 also ran and reached terminal failure after the verdict read; follow-up Dispatch current-head CodeQL scan job 105120402219 remains queued with runner_id=0 and steps=[]. This remains a mixed terminal/pre-checkout admission state on one unchanged leaf head, so I did not rerun, cancel, change runner selectors, or mutate the TEPP foundation branch.

Copy link
Copy Markdown
Contributor Author

Fresh TEPP downstream queue evidence on unchanged PR #538 exact 0a6947b11f2dba9e161f2abe8bdec0f652d4b47a (2026-09-17 KST): CodeQL PR run 35045658451 remains queued. Language detection job 105237102626 completed SUCCESS with exact-head checkout/classification; compatibility jobs 105237101504 (python) and 105237101723 (actions) are both still queued with no reported steps. TEPP leaf did not rerun, cancel, no-op commit, or change runner selectors. This is the current owner-path evidence replacing the earlier attempt-3 terminal/dispatch snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: operations Operability, observability, readiness, SLO, backup, or retention enhancement New feature or request priority: medium Normal-priority or P2 work status: blocked Blocked by conflict, dependency, or required prerequisite type: feature New or expanded product capability

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants