diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index b36cc3199..91cac9a5d 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -2568,3 +2568,4 @@ together (the script's header says why there is no escape hatch). - 2026-09-26 — The lab store is keyed `~/.abcd/lab///`, with one `index.jsonl` registry per root-sha lane beside the lab homes (lane implementer, autonomous run A, on review-lab's third finding against spc-2609212141418943 for itd-2609212137128014). This supersedes two recorded texts: the spec's literal `~/.abcd/lab//` (scope item 1), and the 2026-08-31 lab-convention entry's hand-run keying `~/.abcd/lab/-/` with a single top-level `~/.abcd/lab/index.jsonl`, whose stated divergence from root-sha keying is withdrawn. Why: the intent's scope condition keys the store "as the other machine-scoped stores are", and the worktree and transcript stores key on the repository's root commit, because a checkout moves, is renamed and is cloned twice on one machine while its root commit does none of that; a lab's identity is still its intention, carried by its id `lab--` (the UTC mint time and the pin), so several labs share one baseline inside one lane. The hand-run labs that predate the verb stay where they are, beside the root-sha lanes, and the verb neither reads nor writes them or the top-level registry, so no real lab is moved or migrated by the change. A later text naming `~/.abcd/lab//` (the open spc-2609221011151661's `pairs.jsonl` among them) means the lab home inside its root-sha lane. - 2026-09-26 — Three of the rules loader's security records close, and one stays owed to the product thinker (autonomous run A orchestrator's lane brief, taken by the implementer of lane drainS2). (1) The home directory is never a session's repo root (iss-2609020219198779, answering the owed question "is a home-directory git toplevel a legitimate config scope, or excluded outright?" as the brief rules it): its `.abcd/` is the user layer, so the root walk passes over the home and a toplevel that is the home resolves like a non-repo directory. The lane narrowed the brief's "or an ancestor of HOME": a toplevel that contains the home, the shape of a hermetic harness that points `HOME` inside its checkout, stays the root because git vouched for it and its own `.abcd/` is its own; only the stop at the home is removed. A session whose working directory is the home still reads a `.abcd/` there as the working directory's, the posture question recorded on 2026-09-25. (2) A bundled guardrail that an override withholds is named on every load (iss-174): for COMMITTING, LOAD and PII, each bundled recall keyword, alias or rule missing from a list an override set goes to stderr with the file whose list is in force, and the merge stays per field. The other bundled domains are left out because a repository restates them in its own words, and a note on every restatement would bury the one that matters. Still owed to the product thinker: whether security-bearing lists should union with the bundled entries or take a replace-versus-extend marker instead (itd-117's finer-grained-merging follow-up). (3) The foreign-uid refusal says what it still reads (iss-2609251522588539): the note, the configuration chapter and the install how-to now say that a `.abcd/` at the working directory is read, as AGENTS.md has since 0434d475. (4) iss-2609020219265817 is deferred past v0.11.0, not closed. Every CommonMark heading construct in a rule body (ATX on any line, the first line included, setext, and HTML h1-h6) can be closed only by a code-safe rendering that flattens legitimate structure, or by a fence-aware escaper that is complete only by enumeration and changes the raw text the model reads. So "escaped, fenced, or left to the line-start contract" is the product thinker's ruling. - 2026-09-26 — The build loop's worktree store is keyed on the FULL root sha: a lane lives at `~/.abcd/worktrees//-` with the 40-hex root commit, the form the history, transcript and voyage stores use and the one the store's draft (itd-2609091014076309) specifies. The lanes of autonomous run A made by hand under the abbreviated key (`~/.abcd/worktrees/488a0aa9//`) are the pre-verb convention, not a second form of the store: the loop never reads or adopts a lane under that key, and those worktrees are retired with `git worktree remove` like any other (implementer of fix round fix2-loop2, autonomous run A, on item 4 of the loop2 review; spc-2609202134338445 piece 6). +- 2026-09-28 — Rulings Z, AR and the cancel policy, given by the user as technical facilitator at 15:10:37Z (autonomous run A, recorded by lane cap45 for orchestrator abcd-a8). (Z) The macOS leg of ci.yml's `check` job and the main ruleset's merge-queue `check_response_timeout_minutes` both rise from 30 to 45 minutes, because a 30-minute cap cancelled passing macOS runs (#728, #730 twice, #733; iss-2609281514435020). This amends the standing rule that never raises the check job's timeout or edits the ruleset, for exactly this one change: 45 minutes, those two settings; every other timeout, every required-check name or split and every other ruleset field stays as it is, and the ubuntu leg keeps 30. The workflow and the `.abcd/work/rulesets/main-protection.json` mirror change through a reviewed pull request; the live ruleset is changed with `gh api` by the run's orchestrator after that pull request merges, and until then the live queue still fails a group at 30 minutes. (AR) All three speed-ups of iss-2609261924541555 are built: a test-only switch that skips the disk flush in the atomic write code and the slowest -race package first in the race step (lane ciSpeed), and the scanner's per-identity git calls folded into one (lane scanFold), a trust path that is security-reviewed before it lands. (Cancel policy) The rerun-once rule stands: a check cancelled at the cap is rerun once, and a second cancellation for the same reason stops that pull request and opens a speed lane, never another raise of the timeout; and a step on the macOS leg warns, in the log and the step summary, once the check has run past 35 minutes, without ever failing the job, so a speed lane opens before any cancellation. diff --git a/.abcd/work/issues/resolved/iss-2609281514435020-the-merge-queue-fails-a-group-whose-macos-check-job-runs.md b/.abcd/work/issues/resolved/iss-2609281514435020-the-merge-queue-fails-a-group-whose-macos-check-job-runs.md new file mode 100644 index 000000000..5d2058320 --- /dev/null +++ b/.abcd/work/issues/resolved/iss-2609281514435020-the-merge-queue-fails-a-group-whose-macos-check-job-runs.md @@ -0,0 +1,22 @@ +--- +schema_version: 1 +id: "iss-2609281514435020" +slug: "the-merge-queue-fails-a-group-whose-macos-check-job-runs" +severity: "minor" +category: "process" +source: "review-followup" +found_during: "autonomous run A resumed 2026-09-25: ruling Z" +origin: researcher-authored +production_mode: hand-written +found_at: ".github/workflows/ci.yml" +resolution: "The macOS leg of ci.yml's check job carries a 45-minute ceiling (the ubuntu leg keeps 30) and the ruleset mirror's merge-queue check_response_timeout_minutes records 45, under ruling Z of 2026-09-28; the live ruleset follows with gh api after the merge. A step at the end of the macOS leg warns in the log and the step summary once the check passes 35 minutes, naming the rerun-once rule and the speed lane, and never fails the job. TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds every leg at or below the mirror's cap and TestCheckJobWarnsBeforeTheQueueCap pins the 35-minute threshold below it." +impact: internal +resolved_by: + commit: "b248cb647" +--- + +The merge queue fails a group whose macOS check job runs past 30 minutes, cancelling pull requests whose code is fine: #728 was cancelled at 30.3 minutes, #730 twice at the cap, and #733 once. The check job's timeout-minutes and the main ruleset's merge-queue check_response_timeout_minutes are both 30, while the macOS leg of a source change measures 23 to 30 minutes, so ordinary runner variance cancels a passing run, and nothing warns before the cancellation arrives. Ruling Z (2026-09-28, the technical facilitator) raises both limits to 45 minutes and asks for a warning once the macOS check passes 35 minutes, so a speed lane opens before any cancellation. + +## Grounds + +- pursued: a passing macOS check that runs between 30 and 45 minutes concludes and merges instead of being cancelled by the queue, and one that passes 35 minutes shows the warning; a merge-group run cancelled at 30 minutes after the live ruleset is raised, a check failed by the warning step, or a run past 35 minutes with no warning would show it wrong. diff --git a/.abcd/work/rulesets/main-protection.json b/.abcd/work/rulesets/main-protection.json index 45c51d777..65492bea8 100644 --- a/.abcd/work/rulesets/main-protection.json +++ b/.abcd/work/rulesets/main-protection.json @@ -55,7 +55,7 @@ }, { "parameters": { - "check_response_timeout_minutes": 30, + "check_response_timeout_minutes": 45, "grouping_strategy": "ALLGREEN", "max_entries_to_build": 5, "max_entries_to_merge": 5, diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a96ff0aa9..091a4f398 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -223,24 +223,39 @@ jobs: check: needs: changes if: ${{ !cancelled() }} - # Equal to the merge queue's check response timeout (30 minutes, recorded in - # .abcd/work/rulesets/main-protection.json), which is the outer ceiling on a - # merge-group run: the queue fails the group when this required check has - # not concluded by then, so a job ceiling above it is unreachable there. The - # race step's -timeout lifts go test's 10m per-package default, so a slow - # package runs on to this ceiling instead of failing at ten minutes; it does - # not fit inside the ceiling on the macOS leg, which spends about 18m before - # the slowest package starts, so a hang there is cancelled without a - # goroutine dump. Raising the queue's cap is an admin act on the live - # ruleset, left to the technical facilitator; this job may follow it then. - # TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds this at or below the cap. - timeout-minutes: 30 + # The macOS leg's ceiling is equal to the merge queue's check response + # timeout (45 minutes, recorded in .abcd/work/rulesets/main-protection.json), + # which is the outer ceiling on a merge-group run: the queue fails the group + # when this required check has not concluded by then, so a job ceiling above + # it is unreachable there. Both are 45 under ruling Z (2026-09-28, the + # technical facilitator), because a 30-minute cap cancelled passing macOS + # runs of 23 to 30 minutes (iss-2609281514435020). Raising either again is + # an admin act on the live ruleset, left to the technical facilitator, and + # the remedy for a leg near the cap is a faster suite. The ubuntu leg runs + # well inside 30 minutes and keeps that ceiling. The race step's -timeout lifts go test's 10m + # per-package default, so a slow package runs on to this ceiling instead of + # failing at ten minutes; it does not fit inside the ceiling on the macOS + # leg, which spends about 18m before the slowest package starts, so a hang + # there is cancelled without a goroutine dump. + # TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds every leg at or below the + # cap, and TestCheckJobWarnsBeforeTheQueueCap holds the warning step at the + # end of the job below it. + timeout-minutes: ${{ matrix.os == 'macos-latest' && 45 || 30 }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest] runs-on: ${{ matrix.os }} steps: + # The clock for the warning step at the end of the job. It is the first + # step, so the time the runner spent before it (queueing, set-up) is not + # counted, and the elapsed time the warning reads is a lower bound on + # what the merge queue's clock shows. + - name: Record the job's start + id: started + if: matrix.os == 'macos-latest' + run: echo "epoch=$(date +%s)" >>"$GITHUB_OUTPUT" + # Full history: the record-lint step below arms `-agent-diff` against the # base commit, and a three-dot range needs the base object AND a merge # base, neither of which a depth-1 clone holds — with the default shallow @@ -301,9 +316,9 @@ jobs: # unrelated pull request from the merge queue (iss-2609260319483365). 20m # is about twice the slowest passing run, so a real hang still fails with # a goroutine dump where the enclosing ceiling leaves room for it: in - # release.yml's verify job it does, while this job's 30 minutes, held to - # the merge queue's response cap, are the binding ceiling here (see the - # job header). The Makefile's preflight recipe and release.yml's verify + # release.yml's verify job it does, while this job's own ceiling, held to + # the merge queue's response cap, is the binding one here (see the job + # header). The Makefile's preflight recipe and release.yml's verify # job carry the same flag; TestRaceLaneBudgetIsDeclaredAndFitsItsJob # holds all three. - name: Test (race, internal) @@ -392,6 +407,42 @@ jobs: go run ./cmd/abcd site build --out "$RUNNER_TEMP/site-render-check" go run ./cmd/abcd lint site --out "$RUNNER_TEMP/site-render-check" + # Warn before the cap cancels (ruling Z's cancel policy, 2026-09-28). A + # macOS check that runs past WARN_AFTER_MINUTES is ten minutes from the + # merge queue's cap, and a cancellation there carries nothing but the + # cancellation. The rule for a cancelled check is to rerun it once; a + # second cancellation for the same reason stops that pull request and + # opens a speed lane, never another raise of the timeout. This warning + # moves that signal ahead of the first cancellation: it is emitted at the + # job's end, for a run that finished past the line, so a check that + # passed between 35 and 45 minutes says so before one is cancelled at + # the cap. It never fails the + # job: `always()` runs it however the steps before it ended, and + # `continue-on-error` holds even a broken script to a warning. + # TestCheckJobWarnsBeforeTheQueueCap pins the threshold below the leg's + # ceiling. + - name: Warn when the check nears the queue's cap + if: always() && matrix.os == 'macos-latest' + continue-on-error: true + env: + STARTED_AT: ${{ steps.started.outputs.epoch }} + WARN_AFTER_MINUTES: 35 + run: | + case "${STARTED_AT:-}" in + '' | *[!0-9]*) + echo "::notice::no job start was recorded, so the elapsed-time warning did not run" + exit 0 + ;; + esac + elapsed=$(($(date +%s) - STARTED_AT)) + minutes=$((elapsed / 60)) + echo "check (macos-latest) has run ${minutes}m $((elapsed % 60))s; the warning line is ${WARN_AFTER_MINUTES}m" + if [ "$elapsed" -gt $((WARN_AFTER_MINUTES * 60)) ]; then + msg="check (macos-latest) ran ${minutes} minutes, past the ${WARN_AFTER_MINUTES}-minute warning line and close to the merge queue's cap. If it is cancelled at the cap, rerun it once; a second cancellation for the same reason stops the pull request: open a speed lane to make the suite faster, and do not raise the timeout." + echo "::warning title=macOS check near the queue's cap::${msg}" + printf '### macOS check near the merge queue cap\n\n%s\n' "$msg" >>"$GITHUB_STEP_SUMMARY" + fi + # Secrets never land in history: the history each run would land is scanned — # a pull request's own commits, and the full history of the pushed or queued # commit — never every ref the checkout fetched. Run as a diff --git a/internal/core/lint/racelanebudget_test.go b/internal/core/lint/racelanebudget_test.go index b2adecda8..ecf33682d 100644 --- a/internal/core/lint/racelanebudget_test.go +++ b/internal/core/lint/racelanebudget_test.go @@ -27,13 +27,16 @@ import ( // queue fails the group when a required check has not concluded within the // ruleset's check_response_timeout_minutes. That cap is the outer ceiling on // the event the lane was ejected on, and a job ceiling above it is unreachable -// there: the queue gives up first, with no job failure to read. So the check -// job's timeout-minutes may not exceed the cap the ruleset mirror records. The -// step's -timeout still earns its place under that cap: it lifts the ten-minute -// per-package default, so a slow package fails on the job's clock rather than -// at ten minutes. Raising the queue's cap is an admin act on the live ruleset, -// left to the technical facilitator; this test reads the mirror, so the day the -// cap is raised the check job may follow it. +// there: the queue gives up first, with no job failure to read. So no leg of +// the check job may carry a timeout-minutes above the cap the ruleset mirror +// records. Under ruling Z (2026-09-28, the technical facilitator) the cap is 45 +// minutes and the macOS leg follows it, while the ubuntu leg keeps 30; the job +// declares that as one matrix expression, which checkLegTimeouts reads leg by +// leg. The step's -timeout still earns its place under that cap: it lifts the +// ten-minute per-package default, so a slow package fails on the job's clock +// rather than at ten minutes. Raising the queue's cap is an admin act on the +// live ruleset, left to the technical facilitator; this test reads the mirror, +// so a job ceiling can rise only after the mirror records the raise. // // release.yml's verify job is not a merge-queue job, so its own timeout-minutes // is its ceiling. A package timeout that reaches past it never fires: the @@ -75,14 +78,21 @@ func TestRaceLaneBudgetIsDeclaredAndFitsItsJob(t *testing.T) { if check, _, ok := raceJob(".github/workflows/ci.yml", "check"); ok { const where = ".github/workflows/ci.yml job check" - capMin := jobTimeoutMinutes(t, where, check) queue := mergeQueueResponseTimeout(t, root) - if capMin > queue { - t.Errorf("%s: timeout-minutes is %s, above the merge queue's %s check response "+ - "timeout in %s; on a merge-group run the queue fails the group first, so the "+ - "job ceiling is unreachable there and any budget sized against it is false. "+ - "Raise the live ruleset's cap (an admin act) before the job's", - where, capMin, queue, rulesetMirror) + for _, leg := range checkLegTimeouts(t, where, check) { + if leg.minutes > queue { + t.Errorf("%s: the %s leg's timeout-minutes is %s, above the merge queue's %s check "+ + "response timeout in %s; on a merge-group run the queue fails the group first, "+ + "so the job ceiling is unreachable there and any budget sized against it is "+ + "false. Raise the live ruleset's cap (an admin act) before the job's", + where, leg.os, leg.minutes, queue, rulesetMirror) + } + // Ruling Z raised the macOS leg alone; every other leg keeps 30. + if leg.os != "macos-latest" && leg.minutes != 30*time.Minute { + t.Errorf("%s: the %s leg's timeout-minutes is %s; ruling Z (2026-09-28) raised only "+ + "the macOS leg, and every other leg of the check job stays at 30m", + where, leg.os, leg.minutes) + } } } @@ -102,6 +112,132 @@ func TestRaceLaneBudgetIsDeclaredAndFitsItsJob(t *testing.T) { } } +// checkWarnAfterMinutes is where the macOS leg of ci.yml's check job warns that +// it is nearing its ceiling: ruling Z's cancel policy (2026-09-28) sets it at 35 +// minutes, ten under the 45-minute cap, so a speed lane opens before the queue +// cancels anything. +const checkWarnAfterMinutes = 35 + +// checkWarnStep is the name of that warning step in the check job. +const checkWarnStep = "Warn when the check nears the queue's cap" + +// The macOS leg warns before the merge queue's cap cancels it (ruling Z's +// cancel policy, 2026-09-28). A cancellation at the cap carries nothing but the +// cancellation, and the rule for it is rerun once, then stop that pull request +// and open a speed lane; the warning moves that signal ahead of the first +// cancellation. It is a warning and nothing else: it may not fail the job, it +// runs however the steps before it ended, and its threshold sits below the +// leg's ceiling, or it could never fire before the cancellation it announces. +func TestCheckJobWarnsBeforeTheQueueCap(t *testing.T) { + root := filepath.Join("..", "..", "..") + const where = ".github/workflows/ci.yml job check" + job, ok := workflowJobBlock(readRepoFile(t, root, ".github/workflows/ci.yml"), "check") + if !ok { + t.Fatalf("%s: no such job; the parser or the workflow changed shape", where) + } + step, ok := workflowStepBlock(job, checkWarnStep) + if !ok { + t.Fatalf("%s: no %q step; the macOS leg would reach the merge queue's cap with no "+ + "warning ahead of the cancellation", where, checkWarnStep) + } + + m := regexp.MustCompile(`(?m)^\s+WARN_AFTER_MINUTES: "?(\d+)"?\s*$`).FindStringSubmatch(step) + if m == nil { + t.Fatalf("%s step %q: no WARN_AFTER_MINUTES in its env", where, checkWarnStep) + } + warn, _ := strconv.Atoi(m[1]) + if warn != checkWarnAfterMinutes { + t.Errorf("%s step %q warns after %d minutes; the cancel policy of ruling Z sets %d", + where, checkWarnStep, warn, checkWarnAfterMinutes) + } + warnAfter := time.Duration(warn) * time.Minute + for _, leg := range checkLegTimeouts(t, where, job) { + if leg.os == "macos-latest" && warnAfter >= leg.minutes { + t.Errorf("%s step %q warns after %s, not below the macos-latest leg's %s ceiling; "+ + "the job is cancelled before the warning can fire", where, checkWarnStep, warnAfter, leg.minutes) + } + } + if queue := mergeQueueResponseTimeout(t, root); warnAfter >= queue { + t.Errorf("%s step %q warns after %s, not below the merge queue's %s cap in %s", + where, checkWarnStep, warnAfter, queue, rulesetMirror) + } + + ifLine := regexp.MustCompile(`(?m)^\s+if: (.*)$`).FindStringSubmatch(step) + if ifLine == nil || !strings.Contains(ifLine[1], "always()") || !strings.Contains(ifLine[1], "macos-latest") { + t.Errorf("%s step %q: want an `if:` naming always() and the macos-latest leg, so it runs "+ + "on that leg however the steps before it ended", where, checkWarnStep) + } + if !regexp.MustCompile(`(?m)^\s+continue-on-error: true\s*$`).MatchString(step) { + t.Errorf("%s step %q: want `continue-on-error: true`; a warning may never fail the job", where, checkWarnStep) + } + at := strings.Index(step, "run:") + if at < 0 { + t.Fatalf("%s step %q has no run: script", where, checkWarnStep) + } + run := step[at:] + for _, want := range []string{"::warning", "GITHUB_STEP_SUMMARY", "rerun", "speed lane"} { + if !strings.Contains(run, want) { + t.Errorf("%s step %q: its script never mentions %q; the warning names the rerun-once "+ + "rule and the speed lane, in the log and the step summary", where, checkWarnStep, want) + } + } + if strings.Contains(run, "${{") { + t.Errorf("%s step %q interpolates an expression inside run:; pass values through env", where, checkWarnStep) + } +} + +// checkLeg is one matrix leg's job ceiling. +type checkLeg struct { + os string + minutes time.Duration +} + +// checkLegTimeouts reads the check job's `timeout-minutes:` per matrix leg. The +// value is a plain number, which every leg shares, or the one expression +// `${{ matrix.os == '' && || }}`, which gives its own ceiling +// and every other leg . Legs come from the job's `os: [...]` matrix line. +func checkLegTimeouts(t *testing.T, where, job string) []checkLeg { + t.Helper() + osLine := regexp.MustCompile(`(?m)^\s+os: \[([^\]]*)\]\s*$`).FindStringSubmatch(job) + if osLine == nil { + t.Fatalf("%s: no `os: [...]` matrix line", where) + } + var legs []checkLeg + for _, name := range strings.Split(osLine[1], ",") { + legs = append(legs, checkLeg{os: strings.TrimSpace(name)}) + } + minutes := func(s string) time.Duration { + n, err := strconv.Atoi(s) + if err != nil { + t.Fatalf("%s: timeout-minutes %q: %v", where, s, err) + } + return time.Duration(n) * time.Minute + } + if m := regexp.MustCompile(`(?m)^ timeout-minutes: (\d+)\s*$`).FindStringSubmatch(job); m != nil { + for i := range legs { + legs[i].minutes = minutes(m[1]) + } + return legs + } + m := regexp.MustCompile(`(?m)^ timeout-minutes: \$\{\{ matrix\.os == '([a-z0-9.-]+)' && (\d+) \|\| (\d+) \}\}\s*$`).FindStringSubmatch(job) + if m == nil { + t.Fatalf("%s: timeout-minutes is neither a number nor "+ + "`${{ matrix.os == '' && || }}`", where) + } + named := false + for i := range legs { + if legs[i].os == m[1] { + legs[i].minutes, named = minutes(m[2]), true + } else { + legs[i].minutes = minutes(m[3]) + } + } + if !named { + t.Fatalf("%s: timeout-minutes names %q, which is not a leg of the matrix", where, m[1]) + } + return legs +} + // rulesetMirror is the tree's record of the live branch ruleset on main. const rulesetMirror = ".abcd/work/rulesets/main-protection.json" diff --git a/internal/surface/cli/history_capture_cap_test.go b/internal/surface/cli/history_capture_cap_test.go index 03e671d05..a22f768f7 100644 --- a/internal/surface/cli/history_capture_cap_test.go +++ b/internal/surface/cli/history_capture_cap_test.go @@ -31,7 +31,7 @@ import ( // detector has nothing of this package's to watch, and instrumenting // those passes cost 125s against 6s uninstrumented on the same machine: a third // of this package's race time, on the macOS leg whose job is held to the merge -// queue's 30-minute cap (iss-2609261924541555). The uninstrumented lane asserts +// queue's 45-minute cap (iss-2609261924541555). The uninstrumented lane asserts // the same cap on both CI legs. func TestHistoryCaptureAcceptsWhatTheHooksAccept(t *testing.T) { if raceEnabled {