Skip to content

perf(B5): pr-review-receipt moves off pull_request to ad-hoc dispatch — 39.4% of the merge path for ONE verdict in 30 days - #3052

Open
noahgift wants to merge 4 commits into
mainfrom
bse/receipt-adhoc
Open

perf(B5): pr-review-receipt moves off pull_request to ad-hoc dispatch — 39.4% of the merge path for ONE verdict in 30 days#3052
noahgift wants to merge 4 commits into
mainfrom
bse/receipt-adhoc

Conversation

@noahgift

@noahgift noahgift commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Ticket — PMAT-1091, issue #3049 (bse-001). Claim — B5: the largest measured waste on the merge path, moved so it can actually finish.

Measured before (basis=)

machines/clean-room/ci-occupancy.sh, 30 days of aprender PR-event runs, 302-run newest-first sample. PR-event CI costs 286.56 runner-hours:

job runs med min total h share
pr-review-receipt 76 95.7 112.88 39.4%
workspace-test 80 44.6 67.99 23.7%
guard-runner-labels 58 26.6 29.02 10.1%

What those 112.88 hours bought:

cancelled n=71   110.83 h        success n=1   2.05 h
cancelled durations (min): p25=34.1  med=95.7  p75=150.3  p90=150.4  max=151.1

One verdict in thirty days. And the cancels are not all supersedes: p75 sits on the declared timeout-minutes: 150, and GitHub reports a timeout kill as cancelled, not failed — so 32 of 71 ran the full 2.5 hours holding a clean-room slot and were then killed. ~80 runner-hours a month of pure timeout on a 16-slot fleet.

PRREV-015's own comment above that timeout predicted it: "a job that times out the first time the runner is busy, and a cancelled job reads as a failure nobody can distinguish from a real one." This is that, 32 times a month.

Change

if: github.event_name == 'pull_request'workflow_dispatch && inputs.pr != '', plus a pr input on workflow_dispatch.

gh workflow run ci.yml -f pr=<number>

Moved, not deleted. Deleting removes mutation coverage of the receipt guards. Moving lets it finish.

The trap this change would otherwise have walked into

PR_HEAD_SHA is removed from the job-level env. On workflow_dispatch there is no github.event.pull_request, so it would be the empty string — and the sweep does:

BASE=$(git merge-base origin/main "${PR_HEAD_SHA}" 2>/dev/null) || BASE=""

which tolerates an empty base. The job would have swept nothing and read green: the exact defect class this PR is motivated by, reintroduced by the fix. A first step resolves the head from the PR number, asserts a 40-char sha, asserts it is fetchable, and refuses otherwise. It runs before both PR_HEAD_SHA consumers (verified by line order: resolve at job-relative 61, consumers at 201 and 214).

Queries at the change site

All three are written into ci.yml above the if:, not left in a commit message:

  1. required? gh api repos/paiml/aprender/rules/branches/main["gate","workspace-test"]. Not this job. (The classic branch-protection API says ci / gate — a different job. The ruleset is what blocks.)
  2. needs:-ed? grep over every workflow on origin/main → 7 hits, six are comments, the seventh is its own job key. Nothing consumes it.
  3. still covered? Yes, for 5% of the price: present in pr-review-quorum.yml (pull_request_target, base-owned) is 98 runs, 1.7-min median, 5.93 h, and pr-review-sign still signs on every PR. What moves is the mutation sweep over those guards.

Predicted after

~112.9 runner-hours per 30 days off the PR merge path, ~80 h of it pure timeout. Second-largest leg (workspace-test, 68.0 h) becomes the top row and the next measurement.

Writes

.github/workflows/ci.yml (one job's trigger + one new step + the input), docs/roadmaps/roadmap.yaml (+19/−0, added=1 reserialised=0 deleted=0, guard PASS). No script, no crate, no PP-066 row.

actionlint clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EeUX4ymyt8XhjXbtTuNH8L

… — 39.4% of the merge path for ONE verdict in 30 days (#3049)

MEASURED, machines/clean-room/ci-occupancy.sh, 30 days of aprender PR-event runs
(302-run newest-first sample):

  pr-review-receipt   76 runs   95.7 min median   112.88 runner-h   39.4%
  --------------------------------------------------  of ALL PR-event hours
  cancelled n=71 (110.83 h)      success n=1 (2.05 h)
  cancelled durations (min): p25=34.1 med=95.7 p75=150.3 p90=150.4 max=151.1

PR-event CI costs 286.56 runner-hours over that window and this job is 112.88 of
them. It returned ONE verdict.

The cancels are not all supersedes. p75 sits ON the declared timeout-minutes: 150,
and GitHub reports a timeout kill as `cancelled`, not `failed` — so 32 of the 71
ran the full 2.5 hours holding a clean-room slot and were then killed. ~80
runner-hours a month of pure timeout on a 16-slot fleet. PRREV-015's own comment
above that timeout predicted exactly this — "a job that times out the first time
the runner is busy, and a cancelled job reads as a failure nobody can distinguish
from a real one" — and this is that prediction coming true 32 times a month.

A sweep that completes 1 run in 76 is not mutation coverage. It is the
"0 violations over 0 files" shape aimed at this repo's own mutation gate.

THE THREE QUERIES, recorded at the change site rather than in a commit nobody
re-reads:

  1. required?  the RULESET (not the classic branch-protection API, which names a
                DIFFERENT job) says ["gate","workspace-test"]. Not this one.
  2. needs:-ed? 7 grep hits across every workflow on origin/main; six are
                comments, the seventh is its own job key. Nothing consumes it.
  3. covered?   yes, and for 5% of the price: `present` in pr-review-quorum.yml
                (pull_request_target, base-owned) is 98 runs at a 1.7-min median
                = 5.93 h, and pr-review-sign still signs on every PR. What moves
                is the mutation sweep OVER those guards, nothing else.

MOVED, NOT DELETED: deleting removes mutation coverage of the receipt guards.
Moving lets it finish.

THE TRAP THIS AVOIDS. PR_HEAD_SHA is REMOVED from the job-level env. On
workflow_dispatch there is no `github.event.pull_request`, so it would be the
empty string — and the sweep does

    BASE=$(git merge-base origin/main "${PR_HEAD_SHA}" 2>/dev/null) || BASE=""

which TOLERATES an empty base. The job would have swept nothing and read green:
the exact defect class this change is motivated by, reintroduced by the change
itself. A first step resolves the head from the PR number, asserts a 40-char sha,
asserts it is fetchable, and REFUSES otherwise. It runs before both consumers.

  gh workflow run ci.yml -f pr=<number>

actionlint clean. Refs #3049, PMAT-1091.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUX4ymyt8XhjXbtTuNH8L
@noahgift noahgift added the bse-001 BSE-001 build-system enhancement / unblocker label Sep 8, 2026
@noahgift

noahgift commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): three PASS — agreed

{
 "ticket": "PMAT-1091",
 "head": "0e8dd9df80b5beec44ce621e967f8acd4c364cff",
 "width": 3,
 "executor": "agy",
 "agreed": true,
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

Three independent agy lanes judged the diff against PMAT-1091. AGREED: lane
1=PASS, lane 2=PASS, lane 3=PASS, no findings. All three independently confirmed
the part that matters — that the new resolve step runs BEFORE both PR_HEAD_SHA
consumers and refuses rather than letting an empty sha reach a merge-base that
tolerates an empty base.

Refs #3049, PMAT-1091.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUX4ymyt8XhjXbtTuNH8L
@noahgift
noahgift enabled auto-merge September 8, 2026 12:43
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3052 head=a9d60adebc052dc8d253511caa12fbc1ff5ee0df verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

…ly where a PR subject exists" (#3049)

guard-tree caught the ci.yml change, correctly, and the diagnosis is better than
a parser gap:

    FAIL R4: this guard cannot evaluate
             `github.event_name == 'workflow_dispatch' && inputs.pr != ''`

R4 asserted a fixed pair — TRUE on pull_request, FALSE on push, merge_group and
workflow_dispatch — with the reason stated in its own message: "There is no PR
number on this event". That was TRUE until this PR gave workflow_dispatch a `pr`
input. A rule whose premise has changed is worse than no rule: it refuses the one
wiring that makes the receipt path well-defined.

So R4 now states the property directly, as a QUANTIFIER over events rather than a
list: every event the `if:` selects must carry a subject for
evidence/pr-review/<pr>/<sha>/, and it must select at least one.

  pull_request       always (the pull_request payload)
  workflow_dispatch  iff the `if:` REQUIRES a non-empty input AND that input is
                     DECLARED under on.workflow_dispatch.inputs
  push, merge_group  never, on any wiring

TWO HOLES THE CASE TABLE FOUND, neither of which I set out to fix:

  * An undeclared input is ALWAYS empty, so `inputs.prr != ''` naming an input
    the workflow does not declare is a job that never runs and reports nothing —
    dark, and indistinguishable from a deliberate disable. Now RED by name.
  * `inputs.*` is populated on workflow_dispatch and EMPTY on every other event,
    so `github.event_name == 'pull_request' && inputs.pr != ''` never runs on
    anything. Shape B is admissible only with the workflow_dispatch literal;
    anything else is refused rather than evaluated.

AND ONE IN MY OWN FIRST DRAFT, which is why the property is written as a
quantifier: I initially wrote a want-per-event table with `pull_request -> runs`
hardcoded, reintroducing the exact staleness I was removing. It refused the
dispatch wiring it existed to admit.

The evaluator is still narrow and still refuses rather than guesses — shape B is
a SEPARATE pattern, not a widened regex, per this guard's own header: "Extending
the evaluator is a code change with a new case-table row, not an accident."

29/29 rows (was 25), four new, one per FORM VARIANT. Three mutations, each caught
by the row built for it:

  drop the declared-input check      -> row 17
  allow shape B on any literal       -> row 18
  dispatch with no required input OK -> row 16

Refs #3049, PMAT-1091.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUX4ymyt8XhjXbtTuNH8L
@noahgift

noahgift commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): three PASS — agreed

{
 "ticket": "PMAT-1091",
 "head": "80ea9ce0780fc7a2ac0ae62602350b524182e774",
 "width": 3,
 "executor": "agy",
 "agreed": true,
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

…l merge

The first verdict judged a tree without the check_pr_review_wiring.sh change, so
it was void the moment guard-tree caught R4 and the fix landed. Re-run against
the actual head: lane 1=PASS, lane 2=PASS, lane 3=PASS.

Refs #3049, PMAT-1091.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUX4ymyt8XhjXbtTuNH8L
@noahgift noahgift added this to the 0.68.0 milestone Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bse-001 BSE-001 build-system enhancement / unblocker

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant