69% of development push runs are cancelled: the release loop outruns Code Quality

[rubenvdlinde]

69% of the fleet's development push runs are cancelled — the release loop outruns CI

Measured 2026-08-20, last ≤15 completed Code Quality push runs per app on development:

repo	runs	cancelled	success	failure
openregister	8	7	1	0
opencatalogi	13	12	1	0
openconnector	7	2	4	1
docudesk	10	9	1	0
nldesign	14	1	13	0
launchpad	15	2	12	1
softwarecatalog	11	10	1	0
larpingapp	11	10	1	0
zaakafhandelapp	11	10	1	0
procest	11	10	1	0
pipelinq	11	10	1	0
shillinq	9	8	0	1
scholiq	10	9	1	0
portaliq	12	6	6	0
decidesk	13	12	1	0
openbuild	12	5	4	3
doriath	13	8	5	0
hermiq	9	8	1	0
fleet	200	139 (69%)	55	6

Mechanism — measured, not inferred

quality.yml carries, correctly:

concurrency:
  group: quality-${{ github.ref }}
  cancel-in-progress: true

The group is keyed on the branch, so a new push to development cancels the in-flight verification of the previous one. That is the right design — once commit N+1 exists, verifying N matters less.

The failure is one of cadence, not of the concurrency rule:

• Merges into development arrive every 8–12 minutes. An automated release cycle pushes chore(release): X.Y.Z-unstable.<ts> [skip ci], then a release/vX PR is merged. Observed on decidesk, scholiq, shillinq, hermiq simultaneously.
• A Code Quality run takes 14–23 minutes (decidesk median 14, scholiq median 23; 30–42 jobs including a PHPUnit matrix and Playwright).

Because the run is slower than the interval, the newest commit's run is cancelled too. The branch never converges to a verdict — it only does so during a lull, which is exactly what the 55 successes are.

Why this matters more than it looks

A cancelled run is not a failure, but it is not a pass either. It is no verdict at all, and it renders identically to "nothing wrong here" in every summary that counts failures. Roughly two thirds of what lands on development is currently unverified.

It also undermines the gate-drift routine tracked in #523: even once FLEET_DISPATCH_TOKEN is provisioned, a Friday sweep dispatching Code Quality on development will be cancelled by the next release merge, because a workflow_dispatch run shares the same quality-refs/heads/development concurrency group. Provisioning the token is necessary but not sufficient.

Options (needs a decision — I have not changed anything here)

1. Throttle the release loop — batch the unstable releases (e.g. hourly) so CI has room to finish. Highest signal recovered per unit of change.
2. Give sweeps their own concurrency group — e.g. group: quality-${{ github.ref }}-${{ github.event_name }}, so a dispatched drift sweep is not cancelled by a routine merge. Fixes Fleet gate-drift detection has never run: FLEET_DISPATCH_TOKEN is not provisioned #523's interaction but not the everyday gap.
3. Accept it and verify at the promotion boundary instead — treat development as explicitly unverified and make dev→beta the real gate. Honest, but it moves the discovery of breakage much later.

My recommendation is 1 + 2: the loop is the root cause, and the concurrency-group split keeps the drift sweep meaningful regardless of merge traffic.

[rubenvdlinde]

Direct confirmation: a 60-minute observation window, zero verdicts

I re-ran the failed jobs on the four apps that went red on the CDN 504 (see #522), then watched their development heads for a full hour, polling every 60s and requiring status == "completed".

Result after 60 polls: none of the five ever reached a terminal state.

decidesk     queued     larpingapp   queued     pipelinq   queued
scholiq      queued     shillinq     pending

What happened during that hour, per app:

app	Code Quality runs started	cancelled	success
decidesk	3	2	0
larpingapp	3	2	0
pipelinq	3	2	0
scholiq	3	2	0
shillinq	4	2	0

Sixteen runs started. Ten cancelled. Zero verdicts. The remainder were still queued when the window closed, and will themselves be cancelled by the next release merge.

The heads moved three times during the observation (e.g. decidesk 5cfccae → ba282c0 → 226eea8), each move cancelling the run for the previous one — exactly the mechanism in the issue body.

Why this is worth stating plainly

The four apps in #522 are not red and not green — they are unmeasured, and have been for at least an hour. A dashboard counting failures shows them as fine. So does a dashboard counting successes, by omission.

This also means "get the fleet to all-green on development" is not currently an achievable end state, regardless of code quality: the branch cannot hold still long enough to be verified. Any green claim about development right now is a claim about whichever commit last got lucky with a gap in the release cadence.

Throttling the release loop (option 1 in the issue body) is the only one of the three options that restores routine verification. The concurrency-group split (option 2) rescues the drift sweep but leaves everyday commits unverified.

[rubenvdlinde]

Ruling out the alternative explanation: Actions is healthy, the cadence is the cause

Three hours after the first observation, the same five apps still have no verdict on their development head. Two checks to make sure that is the release cadence and not an Actions incident:

The loop is still pushing — last development commit, measured just now: decidesk 15 min ago, scholiq 15 min ago, shillinq 4 min ago.

Actions is moving normally — in the last 20 minutes: openregister in_progress=8 queued=2 completed=21, decidesk in_progress=2 queued=4 completed=15.

So runners are not starved and the queue is not stalled. Runs start, make progress, and are cancelled by the next merge before they can finish. This is the steady state, not a burst: sustained for at least three hours across five repos.

Worth stating for whoever triages this: the fix is not more runners. Throughput is fine — the branch simply never holds still for the 14–23 minutes a verdict takes.