Skip to content

fix(ci): group runner patch failures by incident - #277

Merged
cbartz merged 7 commits into
mainfrom
fix/group-runner-patch-failures
Nov 13, 2026
Merged

cbartz merged 7 commits into
mainfrom
fix/group-runner-patch-failures

Conversation

@cbartz

@cbartz cbartz commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Repeated patch-update failures currently create one GitHub issue and Jira ticket per run. Group both patch workflows into one open incident tracker, with stable runner-patch-main and runner-patch-release entries containing the latest failure links.

  • Reuse only an open marked tracker. If none exists, create a new incident issue; never reopen closed trackers or adopt unmarked legacy issues. Each new GitHub issue has a distinct URL for Jira tracking.
  • Preserve closed incidents, link new trackers to their most recent predecessor, and ignore events already recorded in earlier incidents.
  • Preserve investigation notes and existing labels; add missing bug/ci-failure/keep labels. Closure remains manual after recovery.
  • Serialize reporting with queue: max; retain targeted per-run reporting and existing source guards for ordinary Runner CI/CD failures.

Existing duplicate cleanup is a separate follow-up. This PR does not close issues or modify Jira tickets.

Validation

  • 33 reporter tests pass, covering open-only reuse, fresh incidents, historical replay protection, later run attempts, prior-incident links, labels, pagination, source guards, ordinary CI/CD deduplication, and API errors. Independent review found no issues.
  • Workflow lint uses the existing narrow exception for actionlint's outdated queue schema; queue: max is documented by GitHub.
  • cd src && ./dev.sh test is blocked at restore by existing NU1903 errors for Microsoft.AspNetCore.WebUtilities 8.0.25 (GHSA-x33c-w48q-qrgc) in Test.csproj and Runner.Worker.csproj. No runtime or dependency files are changed.

AI-generated message.

Copilot AI lite review requested due to automatic review settings November 10, 2026 18:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The migration path preserves the existing bug label but fails to add the required ci-failure label to the adopted tracker, leaving its labels inconsistent with the documented plan.

1 open issue
What changed in this PR

This PR consolidates failures from both runner patch-update workflows into a single reusable GitHub issue while preserving per-run reporting for Runner CI/CD failures.

Changes:

  • Added a CommonJS reporter that creates, updates, reopens, and migrates a shared patch-failure tracker.
  • Replaced inline shell reporting with actions/github-script and serialized workflow runs using queue: max.
  • Added regression tests, a path-filtered test workflow, and design and implementation documentation.
File Description
docs/​superpowers/​specs/​2026-11-10-runner-patch-tracker-design.md Documents the shared tracker behavior, migration rules, and constraints.
docs/​superpowers/​plans/​2026-11-10-runner-patch-tracker.md Records the implementation plan and validation status.
.github/​workflows/​test-failure-reporting.yaml Runs the reporter test suite for relevant workflow and script changes.
.github/​workflows/​create-issue-on-failure.yaml Configures serialized failure reporting and invokes the shared reporter module.
.github/​scripts/​report-workflow-failure.test.cjs Tests tracker migration, deduplication, reopening, pagination, guards, label handling, and API errors.
.github/​scripts/​report-workflow-failure.cjs Implements shared patch tracker management while preserving ordinary per-run failure reporting.

Review effort: Lite

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A moderate active issue remains because ordinary failure handling now paginates through every repository issue and pull request before identifying the workflow class, causing unnecessary API calls, latency, and rate-limit pressure.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Avoid full issue pagination for ordinary workflow deduplication

.github/​scripts/​report-workflow-failure.cjs:22

This full listForRepo pagination happens before the Runner CI/CD branch, so every ordinary failure now downloads every issue (and every pull request) across all pages just to check one run marker. That is a regression from the previous targeted lookup and can add many API calls, rate-limit pressure, and latency as the repository's history grows; determine the workflow class first and keep ordinary deduplication targeted, only doing the full listing for patch-tracker migration.

1 resolved since last review

Review effort: Lite

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain, with the shared failure-reporting changes and their focused tests covering migration, recurring and stale events, labels, guards, and API errors.

No open issues 🎉

Review effort: Lite

@yanksyoon

Copy link
Copy Markdown
Member

Will reopening create another tracking item on jira?

@cbartz cbartz changed the title fix(ci): group runner patch failures in one tracker fix(ci): group runner patch failures by incident Nov 13, 2026
@cbartz
cbartz requested a lite review from Copilot November 13, 2026 08:04
@cbartz

cbartz commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

@yanksyoon Reopening the same GitHub issue would normally reopen its existing Jira ticket, rather than create a new one.

I have updated the PR in 3ac4032 to use one tracker per incident instead:

  • While a marked tracker is open, subsequent failures update that same GitHub issue and Jira ticket.
  • Once it is closed, a new failure creates a new GitHub issue, allowing the sync bot to create a new Jira ticket. Closed trackers are never reopened.
  • The new tracker links to the previous incident, and replayed events already recorded in earlier incidents are ignored.

This keeps repeated failures grouped without reusing resolved tickets for later incidents.

AI-generated message.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the label-repair logic has an unverified edge case that could cause GitHub to reject tracker updates when the standard bug label is missing.

No open issues 🎉

Review effort: Lite

@cbartz

cbartz commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

force merging - CI failures are unrelated and need to be fixed in a separate PR. Force merging should be safe here as we are not touching production code in this PR

@cbartz
cbartz merged commit 56173a2 into main Nov 13, 2026
2 of 8 checks passed
@cbartz
cbartz deleted the fix/group-runner-patch-failures branch November 13, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants