Skip to content

🆘 - Shared Linux runner occupancy delays Domus product CI and Copilot gates #995

Open

Description

@efraespada

Have you checked for existing help or discussions?

  • I have searched existing issues and discussions.

What area do you need help with?

Workflow or CI/CD

Describe your problem or question

Domus product CI and its Copilot gates are waiting for the shared Linux pool while Copilot repository work occupies that pool. Please coordinate capacity with the owning Copilot work so consumer product verification can make progress, and advise whether a supported reservation or concurrency policy should separate substantive consumer CI from Copilot activity.

This records an operational delay, not a proven Copilot scheduling defect. Consumer runner placement is already self-hosted; #964 concerns a different remaining generated-placement contract.

At approximately 2026-11-19 21:25 UTC, Domus PR #145, head 4556aa1612e4eea6cefb3acba4e5371a4475b990, has had its Linux product jobs queued since CI 44613668219 was created around 20:56 UTC. Its amd64 image has passed all 66 browser journeys and fixture cleanup; ARM64 is running. Branch Sync and PR review are also queued.

A read-only audit around 21:04 UTC located these actual Linux jobs in Copilot:

Runner Job Observation
ubuntu-runner-1 CI / Tests and coverage In progress since 20:48:25 UTC
ubuntu-runner-3 CI / Tests and coverage In progress since 20:59:18 UTC
ubuntu-runner-4 Branch Sync In progress since 20:53:08 UTC

ubuntu-runner-2 had also served PR run 44613312322 from 20:58 UTC. These are observations at the recorded times, not claims that all those jobs remain active now. The pool documentation says the four Ubuntu registrations share one VM. Repository-level runner inventory returns zero for Domus; it does not enumerate the organization group, whose inventory was unavailable with the current credential.

Steps or attempts you've already made

  • Followed the exact consumer head with gh, including actual job status rather than only parent run status; a queued parent can contain an active image job.
  • Cancelled only superseded Domus CI, retained its failure evidence, and force-cancelled its remaining always() aggregate only after every product job was terminal. Both superseded Domus runs 44600104441 and 44602865961 are now terminal cancelled.
  • Left active Copilot work untouched and made no runner, group, protection or token changes.
  • Qualified the complete current Domus source journey locally: all 121 cases passed in 375.342132 seconds with zero retries, skipped, unexpected or flaky results. Edge.55 installed acceptance also passed, with the immutable runtime unchanged by the final test-harness refinement. These observations do not replace required same-head product CI.

Environment or context

Domus and ha-ui use the existing vypdev group. Linux jobs request self-hosted, codex, Linux; image jobs use the self-hosted ARM64 Mac and actual Linux images, with amd64 emulation identified separately. Consumer workflows retain exact Action release tags. GitHub-hosted placement remains limited to the authorized npm OIDC publication exception.

Additional context or screenshots

The owner authorized proceeding past a blocking Copilot check when the incident is recorded here. No current product-CI bypass has been applied. Any temporary Copilot-gate exception will be documented separately on the consumer PR with the applicable product evidence. Copilot implementation remains with its owning agent/maintainers; this report does not request a new managed code branch or cancellation of active work without owner coordination.

Activity

  1. added
    questionFurther information is requested
    priority: mediumImportant but not urgent, should be addressed soon.
    helpLabel to detect help request issues
    on Nov 19, 2026
  2. efraespada commented on Nov 19, 2026

    @efraespada
    MemberAuthor

    Fresh read-only observation at 2026-11-19 21:56 UTC: all four compatible Linux runners were occupied by Copilot - Branch Sync jobs in this repository:

    Runner Run / job Started (UTC)
    ubuntu-runner-1 44616045862 / 135884890855 21:24:55
    ubuntu-runner-2 44613240675 / 135875038127 21:28:02
    ubuntu-runner-3 44613299525 / 135875239744 21:17:03
    ubuntu-runner-4 44613039500 / 135874337273 20:53:08

    The actual job API confirms group vypdev (id4) and labels self-hosted, codex, Linux, matching the Domus jobs. Two previously reported CI test jobs have completed successfully and were replaced by these synchronizers. Domus #145 still has seven product jobs queued; both same-head image campaigns passed 66/66 journeys each.

    Runs 44613039500 and 44616045862 target the same #981 branch. The former was triggered at ccefb27627cd627ccdb7d5666fd174c2ac1da07b; the latter is for aa5c03551d283ad62fa36d8d80912090065d212d. Both the current remote ref and PR #994 now identify aa5c03551d283ad62fa36d8d80912090065d212d.

    The oldest job remains in its Run ./ step. The public job-log endpoint returned404 while it is running, so this observation does not establish whether its internal Action is verifying a candidate or waiting for prior runs. No foreign job has been cancelled by this consumer agent. Coordinating cancellation of only the older #981 execution is a smaller recovery option than stopping all current work; the newer exact-head execution and other branches have their own owners.

  3. added theissue type on Nov 19, 2026
  4. moved this to Todo in vypdevon Nov 19, 2026
  5. changed the title [-]Shared Linux runner occupancy delays Domus product CI and Copilot gates[/-] [+]🆘 - Shared Linux runner occupancy delays Domus product CI and Copilot gates[/+] on Nov 19, 2026
  6. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Hi! I’m @vypbot, the Copilot assistant for this repository.

    I can answer questions, explain the codebase, propose implementation and test plans, review issues and pull requests for potential bugs or security problems, and help authorized maintainers apply changes.

    Try /copilot help to see the available commands, or mention @vypbot with your question.

  7. efraespada commented on Nov 20, 2026

    @efraespada
    MemberAuthor

    Capacity has recovered for the Domus consumer. At the user-requested availability check, the current Domus CI at 52ab6e1 had executed Linux product checks on ubuntu-runner-1 and ubuntu-runner-2, and both Branch Sync and the PR review workflow had completed. The frontend job has since passed all 121 source journeys and its catalog/coverage gates. The actual amd64 image job also completed successfully on the shared Apple Silicon pool; backend and ARM64 validation are progressing.

    The previously proposed older Copilot execution 44613039500 is already terminal cancelled, and the newer 44616045862 is terminal success. Those changes occurred externally; this consumer cancelled neither run, and the old scoped cancellation question no longer needs action.

    This is evidence of recovered consumer capacity, not proof that Copilot scheduling or internal wait behavior has been permanently fixed. The current Domus review identified an ordinary product issue which is being corrected independently; it is not attributed to pool availability. No product CI gate was bypassed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

helpLabel to detect help request issuespriority: mediumImportant but not urgent, should be addressed soon.questionFurther information is requested

Type

Projects

  • Status
    Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions