Skip to content

πŸ›πŸ§‘β€πŸ’» - Make the workflow validator reject a CI aggregate gate that ignores worker failuresΒ #978

Description

@efraespada

Is there an existing issue for this?

  • I have searched the existing issues.

Which actions are affected?

Pull Request, Commits

Which platforms are affected?

Linux, macOS

Description

The shipped CI gate correctly fails when any worker fails, but assertSourceCiJobs only requires a string command and the three environment bindings. A future replacement with true can pass the workflow validator without enforcing job results.

Scope and acceptance criteria

  • Protect the aggregate success predicate through a reusable executable contract, without adding a second CI gate or silently accepting a no-op command.
  • At least six fixtures cover all-success, each of the three worker failures, a no-op predicate and a missing result/continue-on-error bypass.
  • Run validate:workflows and targeted validator tests. Preserve the existing self-hosted jobs, fork exclusion, public required check name, independent parallel workers and installation templates.

Reproducing the issue

In a YAML fixture, replace the ci-check command with true while preserving needs, always() and STATIC_RESULT/BUILD_RESULT/COVERAGE_RESULT. Verify the current validator accepts it; retain the positive current-gate control.

Pinned source references:

copilot Version

Unreleased develop at ccefb27627cd627ccdb7d5666fd174c2ac1da07b (PR #951); this is not a released or activated profile.

Additional context and comments

Priority and dependencies

Documentation and implementation boundary

Documentation: docs/development/testing.mdx.
Keep Clean Architecture and current coverage/contract gates. When a later implementation changes a public contract, update its catalogued SDD, tests and docs together and validate specifications; update Graphify after code changes.
Read the effective #950 contract in the owning SDD and its current reconciliation amendments; historical pilot deadlines are not current investigation limits.

Fixed invariants and planning state

Backlog only: no in-progress, branch creation, model execution, deployment or activation requested by this planning step.
Keep BUGBOT_VERIFICATION_MODE=disabled. Evaluated PRs must have zero write attempts; retain frozen pilots and independent evaluator/oracle evidence.
Token usage is optional telemetry and cannot stop investigation or reject a supported verdict. Elapsed time and turn count do not impose research cutoffs; explicit cancellation and evidence/SHA integrity remain mandatory. Post-run latency qualification criteria remain distinct from session deadlines.

Activity

  1. added
    bugSomething isn't working
    bugfixSomething isn't working
    priority: lowNot urgent, can be addressed later.
    on Nov 19, 2026
  2. added theissue type on Nov 19, 2026
  3. moved this to Todo in vypdevon Nov 19, 2026
  4. changed the title [-]Make the workflow validator reject a CI aggregate gate that ignores worker failures[/-] [+]πŸ› - Make the workflow validator reject a CI aggregate gate that ignores worker failures[/+] on Nov 19, 2026
  5. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Hi! I’m @vypbot, the Copilot assistant for this repository.

    I can answer questions, explain the codebase, propose implementation and test plans, review issues and pull requests for potential bugs or security problems, and help authorized maintainers apply changes.

    Try /copilot help to see the available commands, or mention @vypbot with your question.

  6. moved this from Todo to In Progress in vypdevon Nov 19, 2026
  7. added
    branchedIndicates that a branch has been created for this issue.
    on Nov 19, 2026
  8. changed the title [-]πŸ› - Make the workflow validator reject a CI aggregate gate that ignores worker failures[/-] [+]πŸ›πŸ§‘β€πŸ’» - Make the workflow validator reject a CI aggregate gate that ignores worker failures[/+] on Nov 19, 2026
  9. added
    state:workingWork can proceed on the verified branch or without a branch.
    on Nov 19, 2026
  10. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Implementation plan

    Current status: Ready to start. No action is required from maintainers before implementation.

    1. Define the aggregate success contract
      • Add a reusable executable contract for the ci-check predicate and have the validator enforce it.
      • Require all three worker results to succeed; reject no-op predicates and missing-result or continue-on-error bypasses.
    2. Keep the shipped CI gate intact
      • Preserve the public required check name, independent parallel workers, self-hosted jobs, fork exclusion, and installation templates.
    3. Add validator fixtures
      • Cover the positive current gate, all-success, each worker failure, a no-op predicate, and a missing-result or continue-on-error bypass.
    4. Update guidance and verify
      • Update docs/development/testing.mdx with the predicate contract and fixture coverage.
      • Run validate:workflows and the targeted validator tests.

    Acceptance: The positive current-gate fixture and all-success case pass; each worker-failure, no-op, and missing-result or continue-on-error bypass fixture is rejected; validate:workflows and targeted validator tests pass.

    Need something else? Mention the bot with a question or use /copilot help.

  11. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Progress Β· 65% estimated

    Assessed head 0b99506b Β· 2/5 supported
    1 need implementation Β· 2 need verification or clarification

    Implementation still needed

    • R25 Β· In a YAML fixture, replace the ci-check command with true while preserving needs, always() and STATIC_RESULT/BUILD_RESULT/COVERAGE_RESULT. Verify the current validator accepts it; β€” Add the baseline acceptance reproduction and retain a positive current-gate control. Requirement source Β· Evidence at assessed head

    Verification or clarification needed

    • R21 Β· - [ ] Run validate:workflows and targeted validator tests. Preserve the existing self-hosted jobs, fork exclusion, public required check name, independent parallel workers and inst β€” Please provide independent execution evidence for `validate:workflows` and the targeted validator tests. Requirement source Β· Evidence at assessed head
    • R48 Β· Keep Clean Architecture and current coverage/contract gates. When a later implementation changes a public contract, update its catalogued SDD, tests and docs together and validate β€” Please confirm whether Graphify was updated after this code change. Requirement source Β· Evidence at assessed head

    Issue Β· Linked pull request Β· Full assessment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

65%Progress: 65%branchedIndicates that a branch has been created for this issue.bugSomething isn't workingbugfixSomething isn't workingin-progressStart work on an admitted issue.priority: lowNot urgent, can be addressed later.size: SSmall-sized task or issue with minimal effort required.state:workingWork can proceed on the verified branch or without a branch.

Type

Projects

  • Status
    In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions