diff --git a/.github/workflows/strix.yml b/.github/workflows/strix.yml index f15b29f564..4a49ac205b 100644 --- a/.github/workflows/strix.yml +++ b/.github/workflows/strix.yml @@ -369,12 +369,14 @@ jobs: strix: needs: [changed-scope, admit-current-head] if: needs.changed-scope.outputs.code == 'true' && needs.admit-current-head.outputs.admitted == 'true' - # Large, actively-growing repositories (e.g. contextual-orchestrator) can - # legitimately require well over two hours to scan -- this org's own - # standing operating directive accepts that central OpenCode/Strix/Noema - # scans may take more than two hours per model (docs/product-goal-directive.md). - # Inference has no wall-clock deadline; cancellation is reserved for an - # explicit operator action or a superseded head. + # Large repositories can legitimately require more than two hours of model + # work (docs/product-goal-directive.md §8). Total elapsed job deadlines + # therefore remain absent: they would cancel active reasoning, streaming, or + # tool calls without evidence of failure. Runner release is progress-based: + # strix_timeout_compat.py restores the evidence-sourced stream-idle boundary + # after fast-mlsirm holder run 35263416380 demonstrated unbounded no-progress + # occupancy, while STRIX_PROCESS/TOTAL remain 0. Explicit operator cancellation + # and superseded-head concurrency remain separate cancellation authorities. runs-on: ubuntu-24.04 # Least-privilege token scoped to this job (Scorecard alert #43): the scan # exchanges an OIDC token (id-token) and publishes same-repo status evidence diff --git a/docs/adr/0034-review-runner-occupancy-progress-bound.md b/docs/adr/0034-review-runner-occupancy-progress-bound.md new file mode 100644 index 0000000000..5550057bba --- /dev/null +++ b/docs/adr/0034-review-runner-occupancy-progress-bound.md @@ -0,0 +1,160 @@ +# ADR-0034: Review runner occupancy is bounded by progress and admission, never by elapsed inference time + +- **Status:** Proposed +- **Date:** 2026-09-13 (reconstructed 2026-09-15 as #2207; renumbered and extended 2026-09-18 after ADR-0030 was assigned to CI centralization scope; renumbered again 2026-09-18 when open #2260 claimed ADR-0032/0033 for the release-pipeline ADRs) +- **Scope:** `scripts/ci/contextual_orchestrator_review_launcher.py` (serving `ModelClient` construction and `_preflight_with_fallback` continuation), the vendored `contextual_orchestrator` transport, `.github/workflows/noema-review.yml` (deliberately unbounded review job), `.github/workflows/opencode-review-coalesce-tick.yml` (schedule admission under the plan concurrent-job ceiling), `.github/workflows/strix.yml` / `scripts/ci/strix_timeout_compat.py` (sourced Strix stream-idle occupancy bound) +- **Amends:** nothing. Extends ADR-0003's 2026-08-31 amendment ("model inference has no repository- or run-level deadline") by naming what *may* be bounded. ADR-0005's fixed wall-clock budgets remain historical and are not restored. Complements ADR-0030 (centralization cannot lift the plan ceiling) and ADR-0031 (capacity recovery is re-dispatch, not in-job model wait). +- **Numbering note:** Predecessor drafts used ADR-0030, then ADR-0032, for this decision. On current `main`, ADR-0030 is `0030-ci-centralization-scope-given-plan-ceiling.md`, and open #2260 already claims ADR-0032 (`0032-release-pipeline-reusable-workflows.md`) and ADR-0033 (`0033-noema-semver-bump.md`). This record is therefore ADR-0034; content succession from Draft #2140 / earlier #2207 / #2265 heads is the decision and evidence, not the digit. + +## Problem + +Two distinct occupancy failures are easy to confuse, and both look like "the review took hours": + +1. **Transport / model path.** The vendored gateway is about to stop bounding a single provider call, and nothing below GitHub's six-hour job ceiling replaces it. +2. **Actions admission.** Under the org plan concurrent-job ceiling (~60), scheduled and required workflows sit `queued` for hours before a runner is admitted — wall time that is not model work at all. + +`contextual-orchestrator#1118` changed `ModelClient.__init__`'s `timeout` default from `int = 90` to `float | None = None`. That removal is correct: an implicit 90-second cap on the inference path is exactly what ADR-0003's 2026-08-31 amendment forbids, and it is the same class of decision `#1889`, `#1890`, and `#1892` were reverted for reintroducing (`#1891`, `#1895`). The removal is not the problem. What it exposes is. + +Three layers each decline to bound the call, and the combination was never designed: + +| layer | bound after the pin advance | +|---|---| +| `ModelClient` default | `timeout: float \| None = None` | +| `contextual_orchestrator_review_launcher.py:1240` (serving client) | passes no `timeout`, so it takes the library default | +| `.github/workflows/noema-review.yml:261` | *"No job-level `timeout-minutes` here, deliberately."* | +| GitHub job ceiling | 6 h | + +### What the 90-second default is currently doing + +`#1884` head `e85fc437`, [run 34732993973](https://github.com/ContextualWisdomLab/.github/actions/runs/34732993973), sidecar stderr artifact `10310273053`. Serving phase 02:41:17Z → 02:56:27Z, 907.3 s, terminal `HTTP 503`. The job log shows none of this; the uploaded `noema-sidecar-evidence` artifact carries the whole trace, and reading the job log instead is what makes this failure look like a single hung call. + +Fifteen provider attempts inside that window, across two agents on two credential accounts: + +``` +10x nvidia_nim / deepseek-ai/deepseek-v4-flash-0731 + 5x nvidia_nim_sub / deepseek-ai/deepseek-v4-pro-0813 +``` + +Measured attempt-to-failure durations: + +``` + 90.0s 90.0s 22.0s 90.0s 90.0s + 90.0s 90.0s 90.0s 90.0s 0.0s +``` + +Eight of ten land on exactly 90.0 s. That is not providers answering — it is the default expiring. Roughly 720 of the 907 seconds is the cap firing against endpoints that accepted a connection and then delivered nothing. **The cap is the only thing converting a no-observable-progress request into a bounded, diagnosable event**, and it is leaving. + +Two further behaviours the same trace records, both relevant to the decision below: + +- **The breaker resets straight back into the failing route.** `circuit_failure` ×10, `circuit_opened` ×2 (at `failures=3.0 threshold=3`), `circuit_reset` ×2 after `reset_seconds=30.0`, `circuit_cleared` ×1. After each reset the ranking hands back the same agent. `failures=4.0 threshold=3` also appears without a second open, so the counter accounting is worth its own look. +- **Two ready routes were never tried.** `nvidia_nim/meta/llama-3.2-11b-vision-instruct` and `nvidia_nim_sub/meta/llama-3.2-11b-vision-instruct` were both `ready` at preflight and appear zero times in the serving phase, while the two deepseek routes were cycled fifteen times. + +### Why the obvious model-path fix is forbidden + +Passing an explicit `timeout=` at the launcher's serving call site would reinstate the reverted cap under a different name. `docs/product-goal-directive.md` §8 accepts that a model path may take more than two hours and states that speed is not a core consideration; ADR-0005 records that fixed wall-clock budgets "failed for legitimately slow models." This ADR does not reopen that, and the same trace shows why it should not: `google/gemma-4-31b-it` was rejected at preflight after exactly 90.0 s with `TimeoutError`. Nothing in the evidence distinguishes that from a model that was simply going to take 91 seconds. The cap is discarding routes on no evidence of failure. + +So the org currently has both defects at once: a cap that kills slow-but-working models, and, once it is removed, no explicit occupancy boundary for a request that makes no observable transport progress. + +### Why scheduled workflows queue for hours (Actions layer) + +This is a different failure mode with the same wall-clock symptom. Measured on 2026-09-17 +([`docs/doctoring/actions-capacity-root-cause-20260917.md`](../doctoring/actions-capacity-root-cause-20260917.md)): + +- OpenCode Review Dispatch run `34931908846` spent ~13h57m wall time of which **~21 minutes (2.5%)** was job execution and **~13h36m (97.5%)** was inter-job wait for a fresh runner between `needs:` edges. +- The same four-job chain on a lighter day (`34756591400`) finished in 14m40s. Workflow logic did not change; org-wide saturation did. +- Plan concurrent-job ceiling is ~60 (`docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`); censuses show single-digit / low-double-digit `in_progress` against ~10³ `queued`. ADR-0030 already records that workflow consolidation cannot lift that ceiling. + +Schedule delivery compounds this. Cron `*/5` does not guarantee a runner every five minutes: under saturation, GitHub delays schedule delivery itself (documented gaps of tens of minutes to hours in +[`docs/doctoring/coalesce-tick-post-2242-live-verify-20260917.md`](../doctoring/coalesce-tick-post-2242-live-verify-20260917.md) +and `docs/doctoring/actions-queue-24h-remeasurement-20260917.md`). Missed intervals are not backfilled as a stack of five-minute runs. A job that has already been admitted into the shared queue waits behind every other org job competing for the same ~60 slots — including an *inert* coalesce tick if its skip gate is evaluated only after admission (`#2232` → run `35219385415` queued ~3.5h with `OPENCODE_REVIEW_COALESCE_ENABLED=false`; repaired by `#2242`). + +**Diagnosis rule:** multi-hour review duration is first presumed inter-job / schedule admission wait under the plan ceiling, not model API latency, until job-level timestamps prove otherwise. Model-path timeouts remain forbidden regardless of that wait. + +## Constraints + +1. Elapsed inference time must never become a model-failure verdict (ADR-0003 2026-08-31; directive §8). +2. A route must not be dropped, penalised, or circuit-broken for being slow. +3. A review runner must not be held indefinitely by a provider that has stopped responding. +4. Whatever bound exists must be explainable from evidence in the sidecar artifact, not inferred from wall-clock alone. +5. Actions occupancy is repaired at **admission / continuation** boundaries (job-level skip, concurrency coalescing, fail-open re-dispatch) — never by converting queue wait or inference duration into a model-failure verdict. +6. Schedule observability must not force inert work onto the runner queue. + +Constraints 1–2 and 3 are only in tension if "how long has this taken" is the sole available signal. It is not. + +## Decision + +**Bound progress and admission, not elapsed inference time.** + +### A. Transport / model occupancy (gateway and review jobs) + +1. **Idle-socket bound, reset on every byte received.** Within the external job boundary, a response that is actively streaming is never interrupted by the idle-socket bound, regardless of total inference duration — a two-hour generation completes if that external boundary remains available. A job ceiling or a separately classified runner-reclamation event may still terminate the request; neither event is a model-failure verdict and neither may feed route ranking. Before the first response byte, absence of bytes is classified only as a **transport-level no-progress state**. It does not prove that the provider or model failed, and it does not distinguish a legitimately long time-to-first-byte from a stalled transport. If the transport idle bound expires in that state, the event is recorded as no-progress/occupancy release, not as model failure; it must not penalise, circuit-break, or rank the route. The threshold is a transport/runner-occupancy policy derived from observed time-to-first-byte evidence across the pool, not a claim about how long a model should take. + + This is the substantive question the reverts turned on, and it is settled here deliberately rather than in code review: **an idle-socket bound is not a model wall-clock deadline**, because active byte progress resets it and expiry is not a model-failure verdict. A total elapsed model deadline can terminate a progressing request and attribute duration to the model; that remains forbidden. + +2. **Continuation admission.** When the breaker opens on an agent, its reset must not re-offer that agent while equally-ranked, preflight-`ready` alternatives remain untried for this request. The `#1884` trace spent 907 seconds on two routes while two ready routes on the same accounts sat unused. This is a ranking/continuation defect independent of any timeout and would have shortened that run on its own. + +3. **Occupancy release is a job-boundary decision with its own event.** If a runner must be reclaimed, that is an operational decision about the runner, and it must be emitted as such — never attributed to the provider as a model failure, and never recorded in a way that feeds route ranking. This is the "admission/continuation boundary" the directive names. + +Nothing in section A sets a total-duration limit on a review, a model, or a request. + +### B. Actions admission under the plan concurrent-job ceiling (~60) + +4. **Skip before runner admission for inert scheduled work.** When `OPENCODE_REVIEW_COALESCE_ENABLED` is not `true`, the coalesce tick job uses a **job-level** `if:` so the run concludes `completed`/`skipped` without competing for a hosted runner (`#2242`; live post-merge proof `35249460935` in 1s per + [`docs/doctoring/coalesce-tick-post-2242-live-verify-20260917.md`](../doctoring/coalesce-tick-post-2242-live-verify-20260917.md)). + A step-scoped gate that still admits the job (`#2232`) is rejected: it recreates multi-hour `queued` wait for an echo under the same ceiling as real review work. + +5. **Dedicated concurrency group — yes; runner reservation — no (for this tick).** + - **Keep** workflow concurrency group `opencode-review-coalesce-tick` with `cancel-in-progress: false`. The group caps stacking (at most one active + one pending) and protects an in-flight org-wide dispatch from being cut mid-repository. It does **not** mint a private runner pool or bypass the plan ceiling; hosted `ubuntu-24.04` still shares the ~60 org slots. + - **Do not** treat a dedicated concurrency group, a different `runs-on` label, or a "reserved" self-hosted runner as the fix for inert-tick queueing. Those levers do not skip admission, and flipping `cancel-in-progress` to `true` would cancel mid-dispatch rather than shorten the active waiter's admission delay (`docs/doctoring/coalesce-tick-inert-runner-queue-20260917.md`). + - **Do not** invent a model-path or job-level inference timeout to "free" runners stuck behind the ceiling. Capacity recovery for provider exhaustion is bounded re-dispatch (ADR-0031); capacity recovery for Actions saturation is admission shaping and plan-tier headroom (owner-only), not §8-violating model deadlines. + +6. **Enabled ticks may still queue for hours; fail-open covers dispatch liveness.** When coalescing is on, a real tick competes for the same ~60 slots. Scheduler `recent_coalesce_tick_completed()` counts only `conclusion=success` and fail-opens when no fresh successful tick exists (`#2233`), so reviews are not deferred forever while a tick sits `queued`. Re-enable criteria for the repo variable remain operator criteria in the post-`#2242` live-verify record — not automatic with this ADR. + +## Consequences + +- The pin advance in `#2137` becomes safe to land: the implicit 90 s cap goes away and a progress/occupancy boundary replaces it, rather than leaving a six-hour hole. +- Legitimately slow models stop being discarded as model failures. `gemma-4-31b-it`-class routes that need more than 90 s are not penalised merely for elapsed time, which also avoids shrinking the free pool's ready set — relevant to `#1915`, where readiness, not admission, is the concentration point (that run's catalog held 60 admitted routes across 3 accounts; only 4 were ready, all NVIDIA). +- The worst case changes shape rather than disappearing: a provider that dribbles one byte per interval defeats an idle bound. That is accepted. The observed #1884 case is narrower: attempts accepted a connection but emitted no response bytes within the measured window. That observation alone is not promoted to a provider/model-failure verdict. +- A general A.1 threshold for the gateway pool is still a transport/runner-occupancy property and should be derived from observed time-to-first-byte across that pool, not picked as a round number; this ADR does not invent a new gateway-wide constant beyond existing evidence. **Current Strix policy (implemented):** `LLM_STREAM_IDLE_TIMEOUT=90` seconds via `scripts/ci/strix_timeout_compat.py` (`STREAM_IDLE_OCCUPANCY_SECONDS`), sourced from `#1884` head `e85fc437` run `34732993973` / sidecar artifact `10310273053` (eight of ten no-progress attempt-to-failure durations landed on exactly 90.0s). That 90s value is the Strix occupancy release for dead sockets; it is not a model-failure verdict and must not be read as an undecided placeholder. +- Item A.2 can land independently of item A.1 and is the cheaper of the two. +- Agents stop mis-attributing multi-hour schedule/required-check wall time to model latency and stop proposing model timeouts as the repair (`#1889` class). The Actions-layer repair surface is admission (job-level skip, concurrency coalescing, fail-open) plus owner plan capacity — aligning with ADR-0030's "wrong layer" warning. +- Coalesce tick keeps its dedicated concurrency group for mid-dispatch safety; enabling the flag under deep saturation remains an explicit operator choice with documented fail-open acceptance. + +## Alternatives considered + +### Model-path wall-clock / `timeout-minutes` on review jobs + +- **Description:** Cap serving calls or the Noema/OpenCode/Strix job at a fixed elapsed duration (including the reverted 900 s attempts). +- **Rejection:** Forbidden by directive §8, ADR-0003 2026-08-31, and ADR-0005's historical failure mode. Elapsed time is not evidence of model failure. + +### Step-scoped coalesce gate for schedule run-record visibility + +- **Description:** Always admit the tick job; skip work inside a step so every cron produces a visible run. +- **Rejection:** Forces inert jobs onto the shared runner queue (`35219385415`). Job-level skip still produces `completed`/`skipped` run records (`35191169833`, `35249460935`) without admission cost (`#2242`). + +### Runner reservation / self-hosted pool for coalesce tick only + +- **Description:** Dedicate runners so the five-minute tick always admits promptly. +- **Rejection:** Does not address org-wide ceiling pressure on the review chains that dominate queue depth; adds operational surface for a job that must usually be skipped. Dedicated **concurrency group** already provides the stacking/safety property without implying reserved capacity. + +### Raising plan concurrent-job ceiling as the sole ADR outcome + +- **Description:** Treat ~60 as the bug and stop at "buy more concurrency." +- **Rejection:** Owner-only lever; still required for relief under saturation, but does not define transport occupancy semantics or prevent inert work from competing. ADR-0030 already scopes what code changes can and cannot fix. + +## References + +- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). *Site reliability engineering: How Google runs production systems*. O'Reilly Media. +- Fielding, R., Nottingham, M., & Reschke, J. (Eds.). (2022). *HTTP semantics* (RFC 9110). Internet Engineering Task Force. https://doi.org/10.17487/RFC9110 — HTTP defines no client-side completion deadline; how long a client waits is a local policy decision, which is why it has to be made explicitly here rather than inherited. +- Nygard, M. T. (2018). *Release it! Design and deploy production-ready software* (2nd ed.). Pragmatic Bookshelf. — Source of the circuit-breaker and bounded-resource patterns the gateway already implements; item A.2 above is a gap in the continuation half of that pattern, not a new mechanism. +- ADR-0003, 2026-08-31 amendment (model inference has no repository- or run-level deadline) and 2026-09-13 amendment (the `012beaac` pin advance). +- ADR-0005 (historical: fixed wall-clock budgets, superseded). +- ADR-0030 (CI centralization cannot lift the plan concurrent-job ceiling). +- ADR-0031 (Noema transport-capacity recovery via bounded re-dispatch, not in-job model wait). +- `#1889`, `#1890`, `#1892` (900-second caps) and `#1891`, `#1895` (their reverts). +- `#2242` (job-level coalesce skip before admission), `#2244` (post-merge live verify + re-enable criteria). +- `ContextualWisdomLab/contextual-orchestrator#1118` (removal of the implicit 90-second default). +- [`docs/doctoring/actions-capacity-root-cause-20260917.md`](../doctoring/actions-capacity-root-cause-20260917.md) — multi-hour wall time is inter-job queue wait. +- [`docs/doctoring/coalesce-tick-post-2242-live-verify-20260917.md`](../doctoring/coalesce-tick-post-2242-live-verify-20260917.md) — post-`#2242` skipped tick in 1s; schedule lag ≠ model failure. +- [`docs/doctoring/coalesce-tick-inert-runner-queue-20260917.md`](../doctoring/coalesce-tick-inert-runner-queue-20260917.md) — inert tick queueing under the ~60 ceiling. +- [`docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`](../doctoring/actions-plan-concurrency-ceiling-20260903.md). diff --git a/docs/doctoring/strix-unbounded-agentic-occupancy-20260918.md b/docs/doctoring/strix-unbounded-agentic-occupancy-20260918.md new file mode 100644 index 0000000000..aeea97a53c --- /dev/null +++ b/docs/doctoring/strix-unbounded-agentic-occupancy-20260918.md @@ -0,0 +1,109 @@ +# Doctoring record: unbounded Strix agentic occupancy held fast-mlsirm under the plan ceiling (2026-09-18) + +- **Date:** 2026-09-18 +- **Subject:** `ContextualWisdomLab/fast-mlsirm` Actions queue showed ~179–180 + `queued` runs against 1–2 `in_progress` while a single central Strix Security + Scan occupied a hosted runner with no progress or job occupancy bound. +- **Decision records:** ADR-0034 (progress / admission occupancy, not elapsed + inference); ADR-0030 (plan concurrent-job ceiling cannot be lifted by + workflow consolidation); ADR-0028 principle (numeric bounds must be sourced + from measurement, not invented — here applied to Strix occupancy, not the + coalesce-tick max-age). + +## Live holder (re-fetched) + +| Field | Value | +|---|---| +| Run | [35263416380](https://github.com/ContextualWisdomLab/fast-mlsirm/actions/runs/35263416380) | +| Event | `pull_request_target` (required Strix path) | +| PR | fast-mlsirm#1988 @ `3211659557118354c61a8221119c0dc314f0c90b` | +| Run created / started | 2026-09-17T19:11:44Z (~10h wall at measurement) | +| Job | `strix` id `105414661161` | +| Labels | `ubuntu-24.04` (standard hosted; not larger / self-hosted) | +| Job started | 2026-09-18T03:52:48Z | +| Step holding | `Run Strix (quick)` since 2026-09-18T04:18:54Z | +| Concurrent sibling | CI run [35287625813](https://github.com/ContextualWisdomLab/fast-mlsirm/actions/runs/35287625813) (`push`, rust still in progress / python jobs cycling) | + +Timeline split (why "10h" is not 10h of model work): + +1. 19:11Z → ~23:00Z: run admitted; `admit-current-head` / `Detect changed scope` + completed in seconds once they got runners. +2. ~23:00Z → 03:52Z: `strix` job waited for a runner under the org plan + concurrent-job ceiling (~60; see + [`actions-plan-concurrency-ceiling-20260903.md`](actions-plan-concurrency-ceiling-20260903.md)). +3. 03:52Z onward: setup + sidecar (~26m) then unbounded `Run Strix (quick)` with + `STRIX_PROCESS_TIMEOUT_SECONDS=0`, `STRIX_TOTAL_TIMEOUT_SECONDS=0`, + `LLM_STREAM_IDLE_TIMEOUT=0` (forced by `strix_timeout_compat.py`), and **no** + job-level `timeout-minutes` (platform default 360m). + +## Why concurrency looks like ~1 on a public repo + +API sample at measurement (REST `actions/runs?status=`): + +| Repository | `in_progress` | `queued` | +|---|---|---| +| `fast-mlsirm` | 1–2 | ~179–180 | +| `.github` | 17 | ~538 | +| `contextual-orchestrator` | 2 | ~151 | +| `naruon` | 0 | ~76 | + +That shape matches the **org-wide plan concurrent-job ceiling**, not +per-repository Strix serialization and not a larger-runner pool: + +- Central `strix.yml` concurrency group is + `strix-security-scan--` with + `cancel-in-progress: true` — one Strix **per PR**, not one for the whole org. +- Holder labels are `ubuntu-24.04`; org self-hosted runners `total_count=0`. +- Other repos simultaneously hold double-digit `in_progress` counts, so + fast-mlsirm is not uniquely capped at one slot by workflow design. +- Billing plan ceiling is not exposed on the Actions REST API (ADR-0030); the + user-reported 58–60/60 figure remains in the 2026-09-03 doctoring record. + +## Root defect in the central Strix path + +`#1546` / `#1895` correctly removed **model-path** wall clocks (including the +reverted 900s `#1889` cap). The residual hole was treating that as "no bound +of any kind": an agentic scan with stream-idle forced to `0` and no job +occupancy release can hold a shared runner until the platform 360m kill while +hundreds of unrelated jobs stay `queued`. + +Directive §8 still forbids converting elapsed inference into a model-failure +verdict. ADR-0034 names the allowed repairs: **progress (idle-socket)** and +**admission/occupancy release**, with expiry classified as occupancy — not as +"the model was too slow." + +## Repair landed with this record + +1. **No job `timeout-minutes`** on the central `strix` job. The proposed + 180-minute value was derived as measured ~2h plus a 50% margin; that margin + is not a statistical decision rule and would terminate active reasoning or + streaming solely because wall time elapsed. Required check name `strix` + remains unchanged. +2. **`LLM_STREAM_IDLE_TIMEOUT=90`** in `strix_timeout_compat.py` — progress + bound sourced from `#1884` run `34732993973` (8/10 dead-socket attempts at + exactly 90.0s). Request / warm-up elapsed deadlines remain disabled + (`LLM_TIMEOUT=0`, `STRIX_PROCESS/TOTAL_TIMEOUT_SECONDS=0`). +3. **ADR-0034** recorded on `main` so the next change cites the occupancy + decision rather than re-litigating §8. + +## Duplicate "Detect changed scope" contexts (separate, non-blocking) + +On the hung head, four distinct workflow runs each published a check named +`Detect changed scope` (Strix, and peer required workflows). That multiplies +admission slots under the same ceiling. Reduction without weakening gates is +already scoped by open +[ContextualWisdomLab/.github#1962](https://github.com/ContextualWisdomLab/.github/pull/1962) +(fold admit + changed-scope into one metadata job per workflow; keep required +`strix` / scan job names). Do not rename required contexts in this occupancy +PR. + +## Audit trail + +- REST: `repos/ContextualWisdomLab/fast-mlsirm/actions/runs/35263416380` and + `/jobs` (job `105414661161`). +- REST queue samples: `actions/runs?status=queued|in_progress` on + `fast-mlsirm`, `.github`, `contextual-orchestrator`, `naruon`, `OriginWeave`. +- OriginWeave completed-scan sample: run `35178432177` job `105140349882` + (~78.1m job / ~31.1m `Run Strix (quick)`). +- `#1884` / run `34732993973` sidecar evidence cited in ADR-0034. +- `docs/product-goal-directive.md` §8; `#1889` / `#1895`; `#1546`. diff --git a/scripts/ci/strix_quick_gate.sh b/scripts/ci/strix_quick_gate.sh index c7d3667465..96b1faf392 100755 --- a/scripts/ci/strix_quick_gate.sh +++ b/scripts/ci/strix_quick_gate.sh @@ -239,11 +239,13 @@ PY } # Issue #2168: reject "already applied" remediation prose when apply_patch -# missed the materialized scan workspace. Uses scripts/ci/strix_evidence_binding.py. +# missed the materialized scan workspace. Uses the trusted binder beside this +# gate script (SCRIPT_DIR), never STRIX_REPO_ROOT — target/fixture workspaces +# do not ship scripts/ci/strix_evidence_binding.py (#2265 quality-ci). sanitize_remediation_evidence_claims() { local log_file="$1" local report_root="$2" - local binder="$REPO_ROOT/scripts/ci/strix_evidence_binding.py" + local binder="$SCRIPT_DIR/strix_evidence_binding.py" local report_file if [ ! -f "$binder" ] || [ -L "$binder" ]; then diff --git a/scripts/ci/strix_timeout_compat.py b/scripts/ci/strix_timeout_compat.py index 7ddb290654..0e8fbe54aa 100755 --- a/scripts/ci/strix_timeout_compat.py +++ b/scripts/ci/strix_timeout_compat.py @@ -1,11 +1,13 @@ #!/usr/bin/env python3 -"""Launch Strix 1.5.3 with ContextualWisdomLab's unbounded inference contract. +"""Launch Strix 1.5.3 with progress-bounded, elapsed-inference-unbounded policy. Strix 1.5.3 models ``LLM_TIMEOUT`` as an integer and passes it both to request settings and to ``asyncio.wait_for`` during model preflight. ``0`` therefore cancels preflight immediately instead of meaning "no deadline". This trusted, -version-gated launcher keeps Strix's non-model operational timeouts intact while -removing only model-request and model-warm-up wall-clock deadlines. +version-gated launcher keeps Strix's non-model operational timeouts intact, +removes model-request and model-warm-up *elapsed* deadlines (directive §8 / +ADR-0034), and installs a sourced stream-idle occupancy bound so a dead socket +cannot hold a shared runner until GitHub's job ceiling. """ from __future__ import annotations @@ -20,12 +22,19 @@ SUPPORTED_VERSION = "1.5.3" STRIX_DISTRIBUTION = "strix-agent" +# Measured no-progress window from ContextualWisdomLab/.github#1884 head +# e85fc437, run 34732993973 / sidecar artifact 10310273053: eight of ten +# attempt-to-failure durations landed on exactly 90.0s against sockets that +# accepted a connection then delivered no response bytes (ADR-0034 §A.1). +# Expiry is occupancy / transport no-progress release, not a model-failure +# verdict for route ranking. +STREAM_IDLE_OCCUPANCY_SECONDS = "90" def normalize_inference_timeout_environment(environment: MutableMapping[str, str]) -> None: - """Disable Strix request and stream-idle deadlines before settings import.""" + """Clear elapsed request deadlines; keep a sourced stream-idle occupancy bound.""" environment["LLM_TIMEOUT"] = "0" - environment["LLM_STREAM_IDLE_TIMEOUT"] = "0" + environment["LLM_STREAM_IDLE_TIMEOUT"] = STREAM_IDLE_OCCUPANCY_SECONDS class UnboundedInferenceAsyncio: diff --git a/scripts/ci/test_strix_quick_gate.sh b/scripts/ci/test_strix_quick_gate.sh index 150b9102b3..3563e2cc81 100755 --- a/scripts/ci/test_strix_quick_gate.sh +++ b/scripts/ci/test_strix_quick_gate.sh @@ -300,8 +300,12 @@ assert_strix_workflow_pr_trigger_hardened() { assert_file_contains "$workflow_file" "Provision contextual-orchestrator Strix sidecar" "strix workflow provisions the central contextual-orchestrator sidecar" assert_file_contains "$workflow_file" "CONTEXTUAL_ORCHESTRATOR_BASE_URL" "strix workflow uses the sidecar base URL" assert_file_contains "$workflow_file" "CONTEXTUAL_ORCHESTRATOR_TOKEN" "strix workflow uses the sidecar token" - assert_file_not_contains "$workflow_file" "timeout-minutes: 200" "strix workflow job must not cap model inference" - assert_file_not_contains "$workflow_file" "timeout-minutes: 170" "strix scan step must not cap model inference" + assert_file_not_contains "$workflow_file" "timeout-minutes: 200" "strix workflow job must not reinstate the pre-#1546 model inference cap" + assert_file_not_contains "$workflow_file" "timeout-minutes: 170" "strix scan step must not reinstate the pre-#1546 model inference cap" + assert_file_not_contains "$workflow_file" "timeout-minutes: 900" "strix must not invent a 900-minute job bound" + assert_file_not_contains "$workflow_file" "timeout-minutes: 30" "strix must not reinstate the reverted #1889 30-minute job cap" + assert_file_not_contains "$workflow_file" "timeout-minutes: 180" "strix never ends active work on an arbitrary elapsed-job margin" + assert_file_contains "$workflow_file" "35263416380" "strix job timeout cites the measured fast-mlsirm holder run" assert_file_contains "$workflow_file" 'export LLM_TIMEOUT=0' "strix disables the model client inference timeout" assert_file_contains "$workflow_file" 'export STRIX_MEMORY_COMPRESSOR_TIMEOUT=0' "strix disables the memory-compressor inference timeout" assert_file_contains "$workflow_file" 'export STRIX_PROCESS_TIMEOUT_SECONDS=0' "strix disables the scanner process timeout" @@ -491,7 +495,9 @@ assert_changed_file_membership_uses_cached_normalized_paths() { assert_strix_evidence_binding_contract() { assert_file_contains "$GATE_SCRIPT" "sanitize_remediation_evidence_claims" "strix gate sanitizes false already-applied remediation claims" - assert_file_contains "$GATE_SCRIPT" 'scripts/ci/strix_evidence_binding.py' "strix gate binds remediation evidence through the tested Python binder" + assert_file_contains "$GATE_SCRIPT" 'binder="$SCRIPT_DIR/strix_evidence_binding.py"' "strix gate binds remediation evidence through the trusted SCRIPT_DIR binder" + assert_file_not_contains "$GATE_SCRIPT" 'binder="$REPO_ROOT/scripts/ci/strix_evidence_binding.py"' "strix gate must not look for the binder under the scan-target REPO_ROOT" + assert_file_contains "$REPO_ROOT/scripts/ci/test_strix_quick_gate.sh" 'cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py"' "quality-ci fixtures package the evidence binder beside the gate under test" assert_file_contains "$GATE_SCRIPT" "evidence_scope=pr_delta" "strix gate labels PR-delta findings with authenticated provenance" assert_file_contains "$GATE_SCRIPT" "evidence_scope=repository_baseline" "strix gate labels unchanged-path findings as repository_baseline" assert_file_contains "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" 'PR_DELTA = "pr_delta"' "strix evidence binder defines pr_delta scope" @@ -640,7 +646,7 @@ assert_opencode_review_uses_codegraph_and_contextual_orchestrator() { assert_file_not_contains "$workflow_file" 'ref: ${{ github.workflow_sha }}' "opencode trusted checkout never bypasses the validated ref output" assert_file_contains "$workflow_file" "target_repository:" "opencode repository_dispatch can target a repository whose PR does not inherit required workflows" assert_file_contains "$workflow_file" "Materialize pull request merge tree for coverage measurement" "opencode coverage measures the PR merge tree instead of exposing secrets to untrusted checkout actions" - assert_file_contains "$workflow_file" 'TARGET_REPOSITORY: ${{ needs.validate-pr-metadata.outputs.target_repository }}' "opencode coverage fetches exact validated base/head commits from the target repository" + assert_file_contains "$workflow_file" 'TARGET_REPOSITORY: ${{ steps.validate.outputs.target_repository }}' "folded opencode validation job fetches exact validated base/head commits from the target repository" assert_file_contains "$workflow_file" "Exchange OpenCode app token for target repository review reads" "opencode review can read private target repositories through the OpenCode app token before materializing review data" assert_file_contains "$workflow_file" 'GH_TOKEN: ${{ steps.review_read_app_token.outputs.token || secrets.OPENCODE_APPROVE_TOKEN || github.token }}' "opencode materialization prefers the OpenCode app token for private target repository reads" assert_file_contains "$workflow_file" '[ "${GH_REPOSITORY:-}" != "${GITHUB_REPOSITORY:-}" ]' "opencode approval uses the app token for target-repository check lookup" @@ -968,13 +974,14 @@ assert_opencode_review_uses_codegraph_and_contextual_orchestrator() { assert_file_contains "$REPO_ROOT/scripts/ci/run_opencode_review_model_pool.sh" "exponential backoff" "opencode model retry paths use exponential backoff instead of fixed sleeps" assert_file_contains "$workflow_file" '"enabled_providers": ["contextual-orchestrator"]' "opencode review keeps the generated provider set gateway-only" assert_file_contains "$workflow_file" '"model": "contextual-orchestrator/orchestrator/free"' "opencode review keeps the generated model on orchestrator/free" - assert_file_contains "$workflow_file" "coverage-source-tree:" "opencode workflow materializes coverage source before running PR-head tests" + assert_file_not_contains "$workflow_file" "coverage-source-tree:" "opencode workflow does not restore the folded same-trust coverage-source job" assert_file_contains "$workflow_file" "coverage-evidence:" "opencode workflow measures coverage before review" + assert_file_contains "$workflow_file" "needs: [validate-pr-metadata]" "coverage evidence starts only after the folded validation/materialization job succeeds" assert_file_contains "$workflow_file" "Materialize pull request merge tree for coverage measurement" "required OpenCode reviews measure coverage instead of approving skipped coverage evidence" assert_file_contains "$workflow_file" "Exchange OpenCode app token for target repository coverage reads" "coverage source materialization can read private target repositories during central manual dispatch" assert_file_contains "$workflow_file" "Upload materialized pull request merge tree" "coverage source materialization passes only a prepared merge tree artifact to the PR-head coverage job" assert_file_contains "$workflow_file" "Download materialized pull request merge tree" "coverage evidence consumes the prepared merge tree artifact without target-repository credentials" - assert_file_contains "$workflow_file" "Report coverage source materialization failure" "coverage evidence logs source materialization failures as the coverage blocker" + assert_file_not_contains "$workflow_file" "Report coverage source materialization failure" "folded materialization failure stops its producer job instead of scheduling a reporting-only consumer" local coverage_merge_tree_step coverage_merge_tree_step="$( awk ' @@ -3296,6 +3303,7 @@ run_gate_case() { local gate_under_test="$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$GATE_SCRIPT" "$gate_under_test" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$gate_under_test" local fake_strix="$bin_dir/strix" local path_hijack_log="$tmp_dir/path-hijack.log" @@ -7026,6 +7034,7 @@ run_pull_request_target_head_scope_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -7174,6 +7183,7 @@ run_pull_request_target_plaintext_runner_token_fails_closed_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -7296,6 +7306,7 @@ run_pull_request_target_bounded_head_context_scope_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -7401,6 +7412,7 @@ run_pull_request_target_changed_context_scope_uses_pr_head_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -7580,6 +7592,7 @@ run_pull_request_target_changed_backend_context_scope_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -7839,6 +7852,7 @@ run_pull_request_target_frontend_email_context_scope_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -8029,6 +8043,7 @@ run_pull_request_target_shallow_head_merge_base_fallback_case() { cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -8144,6 +8159,7 @@ run_pull_request_target_aborts_on_pr_head_blob_failure_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local real_git @@ -8268,6 +8284,7 @@ run_pull_request_target_rejects_invalid_sha_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -8361,6 +8378,7 @@ run_pull_request_target_irregular_head_entry_fails_closed_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -8444,6 +8462,7 @@ run_pull_request_target_gitlink_is_explicitly_skipped_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -8526,6 +8545,7 @@ run_full_head_scope_skips_gitlink_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -8640,6 +8660,7 @@ run_pull_request_target_rejects_unsafe_changed_path_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" @@ -8732,6 +8753,7 @@ run_timeout_cleanup_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" local child_pid_file="$tmp_dir/child.pid" @@ -8814,6 +8836,7 @@ run_vertex_model_ignores_untrusted_llm_api_base_file_case() { mkdir -p "$repo_root_dir/scripts/ci" "$allowed_input_dir" "$outside_dir" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cat >"$fake_strix" <<'EOF' @@ -8866,6 +8889,7 @@ run_total_timeout_case() { mkdir -p "$bin_dir" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" local output_log="$tmp_dir/output.log" @@ -9193,6 +9217,7 @@ run_llm_api_base_file_outside_input_root_fails_closed_case() { mkdir -p "$repo_root_dir/scripts/ci" "$allowed_input_dir" "$outside_dir" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cat >"$fake_strix" <<'EOF' @@ -9248,6 +9273,7 @@ run_pr_scoped_llm_api_base_file_config_failure_exits_2_case() { mkdir -p "$repo_root_dir/scripts/ci" "$repo_root_dir/src" "$allowed_input_dir" "$outside_dir" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" printf '%s\n' 'print("one")' >"$repo_root_dir/src/one.py" printf '%s\n' 'print("two")' >"$repo_root_dir/src/two.py" @@ -9309,6 +9335,7 @@ run_required_input_file_outside_input_root_fails_closed_case() { mkdir -p "$repo_root_dir/scripts/ci" "$allowed_input_dir" "$outside_dir" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cat >"$fake_strix" <<'EOF' @@ -9379,6 +9406,7 @@ run_input_file_root_override_takes_precedence_over_runner_temp_case() { mkdir -p "$repo_root_dir/scripts/ci" "$explicit_input_root" "$inherited_runner_temp" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cat >"$fake_strix" <<'EOF' @@ -9433,6 +9461,7 @@ run_stale_report_case() { mkdir -p "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" mkdir -p "$stale_report_dir" @@ -9488,6 +9517,7 @@ run_symlink_report_case() { mkdir -p "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" mkdir -p "$external_report_dir" "$repo_root_dir/strix_runs" @@ -9544,6 +9574,7 @@ run_unsafe_target_path_case() { mkdir -p "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cat >"$fake_strix" <<'EOF' @@ -9592,6 +9623,7 @@ run_absolute_outside_target_path_case() { mkdir -p "$bin_dir" "$repo_root_dir/src" "$repo_root_dir/scripts/ci" cp "$GATE_SCRIPT" "$repo_root_dir/scripts/ci/strix_quick_gate.sh" cp "$REPO_ROOT/scripts/ci/strix_model_utils.sh" "$repo_root_dir/scripts/ci/strix_model_utils.sh" + cp "$REPO_ROOT/scripts/ci/strix_evidence_binding.py" "$repo_root_dir/scripts/ci/strix_evidence_binding.py" chmod +x "$repo_root_dir/scripts/ci/strix_quick_gate.sh" local fake_strix="$bin_dir/strix" local call_log="$tmp_dir/calls.log" diff --git a/tests/test_strix_llm_timeout_contract.py b/tests/test_strix_llm_timeout_contract.py index b46630c898..3493d02cfa 100644 --- a/tests/test_strix_llm_timeout_contract.py +++ b/tests/test_strix_llm_timeout_contract.py @@ -1,4 +1,4 @@ -"""Regression contract for unbounded Strix inference through contextual-orchestrator.""" +"""Regression contract for progress-bounded Strix inference (no elapsed model cap).""" from __future__ import annotations @@ -51,15 +51,16 @@ def test_strix_timeout_compat_is_installed_after_the_pinned_runtime() -> None: assert LAUNCHER.is_file() -def test_compat_launcher_disables_request_and_stream_idle_deadlines() -> None: - """The launcher maps central review policy to zero/unbounded settings.""" +def test_compat_launcher_clears_elapsed_request_deadline_keeps_stream_idle() -> None: + """Elapsed request deadlines stay disabled; stream-idle uses the #1884 90s bound.""" launcher = _load_launcher() environment = {"LLM_TIMEOUT": "300", "LLM_STREAM_IDLE_TIMEOUT": "300"} launcher.normalize_inference_timeout_environment(environment) assert environment["LLM_TIMEOUT"] == "0" - assert environment["LLM_STREAM_IDLE_TIMEOUT"] == "0" + assert environment["LLM_STREAM_IDLE_TIMEOUT"] == "90" + assert launcher.STREAM_IDLE_OCCUPANCY_SECONDS == "90" assert launcher.SUPPORTED_VERSION == "1.5.3" @@ -150,7 +151,7 @@ def make_model_settings(*args, **kwargs): assert result is main_module assert launcher.os.environ["LLM_TIMEOUT"] == "0" - assert launcher.os.environ["LLM_STREAM_IDLE_TIMEOUT"] == "0" + assert launcher.os.environ["LLM_STREAM_IDLE_TIMEOUT"] == "90" assert isinstance(scan_setup_module.asyncio, launcher.UnboundedInferenceAsyncio) assert isinstance(main_module.asyncio, launcher.UnboundedInferenceAsyncio) inputs_module.make_model_settings("model", request_timeout=300, other="kept") diff --git a/tests/test_strix_occupancy_bound_contract.py b/tests/test_strix_occupancy_bound_contract.py new file mode 100644 index 0000000000..0e934a3e57 --- /dev/null +++ b/tests/test_strix_occupancy_bound_contract.py @@ -0,0 +1,43 @@ +"""Contract: central Strix releases no-progress, never elapsed-time work.""" + +from __future__ import annotations + +from pathlib import Path +import re + + +ROOT = Path(__file__).resolve().parents[1] +WORKFLOW = ROOT / ".github" / "workflows" / "strix.yml" +COMPAT = ROOT / "scripts" / "ci" / "strix_timeout_compat.py" +DOCTORING = ROOT / "docs" / "doctoring" / "strix-unbounded-agentic-occupancy-20260918.md" +ADR = ROOT / "docs" / "adr" / "0034-review-runner-occupancy-progress-bound.md" + + +def _strix_job_header() -> str: + """Return the strix job header up to its first steps: block.""" + text = WORKFLOW.read_text(encoding="utf-8") + match = re.search(r"^ strix:\n(.*?)(?=^ steps:\n)", text, flags=re.MULTILINE | re.DOTALL) + assert match is not None, "strix job header not found" + return match.group(0) + + +def test_strix_job_has_no_elapsed_occupancy_timeout() -> None: + """Active reasoning or streaming must not end because wall time elapsed.""" + header = _strix_job_header() + assert "timeout-minutes:" not in header + workflow = WORKFLOW.read_text(encoding="utf-8") + compat = COMPAT.read_text(encoding="utf-8") + assert 'STREAM_IDLE_OCCUPANCY_SECONDS = "90"' in compat + assert 'environment["LLM_STREAM_IDLE_TIMEOUT"] = STREAM_IDLE_OCCUPANCY_SECONDS' in compat + assert "export STRIX_PROCESS_TIMEOUT_SECONDS=0" in workflow + assert "export STRIX_TOTAL_TIMEOUT_SECONDS=0" in workflow + assert DOCTORING.is_file() + assert "35263416380" in DOCTORING.read_text(encoding="utf-8") + assert ADR.is_file() + assert "elapsed inference" in ADR.read_text(encoding="utf-8").lower() + + +def test_required_strix_job_id_stays_stable() -> None: + """Keep the required check context name `strix` unchanged.""" + text = WORKFLOW.read_text(encoding="utf-8") + assert re.search(r"^ strix:\n", text, flags=re.MULTILINE)