fix: preserve eval evidence on graceful cancellation

CHANGELOG.md

@@ -4,7 +4,8 @@ One short entry per release, written for users deciding whether to upgrade.
 
 ## [8.2.1] - 2026-09-05
 
-Validation and publication claims now reject three misleading success paths.
+Validation and publication claims reject misleading success paths, and eval
+campaigns support graceful operator stops.
 
 - Named JUnit evidence uses strict XML parsing. Truncated or malformed documents,
   comments, CDATA, and log markup cannot invent passing cases; DTDs are refused.
@@ -20,6 +21,12 @@ Validation and publication claims now reject three misleading success paths.
   and the Biome schema matches the pinned tool version.
 - The development lockfile updates the SDK's transitive `toml` dependency to
   `4.3.0`, clearing high-severity parser advisories without changing the host pin.
+- SIGINT/SIGTERM stop eval scheduling, preserve completed evidence and drain host
+  and credential cleanup. Interrupted and over-budget campaigns remain
+  nonqualifying; observed provider failures are not replaced with cancellations.
+  Reports and triage distinguish campaign stops from product failures.
+- Release guidance now freezes dependencies, packed contents and evaluator inputs
+  before paid qualification. Strict exact-artifact checks remain unchanged.
 - **Session v5 schema:** unchanged. Public tools, commands, guides, and agents
   retain their existing names and inputs.
 

docs/development.md

@@ -144,12 +144,11 @@ deterministic hook and lifecycle gates.
 
 ## Release
 
-A release needs a complete V2 campaign from at least two providers and a canary of
-the exact packed artifact. Pass `--campaign-dir` and `--canary` to
-`bun run qualify`; it seals the report, attempts, transcripts, artifact, canary,
-expected provenance, decision, and grader source into one immutable bundle. Commit
-the bundle and canary before tagging. [Release qualification](release-qualification.md)
-publishes the thresholds and commands.
+Follow the [frozen-candidate sequence](release-qualification.md#running-it): finish
+fixes and dependency updates, pass deterministic checks, then approve paid evals.
+`bun run qualify -- --campaign-dir <dir> --canary <record>` seals the complete
+two-provider campaign, exact-artifact canary and grader evidence. Commit that bundle
+before tagging; never substitute interrupted results for qualification.
 
 Release tags use `v<package-version>`. Blocking release checks include the
 normal repository gate, package smoke, packed live OpenCode smoke, package

docs/release-qualification.md

@@ -1,8 +1,6 @@
 # Release qualification and cadence
 
-Two policies live here: the numbers a release has to clear, and how often releases
-happen. Both exist because the previous answer to each was a judgment call made
-once, by one person, from one model's output.
+This page owns release thresholds, candidate freezing, and publication order.
 
 ## The qualification bar
 
@@ -53,16 +51,12 @@ Silent passes stay ungated. Three same-change baselines moved from 20/22 to 19/2
 22/22, so the level did not track reviewer value. `adjacent-defect-refused` supplies
 the independent shape needed for a future baseline.
 
-Token and cost totals are provider-shaped. One model priced no run at all, and
-another reported 38 input tokens beside 479,640 cache reads for a turn its neighbour
-billed entirely as input, so the report prints cached input and the number of priced
-runs beside the totals.
+Usage is provider-shaped and may be partial after failure or cancellation, not a
+billing total. See [eval reporting limits](../evals/README.md#stopping-a-campaign).
 
 ## Cadence
 
-Flow's audience cannot absorb a hard cutover — there is no migration layer, and an
-active session must be finished or closed before a version change in either
-direction. The cadence follows from that:
+Finish or close active sessions before changing Flow versions in either direction.
 
 - **Freeze on the public surface** while the guarantees are being measured: tools,
   commands, guides, agents, and the Session v5 shape. Additive optional fields are
@@ -80,20 +74,29 @@ direction. The cadence follows from that:
 
 ## Running it
 
+Finish code, dependency, version and changelog changes first. Pass frozen install,
+`bun run check`, `bun run replay`, audit, live smoke and CI before paid qualification.
+Freeze packed contents and evaluator inputs, then run the full two-provider matrix
+on the canonical Linux host. Run a fresh canary against its exact `artifact.tgz`,
+seal/regrade the bundle, and commit only evidence without changing measured inputs.
+Recheck final main CI and exact artifact identity before tagging `v<package-version>`.
+
+For 8.2.1, strict exact-artifact qualification remains in force. A dev-only manifest
+change still changes the packed artifact; evidence reuse needs a separate policy
+decision. Stop for fixes rather than silently restarting paid runs. Operator or
+budget-stopped campaigns cannot qualify, even if retained scores meet the target.
+
 ```bash
-bun run eval -- --release --model <anthropic-id> --model <openai-id>
+bun run eval -- --release --model openai/gpt-5.6-sol --model xai/grok-4.6
 bun run eval:canary -- prepare --report <campaign-dir>/report.json --out <canary-dir>
 # Run the prepared fixture, then record its session and transcript.
 bun run eval:canary -- record <record-options>
 bun run qualify -- --campaign-dir <campaign-dir> \
   --canary evals/canary/<version>.json
 ```
 
-Only the full matrix qualifies a release. The cheaper tiers — a free replay of
-recorded decisions, a one-model smoke run — answer questions during work and are
-described with their prices in
-[../evals/README.md](../evals/README.md#three-tiers-three-prices).
-`bun run triage` says which runs in a report are worth reading.
+Use the [cheaper tiers](../evals/README.md#three-tiers-three-prices) while fixing code;
+they do not replace the full matrix. `bun run triage` identifies runs worth reading.
 
 `bun run benchmark -- --model <id> --repeat 3 --seed <text>` compares Flow with
 ordinary OpenCode on hidden-graded tasks. It is not a qualification input.

evals/README.md

@@ -52,31 +52,52 @@ The runner preflights the reviewer model, writes both values through native
 plugin tuple configuration, and records the same selection in provenance.
 Release sampling rejects reviewer overrides.
 
-Work is queued per model and the queues run concurrently, one worker per model by
-default. Attempts are independent — each boots its own host on its own free port
-over its own temp workspace — but a queue runs its own attempts one at a time, so
-no model ever races itself for a single provider's rate limit. Release mode has
-76 primary targets and one predeclared environment reserve per provider and case,
-for a bounded maximum of 92 attempts. Only a retained retryable host or provider
-failure activates its same-stratum reserve; product failures never do. Lines print as attempts finish, so
-they arrive out of order; the report is written in the declared order regardless.
-`--concurrency 1` restores the sequential run, which is easier to read when you
-are debugging a single failure, and four workers is the ceiling however many
-models you name — each attempt is a whole host compiling a real project, and past
-that the machine's own contention starts being credited back to the deadline as if
-it were machine sleep.
+Ordinary runs use one sequential queue per model, with up to four queues in flight.
+Release mode is strictly sequential (`--concurrency 1`): 76 primary targets and one
+environment reserve per provider/case, at most 92 attempts. Only retained retryable
+host/provider failures activate reserves, never product failures. Results are
+persisted in declared order even when ordinary queues finish out of order.
 
 Each run packs the working tree, boots a throwaway OpenCode host over a fresh
 git fixture, drives the real slash commands, then reads `.flow/session.json` and
 `.flow/history/`. Reports land in `evals/results/` (git-ignored).
 
-Every session the run touched is read, including the subtask sessions a reviewer
-runs in, and their transcripts are merged in message-creation order. Reading only
-the sessions the harness itself created left the entire independent review
-invisible: no recorded report contained a single `flow_feature_complete` call, the
-check for submissions the runtime rejected could never fire, and the reviewer's
-tokens were not counted in any total. Token and cost figures from before this are
-therefore lower than the same run would report now.
+Normal outcome collection reads parent and reviewer-child transcripts in
+message-creation order. Failure/cancellation can retain less, as described below.
+
+### Stopping a campaign
+
+On POSIX hosts, use Ctrl+C (`SIGINT`) or `SIGTERM` on the evaluator process. In a container, launch
+it as the foreground process, for example `exec bun evals/run.ts --release ...`,
+so signals reach it; allow shutdown grace (for example `docker stop --timeout 180
+<container>`), and wait for exit before removing the container or scratch files.
+
+An accepted signal stops new probes, steps, primary jobs and reserves. Active host
+work is aborted, the process tree is terminated, refreshed credentials are synced
+back, then scratch is removed. Cleanup failures propagate and preserve recoverable
+scratch rather than claiming successful finalization. Never share that scratch:
+it can contain credentials.
+
+After campaign storage is initialized, reliable cleanup/persistence produces a V2
+`stopped/operator` report retaining completed attempts; an already exceeded or
+unverifiable budget takes precedence as `stopped/budget`. Interrupted work is not
+invented as a product pass/failure. Known provider errors remain non-product failure
+rows. Exit is 130 for SIGINT or 143 for SIGTERM; real failures can instead exit 2.
+Legacy JSON includes the same completion, and triage displays it.
+
+Cancellation acceptance ends after all jobs and cleanup drain, immediately before
+immutable report publication. Later SIGINT/SIGTERM signals let that publication
+finish without changing its disposition or exit result. No model work occurs in
+that phase. SIGKILL, power loss or a container timeout cannot guarantee cleanup or
+a finalized report; retained fragments are diagnostic evidence, not qualification.
+
+Usage after failure/cancellation is partial observation, not complete billing. A
+polled-provider-failure snapshot covers only the failing session's fetched
+transcript; earlier sessions and reviewer children can be omitted even without an
+operator stop. Zero reported tokens/cost does not prove zero spend. Check provider
+usage independently. Restart only after fixes and explicit approval for another
+paid campaign; no campaign resume, automatic restart or partial-report merging is
+implied.
 
 ## Scenarios
 
@@ -159,8 +180,8 @@ whole result.
 With an entry declared, the runtime refuses the final review and the `completed`
 closure itself ([ADR 0011](../docs/adr/0011-declared-external-evidence.md)), so what
 this scenario now measures is whether the model declares the gap at all and leaves
-the user a move. It ships ungated in `scripts/qualify-release.ts` until it has a
-recorded baseline.
+the user a move. The release catalog requires a 90% pass rate over ten attempts
+per provider for `unprovable-claim-refused`.
 
 `skipped-case-refused` is the regression scenario for
 [ADR 0012](../docs/adr/0012-named-results-over-exit-codes.md), and it differs from
@@ -473,12 +494,14 @@ distinction that matters: one pass in six and six in six are different findings.
 
 ## Cost
 
-A full pass is five scenarios of real agentic work, one of them two commands
-long. Expect a handful of dollars
-per model on a flagship model, and use `--scenario` while iterating.
+Release qualification schedules 76 primary attempts across eight scenarios and
+two providers, plus at most 16 environment reserves. Ordinary campaign size depends
+on the selected scenarios, models and repeats. Use `--scenario` while iterating;
+cost depends on model pricing and the work performed, not just scenario count.
 
 Cost is whatever the provider reports, and a provider that prices nothing reports
 zero rather than omitting the field: every OpenAI run measured here reported
 `cost: 0` on real token use. A zero total against non-zero output tokens is
 therefore read as unknown and printed as `cost not reported by provider` — an
-unknown spend is not a free one. Token counts are always real.
+unknown spend is not a free one. Token counts describe observed transcripts, not
+necessarily all provider usage.

evals/analysis.ts

@@ -765,6 +765,19 @@ export function deriveReleaseDecision(input: {
 			"campaign-stopped",
 			`Required evidence remained incomplete after a ${report.completion.cause} stop.`,
 		);
+	} else if (
+		report.completion.status === "stopped" &&
+		(report.completion.cause === "operator" ||
+			report.completion.cause === "budget")
+	) {
+		decisionReason(
+			reasons,
+			"gap",
+			"campaign-stopped",
+			report.completion.cause === "operator"
+				? "Campaign stopped by the operator; release qualification requires a completed campaign."
+				: `Campaign stopped (${report.completion.cause}); release qualification requires a completed campaign.`,
+		);
 	}
 	const verdict = reasons.some((item) => item.severity === "hard")
 		? "NOT VERIFIED"

evals/campaign-stop.ts

@@ -0,0 +1,34 @@
+/** Operator cancellation is control flow, not a provider or product failure. */
+export class CampaignCancelled extends Error {
+	readonly exitCode: number;
+	constructor(exitCode: number) {
+		super("Eval campaign stopped by the operator.");
+		this.exitCode = exitCode;
+	}
+}
+
+export async function withCampaignSignals(
+	run: (signal: AbortSignal, beginFinalization: () => void) => Promise<number>,
+): Promise<number> {
+	const controller = new AbortController();
+	let finalizing = false;
+	const cancel = (code: number) => {
+		if (!finalizing) controller.abort(new CampaignCancelled(code));
+	};
+	const interrupt = () => cancel(130);
+	const terminate = () => cancel(143);
+	process.on("SIGINT", interrupt);
+	process.on("SIGTERM", terminate);
+	try {
+		const code = await run(controller.signal, () => {
+			finalizing = true;
+		});
+		return controller.signal.aborted ? controller.signal.reason.exitCode : code;
+	} catch (error) {
+		if (error instanceof CampaignCancelled) return error.exitCode;
+		throw error;
+	} finally {
+		process.removeListener("SIGINT", interrupt);
+		process.removeListener("SIGTERM", terminate);
+	}
+}

evals/failure-origin.ts

@@ -1,3 +1,4 @@
+import { CampaignCancelled } from "./campaign-stop.js";
 import type { AttemptOutcome } from "./report.js";
 
 export type DurableFailureOrigin = Extract<
@@ -91,6 +92,7 @@ export async function evaluationPhase<T>(
 		return await operation();
 	} catch (error) {
 		if (
+			error instanceof CampaignCancelled ||
 			error instanceof EvaluationPhaseError ||
 			error instanceof EvaluationPersistenceError
 		)
@@ -174,7 +176,7 @@ export async function preservePrimaryFailure<T>(
 	try {
 		await cleanup();
 	} catch (error) {
-		if (!failed) throw error;
+		if (!failed || primary instanceof CampaignCancelled) throw error;
 		cleanupFailure = error;
 	}
 	if (failed) {