Skip to content

Reject build-only regression proof and save private local receipts - #7

Merged
jerryfane merged 2 commits into
mainfrom
phobos/test-check-proof-receipts-20261007
Oct 7, 2026
Merged

jerryfane merged 2 commits into
mainfrom
phobos/test-check-proof-receipts-20261007

Conversation

@jerryfane

@jerryfane jerryfane commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Assignment

Owner explicitly approved two improvements: (1) old-code compile/import-only failures are inconclusive, never behavioral proof; (2) automatic concise, private, durable local command receipts. Implementation seat: phobos (test-check-proof-fix), receipt filesystem package contributed by independent worker test-check-receipts. No Gitmoot implementation jobs or provider/payment changes.

Frozen final head: 888e88cd951507b433d2d476ca4b0e1def0a3e29; base main: b1a900dd502a4bd2f7d7ac7118bba9090da74802. Prior independent review at 09c7708f7f1ba61f390b3d0bb5d9213075f1264e requested changes; both executed findings addressed and a fresh full final-head review is required.

Changes

  • inconclusive / exit 4 for old build/import failures, explicit reason and existing build flag; per-test aggregates and pieces cannot claim proof/guarded status for build-only failure. Actual Go/Python assertion summaries take precedence over diagnostic words in assertion messages. New-code failures remain fails_on_new.
  • Automatic advice/prove receipts outside checkout with 0700 directories/0600 files, exact available commit and bounded changed-input identity, named-runner evidence, results/warnings, piece rows, and bounded diagnostic links. No deliberate prompts/diffs/env/config collection; command/output literals may still be sensitive.
  • Storage failure preserves original exit/outcome and reports stderr warning + JSON error without a fake path. Controlled cancellation saves only after restoration. Help/list-tests/restore do not create proof/advice receipts.
  • README change notes + packaged skill updated; existing build-warning expectation migrated.
  • Review fixes: missing Python module API becomes inconclusive; advice argv options retained with prompt-title values explicitly redacted. Caller cancellation and test-check-owned deadlines return error after restoration, never child proof/guarded claims. Framework assertion/panic behavior remains valid test failure.

Exercised verification

Go 1.26.4; CGO=0, umask 022 for tests/vet/build/release. CGO=1 only for the required race detector.

  • GOTOOLCHAIN=go1.26.4 CGO_ENABLED=1 go test -race ./internal/receipt passed.
  • GOTOOLCHAIN=go1.26.4 CGO_ENABLED=0 go test ./..., go vet ./..., and actual CLI build passed.
  • Built CLI test-check prove proved seven consumer CLI regressions against original main, and four review-boundary regressions against reviewed head 09c7708: actual old --- FAIL test results, all new pass, build_error_suspected=false per test and aggregate. This does not claim all new package tests are old-compatible or old-red.
  • Actual built CLI in isolated real Git fixtures: 28 scenarios passed. Go/Python behavior/build failures and all earlier restoration/receipt cases, plus missing module API whole/each/pieces, tool-owned deadlines on new and old code in all modes, old-run per-test/piece SIGTERM with no false child proof, and advice options/title-redaction in both flag forms.
  • Cancellation fixture observed old bytes while running, then exact files/index/status/stash restored at receipt publication. No production repository fixtures mutated.
  • Real advice call produced a receipt; validates advice execution path, not model advice accuracy.

Durable local evidence: /root/fleet-tools/state/test-check-improvements-20261007/:

  • proof/final-regression-proof.json (+ stderr)
  • proof/advice.json (+ stderr)
  • integration/final-built-cli/results.json
  • integration/evidence-index.json: copied/read-back private receipts + bounded details under integration/durable-receipts/; not dependent solely on temporary fixture receipt paths.
  • Latest proof: proof/review-fixed-original-base.json and proof/review-obligations-old-red.json.
  • Latest actual CLI scenarios: integration/review-fixed-cli/review-results.json.

Independent formal final-head review, observed Gitmoot merge-gate prerequisites, and GitHub CI are required before merge. No administrator bypass.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18dc25809c553ded-1

Decision: changes_requested
Summary: Reviewed the full diff from b1a900d to exact head 09c7708. No prior verdict for PR #7 appears in the supplied verdicts frozen at 2026-10-07T04:41:10Z. Build, vet, and tests pass with CGO disabled; native commands cannot access /usr/include/stdc-predef.h. A Python missing-attribute failure is still accepted as behavioral proof in all three modes.
Severity: P1

Findings

  • Missing Python attribute is accepted as regression proof (P1)
    • details: oldFailure recognizes selected build/import diagnostics but not AttributeError. In a scratch Git repository, the new test called value.added(), which exists only in the new code. The old run ended with AttributeError: module 'value' has no attribute 'added'; no behavioral assertion ran. The CLI nevertheless returned exit 0 and outcome proven in whole and per-test modes, and guarded in piece mode. The receipt also recorded proven. This defeats the new build-only-proof rejection for a normal Python API addition.
    • evidence_locator: internal/prove/prove.go:618-634; internal/prove/pieces.go:273-279
  • Advice receipts omit the invoked command options (P2)
    • details: The advice receipt command is constructed from only --repo/--pr/--compare. A local invocation with --dir, --base, --title, --model and --json produced a saved receipt whose command was just "test-check". It does not identify the command that generated the evidence, contrary to the documented receipt contract.
    • evidence_locator: cmd/test-check/main.go:94-101; cmd/test-check/receipt.go:31-32

Tests Run

  • go test ./... — failed: runtime/cgo could not read /usr/include/stdc-predef.h (permission denied).
  • go build ./... — failed: same C-header permission denial.
  • go vet ./... — failed: same C-header permission denial.
  • CGO_ENABLED=0 go test -count=1 ./cmd/test-check ./internal/prove ./internal/receipt — passed.
  • CGO_ENABLED=0 go test -count=1 ./... — passed, all six packages.
  • CGO_ENABLED=0 go build ./... — passed.
  • CGO_ENABLED=0 go vet ./... — passed.
  • Built the CLI with CGO_ENABLED=0 and ran it against a scratch Python Git repository: whole and per-test prove returned exit 0/proven; piece prove returned exit 0/proven with a guarded piece, despite an old-run AttributeError. The changed file was restored after each run.
  • Ran the built CLI against the scratch repository with local advice options and no API key: exit 3, receipt saved, recorded command "test-check" despite the supplied options.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18dc262c36cd70f8-1

Decision: approved
Summary: Reviewed the full diff against b1a900d at exact head 888e88c. Both prior findings are resolved at this head. A scratch-checkout CLI run returned inconclusive for a missing Python attribute in whole, per-test, and piece modes; a genuine assertion failure returned proven in all three. Receipts matched those outcomes, and the advice receipt retained command options while redacting the title.

Findings

  • Missing Python attribute no longer counts as regression proof (P1)
    • continues_uid: Reject build-only regression proof and save private local receipts #7-f1
    • detail: Checked at this head with the focused python_attribute test and an independent scratch-checkout CLI run. When the old module lacked value.added, whole, per-test, and piece modes each exited 4 with outcome inconclusive and reason old_code_build_failed; their receipts also recorded inconclusive.
    • evidence_locator: internal/prove/prove.go:640-655; internal/prove/pieces.go:278-284
    • file: internal/prove/prove.go
    • state: answered
  • Advice receipt now retains invoked options (P2)
    • continues_uid: Reject build-only regression proof and save private local receipts #7-f2
    • detail: Checked at this head with TestAdviceReceiptPreservesOptionsWithoutPromptTitle and an independent CLI invocation using --dir, --base, --model, --json, and --title. The saved command contained those options and [redacted] in place of the title; it did not contain the private title text.
    • evidence_locator: cmd/test-check/receipt.go:202-223; cmd/test-check/main.go:94-96
    • file: cmd/test-check/receipt.go
    • state: answered

Tests Run

  • env CGO_ENABLED=0 go build ./... — passed
  • env CGO_ENABLED=0 go vet ./... — passed
  • env CGO_ENABLED=0 go test -count=1 ./cmd/test-check ./internal/prove ./internal/receipt — passed
  • env CGO_ENABLED=0 go test -count=1 -v ./cmd/test-check -run '^TestProofDistinguishesBuildFailureFromBehavior/python_attribute$' — passed
  • env CGO_ENABLED=0 go test -count=1 -v ./cmd/test-check -run '^TestAdviceReceiptPreservesOptionsWithoutPromptTitle$' — passed
  • env CGO_ENABLED=0 go test -count=1 -v ./cmd/test-check -run '^(TestRunnerDeadlinesNeverProveRegression|TestCanceledOldRunNeverClaimsChildProof)$' — passed, including whole/per-test/piece interruption cases
  • env CGO_ENABLED=0 go test -count=1 ./... — passed
  • Scratch-checkout CLI: built cmd/test-check, then ran prove --json with Python unittest in whole, --each, and --pieces modes. Missing attribute: exit 4/inconclusive in each; genuine assertion: exit 0/proven in each. Advice --json with --dir, --base, --model, and --title: exit 3 without an API key; saved receipt retained options and redacted the title.
    Diagnostics: workflow advance failed: review approval held: approval is bound to head 888e88c but the observed pull request head is 09c7708; holding until the observed head catches up, because an approval at a genuinely superseded head must not clear changes_requested

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane
jerryfane merged commit d5f169a into main Oct 7, 2026
2 checks passed
@jerryfane

Copy link
Copy Markdown
Contributor Author

Merged after exact-head independent executed approval, zero unresolved obligations, green PR CI, and native forge observation at the final reviewed SHA before merge (2026-10-07 05:01:44). Manual merge used --match-head-commit and no administrator bypass; native auto-merge setting unchanged.

Merged/source-installed revision: d5f169ab623e8ba0896b727a09f36738612a79e2. Installed Go1.26.4 CGO0 binary reports vcs.revision=d5f169ab623e8ba0896b727a09f36738612a79e2, vcs.modified=false. Packaged skill installed through existing source symlink.

Installed /root/.local/bin/test-check passed 28 isolated real Git CLI scenarios, including both features, all proof modes, zero-selected/error, caller cancellation and owned deadlines with restoration before receipt publication, unsafe receipt roots preserving core results, advice title redaction, and changed-input identity at fixed HEAD. Every saved installed receipt reports the merged tool revision. Seven original-base consumer CLI regressions plus four regressions against the initial reviewed head were behaviorally proven (actual old test failure, new pass, no old build-error flag).

Durable local release/rollback evidence: /root/fleet-tools/state/test-check-improvements-20261007/integration/release.json, installed-cli/review-results.json, evidence-index.json (66 copied/read-back receipt/detail sets), and test-check.rollback-b1a900d. No production repo fixtures or shared provider/daemon/payment settings changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant