diff --git a/CHANGELOG.md b/CHANGELOG.md index 67cda32..cdf43bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased +### Added + +- The README opens with what ProofCLI checks, why an exit code of 0 is not enough, a no-clone try-it, and a still of the broken exporter failing. Those README images use absolute GitHub URLs so they render on PyPI. [docs/ai-agents.md](docs/ai-agents.md) describes using a contract to check changes from an AI coding agent. + ### Changed - `proofcli --help` lists `run`, `validate`, and `init`, each with its own one-line description. `proofcli run --help`, `proofcli validate --help`, and `proofcli init --help` each describe that command instead of sharing one sentence. `proofcli SPEC` still runs a spec, and exit codes are unchanged. diff --git a/README.md b/README.md index 18dc7eb..1e03bfd 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,14 @@ # ProofCLI +ProofCLI runs a command and checks the files it writes. + +Exit code 0 is not proof the output is right. + +Passing starter, no git clone: `python3 -m venv .venv && .venv/bin/python -m pip install proofcli==0.4.2 && .venv/bin/proofcli init demo.json && .venv/bin/proofcli run demo.json` + +![ProofCLI fails the broken exporter. Grace's amount is 999.00, expected 8.25. The exporter's exit code is 0. ProofCLI's process exit is 1.](https://raw.githubusercontent.com/bk1202/proofcli/main/docs/assets/wrong-value-failure.png) + ### A command can exit successfully and still produce the wrong result. Run the command. Check its files. Report what passed and what failed. @@ -13,13 +21,13 @@ Run the command. Check its files. Report what passed and what failed. **Windows · macOS · Linux · JSON contracts · Human and JSON reports** -[Quick start](#quick-start) · [See it fail](#see-it-fail) · [Checks](#what-you-can-check) · [Documentation](#documentation) · [Limitations](#limitations) +[Quick start](#quick-start) · [See it fail](#see-it-fail) · [Checks](#what-you-can-check) · [AI agents](docs/ai-agents.md) · [Documentation](#documentation) · [Limitations](#limitations) --- -![An exporter exits 0 and writes three records, but ProofCLI catches Grace's amount: expected 8.25, actual 999.00. ProofCLI exits 1.](docs/assets/output-contract.png) +![An exporter exits 0 and writes three records, but ProofCLI catches Grace's amount: expected 8.25, actual 999.00. ProofCLI exits 1.](https://raw.githubusercontent.com/bk1202/proofcli/main/docs/assets/output-contract.png) The [broken exporter](examples/export/export.py) prints “Export complete” and exits `0`. Its JSON and CSV contain the wrong amount. The [contract](examples/export/wrong-value.json) checks the records and fails with this difference in the actual report: @@ -76,6 +84,20 @@ Artifact directories must be new. Reports are replaced atomically when explicitl ## See it fail +### Try the broken export without cloning + +The picture at the top is a real run of this contract. The PyPI 0.4.2 wheel does not include the example. Install from the [quick start](#quick-start), put that environment's `proofcli` on `PATH`, and fetch the contract, the exporter, and the fixture from the `v0.4.2` tag: + +```sh +mkdir -p export-demo/fixtures +curl -fsSL -o export-demo/wrong-value.json https://raw.githubusercontent.com/bk1202/proofcli/v0.4.2/examples/export/wrong-value.json +curl -fsSL -o export-demo/export.py https://raw.githubusercontent.com/bk1202/proofcli/v0.4.2/examples/export/export.py +curl -fsSL -o export-demo/fixtures/input.csv https://raw.githubusercontent.com/bk1202/proofcli/v0.4.2/examples/export/fixtures/input.csv +proofcli run export-demo/wrong-value.json +``` + +ProofCLI exits `1`. The exporter exits `0`. `proofcli init demo.json` writes a passing starter. It does not include this exporter. + With ProofCLI [installed](#quick-start), clone the repository and run the failing contract: ```console @@ -232,6 +254,7 @@ ProofCLI can run inside pytest, a shell script, or CI. Choose a general test fra | [Specification](docs/specification.md) | Every contract field, path rule, placeholder, and resource limit. | | [Reports](docs/reports.md) | Statuses, diagnostics, interruption handling, and exit codes. | | [Use cases](docs/use-cases.md) | Choosing checks for exporters, generators, and workflows. | +| [AI agents](docs/ai-agents.md) | Run a contract before an agent calls the work done, and run the same check in CI. | | [Audit](docs/audit-0.3.md) | Defects found, regression coverage, and unresolved risks. | | [Releases](docs/releasing.md) | Build, validation, checksums, and publication. | | [Changelog](CHANGELOG.md) | Version history and compatibility changes. | diff --git a/docs/ai-agents.md b/docs/ai-agents.md new file mode 100644 index 0000000..324b195 --- /dev/null +++ b/docs/ai-agents.md @@ -0,0 +1,54 @@ +# Using ProofCLI with AI coding agents + +An agent can run a command, see exit code 0, and report the change as done while an output file is wrong. Keep a JSON contract for the output the change should produce. Have the agent run that contract before it claims the work is done. Use the same file in CI. + +The commands below are from a checkout of this repository, using the example exporter. In another project, point them at that project's contract. + +## Before calling the work done + +This contract is supposed to fail. ProofCLI exits 1. The exporter exits 0 and writes `999.00` where the contract expects `8.25`: + +```console +proofcli run examples/export/wrong-value.json +``` + +A contract that matches the exporter exits 0: + +```console +proofcli run examples/export/good.json +``` + +`validate` checks the contract and does not run the command: + +```console +proofcli validate examples/export/good.json +``` + +`--json` prints one JSON object on stdout. `--report` writes that report to a file. `--artifacts` copies the declared output files into a directory that must not already exist: + +```console +proofcli run examples/export/good.json --json +proofcli run examples/export/wrong-value.json --report proofcli-report.json --artifacts proofcli-output +``` + +If ProofCLI exits 1, the report names the failed check, the path, the expected value, and the actual value. Exit code 0 means every check in that contract passed. It does not mean the rest of the diff is correct. + +## On pull requests + +The [GitHub Action](github-action.md) checks out nothing by itself. After `actions/checkout`, it runs the ProofCLI source from the selected ref and fails the step when verification fails: + +```yaml +steps: + - uses: actions/checkout@v7 + - uses: bk1202/proofcli@v0.4.2 + with: + spec: examples/export/good.json +``` + +Change `spec` to the contract that should pass for the pull request. The Action exposes `report`, `status`, and `exit-code` outputs. Optional `report` and `artifacts` inputs choose where evidence is saved. Pin a full commit SHA instead of `@v0.4.2` when the workflow needs an immutable action. + +## Limits + +A pass covers the checks in the contract and the files that run produced. If the agent edits the contract so a bad file matches, the check can pass. Review contract changes the same way you review the code. + +ProofCLI is not a sandbox. The command runs with the permissions of whoever starts it. It does not judge prose, visual layout, or whether a business decision is right. Those limits are also listed in the [README](../README.md#limitations). diff --git a/docs/assets/wrong-value-failure.png b/docs/assets/wrong-value-failure.png new file mode 100644 index 0000000..c292d2b Binary files /dev/null and b/docs/assets/wrong-value-failure.png differ