Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Unreleased

### Added

- The README opens with what ProofCLI checks, why an exit code of 0 is not enough, a no-clone try-it, and a still of the broken exporter failing. Those README images use absolute GitHub URLs so they render on PyPI. [docs/ai-agents.md](docs/ai-agents.md) describes using a contract to check changes from an AI coding agent.

### Changed

- `proofcli --help` lists `run`, `validate`, and `init`, each with its own one-line description. `proofcli run --help`, `proofcli validate --help`, and `proofcli init --help` each describe that command instead of sharing one sentence. `proofcli SPEC` still runs a spec, and exit codes are unchanged.
Expand Down
27 changes: 25 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

# ProofCLI

ProofCLI runs a command and checks the files it writes.

Exit code 0 is not proof the output is right.

Passing starter, no git clone: `python3 -m venv .venv && .venv/bin/python -m pip install proofcli==0.4.2 && .venv/bin/proofcli init demo.json && .venv/bin/proofcli run demo.json`

![ProofCLI fails the broken exporter. Grace's amount is 999.00, expected 8.25. The exporter's exit code is 0. ProofCLI's process exit is 1.](https://raw.githubusercontent.com/bk1202/proofcli/main/docs/assets/wrong-value-failure.png)

### A command can exit successfully and still produce the wrong result.

Run the command. Check its files. Report what passed and what failed.
Expand All @@ -13,13 +21,13 @@ Run the command. Check its files. Report what passed and what failed.

**Windows · macOS · Linux · JSON contracts · Human and JSON reports**

[Quick start](#quick-start) · [See it fail](#see-it-fail) · [Checks](#what-you-can-check) · [Documentation](#documentation) · [Limitations](#limitations)
[Quick start](#quick-start) · [See it fail](#see-it-fail) · [Checks](#what-you-can-check) · [AI agents](docs/ai-agents.md) · [Documentation](#documentation) · [Limitations](#limitations)

</div>

---

![An exporter exits 0 and writes three records, but ProofCLI catches Grace's amount: expected 8.25, actual 999.00. ProofCLI exits 1.](docs/assets/output-contract.png)
![An exporter exits 0 and writes three records, but ProofCLI catches Grace's amount: expected 8.25, actual 999.00. ProofCLI exits 1.](https://raw.githubusercontent.com/bk1202/proofcli/main/docs/assets/output-contract.png)

The [broken exporter](examples/export/export.py) prints “Export complete” and exits `0`. Its JSON and CSV contain the wrong amount. The [contract](examples/export/wrong-value.json) checks the records and fails with this difference in the actual report:

Expand Down Expand Up @@ -76,6 +84,20 @@ Artifact directories must be new. Reports are replaced atomically when explicitl

## See it fail

### Try the broken export without cloning

The picture at the top is a real run of this contract. The PyPI 0.4.2 wheel does not include the example. Install from the [quick start](#quick-start), put that environment's `proofcli` on `PATH`, and fetch the contract, the exporter, and the fixture from the `v0.4.2` tag:

```sh
mkdir -p export-demo/fixtures
curl -fsSL -o export-demo/wrong-value.json https://raw.githubusercontent.com/bk1202/proofcli/v0.4.2/examples/export/wrong-value.json
curl -fsSL -o export-demo/export.py https://raw.githubusercontent.com/bk1202/proofcli/v0.4.2/examples/export/export.py
curl -fsSL -o export-demo/fixtures/input.csv https://raw.githubusercontent.com/bk1202/proofcli/v0.4.2/examples/export/fixtures/input.csv
proofcli run export-demo/wrong-value.json
```

ProofCLI exits `1`. The exporter exits `0`. `proofcli init demo.json` writes a passing starter. It does not include this exporter.

With ProofCLI [installed](#quick-start), clone the repository and run the failing contract:

```console
Expand Down Expand Up @@ -232,6 +254,7 @@ ProofCLI can run inside pytest, a shell script, or CI. Choose a general test fra
| [Specification](docs/specification.md) | Every contract field, path rule, placeholder, and resource limit. |
| [Reports](docs/reports.md) | Statuses, diagnostics, interruption handling, and exit codes. |
| [Use cases](docs/use-cases.md) | Choosing checks for exporters, generators, and workflows. |
| [AI agents](docs/ai-agents.md) | Run a contract before an agent calls the work done, and run the same check in CI. |
| [Audit](docs/audit-0.3.md) | Defects found, regression coverage, and unresolved risks. |
| [Releases](docs/releasing.md) | Build, validation, checksums, and publication. |
| [Changelog](CHANGELOG.md) | Version history and compatibility changes. |
Expand Down
54 changes: 54 additions & 0 deletions docs/ai-agents.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Using ProofCLI with AI coding agents

An agent can run a command, see exit code 0, and report the change as done while an output file is wrong. Keep a JSON contract for the output the change should produce. Have the agent run that contract before it claims the work is done. Use the same file in CI.

The commands below are from a checkout of this repository, using the example exporter. In another project, point them at that project's contract.

## Before calling the work done

This contract is supposed to fail. ProofCLI exits 1. The exporter exits 0 and writes `999.00` where the contract expects `8.25`:

```console
proofcli run examples/export/wrong-value.json
```

A contract that matches the exporter exits 0:

```console
proofcli run examples/export/good.json
```

`validate` checks the contract and does not run the command:

```console
proofcli validate examples/export/good.json
```

`--json` prints one JSON object on stdout. `--report` writes that report to a file. `--artifacts` copies the declared output files into a directory that must not already exist:

```console
proofcli run examples/export/good.json --json
proofcli run examples/export/wrong-value.json --report proofcli-report.json --artifacts proofcli-output
```

If ProofCLI exits 1, the report names the failed check, the path, the expected value, and the actual value. Exit code 0 means every check in that contract passed. It does not mean the rest of the diff is correct.

## On pull requests

The [GitHub Action](github-action.md) checks out nothing by itself. After `actions/checkout`, it runs the ProofCLI source from the selected ref and fails the step when verification fails:

```yaml
steps:
- uses: actions/checkout@v7
- uses: bk1202/proofcli@v0.4.2
with:
spec: examples/export/good.json
```

Change `spec` to the contract that should pass for the pull request. The Action exposes `report`, `status`, and `exit-code` outputs. Optional `report` and `artifacts` inputs choose where evidence is saved. Pin a full commit SHA instead of `@v0.4.2` when the workflow needs an immutable action.

## Limits

A pass covers the checks in the contract and the files that run produced. If the agent edits the contract so a bad file matches, the check can pass. Review contract changes the same way you review the code.

ProofCLI is not a sandbox. The command runs with the permissions of whoever starts it. It does not judge prose, visual layout, or whether a business decision is right. Those limits are also listed in the [README](../README.md#limitations).
Binary file added docs/assets/wrong-value-failure.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading