Skip to content

RV-S7: SQL test wrapper exits 0 on failed assertions (run_tests.sh) #181

Closed

Description

@amar-python

What happened

External repo review at commit c7bebd1 reproduced: tests/framework/test_framework.sql records FAIL rows without raising; tests/run_tests.sh exits 0 unless psql itself errors. Scratch run with a forced assertion failure returned exit 0.

Code paths:

  • tests/framework/test_framework.sql L55–78 (assertion impl)
  • tests/run_tests.sh L104–123 (wrapper exit)

Tier S eval parses report results separately, which limits CI exposure, but the documented standalone run_tests.sh verification command is unreliable for automation.

Why this matters

Affects confidence in #60 offline install evidence: the PASS lines in the clean-VM dry run come from this wrapper. Until fixed, a passing exit is not evidence that the assertions passed.

Proposed

  1. Make the SQL orchestrator raise (or exit non-zero) when any assertion recorded FAIL.
  2. Shell wrapper propagates that exit status.
  3. Negative control: a forced assertion failure in a test fixture that CI confirms makes the wrapper exit non-zero.

Done when

  • Forced FAIL assertion makes tests/run_tests.sh exit non-zero.
  • Existing passing suite still exits 0.
  • Negative control committed and runs in CI.

Reference

Tracker row G40. Repo review finding RV-S7. Sub-issue of #60 (offline install evidence trust). Related to RV-S6 (Tier I ON_ERROR_STOP).

Activity

  1. amar-python commented on Nov 9, 2026

    @amar-python
    OwnerAuthor

    Fixed in #202 (merged as 203c8ae).


    Generated by Claude Code

  2. added a commit that references this issue on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions