Skip to content

Gate a deploy on the command health check and receipt it (S-009-2..9, 11, 12) - #180

Merged
shipmill-romamo[bot] merged 1 commit into
mainfrom
feat/164-command-health-gate
Oct 30, 2026
Merged

shipmill-romamo[bot] merged 1 commit into
mainfrom
feat/164-command-health-gate

Conversation

@shipmill-romamo

Copy link
Copy Markdown
Contributor

Summary

Runs a component's healthCheck of type: command (schema from #170) in the cloudfall_deploy role's health gate and records it in the release receipt, per spec S-009 (docs/specs/009-command-health-check.md).

  • tasks/command_health_check.yml (new) runs the check after Switch the current release symlink and Restart the component service, inside "Activate the release behind the health gate", in a transient unit cloudfall-health-<component> (systemd-run --wait --collect --service-type=exec, --uid/--gid of the application user, --working-directory=<installRoot>/current, EnvironmentFile=-<installRoot>/shared/env/<component>.env, RuntimeMaxSec=<timeoutSeconds>, TimeoutStopSec=10, NoNewPrivileges, PrivateTmp). The argv is a list: a relative first element becomes <installRoot>/current/<element>, an absolute one is used as is, and the command is never passed through a shell or templated
  • Up to attempts runs, 3 seconds apart (retries/delay/until); the until ends the loop itself on the last attempt, so the result (attempts, exitCode or null, timedOut from systemd-run's Finished with result: timeout line) is always recorded before a separate task fails the gate
  • Output stays on the server: no --pipe, and the run task is no_log: true. Only rc and the timeout line are read
  • On failure the existing rescue restores the previous release (or stops a first one), then writes the receipt with healthCheckPassed: false and fails with a message naming the check, its exit code or timeout, the attempts and the release it was rolled back to
  • The receipt tasks moved to tasks/release_receipt.yml, included from the pass path and from the rescue, so both write the same receipt; a command component's receipt carries healthCheck, http and none receipts do not
  • release-receipt.schema.json: healthCheckPassed is a boolean; new optional healthCheck (type, command, attempts, exitCode, timedOut, additionalProperties: false); false requires a healthCheck whose exitCode is not 0
  • restart.yml, rollback.yml and health.yml keep refusing a command check, now pointing at Run the command health check in restart, rollback and health (S-009-10) #166; main.yml no longer refuses it

Plan

  • Serves: G-3
  • Files: engine/ansible/roles/cloudfall_deploy/tasks/command_health_check.yml (new), engine/ansible/roles/cloudfall_deploy/tasks/release_receipt.yml (new, the receipt tasks moved out of main.yml so the pass path and the rescue write the same receipt), engine/ansible/roles/cloudfall_deploy/tasks/main.yml, defaults/main.yml (a comment), restart.yml, rollback.yml, health.yml (their refusal now points at Run the command health check in restart, rollback and health (S-009-10) #166), config/schemas/v1/release-receipt.schema.json, engine/tests/test_command_health_gate.py (new), engine/tests/test_deploy_role.py, engine/tests/test_release_command.py, engine/README.md, config/README.md, docs/ci-deploy-guide.md, changelog.d/164-command-health-gate.md
  • Order: the receipt schema first (S-009-9, S-009-11), then command_health_check.yml run for real under ansible-playbook with a stand-in systemd-run (S-009-2, 3, 7, 8, 9), then the gate block, its rescue and release_receipt.yml (S-009-4, 5, 6, 12), then the docs; behaviour as in the spec's "How it runs", "Output stays on the server", "Failure" and "Receipt"
  • Proof: engine/tests/test_command_health_gate.py with one test per criterion S-009-2, S-009-3, S-009-4, S-009-5, S-009-6, S-009-7, S-009-8, S-009-9, S-009-11, S-009-12; the existing test_deploy_role.py and test_release_command.py still pass for http and none

Choices worth a look

  • The receipt schema is stricter than the spec's words in two places: a passing receipt with a healthCheck must have exitCode: 0 and timedOut: false, and inside healthCheck timedOut: true requires exitCode: null (and false an integer). Every receipt written today has no healthCheck, so it is unaffected
  • release_receipt.yml is a task file the spec's "Touches" list doesn't name; it keeps one receipt template for the pass path and the rescue
  • command_health_check.yml is included from main.yml only. The spec also includes it from health_gate.yml; that is Run the command health check in restart, rollback and health (S-009-10) #166's (S-009-10)

Test plan

  • engine/tests/test_command_health_gate.py (new, 39 tests): command_health_check.yml and release_receipt.yml run for real under ansible-playbook against localhost with a stand-in systemd-run that, unlike the real one, passes the command's output on. The S-009-9 marker test runs ansible-playbook -vvvv; with no_log: false it fails, showing the marker in the output. The gate's order, its rescue and the failure message are replayed with Ansible's templating, as test_deploy_role.py does
  • uv run pytest engine/tests sdk/tests/test_resources.py sdk/tests/test_command_health_check.py: 318 passed, 1 skipped
  • uv run ruff check ., uv run mypy, ANSIBLE_CONFIG=engine/ansible/ansible.cfg uv run ansible-lint engine/ansible (as CI runs it): clean
  • The full suite is left to CI
  • Not run: a live deploy on a Debian host with the real systemd-run, so its exit code and Finished with result: timeout line on a timeout are taken from the release command's proven behaviour (S-005), not seen here

Fixes #164

🤖 Generated with Claude Code

… 11, 12)

A component's healthCheck of type: command now runs in the deploy role's
health gate, after the switch and the restart and inside the block whose
rescue restores the previous release. It runs in a transient systemd unit
cloudfall-health-<component> as the application user, in
<installRoot>/current, with the unit's EnvironmentFile=- line and
RuntimeMaxSec= set to timeoutSeconds, up to attempts times 3 seconds apart.
The run task is no_log and the unit has no --pipe, so the command's output
stays in the server's journal; only the exit code and systemd-run's timeout
line are read.

A failed check rolls back (or stops a first release), writes the release
receipt with healthCheckPassed: false and the check's result, and fails the
deploy naming the check and its exit code or timeout. The receipt tasks move
to release_receipt.yml so the pass path and the rescue write the same
receipt. The receipt schema gains healthCheck (no other key) and accepts
healthCheckPassed: false only with a failed command check; receipts written
before still validate. http and none gates and receipts are unchanged.
restart, rollback and health keep refusing a command check until #166.

Fixes #164

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo

Copy link
Copy Markdown
Contributor Author

@romamo Decided without waiting: the release receipt schema refuses a little more than S-009-11 lists (a passing healthCheck must carry exitCode: 0 and timedOut: false, a timed-out one exitCode: null, and exitCode stays within 0..255).

Why: S-009-11 asks the schema to refuse contradictory receipts and accept every receipt the deploy writes today; these rules refuse only contradictions, and the review confirmed every receipt written today still validates.
Not taken: the schema exactly as worded in S-009-11, which would accept a passing receipt whose command check recorded a non-zero exit.

Reply with a comment starting with veto before 2026-10-31 15:48 UTC to reverse it, adding what you want instead if you like. A later veto still counts and gets a follow-up.

@shipmill-romamo shipmill-romamo Bot added the decided An agent decided at notify; the maintainer may veto label Oct 30, 2026
@shipmill-romamo
shipmill-romamo Bot merged commit 3cc2c78 into main Oct 30, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the feat/164-command-health-gate branch October 30, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

decided An agent decided at notify; the maintainer may veto

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gate a deploy on the command health check and receipt it (S-009-2..9, 11, 12)

0 participants