Skip to content

Spec 009: a command health check that gates a deploy - #153

Merged
shipmill-romamo[bot] merged 1 commit into
mainfrom
spec/009-command-health-check
Oct 30, 2026
Merged

shipmill-romamo[bot] merged 1 commit into
mainfrom
spec/009-command-health-check

Conversation

@shipmill-romamo

@shipmill-romamo shipmill-romamo Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Spec for #152

Summary

Adds docs/specs/009-command-health-check.md (S-009, status: approved; merging this PR is the approval). No implementation code.

  • A new healthCheck type: command with command, timeoutSeconds (1 to 300) and attempts (1 to 20), the bounds the http check uses
  • It runs inside "Activate the release behind the health gate" in cloudfall_deploy, after the symlink switch and the restart, through the same transient systemd-run unit release_command.yml uses: the application user, working directory <installRoot>/current, the service unit's EnvironmentFile=- line, RuntimeMaxSec= from timeoutSeconds. A relative command[0] resolves against current/ as service.command does
  • Up to attempts runs, 3 seconds apart; if all fail (non-zero exit or timeout) the existing rescue restores and restarts the previous release, or stops a first release, and deploy --yes exits ENGINE_STEP_FAILED naming the exit code
  • ReleaseReceipt gains an optional spec.healthCheck (type, command, attempts, exitCode, timedOut, nothing else). healthCheckPassed becomes a boolean; a failed command check writes a receipt with healthCheckPassed: false
  • Additive: http and none components keep the same tasks and receipts, and a failed http or none gate still writes no receipt
  • 13 acceptance criteria, S-009-1 to S-009-13; the issue's "Done when" is S-009-5

The number is 009 because open PR #128 already uses 008.

Decisions for you

The spec is written with the first option of each.

  1. Should a failed deploy leave a receipt? Today a release receipt is only written when the health gate passes.
    • (recommended) Yes, for a failed command check: write tmp/releases/<component>.json with healthCheckPassed: false, the check's exit code, and previousRelease naming the release that serves again. The CI guide's upload step (if: always()) then keeps the evidence of the failed run
    • No: keep "a receipt means a live release" and only name the exit code in the deploy error message
  2. What happens to the command's output? It may print the API key it calls with.
    • (recommended) It never leaves the server: it goes to the server's journal (journalctl -u cloudfall-health-<component>), the Ansible task is no_log, and neither the receipt, the Ansible output nor the deploy error shows any of it
    • Show the last 500 characters in the deploy error message only (never in the receipt), as the release command does, so a failure is easier to read from CI
  3. Where does a failed receipt go?
    • (recommended) The same file as a successful one, tmp/releases/<component>.json: it is the record of the last deploy, and previousRelease says what serves now
    • A separate tmp/releases/<component>.failed.json, so <component>.json always names a release that passed
  4. Do restart, rollback and health run the command check too? They share the deploy's health gate today.
    • (recommended) Yes: one declared check means the same thing everywhere, so the smoke check must be safe to run at any time against the live service
    • Only deploy runs it; the others fall back to "the unit is active", like type: none
  5. Is the wait between attempts configurable?
    • (recommended) No, fixed at 3 seconds like the http check
    • Add an optional delaySeconds to the command check

Plan

  • Serves: G-3
  • Files: docs/specs/009-command-health-check.md
  • Order: the spec alone; build issues are split from it once it merges
  • Proof: approved by @romamo's answer on this PR (all five choices as recommended)

🤖 Generated with Claude Code

Spec for #152: a healthCheck of type command runs an argv on the server
after the switch and restart, as the application user in current with
the component's environment file; a non-zero exit or timeout takes the
existing rollback and writes a failed release receipt with the exit
code and never the output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo

Copy link
Copy Markdown
Contributor Author

@romamo Decision needed: does spec 009, a command health check that gates a deploy and rolls it back on a non-zero exit (#152), set the behaviour the way you want, with the five choices under "Decisions for you" in this PR's body?

  1. Take every recommendation (recommended): your answer is the approval; shipmill rebases this PR if it needs it and merges it once it is mergeable and green, then files its build issues under The health gate can only check one HTTP status, so a deploy can't run a smoke check before it keeps a release #152. You may still merge it by hand
  2. Change some of them: reply with the numbers and what you want instead; the spec is revised in this PR and you're asked again, and nothing merges until then

Reply here with a number or your own answer; shipmill takes this up on the tick after your reply.

@shipmill-romamo shipmill-romamo Bot added the needs-decision A shipmill session asked a question here; waits for a reply label Oct 28, 2026
@romamo

romamo commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Decision by @romamo, relayed by Claude Code

  1. Take every recommendation: approved, all five choices as recommended.

@shipmill-romamo shipmill-romamo Bot removed the needs-decision A shipmill session asked a question here; waits for a reply label Oct 30, 2026
@shipmill-romamo
shipmill-romamo Bot merged commit 7a256bc into main Oct 30, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the spec/009-command-health-check branch October 30, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant