Spec 009: a command health check that gates a deploy

docs/specs/009-command-health-check.md

@@ -0,0 +1,162 @@
+# S-009: Command health check
+
+status: approved
+
+## Problem
+
+A component's health gate can check one local URL for an expected status (`healthCheck`
+`type: http`) or only that the unit stays up (`type: none`)
+(`config/schemas/v1/component.schema.json`, `healthCheck` `oneOf`). A smoke check that
+makes authenticated calls to the routes customers use can't gate a deploy: `release.command`
+(S-005) runs before the switch, so it can't test the new release while it serves, and nothing
+after the switch in "Activate the release behind the health gate"
+(`engine/ansible/roles/cloudfall_deploy/tasks/main.yml`) can fail the deploy and take its
+`rescue` (cloudfall-dev/cloudfall#152). Teams run the smoke check as a separate CI step after
+`cloudfall deploy` and call `cloudfall rollback` themselves, so a bad release serves for that
+window and its rollback is not in the release receipt. In the reported case the smoke check
+caught a 500 on every authenticated route that an HTTP check on `/health` passed.
+
+## Behaviour
+
+A component may declare a command health check:
+
+```yaml
+spec:
+  healthCheck:
+    type: command
+    command: [.venv/bin/python, -m, app.smoke]
+    timeoutSeconds: 60
+    attempts: 3
+```
+
+- `command` is required, a non-empty list of non-empty strings like `service.command`. A relative first element resolves against `<installRoot>/current/`, as the unit template (`component.service.j2`) resolves `service.command`; an absolute one is used as is
+- `timeoutSeconds` is required, an integer from 1 to 300, the bound of the `http` check; it limits each run
+- `attempts` is required, an integer from 1 to 20, the bound of the `http` check
+- No other key is accepted (`scheme`, `port`, `path` and `expectedStatuses` belong to `http`)
+- Components with `type: http` or `type: none` deploy, restart, roll back and report health exactly as today
+
+**How it runs**
+
+In the `cloudfall_deploy` role, the check runs inside "Activate the release behind the
+health gate", after `Switch the current release symlink` and `Restart the component
+service`, on each server as that server's gate (the play stays `serial: 1`,
+`any_errors_fatal`). It reuses the transient unit mechanism of `release_command.yml`: a
+`systemd-run --wait --collect --service-type=exec` unit named
+`cloudfall-health-<component>`, with:
+
+- `--uid` and `--gid` of the application's `linuxUser` (`application_user`)
+- working directory `<installRoot>/current`
+- `EnvironmentFile=-<installRoot>/shared/env/<component>.env`, the line the service unit carries, so the command sees the variables the service sees, parsed by systemd's rules; a missing file gives no extra variables
+- `RuntimeMaxSec=` set to `timeoutSeconds`, `TimeoutStopSec=10`, `NoNewPrivileges=true` and `PrivateTmp=true`, as for the release command
+
+It runs up to `attempts` times, 3 seconds apart, the delay the `http` check uses, and stops
+at the first exit 0. Each run is bounded by `timeoutSeconds`; a run still going then is
+stopped and counts as a failed attempt. The check is skipped in check mode, like the other
+gates. The tasks live in one task file, `engine/ansible/roles/cloudfall_deploy/tasks/command_health_check.yml`,
+included from the gate block in `main.yml` and from `health_gate.yml`; it computes its paths
+from `install_root`, because `restart.yml` and `rollback.yml` don't set the deploy's path
+facts.
+
+**Output stays on the server**
+
+The command's stdout and stderr go to the server's journal under the transient unit's name
+(no `--pipe`), where the service's own logs already are; `journalctl -u
+cloudfall-health-<component>` on the server shows them. The run task is `no_log: true`. So
+the command's output never reaches the controller: not the receipt, not the
+`ansible-playbook` output, not the `deploy` error. Cloudfall reads only `systemd-run`'s exit
+code and its own `Finished with result: timeout` line, which tells a timeout from an exit.
+
+**Failure**
+
+When every attempt fails, the gate fails and takes the existing `rescue`: the previous
+release's symlink is restored and the service restarted, or, for a first release, the
+service is stopped. Before the rescue fails the play, it writes a failed release receipt (see
+Receipt). `deploy --yes` exits `ENGINE_STEP_FAILED`, and its `error.message` says the command
+health check of the component failed on that server, with the last attempt's exit code or
+the timeout, the number of attempts, and the release it was rolled back to (or that it was
+stopped). A server that already passed its gate earlier in the same deploy keeps the new
+release, as with an `http` check today.
+
+**Receipt**
+
+`ReleaseReceipt` (`config/schemas/v1/release-receipt.schema.json`) gains an optional
+`spec.healthCheck` object, present only for a component with a command check:
+
+- `type`: `command`
+- `command`: as declared
+- `attempts`: how many runs the gate made, 1 to 20
+- `exitCode`: the last run's exit code, or `null` when it timed out
+- `timedOut`: whether the last run was stopped at `timeoutSeconds`
+
+The object takes no other key (`additionalProperties: false`), so the schema itself refuses a
+receipt that carries output. `healthCheckPassed` changes from `const: true` to a boolean;
+`false` is valid only with a `healthCheck` of `type: command` whose `exitCode` is not 0. On a
+pass the receipt is written where and when it is today, with `healthCheckPassed: true` and
+`exitCode: 0`. On a failed command check the rescue writes the receipt to the same path,
+`<releases>/<component>.json`, with `healthCheckPassed: false`; `release` names the release
+that failed, `previousRelease` names the release now active again (`null` for a first release,
+which was stopped), and `releaseCommand` is there as on a pass when the component declares one.
+The receipt file is the record of the component's last deploy on the server that wrote it last,
+so `previousRelease` is what a reader checks to know what serves after a failure. A failed
+`http` or `none` gate writes no receipt, as today. Nothing in the SDK loads release receipts
+today, so an older `cloudfall` is not affected.
+
+**Restart, rollback and health**
+
+`restart.yml`, `rollback.yml` and `health.yml` include `health_gate.yml`, so `cloudfall
+restart`, `cloudfall rollback` and `cloudfall health` run the command check too: one
+declared check, one meaning, wherever Cloudfall asks whether the component is healthy. They
+write no receipt, as today. A rollback still does not restore anything when its gate fails.
+The check must therefore be safe to run at any time against the live service, as a smoke
+check is.
+
+**Plan**
+
+`deploy`, `restart` and `rollback` without `--yes` say in their `effect` that the component's
+command health check `<command>` runs on each server after its restart. The plan JSON gains no
+field.
+
+Touches `config/schemas/v1/component.schema.json`,
+`config/schemas/v1/release-receipt.schema.json`, `sdk/src/cloudfall/domain.py`
+(`HealthCheckType`), `sdk/src/cloudfall/inventory.py` (`ComponentHealthCheck` and its parser),
+`sdk/src/cloudfall/lifecycle.py` (preview effects),
+`engine/src/cloudfall_engine/ansible_inventory.py`,
+`engine/ansible/roles/cloudfall_deploy/meta/argument_specs.yml`,
+`engine/ansible/roles/cloudfall_deploy/tasks/main.yml`,
+`engine/ansible/roles/cloudfall_deploy/tasks/health_gate.yml`, the new
+`engine/ansible/roles/cloudfall_deploy/tasks/command_health_check.yml`, `engine/README.md`,
+`config/README.md` and `docs/ci-deploy-guide.md`.
+
+## Acceptance criteria
+
+- S-009-1: `config validate` accepts a component with `healthCheck: {type: command, command, timeoutSeconds, attempts}` and exits `CONFIG_INVALID` for an empty `command`, a missing `timeoutSeconds` or `attempts`, a `timeoutSeconds` outside 1 to 300, an `attempts` outside 1 to 20, or an `http` key such as `port` on a command check
+- S-009-2: The command check runs after the `Switch the current release symlink` and `Restart the component service` tasks, inside the block whose `rescue` restores the previous release, as the application's `linuxUser`, with working directory `<installRoot>/current`, the service unit's `EnvironmentFile=-` line and `RuntimeMaxSec=` equal to `timeoutSeconds`
+- S-009-3: A relative first element of `command` runs as `<installRoot>/current/<element>`, and an absolute one runs unchanged
+- S-009-4: A command check that exits 0 lets `deploy --yes` exit 0 with a receipt carrying `healthCheckPassed: true` and `healthCheck` with the command, `attempts: 1`, `exitCode: 0` and `timedOut: false`
+- S-009-5: A command check that exits 3 on every attempt makes `deploy --yes` exit `ENGINE_STEP_FAILED` with an `error.message` naming the command health check and exit code 3; afterwards `current` points at the previous release, the service was restarted on it, and the receipt at `<releases>/<component>.json` validates against the schema with `healthCheckPassed: false`, `healthCheck.exitCode: 3`, `release` the failed release and `previousRelease` the restored one
+- S-009-6: The same failure on a first release stops the service and writes a receipt with `healthCheckPassed: false` and `previousRelease: null`
+- S-009-7: A run still going after `timeoutSeconds` is stopped and counts as a failed attempt; when the last attempt timed out, the deploy fails the same way, the `error.message` names the timeout, and the receipt has `timedOut: true` and `exitCode: null`
+- S-009-8: A check that fails twice and then passes with `attempts: 3` passes and records `attempts: 3`; a check that always fails runs exactly `attempts` times, 3 seconds apart
+- S-009-9: A check that prints a marker string on stdout and stderr and exits 1 leaves the marker in neither the receipt, the `ansible-playbook` output nor the `deploy` error, and the receipt schema refuses a `healthCheck` with any key besides `type`, `command`, `attempts`, `exitCode` and `timedOut`
+- S-009-10: `restart`, `rollback` and `health` of a component with a command check run the same check; `health` reports `unhealthy` when it exits non-zero, and none of them writes a receipt
+- S-009-11: The receipt schema refuses `healthCheckPassed: false` without a command `healthCheck`, or with `exitCode: 0`, and still accepts every receipt the deploy writes today
+- S-009-12: Components with `type: http` and `type: none` run the same gate tasks as before, their receipts have no `healthCheck` key, and a failed `http` or `none` gate writes no receipt
+- S-009-13: `deploy`, `restart` and `rollback` without `--yes` exit 0, name the command health check and its command in `effect` for a component with one, and run no `ansible-playbook`
+
+## Out of scope
+
+- A failed receipt for `http` and `none` gates: kept as today so existing components and their receipts don't change; a later spec can make every failed gate leave one
+- Recording any of the command's output, even a capped or redacted tail: it may hold the secrets it authenticates with. The output is in the server's journal
+- A configurable delay between attempts: fixed at 3 seconds like `http`
+- A `workingDirectory` for the check: it runs in `current`, as the service does; a command can change directory itself
+- Combining checks (an `http` check and a command check together, or several commands): one `healthCheck` per component, as today
+- Running the check from the controller or another server: it runs on the server it gates
+- A history of receipts: `<releases>/<component>.json` keeps only the last deploy, as today; `docs/ci-deploy-guide.md` already shows how to keep each one
+
+## Decisions relied on
+
+- none
+
+## Issues
+
+## Verification