Gate a deploy on the command health check and receipt it (S-009-2..9, 11, 12)

changelog.d/164-command-health-gate.md

@@ -0,0 +1,3 @@
+### Added
+
+- Deploy: a component's `healthCheck` of `type: command` now gates `deploy --yes`. After the switch and the restart it runs on each server in a transient systemd unit `cloudfall-health-<component>`, as the application's user, in `<installRoot>/current`, with the service's env file and `timeoutSeconds` as its runtime limit, up to `attempts` times 3 seconds apart. When every attempt fails or times out, the previous release is restored (a first release is stopped), the deploy fails with `ENGINE_STEP_FAILED` naming the check and its exit code or timeout, and the release receipt is written with `healthCheckPassed: false`. Receipts of a command check carry `healthCheck` (command, attempts, exit code, timed out), never the command's output, which stays in the server's journal. The release receipt schema accepts `healthCheckPassed: false` only with a failed command `healthCheck`, and every receipt written before. `restart`, `rollback` and `health` still refuse a command check until #166 (#164)

config/README.md

@@ -44,7 +44,9 @@ and public HTTP evidence. `deployment-receipt.schema.json` validates receipts
 written only after an Ansible proxy run has applied and validated its final
 configuration. `artifact.schema.json` validates build metadata for hashed
 release tarballs, and `release-receipt.schema.json` validates receipts written
-only after a component release has passed its health gate. Together with server observations, these inputs derive planned,
+after a component release has passed its health gate, or failed a command
+health check (`healthCheckPassed: false`, with the check's attempts, exit code
+and timeout under `healthCheck`, never its output). Together with server observations, these inputs derive planned,
 ready-to-deploy, deployed, configured, and healthy service states.
 
 `examples/` contains a complete valid desired-state set. `tests/invalid/`

config/schemas/v1/release-receipt.schema.json

@@ -41,7 +41,68 @@
           "pattern": "^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{7,40}$"
         },
         "healthCheckPassed": {
-          "const": true
+          "type": "boolean"
+        },
+        "healthCheck": {
+          "type": "object",
+          "properties": {
+            "type": {
+              "const": "command"
+            },
+            "command": {
+              "type": "array",
+              "items": {
+                "type": "string",
+                "minLength": 1
+              },
+              "minItems": 1
+            },
+            "attempts": {
+              "type": "integer",
+              "minimum": 1,
+              "maximum": 20
+            },
+            "exitCode": {
+              "type": [
+                "integer",
+                "null"
+              ],
+              "minimum": 0,
+              "maximum": 255
+            },
+            "timedOut": {
+              "type": "boolean"
+            }
+          },
+          "required": [
+            "type",
+            "command",
+            "attempts",
+            "exitCode",
+            "timedOut"
+          ],
+          "additionalProperties": false,
+          "if": {
+            "properties": {
+              "timedOut": {
+                "const": true
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "exitCode": {
+                "type": "null"
+              }
+            }
+          },
+          "else": {
+            "properties": {
+              "exitCode": {
+                "type": "integer"
+              }
+            }
+          }
         },
         "releaseCommand": {
           "type": "object",
@@ -91,7 +152,47 @@
         "previousRelease",
         "healthCheckPassed"
       ],
-      "additionalProperties": false
+      "additionalProperties": false,
+      "if": {
+        "properties": {
+          "healthCheckPassed": {
+            "const": false
+          }
+        },
+        "required": [
+          "healthCheckPassed"
+        ]
+      },
+      "then": {
+        "properties": {
+          "healthCheck": {
+            "properties": {
+              "exitCode": {
+                "not": {
+                  "const": 0
+                }
+              }
+            }
+          }
+        },
+        "required": [
+          "healthCheck"
+        ]
+      },
+      "else": {
+        "properties": {
+          "healthCheck": {
+            "properties": {
+              "exitCode": {
+                "const": 0
+              },
+              "timedOut": {
+                "const": false
+              }
+            }
+          }
+        }
+      }
     }
   },
   "required": [

docs/ci-deploy-guide.md

@@ -264,7 +264,11 @@ What each step does:
 - **Show the plan.** `deploy` without `--yes` validates the release and
   prints which servers it would change, and changes nothing
 - **Deploy.** Ships the release behind the component's health check, and
-  writes the release receipt `tmp/releases/<component>.json`
+  writes the release receipt `tmp/releases/<component>.json`. A smoke check
+  declared as a `type: command` health check runs on each server after its
+  restart, so it gates the deploy and needs no CI step of its own: when it
+  fails, the server goes back to the previous release, the deploy fails, and
+  the receipt records the failure with `healthCheckPassed: false`
 - **Remove the decrypted files.** GitHub-hosted runners are thrown away
   after the job; on a self-hosted runner this step is what keeps the values
   off its disk
@@ -346,7 +350,9 @@ step keeps, for 90 days:
 
 - `tmp/artifacts/<component>/<release>.json`: the artifact metadata, with the
   commit and the archive's SHA-256
-- `tmp/releases/<component>.json`: the release receipt from `deploy`
+- `tmp/releases/<component>.json`: the release receipt from `deploy`; after
+  a failed command health check its `previousRelease` names the release that
+  serves again (`null` when a first release was stopped)
 - `tmp/env-receipts/<component>.json`: the key names and hash of the
   rendered environment file, never the values
 

engine/README.md

@@ -244,7 +244,26 @@ running. The kept copy stays after a successful deploy and is never
 overwritten; a later hand-made unit that differs from it refuses the deploy.
 Check mode only reports the unit it would keep. Successful runs write a
 schema-valid release receipt on the
-controller. The first slice supports Python components managed by uv on
+controller.
+
+A `healthCheck` of `type: command` (a smoke check, say) is the gate's last
+step: after the switch and the restart, the command runs in a transient
+systemd unit, `cloudfall-health-<component>`, as the application's user, in
+`<installRoot>/current` (a relative first element of `command` resolves
+there), with the service unit's `EnvironmentFile=-` line, and is stopped
+after `timeoutSeconds`. It runs up to `attempts` times, 3 seconds apart, and
+passes at the first exit 0. Its stdout and stderr stay in the server's
+journal (`journalctl -u cloudfall-health-<component>`) and never reach the
+receipt or the Ansible output. When every attempt fails, the rescue above
+restores the previous release (or stops a first one), writes the receipt with
+`healthCheckPassed: false`, the last exit code or the timeout, and
+`previousRelease` naming the release that serves again, and fails the run.
+The receipt of a passing command check records it under `healthCheck` too;
+`http` and `none` checks keep their receipts and their gate as before.
+`restart`, `rollback` and `health` still refuse a command check until
+cloudfall-dev/cloudfall#166 runs it there.
+
+The first slice supports Python components managed by uv on
 x86_64 Debian hosts.
 
 ## Public domain routes

engine/ansible/roles/cloudfall_deploy/defaults/main.yml

@@ -4,7 +4,8 @@ cloudfall_deploy_receipt_directory: ""
 # A component without an HTTP health check passes its gate when its unit is
 # active right after the restart and still active this many seconds later.
 cloudfall_deploy_unit_settle_seconds: 10
-# A release command runs in a transient unit started by this binary.
+# A release command and a command health check run in a transient unit
+# started by this binary.
 cloudfall_deploy_systemd_run: /usr/bin/systemd-run
 # The receipt keeps this much of the release command's output; a failure
 # message keeps less, so it stays inside the end of the Ansible log the

engine/ansible/roles/cloudfall_deploy/tasks/command_health_check.yml

@@ -0,0 +1,116 @@
+---
+# Runs the component's command health check against the release current
+# points at, in a transient systemd unit with the service's user, working
+# directory and EnvironmentFile=, up to `attempts` times 3 seconds apart.
+# Paths come from install_root, so every entry point can include this file.
+#
+# The command's stdout and stderr stay in the server's journal under the
+# transient unit's name (no --pipe), and the run task is no_log: only its
+# exit code and systemd-run's own timeout line are read. The result lands in
+# cloudfall_deploy_command_health_result, then a failed check fails the task
+# that follows, so a block's rescue can still read what happened.
+- name: Compute the command health check invocation
+  ansible.builtin.set_fact:
+    cloudfall_deploy_command_health_current: >-
+      {{ cloudfall_deploy_component.install_root }}/current
+    cloudfall_deploy_command_health_executable: >-
+      {{
+        cloudfall_deploy_component.healthCheck.command[0]
+        if cloudfall_deploy_component.healthCheck.command[0].startswith('/')
+        else cloudfall_deploy_component.install_root ~ '/current/'
+        ~ cloudfall_deploy_component.healthCheck.command[0]
+      }}
+    cloudfall_deploy_command_health_attempts: >-
+      {{ cloudfall_deploy_component.healthCheck.attempts | int }}
+
+- name: Run the command health check
+  ansible.builtin.command:
+    argv: >-
+      {{
+        [
+          cloudfall_deploy_systemd_run,
+          '--unit=cloudfall-health-' ~ cloudfall_deploy_component.id,
+          '--wait',
+          '--collect',
+          '--service-type=exec',
+          '--uid=' ~ cloudfall_deploy_component.application_user,
+          '--gid=' ~ cloudfall_deploy_component.application_user,
+          '--working-directory=' ~ cloudfall_deploy_command_health_current,
+          '--property=EnvironmentFile=-'
+          ~ cloudfall_deploy_component.install_root ~ '/shared/env/'
+          ~ cloudfall_deploy_component.id ~ '.env',
+          '--property=RuntimeMaxSec='
+          ~ cloudfall_deploy_component.healthCheck.timeoutSeconds,
+          '--property=TimeoutStopSec=10',
+          '--property=NoNewPrivileges=true',
+          '--property=PrivateTmp=true',
+          '--',
+          cloudfall_deploy_command_health_executable,
+        ]
+        + cloudfall_deploy_component.healthCheck.command[1:]
+      }}
+  register: cloudfall_deploy_command_health_run
+  changed_when: false
+  failed_when: false
+  # The last attempt ends the loop itself, so running out of attempts is not
+  # a task failure and the result below is still recorded.
+  retries: "{{ cloudfall_deploy_command_health_attempts | int - 1 }}"
+  delay: 3
+  until: >-
+    (
+      cloudfall_deploy_command_health_run.rc == 0
+      and 'Finished with result: timeout'
+      not in cloudfall_deploy_command_health_run.stderr
+    )
+    or cloudfall_deploy_command_health_run.attempts | default(1) | int
+    >= cloudfall_deploy_command_health_attempts | int
+  no_log: true
+
+- name: Record the command health check result
+  ansible.builtin.set_fact:
+    cloudfall_deploy_command_health_result: >-
+      {{
+        {
+          'type': 'command',
+          'command': cloudfall_deploy_component.healthCheck.command,
+          'attempts': cloudfall_deploy_command_health_run.attempts
+          | default(1) | int,
+          'exitCode': (
+            none
+            if cloudfall_deploy_command_health_timed_out
+            else cloudfall_deploy_command_health_run.rc | int
+          ),
+          'timedOut': cloudfall_deploy_command_health_timed_out,
+        }
+      }}
+  vars:
+    cloudfall_deploy_command_health_timed_out: >-
+      {{
+        'Finished with result: timeout'
+        in cloudfall_deploy_command_health_run.stderr
+      }}
+
+- name: Fail the command health check
+  ansible.builtin.fail:
+    msg: >-
+      The command health check of {{ cloudfall_deploy_component.id }} failed
+      on {{ cloudfall_server_id }}: {{
+        cloudfall_deploy_command_health_result.command | join(' ')
+      }} {{
+        'timed out after '
+        ~ cloudfall_deploy_component.healthCheck.timeoutSeconds
+        ~ ' seconds and was stopped'
+        if cloudfall_deploy_command_health_result.timedOut
+        else 'exited with code '
+        ~ cloudfall_deploy_command_health_result.exitCode
+      }} {{
+        'on its only attempt'
+        if cloudfall_deploy_command_health_result.attempts == 1
+        else 'on the last of '
+        ~ cloudfall_deploy_command_health_result.attempts ~ ' attempts'
+      }}; its output is in journalctl -u cloudfall-health-{{
+        cloudfall_deploy_component.id
+      }} on that server
+  when: >-
+    cloudfall_deploy_command_health_result.timedOut
+    or cloudfall_deploy_command_health_result.exitCode != 0

engine/ansible/roles/cloudfall_deploy/tasks/health.yml

@@ -1,12 +1,12 @@
 ---
-# Until cloudfall-dev/cloudfall#164 runs it, a command health check fails here
+# Until cloudfall-dev/cloudfall#166 runs it, a command health check fails here
 # rather than reporting healthy without having run.
 - name: Refuse a command health check this engine does not run yet
   ansible.builtin.fail:
     msg: >-
       Component {{ cloudfall_deploy_component.id }} declares a command health
       check, which this engine does not run yet
-      (cloudfall-dev/cloudfall#164); nothing was changed
+      (cloudfall-dev/cloudfall#166); nothing was changed
   when: cloudfall_deploy_component.healthCheck.type == "command"
 
 - name: Probe the component's declared health check