Run Component build steps in artifact build (S-002)

changelog.d/115-component-build-steps.md

@@ -0,0 +1,3 @@
+### Added
+
+- `cloudfall-engine artifact build` runs a Component's declared `spec.build` steps on the controller before packaging: it checks every `spec.buildTools` entry is on `PATH` at its minimum version before cloning (`artifact_build_tool_missing`, `artifact_build_tool_unsupported`), runs each step in the clean checkout with only `PATH`, `HOME`, `LANG`, `TMPDIR` and `CI=true` in its environment and its output on stderr, fails without writing an archive on a non-zero exit or a step outrunning its own `timeoutSeconds` (`artifact_build_step_failed`), a missing include path (`artifact_include_missing`), an include path that leads out of the checkout through a symlink (`artifact_include_outside_checkout`) or a tracked file a step deleted (`artifact_tracked_file_missing`), packages the tracked files plus every `spec.artifact.include` path (a gitignored `web/dist`, say), and records the steps and tool versions as the new optional `spec.build` of the Artifact metadata. The command's time limit is now the clone time (1800 seconds) plus every declared step timeout, on the CLI and the SDK/MCP path alike; an explicit `--timeout` replaces it, and `--timeout 0` still disables it. A Component without `build` is packaged as before (#115)

config/schemas/v1/artifact.schema.json

@@ -46,6 +46,50 @@
         "sizeBytes": {
           "type": "integer",
           "minimum": 1
+        },
+        "build": {
+          "type": "object",
+          "properties": {
+            "steps": {
+              "type": "array",
+              "minItems": 1,
+              "items": {
+                "type": "object",
+                "properties": {
+                  "workingDirectory": {
+                    "type": "string",
+                    "minLength": 1
+                  },
+                  "command": {
+                    "type": "array",
+                    "minItems": 1,
+                    "items": {
+                      "type": "string",
+                      "minLength": 1
+                    }
+                  }
+                },
+                "required": [
+                  "workingDirectory",
+                  "command"
+                ],
+                "additionalProperties": false
+              }
+            },
+            "tools": {
+              "type": "object",
+              "minProperties": 1,
+              "additionalProperties": {
+                "type": "string",
+                "pattern": "^[0-9]{1,9}(?:\\.[0-9]{1,9}){0,7}$"
+              }
+            }
+          },
+          "required": [
+            "steps",
+            "tools"
+          ],
+          "additionalProperties": false
         }
       },
       "required": [

docs/specs/002-component-build-steps.md

@@ -1,6 +1,6 @@
 # S-002: Component build steps
 
-status: approved
+status: built
 
 ## Problem
 
@@ -139,3 +139,19 @@ validate`, and the `build_artifact` MCP tool.
 - cloudfall-dev/cloudfall#115: S-002-2, S-002-3, S-002-4, S-002-5, S-002-6, S-002-7, S-002-8, S-002-9, S-002-10, S-002-12, S-002-13
 
 ## Verification
+
+- S-002-1: `test_s002_1_component_with_build_steps_validates`, `test_s002_1_existing_component_without_build_is_unchanged` and `test_s002_1_build_step_timeout_accepts_its_bounds` (`sdk/tests/test_component_build_validation.py`) passed, and `cloudfall config validate --project config/examples` still exits 0
+- S-002-2: `test_s002_2_the_tarball_holds_the_gitignored_build_output` runs `cloudfall-engine artifact build --ref <sha>` as a child process with stand-in `node` and `pnpm` on a temporary `PATH`; the tarball held `./web/dist/index.html` although the fixture's `.gitignore` lists `web/dist/`
+- S-002-3: `test_s002_3_the_tarball_holds_every_tracked_file_and_no_node_modules` compared the tarball against `git ls-tree -r` at the commit: every tracked file, no `node_modules` entry, no `.git`. A step that deletes a tracked file would break this, so the build refuses it: `test_s002_3_a_step_that_deletes_a_tracked_file_writes_no_artifact` exited 81 with the code `artifact_tracked_file_missing` naming `web/package.json`, and wrote nothing
+- S-002-4: `test_s002_4_the_metadata_records_the_steps_and_tool_versions` read `spec.build.steps` (both steps in order) and `spec.build.tools` `{"node": "22.3.0", "pnpm": "9.1.0"}` from the versions the stand-ins printed (`v22.3.0`, `9.1.0`), and validated the file against `artifact.schema.json`
+- S-002-5: `test_s002_5_a_missing_tool_fails_before_any_clone` exited 81 with `artifact_build_tool_missing` and `tool: node`; the logging `git` wrapper on `PATH` recorded no `clone`, and no file was written
+- S-002-6: `test_s002_6_an_old_tool_names_the_tool_and_both_versions` (`node` printing `v20.11.1` against `22.0.0`) and `test_s002_6_a_tool_that_prints_no_version_is_unsupported` exited 81 with `artifact_build_tool_unsupported`, naming the tool and both versions
+- S-002-7: `test_s002_7_a_failing_step_names_itself_and_writes_nothing` exited 81 with `artifact_build_step_failed`, `step: 1`, `argv: ["pnpm", "fail"]`, `exitCode: 3` and the step's stderr tail; `test_s002_7_a_step_past_its_timeout_fails_the_build` showed a step's own 2-second `timeoutSeconds` firing as `artifact_build_step_failed`; neither left a `.tar.gz` or `.json` in the output directory. The command's limit is the clone time (1800 s) plus every declared step timeout, on the CLI and on the SDK/MCP path, whose engine call carries no 3600 s cap of its own: `test_s002_7_the_run_limit_covers_every_declared_step` (three 3600 s steps), `test_s002_7_the_sdk_leaves_the_build_limit_to_the_engine`, `test_s002_7_an_explicit_timeout_still_wins` (`--timeout 3` ends in `TIMEOUT`), `test_s002_7_without_timeout_the_computed_limit_applies`, and `test_s002_7_timeout_zero_still_means_no_limit` (`--timeout 0` applies no limit, as before). The command takes `--timeout` as a field of its own, which treaty allows in place of its framework flag, so the build can tell an omitted flag apart from 0
+- S-002-8: `test_s002_8_a_missing_include_path_writes_no_artifact` exited 81 with `artifact_include_missing` naming `web/dist` and wrote nothing; `test_s002_8_an_include_through_a_symlink_out_of_the_checkout_is_refused` exited 81 with `artifact_include_outside_checkout` for an include path under a tracked symlink to a directory outside the checkout, and wrote nothing
+- S-002-9: `test_s002_9_step_stdout_stays_out_of_the_json_envelope` and `test_s002_9_stdout_is_exactly_one_envelope` parsed the command's whole stdout as one JSON envelope, on success and on a failed step; the steps' stdout lines were on stderr
+- S-002-10: `test_s002_10_a_step_sees_ci_and_none_of_the_controller_secrets` set `HCLOUD_TOKEN` and `SOPS_AGE_KEY` for the command; the environment the build step dumped into the archive had `CI=true`, the controller's `PATH`, `HOME` and `LANG`, and neither secret
+- S-002-11: the `test_s002_11_*` tests in `sdk/tests/test_component_build_validation.py` passed (undeclared tool, absolute and `..` paths, `buildTools` or `artifact` without `build`, malformed steps)
+- S-002-12: `test_s002_12_deploy_and_rollback_carry_the_built_files` builds two releases, runs the `cloudfall_deploy` role's own path, digest, unpack and switch tasks and then `rollback.yml`'s switch tasks under `ansible-playbook` against a directory on the controller (ownership dropped, no systemd); `releases/<release>/web/dist/index.html` held each build's page and `current/web/dist/` followed the deploy and the rollback. It needs GNU tar for `unarchive`, so it runs on CI's Linux runner; on macOS the same flow passed with the unpack done by `tar -xzf`. Not run against a live Debian host
+- S-002-13: `test_s002_13_a_component_without_build_packages_the_clean_tree` found the archive equal to the clean checkout's tracked files and directories, no `git ls-files` run and no `spec.build` key; `test_s002_13_the_built_archive_lists_entries_like_the_plain_one` checked a build adds only its include paths
+- S-002-14: `test_s002_14_reserved_runtimes_are_refused_as_config_invalid` (`sdk/tests/test_validation.py`) passed
+- S-002-15: `test_s002_15_node_web_service_is_a_gap_not_a_component` (`sdk/tests/test_importer.py`) passed

engine/README.md

@@ -169,6 +169,21 @@ and writes schema-valid artifact metadata beside it:
 task artifact:build COMPONENT=crm-backend REF=main
 ```
 
+A component that declares `spec.build` (with `spec.buildTools` and
+`spec.artifact.include`, see `docs/specs/002-component-build-steps.md`) is
+built on the controller before it is packaged. Every `buildTools` entry must
+be on `PATH` at its minimum version before anything is cloned; then each step
+runs in order in the clean checkout, with only `PATH`, `HOME`, `LANG` and
+`TMPDIR` from the controller's environment plus `CI=true`, so cloud tokens and
+SOPS keys stay out of `package.json` scripts. Step output goes to stderr. The
+archive holds the files git tracks plus the `artifact.include` paths, such as
+a gitignored `web/dist`, and the metadata records the steps and the tool
+versions under `spec.build`. A failed or timed-out step, a missing include
+path, an include path that leads out of the checkout through a symlink, or a
+tracked file a step deleted writes no archive. The command may take
+1800 seconds for the clone plus every step's `timeoutSeconds`; an explicit
+`--timeout` replaces that limit, and `--timeout 0` disables it.
+
 `ansible/playbooks/deploy.yml` deploys one built release to the component's
 declared servers, one host at a time: