Refuse a data copy the target can't hold before dumping it (S-007-6..11)

changelog.d/125-data-copy-preflight.md

@@ -0,0 +1,3 @@
+### Added
+
+- `cloudfall data copy`, and the `data:<database>` step of `cloudfall migrate`, run a preflight on the target host after the empty-target check and before `pg_dump`: it reads both catalogs and refuses a copy the target can't hold, with nothing dumped, the target untouched and no receipt, exiting `PRECONDITION` with code `data_copy_preflight_refused` and four lists in the error context, each item named: `missingSchemas` (a named schema the source lacks), `extensions` (one the selected schemas need, through `pg_depend` or by living in them, or every source extension but `plpgsql` without `--source-schema`, that the target's `pg_available_extensions` lacks), `roles` (a role a row-level security policy on a selected table names that the target's `pg_roles` lacks) and `authReferences` (a function or policy of the selected schemas that references `auth.` when `auth` isn't copied). The role writes the findings to `<service>-<database>.preflight.json` under `--data-migrations`, so the lists arrive whole. With `--source-schema`, `pg_dump` also gets one `--extension` per extension the schemas need (pg_dump 14 or later), and `pg_restore` now runs with `--single-transaction`, so a failed restore rolls back whole, writes no receipt, exits `ENGINE_STEP_FAILED`, and the next run passes the empty-target check (#125)

changelog.d/126-source-schema-checks.md

@@ -0,0 +1,3 @@
+### Added
+
+- `cloudfall migrate` takes a repeatable `--data-schema DATABASE=NAME` beside `--data DATABASE=PATH` (and the `migrate` MCP tool a `data_schema` list): the `data:<database>` step copies only the named schemas, as `data copy --source-schema` does, and the plan names them. A `--source-schema` or `--data-schema` name must now match `^[a-z_][a-z0-9_]*$` and be at most 63 characters, and may be given once per copy; a name that doesn't, a repeated name, or a `--data-schema` for a database with no `--data` exits `PRECONDITION` (code `data_copy_schema_invalid` or `migrate_data_schema_without_data`) before any playbook runs, where a malformed name was a usage error and a repeat was silently dropped (#126)

changelog.d/159-domain-routes-validation.md

@@ -0,0 +1,3 @@
+### Added
+
+- A Domain can declare `spec.routes`, path routes that each either proxy to an `upstream` (`address`, `port`) or serve a component's release files (`static`: `component`, `directory`, optional `spaFallback`). `cloudfall config validate` refuses a Domain with both `proxy.upstream` and `routes` (`domain_route_source_invalid`), a path that doesn't start with `/` or uses characters outside letters, digits and `._~/-`, or a static path without a trailing `/` (`domain_route_path_invalid`), a path declared twice (`domain_route_path_duplicate`), a path under `/.well-known/acme-challenge` (`domain_route_path_reserved`), a missing static component (`resource_reference_missing`) or one not deployed on the Domain's `proxy.server` (`domain_route_component_not_on_proxy`), and a `directory` or `spaFallback` that is absolute, climbs out with `..`, or, for `directory`, names the release root. Validation only: the domains playbook doesn't render routes yet and still refuses a Domain without `proxy.upstream` before any change. Domains with only `proxy.upstream` validate and render as before (#159)

changelog.d/161-command-health-check-schema.md

@@ -0,0 +1,3 @@
+### Added
+
+- Components: a `healthCheck` of `type: command` (a required non-empty `command` argv, `timeoutSeconds` 1 to 300 and `attempts` 1 to 20, no other key) validates, and `deploy`, `restart` and `rollback` without `--yes` name the check and its command in their plan's `wouldRun`. The engine does not run the check yet: until it does, `deploy`, `restart`, `rollback` and `health` of such a component fail on their first task, before anything changes on the server, instead of passing an unrun gate (#161)

config/schemas/v1/component.schema.json

@@ -218,6 +218,40 @@
                 "type"
               ],
               "additionalProperties": false
+            },
+            {
+              "type": "object",
+              "properties": {
+                "type": {
+                  "const": "command"
+                },
+                "command": {
+                  "type": "array",
+                  "items": {
+                    "type": "string",
+                    "minLength": 1,
+                    "pattern": "^[^\\u0000]+$"
+                  },
+                  "minItems": 1
+                },
+                "timeoutSeconds": {
+                  "type": "integer",
+                  "minimum": 1,
+                  "maximum": 300
+                },
+                "attempts": {
+                  "type": "integer",
+                  "minimum": 1,
+                  "maximum": 20
+                }
+              },
+              "required": [
+                "type",
+                "command",
+                "timeoutSeconds",
+                "attempts"
+              ],
+              "additionalProperties": false
             }
           ]
         },

config/schemas/v1/domain.schema.json

@@ -3,6 +3,90 @@
   "$id": "https://cloudfall.internal/schemas/v1/domain.schema.json",
   "title": "Cloudfall public domain route",
   "x-cloudfall-kind": "Domain",
+  "$defs": {
+    "upstream": {
+      "type": "object",
+      "properties": {
+        "address": {
+          "type": "string",
+          "anyOf": [
+            {
+              "format": "ipv4"
+            },
+            {
+              "format": "ipv6"
+            }
+          ]
+        },
+        "port": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 65535
+        }
+      },
+      "required": [
+        "address",
+        "port"
+      ],
+      "additionalProperties": false
+    },
+    "releasePath": {
+      "type": "string",
+      "pattern": "^(?!/)(?!(?:.*/)?\\.\\.(?:/|$))[A-Za-z0-9._/-]+$",
+      "not": {
+        "pattern": "[\\u0000-\\u001f\\u007f]"
+      }
+    },
+    "route": {
+      "type": "object",
+      "properties": {
+        "path": {
+          "type": "string"
+        },
+        "upstream": {
+          "$ref": "#/$defs/upstream"
+        },
+        "static": {
+          "type": "object",
+          "properties": {
+            "component": {
+              "$ref": "common.schema.json#/$defs/resourceId"
+            },
+            "directory": {
+              "$ref": "#/$defs/releasePath",
+              "not": {
+                "pattern": "^\\.(?:/+\\.)*/*$"
+              }
+            },
+            "spaFallback": {
+              "$ref": "#/$defs/releasePath"
+            }
+          },
+          "required": [
+            "component",
+            "directory"
+          ],
+          "additionalProperties": false
+        }
+      },
+      "required": [
+        "path"
+      ],
+      "oneOf": [
+        {
+          "required": [
+            "upstream"
+          ]
+        },
+        {
+          "required": [
+            "static"
+          ]
+        }
+      ],
+      "additionalProperties": false
+    }
+  },
   "type": "object",
   "properties": {
     "apiVersion": {
@@ -42,30 +126,7 @@
               "$ref": "common.schema.json#/$defs/serviceName"
             },
             "upstream": {
-              "type": "object",
-              "properties": {
-                "address": {
-                  "type": "string",
-                  "anyOf": [
-                    {
-                      "format": "ipv4"
-                    },
-                    {
-                      "format": "ipv6"
-                    }
-                  ]
-                },
-                "port": {
-                  "type": "integer",
-                  "minimum": 1,
-                  "maximum": 65535
-                }
-              },
-              "required": [
-                "address",
-                "port"
-              ],
-              "additionalProperties": false
+              "$ref": "#/$defs/upstream"
             }
           },
           "required": [
@@ -75,6 +136,13 @@
           ],
           "additionalProperties": false
         },
+        "routes": {
+          "type": "array",
+          "items": {
+            "$ref": "#/$defs/route"
+          },
+          "minItems": 1
+        },
         "origin": {
           "type": "object",
           "properties": {

docs/neon-migration-guide.md

@@ -175,13 +175,27 @@ the peer-authenticated socket as the declared owner, verifies per-table
 row counts match the source exactly, removes the staged URL file and the
 dump, and writes a `DataMigrationReceipt` with per-table counts. It
 refuses a target database that already has tables in the copied schemas,
-so a re-run requires deliberately dropping what was restored.
+so a re-run after a finished copy requires deliberately dropping what was
+restored. A restore that fails runs in one transaction and rolls back
+whole, so the next run needs no cleanup.
+
+Before it dumps anything, a preflight reads both catalogs and refuses a
+copy the target can't hold, with `PRECONDITION` and the code
+`data_copy_preflight_refused`, naming every finding at once:
+`missingSchemas` (a named schema the source lacks), `extensions` (one the
+copied schemas need that the target's `pg_available_extensions` lacks),
+`roles` (a role a row-level security policy names that the target lacks),
+and `authReferences` (a function or policy that calls `auth.` when `auth`
+isn't copied). Provide what it names, or rewrite it, and run again.
 
 By default the copy takes every non-system schema of the source, and the
 row-count check and the receipt cover all of them. To take only some,
 name each with a repeatable `--source-schema`, such as
 `--source-schema public`; the plan names the schemas it will copy, and the
-receipt lists them under `spec.schemas`, each table with its `schema`.
+receipt lists them under `spec.schemas`, each table with its `schema`. A
+name must match `^[a-z_][a-z0-9_]*$`, be at most 63 characters, and be
+named once; anything else stops the command with `PRECONDITION` before
+any playbook runs.
 
 The row-count check is the verify step and the receipt is the record's
 entry for the migration. The restore is not done because `pg_restore`
@@ -193,7 +207,8 @@ refusal guards, and cannot skip either.
 One database at a time; repeat per declared database. If the application
 is also moving in the same operation, `cloudfall migrate --data
 app=tmp/neon-app-url` inserts this step into the orchestrated plan
-between services and deploys.
+between services and deploys; `--data-schema app=public` (repeatable)
+narrows it to the named schemas, as `--source-schema` does.
 
 Note the one-time egress: Neon meters the dump like any other transfer,
 so a 14 GB dump costs about a dollar — the last transfer bill this

docs/render-migration-guide.md

@@ -147,13 +147,18 @@ uv run cloudfall migrate --build acme-api=main --data acme=~/.render/acme-db.url
 ```
 
 Without `--yes` the command shows the plan. The engine dumps on the target
-host, restores over the peer-authenticated socket, refuses a target
-database that already has tables in the copied schemas, verifies per-table
+host, restores over the peer-authenticated socket in one transaction,
+refuses a target database that already has tables in the copied schemas,
+refuses before the dump a copy whose schemas, extensions, policy roles,
+or `auth.` references the target can't hold (`data_copy_preflight_refused`,
+each one named), verifies per-table
 row counts, and writes a `DataMigrationReceipt`. The row-count check is the
 verify step: the migration is not done because the restore exited zero, it
 is done because the counts match. The copy takes every non-system schema of
-the source; `data copy --source-schema NAME` (repeatable) takes only the
-named ones.
+the source; `data copy --source-schema NAME` and `migrate --data-schema
+DATABASE=NAME` (both repeatable) take only the named ones. A name must
+match `^[a-z_][a-z0-9_]*$` and be named once; anything else stops the
+command with `PRECONDITION` before any playbook runs.
 
 ## Step 7: cut over DNS
 

docs/specs/007-data-copy-schemas.md

@@ -1,6 +1,6 @@
 # S-007: Data copy schemas
 
-status: approved
+status: built
 serves: G-3
 
 ## Problem
@@ -134,3 +134,20 @@ copy.
 - cloudfall-dev/cloudfall#126: S-007-12, S-007-13
 
 ## Verification
+
+No criterion has run against a live PostgreSQL yet: the role's tasks and queries were rendered with Ansible's own templating and psql's variable quoting replayed, the queries were parsed with PostgreSQL's parser (libpg_query, through pglast) but not executed, and the SDK side ran with a stand-in for the playbook that writes what the role writes. What a real `pg_dump --schema --extension`, the catalog joins of the preflight, and a rolled-back `pg_restore --single-transaction` do on a server is unexercised.
+
+- S-007-1: `test_s007_1_named_schemas_reach_pg_dump_and_the_receipt`, `test_s007_1_a_schema_name_stays_one_quoted_pg_dump_argument` and `test_s007_1_data_migration_plan_passes_the_named_schemas_sorted`, the dump got one quoted `--schema` per name and the rendered receipt `spec.schemas` was `["app", "public"]`
+- S-007-2: `test_s007_2_without_schema_the_dump_is_the_whole_database_as_in_0_7_0`, `test_s007_2_source_schemas_query_skips_only_the_system_schemas` and `test_s007_2_data_migration_plan_without_schemas_is_unchanged`, the dump command and extra vars were 0.7.0's and `spec.schemas` was `["app", "audit", "public"]`
+- S-007-3: `test_s007_3_receipt_tables_carry_schema_table_and_rows`, every rendered entry had `schema`, `table` and `rows`, a quoted name with a comma included
+- S-007-4: `test_s007_4_row_count_query_reads_only_the_selected_schemas` and `test_s007_4_both_sides_pass_the_schemas_to_the_row_count_query`, the schemas reached the query only as the psql variable, ordered by schema and table
+- S-007-5: `test_s007_5_empty_target_check_counts_only_the_selected_schemas`, the count read only the selected schemas and a count of 3 failed naming "already contains 3 tables"
+- S-007-6: `test_s007_6_a_missing_schema_is_refused_before_the_dump`, `test_s007_6_the_source_preflight_takes_names_only_as_psql_variables` and `test_s007_6_a_missing_schema_is_refused_with_its_name`, the refusal comes before `pg_dump` and answered `data_copy_preflight_refused` with `missingSchemas` `["missing"]`
+- S-007-7: `test_s007_7_an_extension_the_target_lacks_is_refused`, `test_s007_7_named_schemas_dump_their_needed_extensions` and `test_s007_7_an_extension_the_target_lacks_is_named`, the target query read `pg_available_extensions` from a psql variable, `extensions` named `citext`, no receipt was written
+- S-007-8: `test_s007_8_policy_roles_the_target_lacks_are_refused` and `test_s007_8_policy_roles_the_target_lacks_are_named`, `roles` was `["anon", "authenticated"]`
+- S-007-9: `test_s007_9_auth_references_name_the_function_and_the_policy` and `test_s007_9_auth_references_are_named`, both `function public.current_tenant()` and `policy notes_owner on public.notes` were named and the copy refused
+- S-007-10: `test_s007_10_every_finding_is_named_in_one_answer`, `test_s007_10_every_finding_comes_whole_in_one_answer` (400 role names, past the 2000-character output tail) and `test_s007_10_the_migrate_data_step_surfaces_the_same_refusal` (`migrate` answers `PRECONDITION` with the same lists)
+- S-007-11: `test_s007_11_the_restore_is_one_transaction_and_the_receipt_follows_it`, `test_s007_11_a_failed_restore_is_an_engine_failure_without_a_receipt`, `test_s007_11_a_stale_findings_file_never_answers_for_a_new_run` and `test_s007_11_a_failed_migrate_data_step_stays_an_engine_failure`, `pg_restore` runs `--single-transaction`, the receipt task follows the block, a failure answered `ENGINE_STEP_FAILED` and the next run went ahead
+- S-007-12: `test_s007_12_data_copy_refuses_a_malformed_or_repeated_schema`, `test_s007_12_cli_migrate_refuses_a_bad_data_schema_before_any_playbook`, `test_s007_12_migrate_refuses_a_data_schema_without_its_data` and `test_s007_12_the_data_tools_refuse_a_bad_schema_as_the_cli_does` (over MCP), each exited `PRECONDITION` with no inventory render and no plan file; a value holding a newline is refused earlier by the CLI framework as `ARG_ERROR`
+- S-007-13: `test_s007_13_migrate_copies_only_the_data_schema_as_data_copy_does` and `test_s007_13_migrate_groups_each_data_schema_under_its_database`, the `data:crm` step's playbook argv equalled `data copy --source-schema public`'s
+- S-007-14: `test_s007_14_data_copy_plan_names_the_selected_schemas`, the plan said "dump schemas app, public" and, without a name, "dump every non-system schema"