[shipmill-romamo]

Summary

Builds S-007-6 to S-007-11 of docs/specs/007-data-copy-schemas.md: the data copy preflight and a restore that leaves no half-restored target. With #169 (S-007-12, S-007-13) every criterion of the spec is delivered, so this PR sets the spec to built and fills its Verification section.

• Preflight (cloudfall_data_migration, after the empty-target check and before pg_dump): one source query and one target query, each answering a JSON object line. Every name reaches SQL only as a psql variable (:'cloudfall_data_schemas', :'cloudfall_data_every_schema', :'cloudfall_data_extensions', :'cloudfall_data_policy_roles'), as Copy only the named schemas in data copy (#124) #134 does; the source side keeps the URL in $(cat <file>), and the target side passes the source's names as command argv, never in a shell
  • missingSchemas: a selected schema pg_namespace lacks
  • extensions: one installed in a selected schema, or whose member object (pg_depend deptype e) an object of a selected schema depends on (relations, types, functions, constraints, column defaults, triggers, view rules, policies), or with no --source-schema every extension but plpgsql; kept when the target's pg_available_extensions lacks it
  • roles: roles in pg_policy.polroles on a selected table that the target's pg_roles lacks
  • authReferences: when auth isn't selected, function <schema>.<name>(<args>) whose prosrc matches \mauth\. or that depends on an object in auth, and policy <name> on <schema>.<table> whose USING or WITH CHECK does
• The role writes the four lists to <service>-<database>.preflight.json under --data-migrations (the controller directory is now created before the block) and fails before the dump on any finding. migrate_data removes a stale file first, reads the file when the playbook fails, and raises data_copy_preflight_refused with the four lists in the error context; data copy exits PRECONDITION, and migrate maps the data:<database> step's refusal to PRECONDITION too (other step failures stay ENGINE_STEP_FAILED). The file is removed after reading
• With --source-schema, pg_dump gets one quoted --extension per needed extension. pg_restore runs with --single-transaction
• LifecycleError gains an optional context, merged into the error context; DataCopyPreflightFindings reads the file strictly (exactly the four keys, lists of strings, else data_copy_preflight_findings_invalid)
• The Neon and Render guides describe the preflight and the one-transaction restore

Plan

• Serves: G-3
• Files: engine/ansible/roles/cloudfall_data_migration/tasks/main.yml, engine/ansible/roles/cloudfall_data_migration/defaults/main.yml, engine/ansible/roles/cloudfall_data_migration/meta/argument_specs.yml, engine/tests/test_data_migration_role.py, sdk/src/cloudfall/lifecycle.py, sdk/src/cloudfall/app.py, sdk/tests/test_data_copy_preflight.py, docs/neon-migration-guide.md, docs/render-migration-guide.md, docs/specs/007-data-copy-schemas.md (status built and Verification), changelog.d/125-data-copy-preflight.md
• Order: failing tests first (the rendered preflight queries and tasks, the --extension and --single-transaction argv, the SDK reading the findings file into data_copy_preflight_refused for data copy and the data:<database> step of migrate), then the role's source and target catalog queries taking every name as a psql variable, the findings file under --data-migrations, the refusal before pg_dump, pg_dump --extension, pg_restore --single-transaction, then the SDK side; Behaviour as in spec 007, "Preflight before dumping" and "No half-failed restore"
• Proof: S-007-6, S-007-7, S-007-8, S-007-9, S-007-10, S-007-11

Test plan

• Engine (engine/tests/test_data_migration_role.py, Ansible's own templating): test_s007_6_* (task order: empty-target check, preflight, findings file, refusal, then pg_dump; names only as psql variables, a hostile name stays one quoted word), test_s007_7_* (target query reads pg_available_extensions from a variable; --extension per needed extension only with named schemas; a hostile extension name stays one quoted word), test_s007_8_*, test_s007_9_*, test_s007_10_* (all findings in one message and one file), test_s007_11_* (pg_restore --single-transaction, the receipt task follows the block)
• SDK (sdk/tests/test_data_copy_preflight.py, a stand-in playbook that writes the findings file and fails): each list reaches the error context whole (400 role names, past the 2000-character tail), no receipt, no findings file left, the source URL's password never in the answer, a stale file never answers for a new run, a malformed file is refused, a failed restore without findings stays ENGINE_STEP_FAILED and the next run goes ahead, and migrate's data step answers PRECONDITION with the same lists
• Every role query, with the psql variables substituted, parses with PostgreSQL's parser (pglast 8.7, libpg_query), run ad hoc, not added to the suite
• uv run ruff check ., uv run mypy, ansible-lint engine/ansible, and the touched test files pass locally; the full suite is left to CI
• specs.py check passes; specs.py coverage --spec 007 lists tests for every criterion but S-007-12 and S-007-13, whose tests are in Refuse a bad --source-schema with PRECONDITION and add migrate --data-schema (S-007-12, S-007-13) #169

Not run against a PostgreSQL server: the catalog queries' results on a real source (an extension type two pg_depend hops away, polroles, pg_get_expr text matching \mauth\.), pg_dump --schema --extension (pg_dump 14 or later), a pg_restore --single-transaction that fails and rolls back, and the whole role over SSH.

Fixes #125

🤖 Generated with Claude Code