Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions docs/specs/007-data-copy-schemas.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ once: platform schemas, Supabase-only extensions, and RLS policies on `anon` and

## Behaviour

**Choosing the schemas.** `cloudfall data copy` gains a repeatable `--schema NAME` flag,
and `cloudfall migrate` gains a repeatable `--data-schema DATABASE=NAME` flag beside its
**Choosing the schemas.** `cloudfall data copy` gains a repeatable `--source-schema NAME`
flag, and `cloudfall migrate` gains a repeatable `--data-schema DATABASE=NAME` flag beside its
`--data DATABASE=PATH`. The `data_copy` MCP tool, generated from the command, gains the
same `schema` list. A flag, not a declared field, because a copy is configured by flags
same `source_schema` list. The flag isn't `--schema` because the CLI framework (treaty)
reserves `--schema` on every command to print that command's schema; `--source-schema` pairs
with `--source-url-file` (#134). A flag, not a declared field, because a copy is configured by flags
today (`--database`, `--source-url-file`; `DataMigrateArgs` in `sdk/src/cloudfall/app.py`)
and the `Service` declares only the target: the source and what to take from it belong to
the one-time action, not to the desired state.
Expand All @@ -31,9 +33,9 @@ doesn't, a name given twice, or a `--data-schema` for a database with no `--data
command with `PRECONDITION` before any playbook runs. The plan shown without `--yes` names
the schemas it will copy, or says it copies every non-system schema of the source.

**What gets copied.** With `--schema`, `pg_dump` gets one `--schema` per name and one
**What gets copied.** With `--source-schema`, `pg_dump` gets one `--schema` per name and one
`--extension` per extension the preflight found the selected schemas need, so only those
schemas and their extensions reach the target. Without `--schema`, the dump command is
schemas and their extensions reach the target. Without `--source-schema`, the dump command is
today's, unchanged, and the selected schemas are every schema of the source except
`pg_catalog`, `information_schema`, `pg_toast`, and the `pg_temp_*` and `pg_toast_temp_*`
families: what `pg_dump` dumps today.
Expand All @@ -50,7 +52,7 @@ at least one table across them.
- `missingSchemas`: a named schema the source doesn't have
- `extensions`: an extension the selected schemas need that is not in the target's
`pg_available_extensions`. "Need" means an object in a selected schema depends on it in
`pg_depend`, or it is installed in a selected schema; without `--schema`, every extension
`pg_depend`, or it is installed in a selected schema; without `--source-schema`, every extension
installed in the source except `plpgsql` counts, since the whole dump creates them all
- `roles`: a role named in a row-level security policy (`pg_policy.polroles`) on a table in
a selected schema that the target's `pg_roles` lacks, such as `anon` or `authenticated`
Expand All @@ -74,17 +76,17 @@ re-run needs no manual cleanup.
the copied schemas in sorted order, and each `spec.tables[]` entry gains `schema` beside
`table` and `rows`. Both are additions; a reader of `table` and `rows` keeps working.

**Compatibility.** Without `--schema`, what is dumped and restored stays the same as 0.7.0:
**Compatibility.** Without `--source-schema`, what is dumped and restored stays the same as 0.7.0:
the whole database. The default stays "every non-system schema", not `public`, because
`public` was only what the checks read, never what the copy covered. Narrowing the default
would silently drop schemas existing migrations copy today (an ORM's migration table, such
as Drizzle's `drizzle.__drizzle_migrations`, or an app's own schemas) while the receipt still
said the counts matched. Two things do change for a copy without `--schema`. The row-count
said the counts matched. Two things do change for a copy without `--source-schema`. The row-count
check now covers every copied schema, so a mismatch outside `public` that passed before now
fails. And the preflight can refuse, but only for extensions and policy roles the target
lacks, which make today's restore fail too; `auth` is a selected schema in this mode
whenever the source has one, so `authReferences` stays empty. A Supabase project is copied
with `--schema public` (or the app's own schemas), the way the refusal points to.
with `--source-schema public` (or the app's own schemas), the way the refusal points to.

Touches `sdk/src/cloudfall/app.py` (`data copy`, `DataMigrateArgs`, `cloudfall migrate`),
`sdk/src/cloudfall/lifecycle.py` (`plan_data_migration`, `migrate_data`,
Expand All @@ -96,20 +98,20 @@ copy.

## Acceptance criteria

- S-007-1: Against a source with tables in schemas `public`, `app`, and `audit`, `cloudfall data copy <service> --database <db> --source-url-file <file> --schema public --schema app --yes` leaves the target with the tables of `public` and `app` only, and the receipt's `spec.schemas` is `["app", "public"]`
- S-007-2: Against the same source, the copy without `--schema` leaves the target with the tables of all three schemas, as 0.7.0 does, and the receipt's `spec.schemas` is `["app", "audit", "public"]`
- S-007-1: Against a source with tables in schemas `public`, `app`, and `audit`, `cloudfall data copy <service> --database <db> --source-url-file <file> --source-schema public --source-schema app --yes` leaves the target with the tables of `public` and `app` only, and the receipt's `spec.schemas` is `["app", "public"]`
- S-007-2: Against the same source, the copy without `--source-schema` leaves the target with the tables of all three schemas, as 0.7.0 does, and the receipt's `spec.schemas` is `["app", "audit", "public"]`
- S-007-3: Each `spec.tables[]` entry of the receipt has `schema`, `table`, and `rows`, and the entries are exactly the base tables of the selected schemas
- S-007-4: The rendered row-count query for schemas `app` and `public` reads only those two schemas and returns one `schema,table,rows` line per base table, ordered by schema and table
- S-007-5: A target with a table in a selected schema refuses the copy with `ENGINE_STEP_FAILED` naming the table count, as today, and a target whose only table is in a schema not selected goes ahead
- S-007-6: `--schema missing` for a schema the source lacks exits `PRECONDITION` with code `data_copy_preflight_refused` and `missingSchemas` of `["missing"]`, and no dump file is written
- S-007-6: `--source-schema missing` for a schema the source lacks exits `PRECONDITION` with code `data_copy_preflight_refused` and `missingSchemas` of `["missing"]`, and no dump file is written
- S-007-7: A selected table using a type from an extension that is not in the target's `pg_available_extensions` exits `PRECONDITION` with code `data_copy_preflight_refused` naming that extension under `extensions`, the target holds no new table, and no receipt is written
- S-007-8: A policy `TO anon, authenticated` on a selected table, with neither role on the target, exits `PRECONDITION` with code `data_copy_preflight_refused` and `roles` of `["anon", "authenticated"]`
- S-007-9: With `--schema public`, a function `public.current_tenant()` whose body calls `auth.uid()` and a policy on `public.notes` whose `USING` calls `auth.uid()` are both named under `authReferences`, and the copy is refused
- S-007-9: With `--source-schema public`, a function `public.current_tenant()` whose body calls `auth.uid()` and a policy on `public.notes` whose `USING` calls `auth.uid()` are both named under `authReferences`, and the copy is refused
- S-007-10: A source with a missing extension, a missing role, and an `auth.` reference at once is refused with all three named in one answer
- S-007-11: A restore that fails after the preflight passes exits `ENGINE_STEP_FAILED`, leaves no table in the selected schemas of the target, writes no receipt, and a second run against the same target passes the empty-target check
- S-007-12: `--schema 'Bad;Name'`, `--schema app --schema app`, or `cloudfall migrate --data-schema crm=public` without `--data crm=<file>` exits `PRECONDITION` and runs no playbook
- S-007-13: `cloudfall migrate --data crm=<file> --data-schema crm=public --yes` copies only `public` into `crm`, as `data copy --schema public` does
- S-007-14: `cloudfall data copy` without `--yes` names the selected schemas in its plan, or says it copies every non-system schema when no `--schema` is given
- S-007-12: `--source-schema 'Bad;Name'`, `--source-schema app --source-schema app`, or `cloudfall migrate --data-schema crm=public` without `--data crm=<file>` exits `PRECONDITION` and runs no playbook
- S-007-13: `cloudfall migrate --data crm=<file> --data-schema crm=public --yes` copies only `public` into `crm`, as `data copy --source-schema public` does
- S-007-14: `cloudfall data copy` without `--yes` names the selected schemas in its plan, or says it copies every non-system schema when no `--source-schema` is given

## Out of scope

Expand Down
Loading