Skip to content

Gated database migrations - #16

Open
carbolymer wants to merge 14 commits into
masterfrom
mgalazyn/feature/gated-migrations
Open

Gated database migrations#16
carbolymer wants to merge 14 commits into
masterfrom
mgalazyn/feature/gated-migrations

Conversation

@carbolymer

@carbolymer carbolymer commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Closes #15.

Until now, Kupo ran every database migration silently at startup, however long it took. A heavy migration could keep the server busy for hours with no warning, and nothing recorded which migrations had run. This PR gives migrations a registry and a kind, and moves the heavy ones out of startup into an explicit operator command.

The fix for the spent input index, #14, stacks on this PR and uses the new data repair kind for its repair.

What changes for operators

At startup, Kupo now checks the registry before it writes anything:

  • Automatic migrations run as before.
  • If a blocking migration is pending, Kupo logs the pending migrations and exits with code 3. Run kupo db migrate, then start Kupo again.
  • If a data repair is pending, Kupo starts and logs a warning at every start.
  • If the database holds a migration that this Kupo does not know, Kupo refuses to run, because a newer Kupo migrated it. This applies to the writer, to read-only replicas and to kupo copy.

Two new commands:

kupo db status  --workdir DIR
kupo db migrate --workdir DIR [--confirm-complete-index]

db status lists every migration with its kind, state and summary, plus an estimate and a resume cursor for a repair that is not done. It never writes to the database. It exits with code 3 when a blocking migration is pending, and with code 1 when a newer Kupo migrated the database, like Kupo at startup.

db migrate applies every pending migration, blocking ones included, then runs the pending data repairs. It exits with code 1 when a repair stays pending because its precondition failed. --confirm-complete-index tells repairs that the index matches * and was synchronised from origin, which the database cannot prove by itself.

Both commands work only on a SQLite database given with --workdir. They do not support --in-memory or a remote (PostgreSQL) database.

Design

  • Each embedded SQL migration now lives in its own module, with a stable id, a kind and a summary. Kupo.App.Database.Migration holds the ordered list and the runner. Each migration's SQL now runs as one script through SQLite's own parser, so a ; inside a statement (a trigger body, a string literal) no longer splits it. Transactions and user_version values are unchanged.
  • A migration is a sum type. Automatic and Blocking carry SQL, and Repair carries a resumable batch repair. A repair with SQL, or a schema migration with a batch, therefore cannot be written.
  • A new migrations table records each migration as applied, pending or skipped. The migration that creates it adds an applied row for every earlier migration.
  • There are three kinds. Automatic runs at startup. Blocking stops startup until the operator runs kupo db migrate. Repair is a batch data repair: a precondition, a cheap estimate, and a function that repairs one batch from a cursor.
  • A pure planner decides what runs. A fresh database applies automatic and blocking migrations, and records repairs as skipped, because there is nothing to repair. An existing database applies automatic migrations, registers repairs as pending with a cutoff slot, and stops at a blocking migration. The operator mode of the planner applies blocking migrations too.
  • Crash safety: a repair runs in batches, and each batch commits together with its cursor. An interrupted repair resumes from the cursor and never repeats a batch. The last batch returns a summary, which the final log line shows.
  • Lock: kupo db migrate takes an exclusive lock on the database, so it refuses to run while Kupo is running.
  • Downgrade guard: Kupo checks the registry for unknown ids before it writes anything.

Failure handling

  • A migration that fails rolls back completely: its statements, its user_version bump and its registry row. Migrations before it stay applied, and later ones do not run.
  • Kupo logs DatabaseMigrationFailed (or DatabaseRepairFailed for a failed repair batch) at Error level, naming the migration.
  • A failed repair batch rolls back with its cursor, and the next kupo db migrate resumes from the last committed batch.
  • Tests cover the rollback.

Testing

  • 395 examples pass, and the build is warning-free.
  • The planner is pure, so its rules are covered by unit tests.
  • End to end with the built binary, on a database made by an older Kupo: db status reports the pending migrations, db migrate applies them, and a second run reports the database up to date.

Not in this PR

  • Repair by re-fetching blocks from the node. The repair kind has no such source yet.
  • Reporting pending migrations in /health.
  • Online repair, that is, repairing while Kupo runs. kupo db migrate needs Kupo to be stopped.
  • PostgreSQL. The postgres build flag is an unfinished copy of the SQLite backend that keeps its own older migration list, so it gets none of this (no registry, gate, repair or kupo db support).

References

@carbolymer carbolymer self-assigned this Nov 11, 2026
@carbolymer carbolymer mentioned this pull request Nov 12, 2026
Open
@carbolymer
carbolymer force-pushed the mgalazyn/feature/gated-migrations branch 2 times, most recently from 1aaa00c to 804b7de Compare November 13, 2026 10:30
@carbolymer
carbolymer marked this pull request as ready for review November 13, 2026 12:23
@carbolymer
Move withTransaction to its own module
9dbc6b1 
Pure move, no change.

The migration runner that the next commits add needs withTransaction,
and SQLite.hs will import that runner, so the function cannot stay in
SQLite.hs without an import cycle.
@carbolymer
Describe database migrations as data
15c6a78 
Add the types that describe a migration: a stable id such as
v2.10.0/001, a one-line summary and its action.

The action says when a migration may run. An automatic migration runs
at startup. A blocking one runs at startup only on a fresh database,
and otherwise needs the operator. Both run SQL. A repair fixes existing
data, and never runs at startup.

A repair runs in batches. It has a precondition, a cheap estimate, and
a function that repairs one batch from a cursor.

Nothing uses these types yet.
@carbolymer
Add a pure planner for database migrations
5852d6b 
planMigrations decides what runs, from the migration list and what the
database records about it. It never touches the database, so the unit
tests cover each case without one.

On a fresh database, it applies automatic and blocking migrations, and
records repairs as skipped. On an existing database, it applies
automatic migrations, registers repairs as pending, and stops at the
first blocking migration. The operator mode applies blocking migrations
too. A repair is skipped when the same plan applies the migration that
introduced its data, because there is nothing to repair yet.

Until the database has a registry, the first user_version migrations
count as applied.

Nothing calls the planner yet.
@carbolymer
Give each database migration its own module
f34d998 
Mechanical. Each SQL file under db/ gets a module that embeds it, with
a stable id, a kind (all automatic) and a one-line summary.
Kupo.App.Database.Migration lists them in the same order as the list in
SQLite.hs, which still runs its own copy until the next commit.

The tests pin the ids and their order: the position of a migration in
the list is its user_version, so the list is append only.
@carbolymer
Run database migrations through the planner
abb53c7 
SQLite.hs now calls the runner in Kupo.App.Database.Migration, and its
own list and runner go away. The runner reads the database state once,
plans, and runs each planned step in its own transaction, together with
the new user_version. Each migration's SQL now runs as one script
through SQLite's parser (direct-sqlite's exec), so a ; inside a
statement no longer splits it. When a step fails, the runner logs the
migration and the error, then rethrows.

Once a migrations table exists, the runner also records each step in
it, and backfills the earlier migrations in the step that creates the
table. The next commit adds that table. Until then, the runner applies
the same statements and sets the same user_version values as before.

After the steps, the runner logs a warning about pending repairs, and
refuses to go past a pending blocking migration: it logs the migrations
that need the operator and throws. No shipped migration is a repair or
blocking yet.

copy now passes its tracer to the runner, so it logs the same messages
as Kupo does at startup.
@carbolymer
Record database migrations in a registry table
f6f5780 
Add migration v2.12.0/001, which creates the migrations table. Each row
holds the state of one migration (applied, pending or skipped), the
cutoff slot and cursor of a data repair, and when and by which version
Kupo wrote it. The runner from the previous commit fills the table: the
step that creates it also adds an applied row for each earlier
migration, without a version or an apply time.

db/README.md documents the table.

The runner tests come with this commit because they read the registry:
the backfill, a repair skipped on a fresh database and pending with a
cutoff slot on an existing one, a blocking migration that stops the
runner, a second run that changes nothing, a trigger body that holds a
semicolon, and a failing migration, which rolls back and which the
runner logs.
@carbolymer
Check migrations on read-only replicas too
a615635 
A read-only replica never migrates. It now plans like the writer when
it creates its pool, in a read-only transaction, and writes nothing.
It logs the migrations that the writer has not run yet and carries on,
logs the same warning about pending repairs, and refuses the same
databases as the writer: those with a pending blocking migration.
@carbolymer
Refuse databases that a newer Kupo migrated
40cd372 
The planner now lists the registry ids that this version does not know.
They come from a newer Kupo, and an older one must not touch what those
migrations changed. Before it writes anything, the runner logs the ids
and throws. The replica check does the same, and so does copy, through
the runner.
@carbolymer
Exit with code 3 when a migration blocks startup
bfa2e08 
When the database layer refuses a database, it has already logged why,
so Kupo now stops without logging an unexpected error on top. It exits
with code 3 when a blocking migration is pending, so that scripts and
supervisors can tell this case apart, and with code 1 when a newer
Kupo migrated the database.

Read-only replicas check migrations while they create their pool,
before the handler that stops Kupo applies, so pool creation now
handles both refusals itself.
@carbolymer
Add a resumable runner for data repairs
dbeffa8 
migrateDatabase is what the operator runs. It applies every migration
that the plan allows to an operator, blocking ones included, then runs
each pending repair from its stored cursor, and returns the repairs
that stay pending.

A repair first checks its precondition, which gets the cutoff slot
recorded at registration and whether the operator confirms a complete
index. When the precondition fails, the repair stays pending and the
runner logs why. Otherwise the repair runs in batches, and each batch
commits together with its cursor, so an interrupted repair resumes
without repeating a batch. The last batch marks the repair as applied
and returns a summary for the log. When a batch fails, the runner logs
the repair and the error, then rethrows.

No command calls it yet.
@carbolymer
Report the state of every migration
e8b972c 
readStatus reads what the registry records about each known migration,
in one read-only transaction, and plans as Kupo would at startup. A
repair that is not done also gets an estimate of the work left, from
cheap statistics, and the cursor it would resume from.

renderStatus prints one aligned line per migration, the details of a
repair below its line, and hints at the end. statusExitCode gives the
exit code that Kupo would have at startup: 1 when a newer Kupo migrated
the database, and 3 when a blocking migration is pending.

No command calls them yet.
@carbolymer
Add the actions behind kupo db status and db migrate
18acd54 
databaseStatus prints the status report and returns the exit code that
Kupo would have at startup. It opens the database read-only, never
creates it, and fails while kupo db migrate holds its lock.

databaseMigrate first takes an exclusive lock, so it fails while Kupo
or any other connection has the database open, and keeps them out until
it is done. It then runs migrateDatabase, and returns 1 when a repair
stays pending.

Both print any failure and return 1. The next commit adds them to the
command line.
@carbolymer
Add the kupo db status and db migrate commands
51bbb5c 
Add both actions to the command line. Each takes --workdir, the
directory that holds the database. db migrate also takes
--confirm-complete-index, which tells repairs that the index matches *
from origin, a fact that the database cannot prove. db migrate logs
from the Info level up.

The man page documents both commands, the new flag and the exit codes:
3 when a blocking migration is pending, and 1 in the new failure cases.
@carbolymer
Describe the migration registry in the changelog
3fd8a76 
Sum up for operators what the previous commits change: the registry,
the startup gate and its exit code, the warning about pending repairs,
the two db commands, and the refusal of databases that a newer Kupo
migrated.
@carbolymer
carbolymer force-pushed the mgalazyn/feature/gated-migrations branch from 804b7de to 3fd8a76 Compare November 13, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@carbolymer carbolymer

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Gated and resumable database migrations (kupo db status and kupo db migrate)

1 participant

@carbolymer