Skip to content

feat(cli): migrate the database before replacing app-api #719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Medformatik merged 1 commit into from
Oct 31, 2026
Merged

feat(cli): migrate the database before replacing app-api #719

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions apps/api/esbuild.config.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,9 @@ const noRelativeIntegrationsImports = {
};

await build({
entryPoints: ["src/server.ts"],
// migrate.ts is the one-shot migration step `openmapx services update` runs
// before it replaces app-api.
entryPoints: ["src/server.ts", "src/migrate.ts"],
bundle: true,
platform: "node",
target: "node24",
Expand All @@ -68,6 +70,6 @@ await build({
define: {
__OPENMAPX_PRIVACY_SOURCE_FINGERPRINT__: JSON.stringify(privacySourceFingerprint),
},
outfile: "dist/server.js",
outdir: "dist",
plugins: [noRelativeIntegrationsImports],
});
66 changes: 66 additions & 0 deletions apps/api/src/migrate.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/**
* Applies the pending database migrations and exits.
*
* `openmapx services update` runs this in a one-off app-api container before
* it replaces app-api, so a long migration is its own step with its own output
* rather than part of app-api's startup and healthcheck. app-api still applies
* any pending migrations when it starts; after this step there are none.
*/
import { existsSync } from "node:fs";
import { join } from "node:path";
import { envString } from "@openmapx/core/server-env";
import { openMigrationSession, sql } from "./db/index";
import { type MigrationRun, startMigrations } from "./db/migration-runner";
import { applyRequiredMigrations } from "./startup-migrations";
import { createSafePinoOptions } from "./utils/safe-log-fields";

const { default: pino } = await import("pino");
const log = pino(createSafePinoOptions(envString("LOG_LEVEL", "info")));

// The one-off container runs this as PID 1, which ignores SIGTERM without a
// handler. Stopping the container must stop the migration in Postgres too.
let run: MigrationRun | undefined;
let stopping = false;
const stop = (signal: NodeJS.Signals) => {
if (stopping) return;
stopping = true;
log.warn({ signal }, "Stopping the database migration");
void (run?.cancel() ?? Promise.resolve())
.catch((err) => log.warn(err, "Could not stop the running database migration"))
.finally(() => process.exit(signal === "SIGTERM" ? 143 : 130));
};
process.on("SIGTERM", stop);
process.on("SIGINT", stop);

const migrationsDir = join(import.meta.dirname ?? ".", "db", "migrations");
const startedAt = Date.now();
try {
let applied: string[] = [];
await applyRequiredMigrations({
migrationsDirectory: migrationsDir,
directoryExists: existsSync,
migrate: async () => {
run = startMigrations({
session: openMigrationSession(),
control: sql,
migrationsFolder: migrationsDir,
log,
});
applied = await run.done;
},
});
log.info(
{ migrations: applied, durationMs: Date.now() - startedAt },
"Database migrations applied",
);
await sql.end({ timeout: 5 });
process.exit(0);
} catch (err) {
if (stopping) {
log.warn("Database migration stopped before it finished; it rolled back");
// The stop handler exits once the migration has rolled back.
await new Promise<never>(() => {});
}
log.fatal(err, "Database migration failed");
process.exit(1);
}
32 changes: 16 additions & 16 deletions docs/docs/developer/cli-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,22 +62,22 @@ Discover, inspect, select, build, and run the backend services that make up a
deployment. The selection, preset, build, and lifecycle semantics are documented
in full under [Managing services](../install/managing-services.md).

| Command | Description |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `services list` | List discovered services as a table. Filters: `--capability <cap>`, `--quality <built-in\|community-verified\|community>`, `--enabled`. |
| `services get <id>` | Print one service's full manifest as JSON. |
| `services selected` | Show the requested roots and the effective set after dependency expansion. |
| `services enable <ids...>` | Persistently add root services to `infra/docker/state/ops-agent/service-selection.json`. |
| `services disable <ids...>` | Persistently remove root services from the selection file. |
| `services build <ids...>` | Build prepared artifacts for the named services, then re-render and re-link. Flags: `--region <region>`, `--continue-on-error`, `--import` (MOTIS only: also run `motis import`). |
| `services build-all` | Build every service that declares a build, in order. Flags: `--region <region>`, `--fail-fast`. |
| `services start [ids...]` | Render, apply hardlinks, then `docker compose up -d` the named services. Flag: `--preset <names>`. |
| `services stop [ids...]` | `docker compose stop` the named services. Flag: `--preset <names>`. |
| `services restart [ids...]` | In-place reboot (`docker compose restart`); does **not** re-render. Flag: `--preset <names>`. |
| `services update [ids...]` | Re-render, link, pull the configured images, then replace containers with `up -d --force-recreate`. Flag: `--preset <names>`. For core OpenMapX app releases, use the aggregate release-manifest procedure in [Upgrading](../install/upgrading.md). |
| `services status [id]` | Container status table for one or all services (`docker compose ps`). |
| `services logs <id>` | Stream service logs. Flags: `--tail <n>` (default `100`), `--follow`. |
| `services capabilities` | Inventory the capability and data-type vocabulary across the registry. Flag: `--unrecognised`. |
| Command | Description |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `services list` | List discovered services as a table. Filters: `--capability <cap>`, `--quality <built-in\|community-verified\|community>`, `--enabled`. |
| `services get <id>` | Print one service's full manifest as JSON. |
| `services selected` | Show the requested roots and the effective set after dependency expansion. |
| `services enable <ids...>` | Persistently add root services to `infra/docker/state/ops-agent/service-selection.json`. |
| `services disable <ids...>` | Persistently remove root services from the selection file. |
| `services build <ids...>` | Build prepared artifacts for the named services, then re-render and re-link. Flags: `--region <region>`, `--continue-on-error`, `--import` (MOTIS only: also run `motis import`). |
| `services build-all` | Build every service that declares a build, in order. Flags: `--region <region>`, `--fail-fast`. |
| `services start [ids...]` | Render, apply hardlinks, then `docker compose up -d` the named services. Flag: `--preset <names>`. |
| `services stop [ids...]` | `docker compose stop` the named services. Flag: `--preset <names>`. |
| `services restart [ids...]` | In-place reboot (`docker compose restart`); does **not** re-render. Flag: `--preset <names>`. |
| `services update [ids...]` | Re-render, link, pull the configured images, then replace containers with `up -d --force-recreate`. When `app-api` is among them, it first stops `app-api` and applies the new image's database migrations in a one-off container. Flag: `--preset <names>`. For core OpenMapX app releases, use the aggregate release-manifest procedure in [Upgrading](../install/upgrading.md). |
| `services status [id]` | Container status table for one or all services (`docker compose ps`). |
| `services logs <id>` | Stream service logs. Flags: `--tail <n>` (default `100`), `--follow`. |
| `services capabilities` | Inventory the capability and data-type vocabulary across the registry. Flag: `--unrecognised`. |

`services enable` / `disable` refuse to edit the selection file when
`OPENMAPX_ENABLED_SERVICES` is set, so the source of truth stays unambiguous.
Expand Down
40 changes: 21 additions & 19 deletions docs/docs/install/upgrading.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,14 +391,15 @@ dedicated hostname or port.
## 5. Database migrations apply automatically

OpenMapX does not have a separate "run migrations" command for a self-host
upgrade. The `app-api` container **applies pending Drizzle migrations on every
boot**: at startup it runs the migration files bundled in the image against the
database, and the migrator is **idempotent**; it skips migrations that are
upgrade. When `services update` replaces `app-api` (step 4), it first stops the
old `app-api` and applies the new image's pending Drizzle migrations in a one-off
container. Their output is part of the update's output. Only then does it start
the new containers. The migrator is **idempotent**; it skips migrations that are
already applied and only runs the new ones.

So once the new `app-api` image starts (step 4), any schema changes that shipped
with the release are applied automatically. You can confirm it ran cleanly in the
API logs:
`app-api` also applies pending migrations whenever it starts, so a container
started any other way catches up too. After the update's migration step there
are none left. You can confirm it ran cleanly in the API logs:

```bash
pnpm openmapx services logs app-api --tail 100
Expand All @@ -418,15 +419,13 @@ the deployment contains state you cannot readily recreate.

### While a migration runs

`app-api` logs `Applying N database migrations` with their names before it
starts. It answers its healthcheck only after they finish, so the container
shows as `starting` meanwhile, and `app-web` waits for it. Docker allows it 20
minutes before it counts failed healthchecks. A migration that changes many rows
can take a while. To follow it:
The migration step logs `Applying N database migrations` with their names before
it starts. The API is unavailable until it finishes and the new containers have
started. A migration that changes many rows can take a while. To follow it:

- Every minute, `app-api` logs `Still applying database migrations` with the
statement it is running and for how long. If that statement is waiting for a
lock another database session holds, it logs that instead.
- Every minute, it logs `Still applying database migrations` with the statement
it is running and for how long. If that statement is waiting for a lock another
database session holds, it logs that instead.
- In PostgreSQL the migration's session is named `openmapx-api-migrations`.
From `psql` in the `postgis` container, connected to the `openmapx` database:

Expand All @@ -436,11 +435,14 @@ can take a while. To follow it:
```

Let a migration that is making progress finish. All pending migrations run in
one transaction, so stopping `app-api` rolls every one of them back, and the next
start repeats them from the beginning. The admin-panel updater does the same
when `app-api` is not healthy in time: it returns the services to their previous
images, which stops the migration. A second `app-api` started meanwhile waits for
the first to finish instead of migrating the database at the same time.
one transaction, so stopping the update rolls every one of them back. The
previous `app-api` then starts again, nothing is replaced, and the next update
repeats the migrations from the beginning. A migration that fails does the same.

An update from the admin panel stops after 30 minutes, including its backup and
image pull, and its migration with it. Apply a release whose migration needs
longer from the host with `pnpm openmapx services update`, which has no time
limit.

## 6. Verify the upgrade

Expand Down
Loading
Loading