Skip to content

fix(deploy): close five self-host installer gaps; mmctl upgrade --to; mm-core's own Synapse account - #157

Merged
argiad merged 9 commits into
developmentfrom
fix/installer-gaps
Nov 8, 2026
Merged

argiad merged 9 commits into
developmentfrom
fix/installer-gaps

Conversation

@argiad

@argiad argiad commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes five self-host installer gaps that an independent review of the install guides (#155) turned up. Each one was documented in the guides as a workaround; this PR fixes them and drops those workarounds from the docs.

Builds on #155 (merged): the doc edits are on top of its guides.

The five gaps

  1. Moving to a new release (owner's call: mmctl upgrade --to <tag>). Plain mmctl upgrade only re-pulled the pins already installed. --to first downloads the tag, so a wrong tag changes nothing. It refuses a release older than the installed one, because the database migrates forward only. Then it takes and verifies a backup, swaps /opt/mm/src, and runs that release's installer with the install's own domain, email and mode. --to main moves to the newest release. Separately, install.sh now backs up a completed install before re-running over it (the one-liner fetches main, which migrated the DB with no backup). A failed backup stops it; MM_SKIP_BACKUP=1 overrides.
  2. A re-run failed after the owner changed the generated password. Fixing this turned up a bigger bug (below), and the fix is a dedicated admin account for mm-core.
  3. The preflight AAAA check never ran: install.sh set MM_DOMAIN after preflight. The domain is now passed in. The check now also ignores the IPv4-mapped ::ffff: answers glibc returns for A-only names; without that it would have warned on every IPv4-only domain.
  4. No CPU architecture check. On ARM, preflight now stops before installing anything and names the architecture it found.
  5. Backups. Files are written 0600 in a 0700 dir, and older 0644 backups are tightened. Sets older than MM_BACKUP_KEEP_DAYS (default 14, 0 keeps all) are pruned, only after a fully successful backup. The config tarball now carries Synapse's signing key and versions.env + docker-compose.yml, so mmctl restore after a release upgrade brings back the old image pins. mmctl restore also puts the key back. mmctl backup --media (opt-in, owner's call) archives the synapse-data and recordings-data volumes.

The bigger bug behind gap 2: mm-core's admin token was the owner's session

MM_SYNAPSE_ADMIN_TOKEN was a session of the owner's account. A Matrix password change logs out the user's other devices by default (logout_devices defaults to true), so an owner who changed the generated password, as the guides told them to, revoked mm-core's token. After that:

  • the Operator Console signed them in as demo (check_synapse_admin(...).unwrap_or(false));
  • membership gates and moderation failed closed;
  • every re-run died with could not capture admin token.

Every use of the token is a Synapse admin API call, so with the owner's OK it now belongs to @mm-core:<domain>, an admin account with a generated password (MM_SYNAPSE_SERVICE_PASS) that no person signs in with.

  • Re-runs keep a token that still works (whoami) or sign @mm-core in again. They never need the owner's password, and leave the owner account alone unless --admin-user/--admin-pass is passed.
  • An existing install switches over on its next install.sh re-run. The old owner session is signed out only after mm-core is healthy on the new token.
  • mmctl rotate MM_SYNAPSE_ADMIN_TOKEN signs @mm-core in again without a prompt. It signs the old token out after phase 3, so the phase-0 rollback still works.
  • Passwords and tokens reach curl and register_new_matrix_user on stdin, never on a command line.

Live installs where the owner already changed the password are affected today. One re-run of the new installer fixes them.

Tests

  • TDD throughout: each new test was watched failing first. The suite went from 274 to 347 bats tests, all green, and shellcheck is clean (macOS bash 3.2 / BSD tools locally; CI runs them on Ubuntu/GNU).
  • An independent review pass found six issues, all fixed in the last commit: token retirement order, the password on argv, the AAAA false positive, the in-place key write, ordering in upgrade --to, and stale stage dirs.

Not verified on a real server. Creating the Hetzner test server was blocked in this session (auto-mode classifier), so @mm-core provisioning, --password-file /dev/stdin, the switch-over from an owner token, and upgrade --to against a live stack are covered by unit tests only. A fresh-server run is worth doing before the next release. The plan is in the session notes.

CI fix included (separate commit)

ci(test): give each cargo-test run its own database. The mm-runners jobs share one host Postgres, and every run reset the same mm_test. Overlapping runs broke each other in one of two ways: dropdb was refused (this PR, twice), or it dropped the database under live tests (development's push run after #155). Each run attempt now uses mm_test_<run_id>_<attempt> and drops it at the end. cargo-test is a required check, so this was blocking every PR.

Docs

  • docs/install.md, docs/install-agent.md and deploy/README.md drop the workarounds: the manual backup plus pinned re-run, --admin-pass on re-runs, "write the new password down", the find -delete cron line, and the manual AAAA check. They now document upgrade --to, backup --media, retention, @mm-core, and new troubleshooting rows.
  • deploy/docs/secrets-inventory.md and rotation-runbooks.md are updated for @mm-core. docs/upgrade-guide.md points single-server installs at upgrade --to. CHANGELOG has Unreleased entries.

🤖 Generated with Claude Code

argiad and others added 8 commits November 8, 2026 03:27
…chitecture

install.sh set MM_DOMAIN only after preflight ran, so the AAAA check (DNS
advertising IPv6 the host cannot serve) never ran on any install. preflight
now takes the domain as an argument; install.sh resolves it first (except
--no-domain's, which comes from the public IP preflight finds).

First-party images are linux/amd64 only, so an ARM host failed at the image
pull minutes in with "no matching manifest". preflight now stops first, before
installing anything, and names the architecture it found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pins

- Backups hold every secret but were written 0644. Every file is now created
  0600 inside a 0700 backups/, and a backup run tightens files an older
  release left 0644.
- Nothing ever deleted an old backup. A successful backup now deletes backup
  files older than MM_BACKUP_KEEP_DAYS (default 14, 0 keeps all, kept across
  install.sh re-runs). Only backup()'s own file names are touched, and a
  failed backup deletes nothing.
- The config tarball now carries Synapse's signing key (the federation
  identity, which lives in the synapse-data volume no backup covered), and
  mmctl restore writes it back. It also carries versions.env and
  docker-compose.yml, so restoring after a release upgrade brings back the old
  image pins instead of starting the new images on the old database.
- mmctl backup --media also archives the synapse-data and recordings-data
  volumes (opt-in: they can be large).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A re-run signed the owner in again with the password generated at the first
install. Once the owner had changed it, as the guides tell them to, every
re-run died with "could not capture admin token" although the stored token
still worked.

The owner bootstrap moves out of install.sh into bootstrap_owner
(lib/bootstrap.sh). It first asks Synapse who the stored
MM_SYNAPSE_ADMIN_TOKEN belongs to (whoami, token passed to curl on stdin,
never on its command line). A token that still works is kept: no
registration, no sign-in, no prompt, and admin.credentials is left as it is.
Only a missing or rejected token, or an --admin-user naming another account,
takes the old register-and-sign-in path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k up first

mmctl upgrade only re-pulled the pins already in /opt/mm/versions.env, and
only install.sh copies a new set in. So the one way to a new release was
re-running the one-liner, which fetches main and migrated the database with
no backup.

- mmctl upgrade --to <tag> (or --to=<tag>; main = the newest release):
  downloads the tag first, so a wrong tag changes nothing; refuses a release
  older than the installed one (the database migrates forward only); takes
  and verifies the backup; then replaces /opt/mm/src and runs that release's
  installer as a re-run of this install (domain, email, --demo/--no-domain
  read from .env). A failure names the restore command. Plain mmctl upgrade
  is unchanged.
- install.sh backs up before it changes a completed install (one whose
  admin.credentials exists), right after preflight and before it copies in
  the new pins. A failed backup stops it; MM_SKIP_BACKUP=1 overrides. A backup
  mmctl upgrade --to just took (MM_BACKUP_TAKEN) is not taken twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mm-core's MM_SYNAPSE_ADMIN_TOKEN was a session of the OWNER's account: the
installer signed the owner in and kept the token. A Matrix password change
logs the user's other devices out by default (logout_devices), so an owner who
changed the generated password, as the guides say to, revoked it. The
Operator Console then signed them in as "demo" (mm-core decides admin with
this token), membership gates and moderation failed closed, and every re-run
died with "could not capture admin token".

Every use of the token is a Synapse admin API call, so it now belongs to
@mm-core:<domain>, an admin account with a generated password
(MM_SYNAPSE_SERVICE_PASS) that no person signs in with.

- Re-runs keep a token that still works (whoami), or sign @mm-core in again.
  They never need the owner's password, and leave the owner account alone
  unless --admin-user/--admin-pass is passed.
- An install whose token was the owner's session switches to @mm-core on its
  next re-run and signs that old session out (backups taken before still
  hold it).
- A first install checks the owner account can sign in (and signs that
  session straight out), since nothing else does now.
- mmctl rotate MM_SYNAPSE_ADMIN_TOKEN signs @mm-core in again, no prompt, and
  signs the old token out.
- Passwords and tokens reach curl on stdin, never on its command line, and a
  password with quotes or backslashes is sent as valid JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Upgrading to a new release is mmctl upgrade --to <tag>, not a manual
  backup plus a pinned re-run of the one-liner.
- Re-runs back up a completed install themselves, and never need the owner
  password: drop the --admin-user/--admin-pass advice and the "write the new
  password down" warning. Mention @mm-core, the server's own admin account.
- Backups: say what they hold now (signing key, pins), that they are
  root-only and pruned after MM_BACKUP_KEEP_DAYS (14), that --media adds
  uploads and recordings, and how to restore a media archive. The nightly
  cron loses its find -delete line.
- The installer now stops on ARM and warns about a stray AAAA record; new
  troubleshooting rows for those and for a failed pre-run backup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…napse account

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Sign the replaced admin token out only once mm-core runs healthy on the new
  one (install.sh after wait_healthy, rotate after phase 3). Signed out at
  once, the running mm-core held a revoked token if a step in between
  failed, and rotate's phase-0 rollback restored a revoked token.
- register_new_matrix_user gets the password on stdin
  (--password-file /dev/stdin), never on docker's command line.
- check_aaaa ignores the IPv4-mapped (::ffff:) answers glibc gives an A-only
  name; the AAAA check that now runs warned on every IPv4-only domain.
- Restoring the signing key replaces the live key only once the whole new
  key has arrived and is not empty, keeping its owner and mode. The key
  lookup no longer uses grep -q under pipefail (tar's SIGPIPE read as
  "no key").
- mmctl upgrade --to builds the installer flags before the backup (a broken
  .env changes nothing), and warns when an image pin in .env outranks the
  release's.
- A backup clears stage dirs (with a copy of the signing key) that a killed
  backup left, once they are an hour old.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@argiad
argiad force-pushed the fix/installer-gaps branch from 4be394b to 32b80d9 Compare November 8, 2026 11:29
@argiad
argiad marked this pull request as ready for review November 8, 2026 11:30
@argiad
argiad enabled auto-merge (rebase) November 8, 2026 11:39
The mm-runners jobs share one host Postgres, and every run reset the same
"mm_test". When two runs overlapped, the second one's dropdb either
failed ("database is being accessed by other users", PR #157 twice) or
dropped the database under the first one's live tests, which then panicked
in test_support ("connecting to Postgres failed", development's push run
after #155). Each run attempt now uses mm_test_<run_id>_<attempt> and drops
it at the end, cancelled runs included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@argiad
argiad merged commit 5565865 into development Nov 8, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant