Harden updates and settings and expand installation parity checks

ansible.cfg

@@ -1,6 +1,7 @@
 [defaults]
 inventory = inventory/hosts.yml
 roles_path = roles
+library = image/library
 callback_plugins = plugins/callback
 interpreter_python = auto_silent
 retry_files_enabled = False

assets/scripts/cybexos-runtime

@@ -17,6 +17,7 @@ Usage:
   cybexos-runtime dev enable CHECKOUT
   cybexos-runtime dev disable
   cybexos-runtime dev status
+  cybexos-runtime shell status|safe|recover
   cybexos-runtime plugin add|update|clone|remove|list|enable|disable|set|reload|restart [arguments]
 
 The development switch only records a validated checkout. It never fetches,
@@ -243,6 +244,9 @@ exec_runtime() {
       export CYBEXOS_PLUGIN_ROOT=$data_root/plugins
       export OMARCHY_PATH=$selected/compat/omarchy
       export PATH=$OMARCHY_PATH/bin:$PATH
+      if [[ -f $selected/scripts/shell-recovery.py ]]; then
+        exec python3 "$selected/scripts/shell-recovery.py" run "$selected"
+      fi
       exec /usr/bin/qs -p "$selected"
       ;;
     hypridle|hyprlock)
@@ -280,6 +284,9 @@ exec_runtime() {
 ipc_call() {
   local selected
   selected=$(runtime_path quickshell)
+  if [[ -f $selected/scripts/shell-recovery.py ]]; then
+    selected=$(python3 "$selected/scripts/shell-recovery.py" path "$selected")
+  fi
   # --any-display: a caller started by systemd (a reminder timer) need not
   # carry the session's display. The -- keeps function names that shadow qs
   # subcommands (e.g. show) positional.
@@ -303,6 +310,21 @@ case $command_name in
     ipc_call "$@"
     ;;
   dev) dev_command "$@" ;;
+  shell)
+    [[ $# -eq 1 ]] || { usage >&2; exit 2; }
+    case $1 in status|safe|recover|record-stop) ;; *) usage >&2; exit 2 ;; esac
+    selected=$(runtime_path quickshell)
+    # A previous release without recovery support remains rollback-compatible.
+    [[ -f $selected/scripts/shell-recovery.py ]] || {
+      [[ $1 != record-stop ]] || exit 0
+      printf 'cybexos-runtime: this desktop release has no recovery mode\n' >&2
+      exit 1
+    }
+    python3 "$selected/scripts/shell-recovery.py" "$1" "$selected"
+    if [[ $1 == safe || $1 == recover ]]; then
+      exec systemctl --user restart quickshell.service
+    fi
+    ;;
   plugin)
     selected=$(runtime_path quickshell)
     if [[ ${1:-} == reload ]]; then

docs/architecture/ownership.md

@@ -57,6 +57,16 @@ installed runtime. The command records only the canonical path and reloads
 managed desktop components; it never fetches, resets, merges, commits, or
 writes inside the checkout.
 
+The Hyprland startup hook must also work when the development checkout is
+selected on an ISO installation. Checkout installs place the ordered session
+starter in `/usr/local/libexec`; ISO/RPM installs place it in `/usr/libexec`.
+`autostart.lua` resolves an executable helper at login, including when the
+checkout is loaded verbatim without the image packager's path rewriting.
+Otherwise Hyprland can start with `hyprland-session.target` inactive, leaving
+Quickshell, wallpaper, idle handling and desktop portals unavailable. The
+session startup fixtures exercise checkout, packaged and ISO development
+layouts in both source gates.
+
 Use `cybex dev status` to show the active source and `cybex dev disable` to
 return to the verified vendor runtime. Internet updates continue to stage and
 activate releases while development mode is on; they do not modify the selected
@@ -90,23 +100,43 @@ preserve unknown fields, retain a recoverable original, and avoid downgrading
 data on rollback. A new API or schema needs a compatibility plan and
 upgrade/rollback fixtures before it is released.
 
-That target is not yet enforced for every application. Remaining work:
-
-- Shell settings currently normalize to known keys and have visual migrations
-  that infer an untouched value from equality with a previous default. Replace
-  that inference with explicit override tracking; treat legacy stored choices
-  conservatively. Keep unsupported future schemas read-only on older hosts.
-- Personal-dotfile deployment still replaces files such as Fastfetch,
-  Voxtype, MIME associations, and XDG user directories. Move defaults into
-  vendor fragments where supported, or seed only absent user files. Migrate
-  adopted files using a last-installed baseline and preserve conflicting edits.
-- Includes need application-specific precedence tests. Git and Kitty commonly
-  use later values; SSH commonly uses the first obtained value. The current
-  SSH include at the beginning can take precedence over personal choices.
-- Extend release checks beyond file sentinels: verify settings behavior, an
-  enabled API fixture, service overrides, app defaults, failed updates, and
-  rollback against supported previous releases. Preserve user-created package
-  and service additions when optional distro features change.
-
-Until those changes land, the widget contract does not imply that every
-existing application setting already survives distro convergence unchanged.
+Shell settings now use a sparse schema-27 document: the presence of a known
+key records an explicit choice, including a choice equal to the current
+default. Reset removes the override; Undo restores its ownership as well as
+its value. Legacy stored values are conservatively treated as explicit. Visual
+redesigns no longer infer an untouched preference from equality with an old
+default. Unknown JSON fields survive edits and resets, and a newer schema is
+read-only on an older shell.
+
+The asynchronous settings writer merges independent external edits, rejects
+conflicting writes, and confirms fsync and atomic publication before reporting
+success. A rejected edit is retained in a `shell.json.conflict-*` sidecar before
+the form reloads the external values. The first schema migration retains the
+exact original in `shell.json.before-migration-*`. These files live beside the
+user's settings and are retained for recovery; the shell never prunes them.
+The production Qt document component has real-engine lifecycle tests for
+queued changes, retries and unknown data, alongside filesystem transaction tests.
+
+Personal application stores (Fastfetch, Voxtype, Oh My Posh, MIME associations,
+XDG directories and npm configuration) are seeded only when absent. Shared
+Fish, Kitty, Git and SSH fragments use the same ownership ledger on checkout
+and ISO paths. A fragment advances only if it still matches its last installed
+bytes; conflicting edits, symlinks and explicit deletions survive. Adopted
+bytes are backed up before replacement. Unknown legacy fragments remain
+user-owned instead of being guessed at from their filename.
+
+Managed includes follow each application's precedence: Git/Kitty defaults
+come first, while SSH fallbacks come last in an explicit `Host *` scope.
+Git credential helpers accumulate instead of overriding, so vendor credential
+helpers are omitted when personal configuration provides its own chain. The
+SSH vendor fragment lives outside `.ssh/config.d` so wildcard includes cannot
+accidentally give it priority over a personal host. Moving an old unedited
+include retains its original file; edited include blocks are left intact.
+
+Remaining release coverage should exercise service overrides, optional package
+and service additions, and supported previous releases across failure and
+rollback, beyond file sentinels. Application ownership is scoped to these
+managed fragments; independent application databases remain the application's
+responsibility.
+
+The widget contract does not imply ownership of unrelated application state.

docs/fedora-major-upgrade.md

@@ -1,5 +1,107 @@
 # Fedora major-upgrade runbook
 
+## Guided upgrade workflow
+
+`cybex upgrade-system` now coordinates preflight, a durable background DNF5
+**download**, an explicit offline reboot, target convergence and rollback. It
+requires a separately reviewed target release; the current source manifest
+supports **Fedora 44 only**. This workflow does not qualify or advertise Fedora
+45. Incrementing a release number is insufficient.
+
+Select either a clean reviewed target checkout/release directory whose
+`release-manifest.json` **and** inventory declare the target, or a signed
+`cybexos-desktop` RPM for that release and architecture which provides
+`cybexos-supported-fedora = <next>`. The RPM signature must verify against the
+installed trusted keyring. Selecting local source is an explicit administrator
+trust decision, as with bootstrap; the workflow verifies compatibility and
+freezes the selected bytes, but does not claim a local checkout hash proves its
+author. A Git checkout must be clean and only tracked files enter the frozen
+payload. No branch is pulled, reset or switched.
+
+Make and test an external backup first. Supply a JSON receipt beside its tar
+archives, with paths relative to the receipt:
+
+```json
+{
+  "v": 1,
+  "createdAt": "2030-01-01T12:00:00Z",
+  "archives": [{
+    "path": "workstation.tar.zst",
+    "sha256": "REPLACE_WITH_THE_ARCHIVE_SHA256",
+    "covers": ["/home/john", "/etc", "/var/lib/xps-hardware", "/etc/pki/akmods"]
+  }]
+}
+```
+
+Use the actual backup timestamp, account home and checksum. The validator
+requires a backup from the last seven days on another filesystem UUID, hashes
+each archive, reads it completely through tar and verifies its declared
+coverage. The account home and `/etc` are mandatory, plus the hardware/signing
+paths when present. Archive members should be root-relative (`home/john/…`,
+`etc/…`). This verifies integrity and coverage; the independent restore test
+remains part of preparing the backup.
+
+```sh
+cybex upgrade-system check --target <next> --source /path/to/reviewed-release --backup /mnt/backup/receipt.json
+cybex upgrade-system prepare --target <next> --source /path/to/reviewed-release --backup /mnt/backup/receipt.json
+# On an RPM installation, use --rpm /path/to/signed-target.rpm instead.
+cybex upgrade-system status
+# Once status is ready, close applications and explicitly start the offline upgrade:
+cybex upgrade-system reboot
+```
+
+`prepare` returns after starting a durable root service. It does not reboot or
+install packages into the running OS. Status becomes `ready` only after the
+DNF download succeeds and the rollback checkpoint is armed. Closing the terminal
+does not cancel the service. Its journal is under the `downloadUnit` named by
+status. The helper never adds `--allowerasing` or disables signatures.
+
+Preflight requires the normal managed Btrfs root, free space on root/var/boot,
+a clean RPM database, completed current-release updates, the default current
+kernel, no pending hardware reboot, usable signed repositories, and healthy
+installed camera ABI/userspace checks. Enabled repository URLs must follow
+`$releasever` or be release independent; a URL pinned to the old Fedora release
+must be reviewed first. Do not point the running system at target-only package
+repositories. For an RPM target, the old CybexOS channel is omitted from the
+offline download and the explicitly signed target RPM is applied after boot.
+
+`cybex upgrade-system cancel` is available after a completed or failed download,
+before reboot is scheduled. It checks the saved metadata fingerprint before
+cleaning DNF's offline state. It refuses to interrupt active DNF work or delete
+an offline transaction that another operation replaced. An interrupted worker
+retains diagnostic state rather than guessing which cached transaction to erase.
+A normal reboot before scheduling the upgrade does not strand the download:
+cancel or schedule it afterwards if the installed Fedora release and the saved
+offline metadata are unchanged and no offline reboot is already scheduled.
+
+The first target boot converges the frozen source with the saved installation
+choices, or installs/reconciles the staged RPM. RPM convergence requires the
+current package's durable reconciliation status and every account to be ready;
+a successful command exit alone is insufficient. Recovery waits for mounts,
+the system bus and network availability, with login ordered after recovery.
+Failed convergence, root health,
+kernel or signing checks select the previous root and vendor desktop checkpoint
+and reboot before allowing login. Successful root checks produce
+`awaiting-desktop`, not success: the validation timer waits for an actual
+Hyprland session, then verifies the managed shell through the transaction health
+checks. A recovery bar does not count as a healthy desktop. A failed desktop
+check selects rollback and records `restartRequired`; the active session is not
+abruptly rebooted. Use `cybex upgrade-system reboot` to enter that restored root.
+
+Status is private under `/var/lib/cybexos/major-upgrade/`. Frozen release payloads
+are under `/var/lib/cybexos/major-upgrade-payloads/<id>/`; source is retained for
+reproducibility, while a committed RPM payload or explicitly cancelled payload
+is removed. The transaction journal and checkpoint are independently stored in
+the Btrfs recovery store. Personal files and settings are never reverted by the
+vendor checkpoint. The backup covers data outside that checkpoint.
+
+`tests/major-upgrade.py` exercises artifact compatibility, signature gates,
+staging, the download/reboot boundary, cancellation ownership, backup checks,
+boot failure/rollback, deferred desktop validation and process-group cleanup
+using fixtures. It is not an end-to-end Fedora major-upgrade qualification.
+
+## Release engineer preparation
+
 The playbook supports exactly the release named by `fedora_release`; this is a
 safety boundary, not a default. Prepare and test repository support for the
 next Fedora release before upgrading the workstation. Do not change the value
@@ -21,8 +123,9 @@ branch for all compatibility changes.
 4. Make and test a backup that covers the user's home, repository checkout,
    `/etc`, `/var/lib/xps-hardware`, and `/etc/pki/akmods`. Also record
    `rpm -qa`, `flatpak list --system`, enabled repositories, and the current
-   kernel. The rollback for a failed major upgrade is restore/reinstall, not an
-   attempted mass package downgrade.
+   kernel. The guided workflow restores its pre-upgrade root/vendor checkpoint on
+   failure; the external backup and recovery media cover unsupported layouts
+   and personal data. Never attempt a mass package downgrade.
 5. Ensure the prepared target-release branch and recovery media are available
    without relying on this machine's graphical session.
 

docs/installation-parity.md

@@ -15,6 +15,7 @@ Fresh installations use these ISO defaults:
 | Docker administrator access | Sudo required |
 | Desktop automatic login | Enabled only after complete root encryption is verified |
 | Additional local-network firewall ports | Disabled; LocalSend retains its shared ports |
+| Files SMB workgroup | `WORKGROUP`; explicit per-user workgroups take precedence |
 | Machine identity | Preserve the identity already configured by Fedora/Anaconda |
 
 `inventory/group_vars/all.yml` is the common default input.
@@ -34,8 +35,9 @@ identity change in the checkout questionnaire still applies that choice.
 Neither parity nor a release update authorizes repartitioning an existing
 Fedora installation or resetting user settings to match a clean account.
 Fastfetch, Voxtype, Oh My Posh, MIME associations, and npm configuration are
-seeded only when absent, as on the ISO. Managed Fish and Kitty fragments remain
-updateable independently of those personal files.
+seeded only when absent, as on the ISO. Managed Fish, Kitty, Git and SSH fragments remain updateable while their bytes
+match the shared ownership ledger. Conflicting edits and deletions are preserved
+on both paths. Git/Kitty includes precede user values; SSH fallbacks follow them.
 
 The checkout path installs onto existing Fedora and retains source-release
 updates and uninstall; the ISO uses Anaconda for disk/account creation and RPM
@@ -65,3 +67,68 @@ revision and artifact digests in the qualification evidence. Fixture checks
 compare the installation contract; they are not evidence that fresh physical
 or VM installations have been performed. Do not use an older ISO qualification
 as evidence for a newer checkout.
+
+## Real installed-outcome gate
+
+`image/release-gate` requires two full checkout installations, `plain-us` and
+`plain-nl`, in addition to the existing four graphical ISO installations and
+prior-release RPM upgrade/recovery check. They use a checksum-pinned Fedora 44
+Cloud image, the same QEMU hardware/UEFI configuration as the ISO guests, all
+normal feature defaults, the public `./install --non-interactive` entry point,
+and real SDDM password login after reboot. `tests/fedora-vm-convergence` remains
+a separate convergence/uninstall test; its feature opt-outs cannot satisfy
+this gate.
+
+The builder includes the complete reviewed source set in `source.tar.gz`
+alongside its ISO/RPM artifacts. A canonical digest covers file names, content
+and executable bits, including intentional non-ignored working-tree changes.
+It is embedded in the ISO's existing `build.json`. The builder verifies that
+the archived content matches, so an edit during source capture aborts the
+build. Checkout qualification extracts this exact archive, validates its
+checksum and content, and requires the ISO qualification to identify that
+exact source and ISO digest. Extraction rejects links, traversal and duplicate
+paths. The runner's newer checkout cannot substitute for the tested source.
+
+Each guest produces `outcomes-fresh.json`, then saves explicit shell/input
+preferences, a valid personal Hyprland override and unknown future settings
+fields. It reapplies its own installation path, reboots, verifies those values
+survived and produces `outcomes-saved.json`. Captures require a running desktop
+and exactly one Quickshell process owned by `quickshell.service`. The managed
+Settings lifecycle test exercises Network, Sound, Online Accounts, Keyboard,
+Touchpad and Region, including watcher cleanup and the current QML journal.
+
+`image/installed_outcomes.py` compares effective shell/input settings, saved
+installer choices, login policy, actual sudo authorization, Polkit policy,
+account groups/shell, required RPM versions, Flatpaks, application commands and
+associations, service enablement/activation, firewall policy, SELinux, recovery
+support, filesystem and personal-file identities. It retains the complete RPM
+inventories. Every additional package difference needs a reasoned entry in
+`image/parity-exceptions.json`; required package/version differences always
+fail. Initial exceptions cover only the ISO delivery RPM and Cloud provisioning
+tools. Review new actual baseline differences before extending that file.
+
+`parity-fresh.json` and `parity-saved.json` name mismatches. The release gate
+embeds passed same-source checkout evidence in both plain ISO reports.
+`image/prepare-github-release` also rejects missing, stale or fixture-only
+parity evidence when invoked independently.
+
+To rerun checkout comparison against a completed candidate (its corresponding
+ISO qualification must have used `--capture-outcomes`):
+
+```sh
+image/qualify-checkout --execute-vm --scenario plain-us \
+  --artifacts /path/to/build/artifacts \
+  --iso-results /path/to/qualification/plain-us \
+  --output /path/to/new-task-specific-checkout-output
+```
+
+The runner removes task-owned VM disks, SSH keys, cloud seeds, transient logs
+and screenshots on success/failure, retaining compact reports.
+`--keep-artifacts` retains unresolved diagnostics, but never the cloud seed or
+SSH private key. Testing OS ISOs still use `/data/pxe/iso` and the existing
+checksum/iVentoy workflow. Existing Fedora hosts are never repartitioned.
+
+`image/test_installed_outcomes.py` tests content identities, archive validation,
+comparison guards and guest workflow construction without booting a VM. These
+source tests do not qualify an installation or imply the expanded matrix ran.
+New release evidence must come from executing the gate.