Skip to content

Add durable maintenance prepare and resume operations #77

Description

@thymikee

Part of #70. See ADR 0021.

Purpose

Add a durable maintenance mode that stays active after a process restart.

The current reset flow uses an in-memory fence. It releases leases, resets devices, and then reopens admission.

Host needs one prepare operation and one resume operation. Host must not call Simlock's internal cleanup steps.

Client API

type MaintenanceState =
  | "preparing"
  | "drained"
  | "resuming"
  | "awaiting-external-fence"
  | "ready"
  | "failed";

interface MaintenanceStatus {
  epoch: string;
  state: MaintenanceState;
  activeLeaseCount: number;
  activeMutationCount: number;
  activation?: {
    deviceId: string;
    generation: number;
  };
  error?: SimlockError;
}

interface SimlockAdminClient {
  prepareMaintenance(input: { epoch: string }): Promise<MaintenanceStatus>;
  getMaintenanceStatus(input: { epoch: string }): Promise<MaintenanceStatus>;
  confirmMaintenanceActivation(input: {
    epoch: string;
    deviceId: string;
    generation: number;
  }): Promise<MaintenanceStatus>;
  resumeMaintenance(input: { epoch: string }): Promise<MaintenanceStatus>;
}

Prepare operation

prepareMaintenance must do these steps:

  1. Store the epoch and close lease and background-lifecycle admission in one serialized durable transition.
  2. Stop every background lifecycle producer.
  3. Release all leases.
  4. Wait for lifecycle mutations to finish.
  5. Reconcile only owned devices.
  6. Return drained with both active counts at zero.

Read-only catalog and status operations can stay available.

Resume operation

resumeMaintenance is a durable loop:

  1. Load the stored epoch.
  2. Complete startup reconciliation.
  3. Advance the configured warm target from Add configured warm targets and convergence status #76 through its single core convergence owner.
  4. If a reusable identity needs an execution fence, persist and return awaiting-external-fence with its device ID and generation. Keep Simlock drained.
  5. Use the active epoch to confirm the same core activation transition that Add configured warm targets and convergence status #76 defines.
  6. After idempotent confirmation, repeat resumeMaintenance for the same epoch and continue convergence.
  7. Return ready and reopen lease admission only after warm convergence settles.

Epoch rules

  • Repeating an operation for the same epoch returns the same progress or result.
  • A different epoch returns MAINTENANCE_EPOCH_ACTIVE without a mutation.
  • A stale resume or activation message cannot change a newer epoch or generation.
  • A non-epoch warm mutation cannot advance an identity while the epoch is active.
  • A client disconnect stops only that client's wait.
  • A daemon restart keeps the epoch active and lease admission closed.
  • The epoch has no TTL. Failure or supervisor death keeps Simlock drained.
  • Simlock can clear the active epoch only after resumeMaintenance returns ready.
  • Keep the terminal result for a configured recovery period so a lost response can be replayed.

Scope

Host owns session shutdown, artifact cleanup, and process restart order. Simlock owns allocator drain and device recovery.

The Host front-end must not expose these methods. Only the local managed adapter can call them through the Host admin boundary and the authenticated admin client from #71.

The adapter can call prepareMaintenance only while it holds an unforgeable in-process capability. Host creates this capability after it fences and drains device mutations, completes canonical session teardown, releases mapped Simlock leases, and proves zero live runner. Simlock receives only the epoch.

Completion conditions

  • Prepare cannot admit a lease or background mutation after it stores the epoch.
  • Prepare cannot return drained while a lease or lifecycle mutation exists.
  • Background refill, health recovery, quarantine retry, and the cleanup reaper cannot start work after the fence.
  • Restart during maintenance does not reopen lease admission.
  • Repeated prepare and resume calls do not repeat destructive work.
  • Cleanup cannot target a leased, foreign, or unverified device.
  • Resume stays drained while an external fence is pending.
  • Non-epoch convergence and activation fail without mutation while maintenance is active.
  • Resume reopens admission only after reconciliation, warm convergence, and required fence confirmation.

Depends on #71, #73, and #74. Reusable-identity activation also depends on #72. Warm convergence depends on #76.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions