Let server resources hold a stop until they have saved, in contract 7

README.md

@@ -11,10 +11,11 @@ server events, and your own resources decide what to do with them.
 > **Status: proof of concept.** Settings, the panel and the integration contract may still change
 > between versions.
 
-- **Supervision.** Start, stop and restart with a graceful stop. Crashes and hangs are detected and
-  recovered from, a crash loop stops the retries (5 failures within 10 minutes, or 5 starts in a row
-  that never got ready), and FXServer's whole process tree stops with it. A server stuck for 6 minutes
-  is restarted; the delay can be raised, or hang detection turned off.
+- **Supervision.** Start, stop and restart with a graceful stop, which waits for the resources that
+  ask it to until they have saved. Crashes and hangs are detected and recovered from, a crash loop
+  stops the retries (5 failures within 10 minutes, or 5 starts in a row that never got ready), and
+  FXServer's whole process tree stops with it. A server stuck for 6 minutes is restarted; the delay
+  can be raised, or hang detection turned off.
 - **Live console.** Search and filters, commands gated by permission, secrets masked, daily log files.
 - **Resources.** Every resource of the running server with its state, live, and a start, a stop or a
   restart of each, the resources that stop with it named before you confirm.
@@ -217,6 +218,30 @@ detection back on never kills a server that was silent before. A crash is anothe
 FXServer's process exits on its own, Observer restarts it, whatever these settings say, until a crash loop
 stops the retries.
 
+### Stops that wait for your resources
+
+Before FXServer quits, Observer raises `observer:shuttingDown` in your resources, waits 5 seconds,
+then sends `quit`. A resource that saves for longer can hold the stop: it calls the bridge's
+`holdShutdown` export once, as it starts, and `releaseShutdown` each time it has saved. Observer then
+sends `quit` once every resource that holds the stop has released it. The
+[integration contract](resource/CONTRACT.md#holding-a-stop) has the two calls and each case.
+
+The wait has a limit, **Server → Wait for resources at a stop**: 60 seconds by default, from 5
+seconds to 10 minutes, the owner's to set, applied from the next stop. Past it FXServer quits
+anyway, and the console and the audit log name the resources that had not released the stop. Set
+to 5 seconds, no resource is waited for longer than before.
+
+While a stop waits, the status bar above every page names the resources it waits for and how long
+is left, and whoever may start and stop the server can have FXServer **Quit now**; the audit log
+keeps who did, with the resources left behind. The launch settings list which resources of the
+running server hold its stops.
+
+This holds for every graceful stop: a stop or a restart from the panel or the host's window, the
+schedule's restarts and the single ones someone planned, a build applied by a restart, and Observer
+stopping, for an update or by itself. It does not hold when there is no one to wait for or no time:
+after a crash, on a server that stopped responding, and when Windows closes Observer's console or
+ends the session, which leaves a few seconds only.
+
 ## systemd
 
 `observer service print-unit` prints a unit that runs this binary with this configuration, as the
@@ -238,7 +263,9 @@ The unit:
   update (`RestartForceExitStatus=75`, `SuccessExitStatus=75`); a unit printed before updates
   existed restarts it too, as a failure;
 - on stop, sends SIGTERM to Observer alone, which stops FXServer gracefully, then the rest of the
-  group (`KillMode=mixed`, `TimeoutStopSec=45`);
+  group (`KillMode=mixed`, `TimeoutStopSec=45`). While resources hold the stop, Observer asks
+  systemd for the time it waits (`EXTEND_TIMEOUT_USEC`, systemd 236 or later), so a unit already
+  installed needs no change; an older systemd ends the stop after its 45 seconds;
 - makes `/usr`, `/boot`, `/efi` and `/etc` read-only (`ProtectSystem=full`), with a private `/tmp`:
   keep Observer's folder and the server's elsewhere, such as `/srv` or a home folder.
 
@@ -981,7 +1008,8 @@ Observer can back up the game server's database, the one oxmysql or mysql-async
   are copied as the dump reaches them.
 - **When**: before the schedule's restarts and the single restarts someone planned (the default), or
   before every restart; at fixed times, in the restart schedule's zone; and on request. The backup
-  before a restart is dumped once FXServer has stopped, when resources have saved what they held,
+  before a restart is dumped once FXServer has stopped, when resources have saved what they held
+  and released the stop if they hold it,
   and FXServer waits for it one minute at most by default (up to ten): past that it starts anyway,
   the dump goes on, and the run says so. Copies go out while FXServer runs. A crash restart never waits
   for a backup.
@@ -1015,6 +1043,10 @@ an event, an export or one of txAdmin's events by its name. Any account reads it
 see the dashboard, it also says what this server does now: the contract's version, the name the bridge
 took for this start, and whether txAdmin's events are raised, now and from the next start.
 
+A resource that saves when the server stops can hold the stop until it has saved, through the
+bridge's `holdShutdown` and `releaseShutdown` exports: see
+[Stops that wait for your resources](#stops-that-wait-for-your-resources).
+
 Your resources can also give each connected player a label, a roleplay character's name or an ID of
 your own, through the bridge's `setPlayerLabel` export. The panel shows it beside the player's FiveM
 name, marked as your resources' word, keeps each label on the player's profile as long as their names,

crates/fake-fxserver/src/main.rs

@@ -87,6 +87,14 @@
 //!   sweep costs for each entity and for listing them, in microseconds, which cuts a sweep short
 //!   past its budget and spaces the next ones out; `set gamename rdr3` and `onesync off` each
 //!   leave the server without entities to list;
+//! - `fake_holders <names>`, a list separated by commas: each of them asks the bridge to hold the
+//!   stop as it starts, as a resource calling `holdShutdown`, and the bridge reports who holds once
+//!   it is ready and whenever that changes. Once the bridge raised `observer:shuttingDown`, each
+//!   holder releases the stop `fake_release_ms` later, or `fake_release_ms:<name>` for one of them,
+//!   and never without either: a resource that forgot its `releaseShutdown`. On stdin,
+//!   `fake_hold <resource>` and `fake_release <resource>` are the two calls at the moment they are
+//!   typed. A holder that is stopped holds nothing any more, and a bridge started again holds
+//!   nothing of what the one before held;
 //! - on stdin: `fake_debug <resource> <list> <fields...>`, one row of a hook's report (`ev`, `np`,
 //!   `un`, `sb`, `sbl` or `http`) printed as that resource;
 //! - on stdin: `fake_crash [code]`, `fake_hang`, `fake_spin` (a busy loop), `fake_print <text>`,
@@ -114,7 +122,7 @@ mod resources;
 #[allow(unsafe_code)]
 mod traces;
 
-use std::collections::HashMap;
+use std::collections::{BTreeSet, HashMap};
 use std::io::{self, BufRead};
 use std::net::{TcpListener, TcpStream, UdpSocket};
 use std::path::{Path, PathBuf};
@@ -143,6 +151,8 @@ const SV_MAIN: &str = "UV loop: svMain";
 const RESOURCE_EVENTS_MAX: usize = 64;
 /// How long the bridge waits after a start or a stop before it reports, unless its script says.
 const RESOURCES_SETTLE_MS: u64 = 500;
+/// The bridge's `HOLDS_MAX`: the most resources that hold a stop at once.
+const HOLDS_MAX: usize = 64;
 
 /// What the HTTP threads share with the main loop.
 pub struct Server {
@@ -301,6 +311,10 @@ struct Fake {
     lists: u64,
     /// Every resource started, in order.
     started: Vec<String>,
+    /// The resources that hold the stop, as the bridge keeps them.
+    holds: BTreeSet<String>,
+    /// Once the bridge raised `observer:shuttingDown`: when each holder releases the stop by itself.
+    stopping: Option<Vec<(Instant, String)>>,
     quit_at: Option<Instant>,
     /// What `fake_spawn` started, reaped as they end.
     children: Vec<Child>,
@@ -421,6 +435,8 @@ fn main() {
             unreported: Unreported::default(),
             lists: 0,
             started: Vec::new(),
+            holds: BTreeSet::new(),
+            stopping: None,
             quit_at: None,
             children: Vec::new(),
             connections: Vec::new(),
@@ -536,6 +552,9 @@ impl Fake {
             self.report_resources(now);
             self.report_entities(now);
             self.beat(now);
+            if self.release_due(now) {
+                self.report_holds(&[]);
+            }
             self.count(now);
             self.server.tick();
             thread::sleep(TICK);
@@ -616,6 +635,8 @@ impl Fake {
                 }
             }
             ("fake_label", Some(id)) => self.label(id, &args[1..]),
+            ("fake_hold", Some(resource)) => self.hold(resource),
+            ("fake_release", Some(resource)) => self.release(resource),
             ("fake_drop_before_join", Some(id)) => self.drop_before_join(id),
             ("observer_cmd", Some(hex)) if self.bridge.is_some() => self.bridge_command(hex),
             // A convar's name followed by a value sets it, as FXServer's console does for a convar
@@ -709,11 +730,7 @@ impl Fake {
         }
         console::line(RESOURCES, &format!("Stopping resource {resource}"));
         self.resource_event(resource, "stop");
-        if self.bridge.as_ref().is_some_and(|bridge| bridge.name == resource) {
-            self.bridge = None;
-            self.listing = None;
-            self.unreported = Unreported::default();
-        }
+        self.stopped(resource);
     }
 
     /// FXServer's `restart`: a stop then a start, of a started resource only.
@@ -787,8 +804,13 @@ impl Fake {
         if !self.started.iter().any(|started| started == resource) {
             self.started.push(resource.to_owned());
         }
+        if listed("fake_holders") {
+            self.hold(resource);
+        }
         let script = self.bridge_script(resource);
         if let Some(contract) = script.as_deref().and_then(contract_version) {
+            self.holds.clear();
+            self.stopping = None;
             let settle = script
                 .as_deref()
                 .and_then(|script| constant(script, "RESOURCES_SETTLE_MS"))
@@ -881,6 +903,11 @@ impl Fake {
                 &bridge.name,
                 &json!({ "type": "observer", "kind": "resources", "started": self.started }),
             );
+            let mut holds = json!({ "type": "observer", "kind": "holds", "stopping": false });
+            if !self.holds.is_empty() {
+                holds["holding"] = json!(self.holds);
+            }
+            self.traces.script(&bridge.name, &holds);
             let name = bridge.name.clone();
             self.report_resource_list();
             let roster = self.server.players().roster(self.server.uptime_ms());
@@ -906,6 +933,86 @@ impl Fake {
         }
     }
 
+    /// The bridge's `holdShutdown`, called by `resource`: nothing without the bridge to call.
+    fn hold(&mut self, resource: &str) {
+        let full = self.holds.len() >= HOLDS_MAX;
+        if self.bridge.is_none() || full || !self.holds.insert(resource.to_owned()) {
+            return;
+        }
+        self.report_holds(&[]);
+    }
+
+    /// The bridge's `releaseShutdown`, called by `resource`: nothing while no stop is under way.
+    fn release(&mut self, resource: &str) {
+        if self.stopping.is_some() && self.holds.remove(resource) {
+            self.report_holds(&[]);
+        }
+    }
+
+    /// A resource stopped: the bridge forgets its hold, and the bridge itself forgets them all.
+    fn stopped(&mut self, resource: &str) {
+        if self.bridge.as_ref().is_some_and(|bridge| bridge.name == resource) {
+            self.bridge = None;
+            self.listing = None;
+            self.unreported = Unreported::default();
+            self.holds.clear();
+            self.stopping = None;
+            return;
+        }
+        if self.holds.remove(resource) {
+            let gone = [resource.to_owned()];
+            self.report_holds(if self.stopping.is_some() { &gone } else { &[] });
+        }
+    }
+
+    /// The bridge raised `observer:shuttingDown`: every holder's handler runs, and one that takes
+    /// no time has released the stop before the bridge reports who holds it.
+    fn shutting_down(&mut self) {
+        let now = Instant::now();
+        if self.stopping.is_none() {
+            let releases = self
+                .holds
+                .iter()
+                .filter_map(|resource| {
+                    let own = self.server.number(&format!("fake_release_ms:{resource}"));
+                    let after = own.or_else(|| self.server.number("fake_release_ms"))?;
+                    Some((now + Duration::from_millis(after), resource.clone()))
+                })
+                .collect();
+            self.stopping = Some(releases);
+        }
+        self.release_due(now);
+        self.report_holds(&[]);
+    }
+
+    /// Releases the stop for every holder whose time has come, and says whether one did.
+    fn release_due(&mut self, now: Instant) -> bool {
+        let Some(pending) = &mut self.stopping else {
+            return false;
+        };
+        let (due, later): (Vec<_>, Vec<_>) = std::mem::take(pending)
+            .into_iter()
+            .partition(|(at, _)| now >= *at);
+        *pending = later;
+        let mut released = false;
+        for (_, resource) in due {
+            released |= self.holds.remove(&resource);
+        }
+        released
+    }
+
+    /// The bridge's report of who holds the stop, an empty list left out as the bridge leaves it.
+    fn report_holds(&self, stopped: &[String]) {
+        let mut report = json!({ "type": "observer", "kind": "holds", "stopping": self.stopping.is_some() });
+        if !self.holds.is_empty() {
+            report["holding"] = json!(self.holds);
+        }
+        if !stopped.is_empty() {
+            report["stopped"] = json!(stopped);
+        }
+        self.bridge_report(&report);
+    }
+
     fn quit(&mut self, reason: Option<&str>) {
         if self.server.flag("fake_ignore_quit") || self.quit_at.is_some() {
             return;
@@ -1296,14 +1403,19 @@ impl Fake {
             }
             Some("serverName") => self.server_name(payload["name"].as_str()),
             Some(_) => {}
-            None => console::script(
-                &bridge,
-                &format!(
-                    "observer:{} {}",
-                    command["event"].as_str().unwrap_or("?"),
-                    payload
-                ),
-            ),
+            None => {
+                console::script(
+                    &bridge,
+                    &format!(
+                        "observer:{} {}",
+                        command["event"].as_str().unwrap_or("?"),
+                        payload
+                    ),
+                );
+                if command["event"].as_str() == Some("shuttingDown") {
+                    self.shutting_down();
+                }
+            }
         }
     }
 

crates/fake-fxserver/tests/artifacts.rs

@@ -241,6 +241,7 @@ async fn seed(db: &Database, servers: &Servers, owner: &Caller, data: &Path) {
         }],
         hang_detection: true,
         hang_timeout_seconds: 360,
+        stop_hold_seconds: 60,
     };
     servers
         .change_launch_settings(owner, Servers::DEFAULT, launch, None)
@@ -294,6 +295,7 @@ async fn supervise(
                 quit: Duration::from_secs(3),
                 shortened: Duration::from_secs(1),
                 fatal_exit: Duration::from_secs(1),
+                hold: None,
             };
             settings.recovery = RecoveryPolicy {
                 backoff: vec![Duration::from_millis(500)],

crates/fake-fxserver/tests/backups.rs

@@ -219,6 +219,7 @@ fn launch(data: &Path) -> LaunchSettings {
         extra_arguments: Vec::new(),
         hang_detection: true,
         hang_timeout_seconds: 360,
+        stop_hold_seconds: 60,
     }
 }
 

crates/fake-fxserver/tests/changes.rs

@@ -97,6 +97,7 @@ impl Lab {
                 quit: Duration::from_secs(3),
                 shortened: Duration::from_secs(1),
                 fatal_exit: Duration::from_secs(1),
+                hold: None,
             };
         })
         .await
@@ -299,6 +300,7 @@ async fn configure(db: &Database, clock: &Clock, data: &Path) -> (Accounts, Serv
         extra_arguments: Vec::new(),
         hang_detection: true,
         hang_timeout_seconds: 360,
+        stop_hold_seconds: 60,
     };
     servers
         .change_launch_settings(&owner, Servers::DEFAULT, launch, None)