Skip to content

Let server resources hold a stop until they have saved, in contract 7 - #55

Merged
ImBaphomettt merged 3 commits into
mainfrom
shutdown-hold
Nov 11, 2026
Merged

ImBaphomettt merged 3 commits into
mainfrom
shutdown-hold

Conversation

@ImBaphomettt

@ImBaphomettt ImBaphomettt commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Before a stop, Observer raised observer:shuttingDown, waited a fixed 5 seconds, then sent quit. A resource that saves for longer, such as inventories, vehicles or its own data, was cut off halfway through its saves. Server resources can now hold the stop: a resource asks once, as it starts, and releases each stop once its saves are done. Observer sends quit once the 5 seconds have passed and every resource that holds the stop has released it, and at the owner's limit at the latest, naming who did not release it. A resource that never asks changes nothing.

local bridge = GetConvar('observer_bridge', '')
exports[bridge]:holdShutdown()

AddEventHandler('observer:shuttingDown', function(payload)
    -- your saves go here; they may wait on your database
    exports[bridge]:releaseShutdown()
end)

This branch holds main up to #60: contract 7 follows #58's 5 and #60's 6, and its migration is 0022.

What changed

  • Contract 7. The bridge has two new exports, holdShutdown and releaseShutdown, and observer:shuttingDown a new field, maxDelayMs: when quit comes whoever still holds the stop. resource/CONTRACT.md has a new section, "Holding a stop": what the exports return (true, or false with tooMany, invalid, notStopping or notHeld), how long a hold lasts, and a table of what waits for the resources and what does not. The example resource holds each stop for as long as its observer_example_save_ms convar says, 0 by default, and releases it even when its save fails.
  • Bridge. It keeps the resources that hold the stop by GetInvokingResource(), 64 at most, and forgets a resource's hold when the resource stops; a bridge started again holds nothing. It reports the whole list: once after ready, grouped within 500 ms before a stop, and at once during one. Right after it raises observer:shuttingDown, it reports who still holds the stop, once every handler has run up to its first wait.
  • Supervisor. The notice now ends at the later of its 5 seconds and the last release, and at the limit at the latest; past the limit, the console names the resources that did not release the stop, and quit follows as before. A stop that overtakes another (a stop during a restart, Observer stopping during a scheduled restart) keeps the first one's wait and limit, and its observer:shuttingDown gives what is left of both. A crash, a hang, a quit typed in the console and a closing Windows console wait for no one, and the console says who was left when a resource held the stop. The names the bridge reports lose their control characters and are cut at 64 characters before anything shows them.
  • The limit. A new launch setting, the owner's: Settings → Launch → Stop, "Wait for resources at a stop", 60 seconds by default, from 5 to 600. It applies from the next stop and is audited before and after like the other launch settings. Migration 0022_stop_hold.sql adds the column, with its default and its check. At 5 seconds, no resource is waited for longer than before.
  • Quit now. POST /api/servers/default/stop/now, behind the server control permission, has the stop under way go on without the resources that still hold it; the 5 seconds still run to their end. With no resource holding a stop, it answers stop_not_held.
  • Audit. server_stop_hold_expired, by the system, a failure: the resources that held the stop past their limit, or when Windows closed the console, the reason and how long Observer waited. server_stop_hold_skipped: who had FXServer quit without waiting, and the resources left behind.
  • Panel. While a stop waits, the status bar above every page reads "Stopping · for 14 s · waiting for 2 resources", and a line under it names the resources, says when FXServer quits at the latest and has Quit now, with a confirmation, for whoever may stop the server. The Stop and Restart confirmations name the resources that hold each stop. Settings → Launch lists which resources of the running server hold its stops.
  • systemd. While a resource holds the stop, Observer sends EXTEND_TIMEOUT_USEC every 5 seconds, and for 40 seconds after the wait, so a unit already installed with TimeoutStopSec=45 needs no change (systemd 236 and later). A stop no resource holds asks systemd for nothing, as before.
  • Fake FXServer. fake_holders <names>, fake_release_ms and fake_release_ms:<name> in the cfg, and fake_hold <resource> and fake_release <resource> on stdin: resources that hold each stop and release it after a time, or never.
  • README. "Stops that wait for your resources", and the systemd, Integrations and database backups sections.

Why

A fixed delay cannot fit every server: a big one saves for longer than 5 seconds, and data cut off halfway is lost or corrupted. txAdmin has nothing more than the same fixed delay (shutdownNoticeDelayMs). Any server resource can hold a stop, as any can give a label: the owner's limit bounds what one can cost, and Observer names every resource that holds one.

Notes for review

Verified

  • On Windows, on e767c21, this branch with main merged in: cargo xtask check in one run, under Node 26 (the CI's Node 24 is not on this host): fmt, icons, clippy -D warnings with and without observer/test-catalog, the workspace's 2,986 tests, bindings unchanged, then in web/ format, lint, type check, 960 tests and the build. The merge commit alone passed cargo test --workspace --no-fail-fast (2,986 tests, none failed), the panel's format, lint, type check, 960 tests and build, and cargo xtask bindings, whose output differs from main only by this branch's two new types and five changed ones.
  • The CI's browser flows on e767c21, run as cargo xtask e2e runs them but on the Chromium already on this host (the build with observer/test-catalog, Playwright, ports of their own): 12 passed, among them FXServer restarted, stopped and started from the top bar and Kick a player or everyone, and raise announcements and direct messages for the server's resources, in contract 5 #58's kick from the players column; the 2 that run on Linux only were skipped.
  • Against the fake FXServer, in crates/fake-fxserver/tests/supervisor.rs, one test per case: a stop held then released; a stop no resource holds, as before; a resource that never releases, named at the limit; the schedule's restarts and a single restart someone planned; Observer stopping, by itself and in the place of a scheduled restart; a new launch applied by a restart; a crash and a hang, which wait for no one; a closing console, before and during the wait; a stuck main loop, waited for until the limit, then killed; a resource that stops or restarts; the bridge started again; a stop that overtakes a restart; Quit now and its refusal; a resource that asks during the notice; a typed quit; the owner's limit and its change. runtime.rs: the audit of a stop past its limit and of a skipped wait, and the status the panel reads. pre_launch.rs: the database backup before a restart begins once the resources released the stop and FXServer exited.
  • Unit tests: the contract documents the exports, refusals and fields the bridge checks; the holds report; the names cleaned; the limit's range; the migration from 0021; the launch audit; when Observer asks systemd for time; and in the panel the status line, Quit now and its refusal, the setting's save and range, the live status validator and the audit details. Show the integration contract in the panel, on a Documentation page drawn from the file the binary ships #59's contract tests pass on contract 7: every event has its section and its row, every export is shown called, the example listens to every event.
  • In a lab set up afresh on the merge (the fake FXServer as build 35245, migrations up to 0022 applied in order, two resources that hold each stop, one released after 6 seconds): the status card with its wait, its countdown and Quit now, at 1440 and 375 px in both themes, its controls and the line of the wait measured within the card at 375 px (317 px of 317); the same card online, with Kick a player or everyone, and raise announcements and direct messages for the server's resources, in contract 5 #58's menu and Let in approved players only, and close the server for maintenance to all but a list, under another name in the server list, in contract 6 #60's Maintenance; Quit now's confirmation, then the stop, and server_stop_hold_skipped in the audit log with the resource left behind; Settings → Launch with the limit and the resources that hold each stop; the Documentation page, which shows contract 7, its section "Holding a stop" and maxDelayMs in the card of observer:shuttingDown. No console error and no sideways scroll on any of them.
  • The Lua of the bridge and of the example parses after the merge (lua-language-server, Lua 5.4; a syntax error planted in a copy is caught): syntax only.
  • GitHub Actions does not start the jobs at the moment (the organisation's billing): everything above ran locally.

Not verified

  • Nothing ran against a real FXServer: the Lua of the bridge and of the example ran in no interpreter. FiveM's sources say that TriggerEvent runs every handler before it returns (EventScriptFunctions.cpp, ResourceEventComponent.cpp), that an export returns every value to a Lua caller (scheduler.lua) and that GetInvokingResource() names the calling resource (ResourceScriptFunctions.cpp); none of it was exercised here, nor onResourceStop dropping a hold during a stop.
  • Linux: crates/observer/src/systemd.rs was not compiled, as no WSL distribution or cross C compiler was available, and no real systemctl stop was held past 45 seconds. When Observer asks systemd for time is tested on Windows; the lines that send the ask are not. main did not touch the file, so the merge left it as it was.
  • A real Windows console closing; the panel at 1024 px and 320 px.
  • The host's status window shows the stop but does not name who holds it, and observer fxserver run has no option for the limit: it waits 60 seconds.

# Conflicts:
#	README.md
#	crates/fake-fxserver/src/main.rs
#	crates/fake-fxserver/tests/runtime.rs
#	crates/observer-domain/src/audit.rs
#	crates/observer-fxserver/src/bridge.rs
#	crates/observer-fxserver/src/supervisor.rs
#	crates/observer-store/src/migrations.rs
#	crates/observer-web/src/error.rs
#	resource/CONTRACT.md
#	resource/observer/server.lua
#	resource/observer_example/server.lua
#	web/src/api/generated/AuditAction.ts
#	web/src/api/generated/ErrorCode.ts
#	web/src/audit/details.ts
#	web/src/shell/ServerControls.test.tsx
#	web/src/shell/ServerControls.tsx
…od of its own, for the launch to stay within clippy's length
@ImBaphomettt
ImBaphomettt merged commit 5bf5f39 into main Nov 11, 2026
2 of 3 checks passed
@ImBaphomettt
ImBaphomettt deleted the shutdown-hold branch November 11, 2026 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant