Observer

A server manager for FiveM and RedM, written in Rust.

Observer starts and supervises FXServer, installs its builds and serves a web panel for your staff, from one executable that keeps its state in SQLite. It never shows anything to players, but for what you write yourself: the refusal messages for bans, for maintenance and for the allowlist, the kick message, and the name your server takes in the server list during maintenance. It raises documented server events, and your own resources decide what to do with them.

Status: proof of concept. Settings, the panel and the integration contract may still change between versions.

• Supervision. Start, stop and restart with a graceful stop, which waits for the resources that ask it to until they have saved. Crashes and hangs are detected and recovered from, a crash loop stops the retries (5 failures within 10 minutes, or 5 starts in a row that never got ready), and FXServer's whole process tree stops with it. A server stuck for 6 minutes is restarted; the delay can be raised, or hang detection turned off.
• Live console. Search and filters, commands gated by permission, secrets masked, daily log files.
• Resources. Every resource of the running server with its state, live, and a start, a stop or a restart of each, the resources that stop with it named before you confirm.
• Scheduled restarts. Fixed times or an interval in the time zone you choose, with warnings to your resources, skip, delay and one-off restarts.
• FXServer builds. The Cfx.re catalog, verified downloads, a switch at the next start or restart, probation, and a return to the last healthy build.
• Statistics and diagnostics. Players, tick times, CPU and memory of FXServer and the host, FXServer's threads and connections, crash dumps and hangs.
• Live map. Where the connected players stand, over Rockstar's map of GTA V or an image you add, coloured by play time, a click away from their profile, a message, a kick and a ban.
• Entities. The server's vehicles, peds and objects, each with the resource that created it and the player whose game simulates it, found on the live map and deleted one at a time or a hundred.
• Debug. What each resource sends and receives, writes into state bags and calls over HTTP, counted inside the resources: the event that floods the clients, the client that sends far more than the others, the resource that calls a host it never called. Off until the owner turns it on.
• Bans. A ban holds a player's licence, other identifiers and hardware tokens, for a time or for good. The server refuses a banned player at connection with your message, and drops one banned while playing.
• Kicks, announcements and messages. The staff kicks a connected player, or everyone, with a reason and your kick message. An announcement to the server and a message to one player are server events, which your resources show.
• Allowlist. Open the server to approved players only: a refused player reads your message with the ID of their request, which the staff approves or denies in the panel.
• Maintenance mode. One switch closes the server to everyone but the players you list, with your message, through restarts, until someone ends it.
• Accounts and audit. An owner and sub-users with fine-grained permissions, and an audit log of every action.
• Updates. A newer release shows in the panel with its notes; the owner installs it in one click, now or at the next scheduled restart, and Observer restarts into it, or back into the previous version if the new one does not start.

Install on Windows

Windows 10 or Windows Server 2016 and later, x64.

1. From the release, download observer-<version>-windows-x86_64.zip and SHA256SUMS, and check the archive: Get-FileHash .\observer-<version>-windows-x86_64.zip must print the hash written next to its name in SHA256SUMS.
2. Extract it into a folder of its own, such as C:\Observer. It holds observer.exe and THIRD-PARTY-LICENCES.txt. Keep the path short: FXServer fails on paths over 260 characters, and its deepest files sit 125 characters below the build folders Observer keeps under data\artifacts\.
3. Start observer.exe from that folder. Observer reads observer.toml from the folder it is started from, or the file --config names, and keeps data\ next to it: from a terminal elsewhere, run C:\Observer\observer.exe --config C:\Observer\observer.toml.

The executable is not signed: on its first run, SmartScreen says "Windows protected your PC". Choose More info, then Run anyway.

Observer then shows three things:

• The console window, with Observer's log. Ctrl+C in it stops FXServer gracefully, then Observer. Closing it stops both too, but Windows leaves only a few seconds, so FXServer quits at once.
• The status window: the server's state, players, build, next restart and the panel's address, with Start, Stop, Restart, Open the panel and Open the data folder. Until the owner account exists, it also holds the setup link. Closing the window leaves Observer running.
• The tray icon, whose menu reopens the window and has the same controls. Quit Observer… asks for confirmation, then stops FXServer and Observer.

The window and the tray sign nobody in: whoever uses the Windows session already holds the data folder, and with it every account. The audit log shows their actions as Host window, and the events your resources receive name them local.

observer.exe run --no-window starts Observer without the window and the tray, for a host where nobody watches the desktop. The panel listens on every interface by default: allow Observer through the firewall only on the networks your staff use, or see HTTPS.

Install on Linux

x86_64 with glibc 2.35 or later (Ubuntu 22.04, Debian 12 and later).

sha256sum -c --ignore-missing SHA256SUMS      # where the archive and SHA256SUMS were downloaded
mkdir -p ~/observer
tar -xzf observer-<version>-linux-x86_64.tar.gz -C ~/observer
cd ~/observer
./observer

The archive holds observer and THIRD-PARTY-LICENCES.txt. Run Observer as the account that owns the server folder, not as root: FXServer runs as the same account. As on Windows, observer.toml is read from the folder Observer is started from, unless --config names it, and data/ sits next to it.

./observer is short for ./observer run. It runs in the foreground: Ctrl+C, SIGTERM and SIGHUP (a closed SSH session) stop FXServer gracefully, then Observer. A server that must stay up runs under systemd.

First run

On its first start, Observer prints the link that creates the owner account:

Create the owner account within 30 minutes, from this link:
http://localhost:40125/setup#<token>

• The link goes to the terminal only, never to the log files: whoever holds it can take the server. On Windows, the status window shows it too, with Open the setup page and Copy the link.
• It works once, within 30 minutes. While no account exists, observer setup-link prints a new one once Observer has started a first time, even while it runs. Under systemd nobody sees the terminal, so it is the only way there.
• When Observer listens on every interface, the link names localhost, which only a browser on the host itself reaches. From another machine, put the host's address in its place and keep the rest: http://192.0.2.10:40125/setup#<token>.
• The API answers to addresses and localhost. A name such as panel.example.com must be listed in http.hosts: until it is, the panel says Observer does not answer to this name.

Once the owner exists, the panel opens the server setup. Other accounts are added on the Users page, each with a temporary password that works for 72 hours and is changed at the first sign-in.

The panel's pages are grouped by what one comes for: Players (the players, the live map, the bans), Server (restarts, the cfg, builds, backups, file changes), Insights (statistics, drops, entities, diagnostics) and Maintenance (tools, the audit log). The search button beside the logo, or Ctrl K (⌘ K on a Mac) from any page, opens a search of the pages, of the settings by their name, of the events and exports the Documentation page describes (see Integrations), and of the connected players by name or ID. Each account finds only what it may open.

From the host, even while Observer runs:

observer user list                    # accounts, roles and permissions
observer user reset-password <name>   # a temporary password, and the account's sessions end; the owner too
observer user unlock <name>           # lift the login locks on an account

The server setup page

Observer runs an existing server in place. The setup opens by itself for the owner while nothing is configured, and again from Server → Run the setup. It has four steps:

1. Server folder. The full path on the host of the folder that holds resources and the cfg, such as C:\FXServer\server-data or /srv/fxserver/server-data. Network paths are not supported. FXServer runs from this folder. Observer copies nothing into it, and writes there only what people upload over SFTP, the fixes of the server tools the owner runs or turns on, which the Tools page lists each with its backup, and the cfg when the owner saves it from the server.cfg page.
2. Config file. The cfg FXServer executes, relative to the folder, such as server.cfg. The setup reads it and the cfgs it runs in turn, never changes them, and lists what it finds with file and line: blocking problems, warnings and notes, and what the cfg sets (game, endpoints, slots, OneSync, hostname, whether a licence key is set). Fix your files, then Check again; only a blocking problem stops the setup.
3. Build. Install the build Cfx.re recommends, or import a folder of FXServer you already have. On Windows, FXServer.exe must carry a valid Rockstar Games signature. Skip for now leaves it to the Artifacts page.
4. Start. OneSync, Start FXServer with Observer, Send txAdmin's events too, and a summary. Save and start (Save and restart when FXServer runs) saves the settings and starts FXServer, and the First launch checklist follows it: started, port open, licence accepted, Observer bridge connected, server ready. Without a build, Save only keeps the settings for later.

The Server page holds the same settings, hang detection, and convars that go on the command line as +set <name> <value>. A name field suggests the convars FXServer declares, from a catalog read in its sources, and a convar picked there starts at its default; one the catalog does not know is passed as typed. FXServer applies every +set before it runs the cfg, wherever it stands on the command line, so a convar the cfg sets too keeps the cfg's value, OneSync's included: the page says where the cfg sets it and which value applies, and lists what only the cfg sets. It refuses secrets on the command line, since anyone on the host can read it: sv_licenseKey, passwords and connection strings belong in the cfg.

Builds go to data/artifacts/<build>/. The Artifacts page lists the catalog, installs and imports builds, chooses the one FXServer runs, and cleans up old ones. A new build is on probation: until it has stayed up 10 minutes once ready, a crash or a hang sends FXServer back to the last healthy build. A server whose heartbeat or HTTP is late when those minutes end passes only once it answers again, so a hang that began within them fails the probation however long the hang delay. With hang detection off, only a crash fails it, and a silent server still waits to answer before it passes.

Following a start

While FXServer starts, the Overview shows how far it has come: how many resources are up, the command of the cfg it is at (ensure [clothes], server.cfg line 226, 12 of its 150 resources up), the last resource started and how long ago, the resources that did not start, and the five slowest so far. The status bar above every page counts them too.

Observer reads this from FXServer's own console: FXServer prints Started resource <name> once a resource is up, and nothing while it starts one. The command it is at is worked out from the resources each ensure and start of the cfg starts, a category being every resource under the folder of that name. When Observer cannot read every command of the cfg, as when it executes a file inside a resource, the page says so rather than guess.

Once no resource has started for half a minute, the card says how long Observer still waits before it takes the start for stuck (see hang detection below). Whoever may start and stop the server can give that start 15 minutes more, as often as needed; the audit log keeps who did. The hang delay itself is not changed.

Hang detection

Observer watches two signals while FXServer runs: the heartbeat of its bridge resource, which only comes while the main loop turns, and FXServer's HTTP endpoints. Once the server is up, one of them silent for 10 seconds marks it degraded. Silent for the delay under Server → Hang detection, 6 minutes by default, the server is hung: Observer writes a dump of it (a report of its threads on Linux), kills its process tree and starts it again.

A start gives no heartbeat: FXServer runs its cfg before any thread of a resource, the bridge's included. Until the cfg has run, each resource FXServer starts counts in its place, so a start is hung once it has started no resource for the delay, however long it takes in all, and the console names the last resource it started. One resource alone can take minutes on a big server's first start, such as a clothing pack whose files are read for the first time. Raise the delay, from 1 to 60 minutes, or turn hang detection off: FXServer is then left running however long it stays silent, until someone restarts it. Either change applies at once, to a server already starting too, and the delay then counts from the change: turning detection back on never kills a server that was silent before. A crash is another matter: whenever FXServer's process exits on its own, Observer restarts it, whatever these settings say, until a crash loop stops the retries.

Stops that wait for your resources

Before FXServer quits, Observer raises observer:shuttingDown in your resources, waits 5 seconds, then sends quit. A resource that saves for longer can hold the stop: it calls the bridge's holdShutdown export once, as it starts, and releaseShutdown each time it has saved. Observer then sends quit once every resource that holds the stop has released it. The integration contract has the two calls and each case.

The wait has a limit, Server → Wait for resources at a stop: 60 seconds by default, from 5 seconds to 10 minutes, the owner's to set, applied from the next stop. Past it FXServer quits anyway, and the console and the audit log name the resources that had not released the stop. Set to 5 seconds, no resource is waited for longer than before.

While a stop waits, the status bar above every page names the resources it waits for and how long is left, and whoever may start and stop the server can have FXServer Quit now; the audit log keeps who did, with the resources left behind. The launch settings list which resources of the running server hold its stops.

This holds for every graceful stop: a stop or a restart from the panel or the host's window, the schedule's restarts and the single ones someone planned, a build applied by a restart, and Observer stopping, for an update or by itself. It does not hold when there is no one to wait for or no time: after a crash, on a server that stopped responding, and when Windows closes Observer's console or ends the session, which leaves a few seconds only.

systemd

observer service print-unit prints a unit that runs this binary with this configuration, as the account that runs the command. Run it as the account meant to own the server: it writes nothing, and prints on stderr the commands that install the unit, with the binary's and the configuration's full paths, such as:

/home/fivem/observer/observer --config /home/fivem/observer/observer.toml service print-unit | sudo tee /etc/systemd/system/observer.service
sudo systemctl daemon-reload && sudo systemctl enable --now observer
/home/fivem/observer/observer --config /home/fivem/observer/observer.toml setup-link

The unit:

• is Type=notify: systemd counts Observer as started once the panel answers, whatever FXServer does;
• runs <binary> --config <observer.toml> run as that account, from the configuration's folder;
• restarts Observer on failure, after 5 seconds, and once it exits with 75 after installing an update (RestartForceExitStatus=75, SuccessExitStatus=75); a unit printed before updates existed restarts it too, as a failure;
• on stop, sends SIGTERM to Observer alone, which stops FXServer gracefully, then the rest of the group (KillMode=mixed, TimeoutStopSec=45). While resources hold the stop, Observer asks systemd for the time it waits (EXTEND_TIMEOUT_USEC, systemd 236 or later), so a unit already installed needs no change; an older systemd ends the stop after its 45 seconds;
• makes /usr, /boot, /efi and /etc read-only (ProtectSystem=full), with a private /tmp: keep Observer's folder and the server's elsewhere, such as /srv or a home folder.

Printed as root, it warns that Observer and FXServer would run as root.

systemctl status observer      # Status: "FXServer online · 12 / 64 players"
journalctl -u observer -f      # the log, without colour codes
sudo systemctl stop observer   # FXServer stops gracefully first

Under systemd the setup link is never printed at start: while no account exists, the log says to run observer setup-link.

Configuration

Observer reads observer.toml from the folder it is started in, or the file --config names; the option works with every command. A missing file means the defaults. An unknown key stops Observer with an error rather than being ignored. Observer reads the file at start only: restart it after a change. Everything else, such as the launch settings, the schedule and the accounts, is set in the panel and kept in the database.

Every key, with its default:

# The database, its backups, the logs and the FXServer builds. A relative path is resolved against
# the folder of observer.toml.
data_dir = "data"

[http]
# The panel's address and port. 40120 and 40122 stay free for txAdmin.
listen = "0.0.0.0:40125"
# The names the panel is reached by, without a port, such as "panel.example.com". Addresses and
# localhost always work; the API refuses any other name, which a stranger's page could point at
# this host.
hosts = []
# The reverse proxies whose X-Forwarded-For and X-Forwarded-Proto are believed: addresses and CIDR
# blocks, IPv4 or IPv6, such as "127.0.0.1" or "172.18.0.0/16". An invalid entry stops Observer
# with an error that names it.
trusted_proxies = []

[log]
# A tracing filter, such as "debug" or "info,observer_web=debug".
level = "info"

# Where SFTP listens. Without this section the owner gives the address in the panel instead; with
# it, the panel cannot change it (see "File access over SFTP").
# [sftp]
# listen = "0.0.0.0:40126"

# Optional features the host keeps off, which the panel cannot turn on (see "Turning features off").
# [features]
# sftp = "off"

Two environment variables apply to the log of observer run and observer fxserver run:

• OBSERVER_LOG replaces log.level, with a filter of the same form.
• NO_COLOR, set to anything but an empty value, turns off colour codes in the terminal. There are none when the log does not go to a terminal anyway, and never in the log files.

Reference

Data folder

data/
├── observer.sqlite3       the state: accounts, sessions, audit log, settings, schedule, builds
├── metrics.sqlite3        the statistics, never backed up
├── changes.sqlite3        what each start of FXServer ran from, never backed up
├── observer.lock          held by the Observer that uses this folder
├── backups/               observer-<UTC time>-<kind>.sqlite3
├── logs/
│   ├── observer.<date>.log           Observer's log, one file a day, the last 14 kept
│   └── console/console-<date>.log    FXServer's console, 14 days and 2 GiB at most
├── artifacts/
│   ├── <build>/           a build of FXServer
│   ├── .staging/          downloads and extractions in progress
│   └── .trash/            deleted builds on their way out
├── crashes/               FXServer's crash dumps and hang snapshots: the last 10, 20 GiB at most
├── map/                   the live map's image, as the owner gave it
├── tools/backups/         the originals of the files the server tools changed, each with a .json
├── database-backups/.work/ the game database's backup under way, emptied at every start
├── updates/               the release being installed (staging/), how the last install ended
│                          (last.json), and while one restarts Observer, handover.json
└── fxserver.pid           Linux: the running FXServer, which the next start ends if Observer died

While Observer runs, each database also has its -wal and -shm files. A console log file past 256 MiB goes on in console-<date>.<n>.log.

The backups are complete snapshots of the state database:

Kind	When	Kept
daily	at start when none is younger than a day, then once a day	7
pre-migration	before a new version changes the database's schema	3
manual	observer db backup, while Observer is stopped	10
pre-update	just before an update swaps Observer's executable	3

Ports

Port	What
40125/TCP (http.listen)	The panel, its API and its websocket (/api/ws).
40126/TCP, or the port the owner gives	File access over SFTP. Closed until the owner gives its address in the panel or [sftp] names it in observer.toml, and while SFTP is turned off.
The cfg's endpoints, such as 30120 TCP and UDP	FXServer's own. Observer reads FXServer's HTTP endpoint (/dynamic.json, /info.json, /perf/, /players.json) on the cfg's first endpoint_add_tcp, through loopback when that one listens on every interface.

Outbound connections

Observer itself talks to Cfx.re, over HTTPS on port 443, when someone lists or installs builds from the panel, the setup or observer artifacts, and to GitHub, every 6 hours and when the owner checks or installs an update, unless updates are turned off. It follows redirects only to the same hosts and ignores proxy settings. FXServer's own traffic is FXServer's. The database backups reach the database server and the SFTP hosts and S3 endpoints the owner adds as destinations, nothing else.

Host	For
docs.fivem.net	the download page: recommended and latest builds
changelogs-live.fivem.net	recommended and latest builds, end of support and end of life
runtime.fivem.net	the build listings of each branch, and the archives
api.github.com	Observer's latest release, without an account: 60 checks an hour per address
github.com, release-assets.githubusercontent.com, objects.githubusercontent.com	a release's archive, its checksums and their signature

The panel, in the browser, loads everything from Observer, with one exception: the live map loads Rockstar's map of GTA V from s.rsg.sc unless its settings say My image or Grid. Observer itself never talks to that host.

Cfx.re's Platform License Agreement covers the server builds you download.

Command line

Command
observer [run]	Run Observer in the foreground. --no-window on Windows: no status window, no tray.
observer setup-link	A new link that creates the owner, while no account exists.
observer user list	Accounts, roles and permissions.
observer user reset-password <name>	A temporary password, and the account's sessions end.
observer user unlock <name>	Lift the login locks on an account, from every network.
observer artifacts list	This platform's builds with channel, dates and support. --platform windows|linux, --branch <name> for a test branch, --all for old builds only the support policy names.
observer artifacts install <build>	Download, verify and install a build into data/artifacts/. --branch <name>, --allow-end-of-life.
observer db check	The database's integrity and schema version, while Observer is stopped.
observer db backup	A manual backup, while Observer is stopped.
observer fxserver run --build <folder> --data <folder>	FXServer under the supervisor in the foreground, without the panel or the database, while Observer is stopped. --cfg, --onesync, --port, --set NAME=VALUE, --hang-timeout <minutes> from 1 to 60 (6 by default) or --no-hang-detection, see Hang detection.
observer service print-unit	Linux: the systemd unit, see systemd.
observer tools run duplicate-archetypes	Find duplicate archetypes and print the report. --fix while Observer is stopped, --json, --data <folder>, --cfg <file> or --all-resources. Exits with 2 while duplicates are left, 1 on an error. See Server tools.
observer tools undo duplicate-archetypes	Put back what fixes took out, while Observer is stopped: --run <id> or --all, and --data <folder>.
observer tools config duplicate-archetypes	Print the tool's settings, or change them: --at-start, --differences, --escrowed, --exclude <name>, --include <name>.

The user, setup-link and artifacts commands, tools run without --fix and tools config work while Observer runs. --version names the commit a release was built from.

HTTPS

Observer serves plain HTTP and no TLS of its own. Passwords and console commands cross the network in clear text unless the panel stays on the host, sits behind a reverse proxy that terminates HTTPS, or is reached over a VPN.

• At start, when listen is not a loopback address and no proxy is trusted, Observer logs: the panel is served over plain HTTP on 0.0.0.0:40125; put it behind an HTTPS proxy or reach it over a VPN.
• Opened over plain HTTP at a name other than localhost or a name under it, such as observer.localhost, an address in 127.0.0.0/8 or [::1], the panel shows a warning that cannot be dismissed: Unencrypted connection. It goes once the page is served over HTTPS.

Behind a proxy, set three keys:

[http]
listen = "127.0.0.1:40125"           # the proxy on the same host: no direct plain-HTTP access
hosts = ["panel.example.com"]        # the public name
trusted_proxies = ["127.0.0.1"]      # the proxy, as Observer sees it

• trusted_proxies holds the proxy's address as Observer sees it: 127.0.0.1 for a proxy on the same host. When Traefik runs in Docker and Observer on the host, it is the Docker network's block, such as 172.18.0.0/16 (docker network inspect shows it); Observer must then listen on an address the container reaches, with the port closed to the outside. Trust nothing wider.
• Trusting 127.0.0.1 trusts every process on the host, FXServer's resources included (PerformHttpRequest): any of them can claim any client address, which weakens the login throttling by address and the address in the audit log. A proxy in Docker, trusted by its network's block, avoids this.
• X-Forwarded-For and X-Forwarded-Proto are read from trusted peers only, and the client's address is the first entry from the right that is not a trusted proxy. The proxy must write those headers itself, not pass on the client's. X-Forwarded-Host and Forwarded are never read.
• The proxy must pass the browser's Host unchanged: Observer accepts a change, and the websocket's opening, only from the origin <scheme>://<Host>, exactly.
• Behind HTTPS, the session cookie gets Secure, and every response carries Strict-Transport-Security: max-age=31536000, without includeSubDomains. A proxy that adds its own sends the header twice.
• A browser that saw that header reaches the name over HTTPS only, on every port, for a year. Give the panel a name of its own, which no plain-HTTP service shares: txAdmin on port 40120 under the same name would no longer open in that browser.
• The setup link names the listen address. Behind the proxy, open the same path and fragment on the public name: https://panel.example.com/setup#<token>.

Traefik

Traefik passes the original Host (passHostHeader, on by default: leave it on), sets X-Forwarded-For and X-Forwarded-Proto, and upgrades the websocket by itself. It drops the X-Forwarded-* headers a client sends unless the entry point trusts it: never set forwardedHeaders.insecure. A router in the file provider's dynamic configuration:

http:
  routers:
    observer:
      rule: Host(`panel.example.com`)
      entryPoints: [websecure]
      service: observer
      tls:
        certResolver: letsencrypt
  services:
    observer:
      loadBalancer:
        servers:
          - url: http://127.0.0.1:40125

nginx

# In the http block.
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 443 ssl;
    server_name panel.example.com;
    ssl_certificate     /etc/letsencrypt/live/panel.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/panel.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:40125;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

When the public address carries a port, such as https://panel.example.com:8443, use proxy_set_header Host $http_host;: $host drops the port, and the origin would no longer match. Observer pings the websocket every 20 seconds, well within nginx's default 60-second proxy_read_timeout. The panel opens wss:// by itself when the page is HTTPS.

Editing server.cfg

Server → server.cfg, for the owner alone, edits the cfg of the launch settings in two views of the same text.

• Guided gives a field to the lines most servers change: the server name, the project name and description, the tags, the language, the slots, the game build, the game address, the licence key and the Steam Web API key. Each field names the line it is written on, or says that a save adds one. Below them, every line that starts a resource or a category has a switch: off comments the line out, on puts it back, and the order stays the order of the file. A line that starts something neither the server folder nor the build holds is flagged.
• File is the whole text, with line numbers, and the checks of the server setup read again as you type: what blocks a start, what FXServer will skip, with the file and the line.

A save rewrites only what you changed: comments, blank lines, order, line ends and a byte order mark stay as they were. It asks for your password, applies at the next start of FXServer, and is refused when the file on disk is no longer the one you started from. See the changes shows the lines a save would change before you save.

Secrets never reach the browser. The values of sv_licenseKey, mysql_connection_string, rcon_password, sv_tebexSecret, steam_webApiKey, a Discord webhook's token and anything else the file changes hide stand in the text as <<secret-1>>, <<secret-2>>. A save puts each one back where its mark still is; typing over a mark replaces the secret. The comparisons show them as [REDACTED].

Before each save, Observer keeps the file as it was in its own database: the latest 20 versions, with who replaced each and when, secrets included as the file held them. Earlier versions compares any of them with the file and puts one back, which keeps the current file as a version too. The audit log records each save and each version put back with the number of lines added and removed, never their text.

The page edits a plain UTF-8 file of at most 512 KiB. It refuses a link, a read-only file, a file in another encoding, which a save would damage, and the files the cfg runs with exec: those are edited on the host or over SFTP, and the checks still read them.

Resources

The Resources page lists every resource FXServer knows while the server runs: the server's own under their category folder ([core]/[ox]), then those of the build (system_resources). Each has its state, its manifest's version and what last happened to it since the start: started, stopped or restarted, how long ago, by whom when it was asked on this page, and with which resource when FXServer stopped it along with another. A resource FXServer said it could not start is marked Did not start. One it warned about as it started it carries the count of its warnings, and says what they were: streamed files that are heavy or oversized, files its manifest names and does not hold. The filters count each kind, the search narrows the list by name, a folder folds under the line that counts what it has started, which the browser remembers, and the page follows the server as it goes: nothing to reload.

The list is FXServer's own. The bridge reports it once the cfg has run, and again after each rescan: while the server starts, the page says how many resources are up so far, and a stopped server lists nothing.

With Start and stop resources, each row gets Start, or Restart and Stop, and the page gets Rescan. Each is FXServer's own console command, start, stop, restart or refresh, which Observer writes with the name FXServer listed. The line shows in the live console under the account's name, and every order is in the audit log. Until FXServer has answered an order, its row says who asked for it, to everyone who has the page open, and takes no other.

• What stops with a resource. Before it stops a resource, FXServer stops every started resource that depends on it, by the dependency and dependencies of their manifests, and it starts none of them again. A restart is a stop and a start, so it does the same. The page names them before a stop or a restart that takes others along, and a restart offers to start them again once the resource is back, which Observer does with one start each.
• Start starts what the resource depends on first, as FXServer does.
• Rescan has FXServer read its resource folders again: a folder added since the start becomes a resource, a manifest that changed is read again. Nothing starts or stops, and the page says what FXServer found.
• Until the next restart. What is started or stopped here lasts as long as the run: the cfg decides what starts with the server (see Editing server.cfg).

Whoever has View the dashboard or Start and stop resources reads the page. Its buttons take the second, a permission of its own, and sensitive: a stopped resource is gone for every player.

Limits.

• Names the console cannot take. An order is a console line, so Observer only writes a name made of letters, digits, _, - and ., 64 characters at most. A resource named otherwise is listed with Console only in place of its buttons.
• Observer's bridge is not listed, and no order reaches it: it is how Observer hears the server.
• A folder deleted while the server runs stays listed until the server restarts, since FXServer keeps the resource it read. A folder whose manifest is missing or broken is no resource to FXServer, and is not listed.
• The first 5,000 resources are listed; the page says so when FXServer knows more.

File access over SFTP

Observer can let people and CI pipelines read and change chosen parts of the server folder over SFTP, the file transfer of SSH. Its port stays closed until it has an address to listen on, which the owner gives in the panel, under Settings → File access, with the password again: every address of the host, the host itself only (for a tunnel), or one of its addresses, and a port. The port opens, moves and closes at once, without a restart of Observer or of FXServer, and each change is in the audit log. A port that does not open, because another program holds it for one, never stops Observer: the panel says what the host answered, and SFTP waits until the address is given again or changed.

A host that wants the address out of the panel's reach names it in observer.toml:

[sftp]
listen = "0.0.0.0:40126"

The panel then shows that address and cannot change it; it is read at Observer's start, which stops if the port is taken. GET /api/settings/sftp (View settings) says where SFTP listens, whether the file names the address, whether the port is open and the fingerprint of its host key; PUT /api/settings/sftp is the owner's.

The first time the port opens, Observer makes the server's SSH host key. Its fingerprint is beside the address in the panel and in the log (SFTP listening ... fingerprint=SHA256:...); clients compare it on their first connection. Expose the port directly: behind a TCP proxy every client would share the proxy's address, which the lockouts and the pipelines' allowed networks rely on. Allow the port in the firewall only for the networks that need it.

In the panel, under Files:

• Groups say what is reachable, folder by folder: allow or deny rules with rights (list, read, create, modify, rename, delete, make and remove folders), name filters such as *.ytd, and an optional largest file. A deny always wins. Each group shows its reach: Read, Assets (changes only textures, models, images, audio and fonts) or Code (anything else, which runs code on the host). Only the owner writes groups.

• People log in with their username and SSH keys they add themselves under My account (Ed25519 or ECDSA; ssh-keygen -t ed25519 makes one). The owner, or someone with the permission to manage file access, gives them groups; Code groups stay the owner's to give.

• Pipelines log in as +name with a key Observer makes and shows once, valid 90 days by default and 365 at most, optionally from allowed networks only. Store the private key in the CI's secrets; a deploy step then runs, for example:

  sftp -b deploy.txt -P 40126 -i deploy.key -o StrictHostKeyChecking=yes +deploy-main@panel.example.com

Whatever the groups say, no one reaches the cfg files (*.cfg, which hold the licence key and the database string), cache/, Observer's own folders and observer.toml, .git, .env files and key files. Every change is in the audit log; uploads are written to a temporary file and swapped in only when complete, so FXServer never reads half a file. Observer never restarts a resource after an upload: restart it on the Resources page, or let your own resource react.

What changed since the last start

At each start of FXServer, Observer records what the server runs from: the build and the game build, the launch settings, the cfg files exec reaches, and every resource under resources/ with its manifest's version, its git head, its npm packages and a digest of each file. The server never waits for it: the folder is listed as FXServer boots, and the files that changed are read once the server is online, at 64 MB/s at most. The first start reads everything once, in the background; later starts read only what changed.

The Changes page compares the last two starts, any two, or the last start with the disk now (Check the disk: what the next restart will apply). When a restart cut the previous start's reading short, the last start is compared with the latest one that was read in full. It shows what txAdmin cannot: a resource whose files changed while its version did not. Text files (Lua, JS, JSON, cfg, YAML and the like, up to 1 MiB) get line diffs; escrow-encrypted, binary and minified files are compared by size and hash.

• Secrets. Contents are masked before they are stored: convar secrets, webhook URLs, bot tokens, API keys, private keys, Authorization headers, credentials in connection strings and values given to names such as password or token (in code, JSON, YAML, TOML or XML) become [REDACTED], and the diff only says when one changed. .env, key files and txData are never stored. A secret in a shape no rule knows stays visible to whoever may read contents, so the owner can hide more files with patterns and purge what a file left in the history.
• Who sees what. View changes shows the history, versions, files and counts; Read changed files adds the diffs and whole files and is critical: it is the server's code. Reading contents is in the audit log. The cfg files and the launch configuration stay the owner's alone, as they are in the launch settings.
• Storage. changes.sqlite3 keeps starts 180 days and 2 GB of compressed text by default, the oldest contents dropped first; it is never in the daily backup, and a copy that cannot be read is set aside and started again, as the statistics are. Every file is read again once a week to catch a copy that kept both size and time.
• For resources. observer:startChanges tells your resources what changed once a start's files are read: post it to your staff's channel your way (see the contract).

Server tools

Server tools look in the server's resources for problems FXServer does not report, or reports where nobody reads, and repair those they can without changing what players see. A tool runs when the owner asks, on the Tools page or from the command line, and before every start of FXServer once the owner turns that on. Nothing is written into the server folder until the owner runs a fix, stages one or turns fixes on.

The Tools page, the owner's, gives each tool a card: what it found last, when, and the way in. Needs attention and Never run filter them. Beside the cards:

• Staged for the next restart: the fix that waits for the next start of FXServer, with the way to drop it.
• Before each start: what each tool does before FXServer starts, changed in place.
• Latest runs: the last run of each tool; every run that wrote is in the audit log.

Four tools are listed:

• Duplicate archetypes, below, which scans, fixes and undoes.
• cfg check: the checks of the server.cfg page, counted, with the way to the editor.
• Heavy assets: the streamed models and textures FXServer warned about as it last started their resource, heaviest first, with the size it measured and whether it called them oversized. Observer does not rewrite textures: the list is what to work from in a texture editor.
• Missing files: the files a manifest names that its resource does not hold, with the manifest and the line that name each.

The last two are read from FXServer's console as it starts its resources, and kept in memory until the next start: they are empty until Observer has started FXServer once. FXServer prints these warnings before the line that says a resource started, which is how each is given its resource.

Duplicate archetypes

An archetype is the definition of a prop, a vehicle, a weapon or a ped model, which the FiveM client registers by the hash of its name. When a resource defines one that is already registered, the client logs, once per archetype and session:

Duplicate Archetype 'prop_gift_box' (F032591F), seen in 'gift_props2.ytyp' and 'gift_props.ytyp'

The line goes to the client's log, on the player's machine: the server never sees it, and neither does Observer's console. Extra in place of a file stands for a weapon, vehicle or ped .meta; the same file named twice means two copies inside it.

What it reads. The cfg and the manifests, to predict which resources start and in which order; the .ytyp files the started resources stream, one per name, as the client loads them; and the .meta files they load as WEAPON_METADATA_FILE, VEHICLE_METADATA_FILE or PED_METADATA_FILE. It needs no GTA V install and decrypts nothing: a file encrypted by the asset escrow is listed as unreadable, and archives loaded as RPF_FILE are not opened. The archetypes of the game itself and of Rockstar's updates are not read, so a resource that defines one of them again is not seen.

What it fixes. The copies of one archetype form a group. By default, the tool fixes a group, by taking out every copy but one, only when all of these hold:

• the copies are identical, so which one the game uses cannot matter;
• they all lie in one resource, so the kept copy starts and stops with those taken out;
• that resource is not protected by the asset escrow (a .fxap at its root): its author ships the fix, and the report names the files and entries to pass on;
• none defines an interior (an MLO), and .ytyp copies do not meet .meta ones;
• no other file can take the place of a .ytyp involved: no other .ytyp of the same name in the stream/ of any resource, its own included, started or not, and no other resource requesting or listing its name (DLC_ITYP_REQUEST, a .ymf manifest or _cache_y.dat); for copies in two files, no manifest of their resource lists them;
• every file can be changed: a layout the tool knows how to change, not read-only, not reached through a link, not in an excluded resource;
• the fix leaves exactly one copy: nothing is taken out of a group it cannot fully fix.

The kept copy is the last one registered. Every group the tool leaves is reported with every reason, so the report says which setting would change the answer, or what to pass on to a resource's author. With the default settings no fix depends on the prediction: it only orders the report and picks which of identical copies stays.

How it writes. A fix takes the extra entries out of a .ytyp's archetype list, or the extra Items out of a .meta, and leaves the rest of the file as it was. The new file is written under a temporary name beside the old one, then renamed over it, never written in place: FXServer hard-links every stream file into its cache/, and a write in place would change the cached copy too. A file changed since the scan is left alone. A change takes effect the next time FXServer starts the resource.

Each write is recorded before it starts: whatever stops Observer midway, its next start tells from the file's contents whether the write landed, and removes a temporary file left behind. A run leaves 64 MiB free on the server folder's volume and on the data folder's, and stops at the first file that would take more.

Backups, heal and undo.

• Before it first changes a file, Observer keeps it as it was in data/tools/backups/<sha256>, with <sha256>.json naming the server folder, the file and the time, so that it can be copied back by hand even without the database. A backup is verified before every restore, and kept while a fix relies on it and for 30 days after the last one ends.
• Heal runs before every start Observer makes of FXServer, whatever the settings and the cause of the start, an automatic restart after a crash included, and before every fix. It puts back every entry whose kept copy no longer stands in for it: its file gone or no longer defining the archetype, another file now able to take its place, its resource excluded, or escrowed while escrowed resources are skipped. When unsure, it puts the entry back: a duplicate warning is harmless, a missing prop is not. A file someone else has changed since is left as it is, and no longer counted as fixed. A scan heals nothing: it says how many files hold a fix.
• Undo puts back what fixes took out, by rebuilding each file from its backup: what one run took out, one file's entries, or everything. A file changed since Observer wrote it is left alone.

Settings, per server, on the Tools page or with observer tools config:

• At start: Off (the default), the tool runs only when asked; Report, it scans before every start and writes to the console when the result has changed; Fix, it scans and applies the fixes before every start. Heal runs whatever this says.
• Differences: Identical only (the default) fixes identical copies only; Keep the latest copy fixes copies that differ too, keeping the last one registered. The client keeps every copy, and which one the game uses when they differ is not known for sure: "the last one" is a prediction. If the game uses another, Keep the latest copy changes the model, the textures or the draw distance players see.
• Escrowed resources: Skip (the default) only reports escrowed resources; Allow fixes their plain files too.
• Excluded resources: resources never changed, by name as FXServer knows them, case included. Put there those a pipeline or a file group keeps: otherwise each deploy brings their duplicates back, and the next fix changes the file again.

Excluding a resource puts its entries back at the next heal, and so does setting Escrowed resources back to Skip for escrowed resources.

The Tools page is the owner's. Last run shows the latest run and what it found or changed. Groups lists every group: its copies, how they differ (only in draw distances and bounds, or in more), what the tool does and why. Files changed lists the files the fixes changed, with their backups. Scan now makes a new report and writes nothing, even while FXServer runs. Fix applies the plan on screen: if a fresh scan plans something else, it applies nothing (the heal it runs first may have put entries back) and the page shows the new plan instead. Once an undo or a heal has put entries back, the page says the report on screen is out of date: scan again before fixing. Undo puts back the entries taken out of one file, or by one run, in the folder they were taken out of; Undo all those of every file in the folder of the launch settings. A change made in a folder that is gone, or is another folder now, is listed without Undo: its backup puts it back by hand. Fix, Undo, Undo all and the settings ask for the owner's password again, and the three buttons run only while FXServer is stopped. Writes and settings changes are in the audit log, those of a run that Observer's stop cut short at its next start; report runs are not.

While FXServer runs, the plan on screen can be staged for the next restart instead. Before the next start that is not a recovery from a crash, Observer scans the folder again: it writes the fix if the scan plans exactly what was staged, and writes nothing if it plans anything else, saying so in the console. Either way the staged fix is gone after that start, and so it is once any run has changed files. Staging and dropping a staged fix ask for the password and are in the audit log. With At start on Fix there is nothing to stage: every start fixes.

The tool's page counts the duplicates it leaves by reason, with a switch for the two the owner decides: Change escrowed resources and Fix them too, keeping the latest copy are the settings below, saved and followed by a new scan.

Before a start, FXServer waits for the tool. Past 60 seconds, the tool is told to stop between two files, and FXServer starts once it has; a stop in the meantime cancels the start. FXServer never starts while a tool writes: a start asked during a fix or an undo waits for it to end, even past 60 seconds. On an automatic restart after a crash, the tool only heals. Its console lines appear under tools, a summary and at most 10 more, and the Tools page has the rest; a report, or a fix that wrote nothing, says nothing when its result is that of the last report. Uploads, renames and deletions over SFTP wait while a tool writes. observer fxserver run runs no tool, not even the heal, and says so.

From the host:

observer tools run duplicate-archetypes [--fix] [--json] [--data DIR] [--cfg FILE | --all-resources]
                                        [--differences D] [--escrowed E]
observer tools undo duplicate-archetypes --run ID | --all [--data DIR]
observer tools config duplicate-archetypes [--at-start M] [--differences D] [--escrowed E]
                                           [--exclude NAME]... [--include NAME]...

• run reads the server folder and the cfg of the launch settings, or the folder --data names with the cfg --cfg names in it (server.cfg by default); --all-resources reads no cfg and takes every resource FXServer accepts as started, in name order, and still reads the cfg for the game it runs (without one, the server is taken to run GTA V). --differences and --escrowed change the saved settings for this run only, but --fix takes --escrowed allow only once it is saved with config: the next heal would put back what the fix took out of escrowed resources while the saved setting skips them. It prints the report, as JSON with --json.
• Without --fix, run writes nothing, the database included: it heals nothing, and its report is not in the Tools page's Last run. It needs no database either: with --data, it works before Observer's first start, with the default settings, and so it does with a database this build cannot read yet, before its first start after an upgrade, as it says.
• With --fix, run applies the plan and prints what became of each file, and the undo that puts it back, with the run's number and the --data it was given. The run is recorded and audited like a fix from the Tools page.
• run exits with 0 when no duplicate is left once it ends, and with 2 when some are: for a report, every group it finds, fixable or not, since it fixes none; with --fix, every group it did not fix. It exits with 1 when it cannot run, or when a fix stops before the end, such as on a full volume, once it has printed what it wrote. A mistyped command also exits with 2, and prints its usage.
• undo --run ID puts back what that run took out, in the folder it changed, as long as that still is the same folder; undo --all puts back everything in the folder of the launch settings. --data DIR undoes in that folder instead, for either.
• config prints the settings and changes those its options name: --at-start (off, report or fix), --differences (identical or keep-latest), --escrowed (skip or allow), --exclude NAME to add a resource to the excluded ones and --include NAME to take one off, both repeatable. A change takes effect at the next start.
• undo and config exit with 0, or 1 on an error.

run without --fix takes no lock, and config checks that no one changed the settings meanwhile: both work while Observer runs. --fix and undo take Observer's lock, so stop Observer first, or use the Tools page.

Limits.

• Runtime starts. Resources that scripts or the console start, stop or restart while the server runs are not seen, nor constraints such as /assetpacks that only FXServer decides: the report may list groups the clients never meet, or miss some. No fix depends on them, since the copies of a fixed group share one resource.
• RedM is not supported: when the cfg sets gamename rdr3, a scan or a fix is refused, with --all-resources too, and the next start puts back whatever an earlier fix took out. An undo still works.
• File metadata. A replaced file is a new file: on Linux it keeps its mode, not its ACLs or extended attributes; on Windows it takes the permissions its folder gives new files. A read-only file is never changed.
• Folder order on Linux. Windows lists folders in a set order, Linux in none: where a resource holds two stream files of the same name, which one FXServer sends may differ from the prediction. The report notes it, and such files are never changed.

Debug

The Debug group of the panel counts what the server's resources do, inside the resources themselves: the events they send and receive, the state bags they write and the HTTP requests they send out. It answers which event floods the clients, which client sends far more than the others, which key is written sixty times a second, and which resource started calling a host it never called. It is off until the owner turns it on, on any of its pages or on the Settings page (Features), and it counts from the next start of FXServer.

What turning it on changes. At each start of FXServer, Observer adds one line at the top and one at the end of the build's citizen/scripting/lua/scheduler.lua, the file every server Lua runtime runs first, and keeps the file as it was beside it, as scheduler.lua.original. The first line loads hook.lua from the bridge's folder into the resource; the last hands it the scheduler's event handlers. Nothing is added to the server's resources. Once Debug is off, Observer stops reading the counts at once, and the next start takes the two lines out: the file is byte for byte what it was. Until that start, the resources of the server that runs still count. A build whose scheduler Observer does not recognise is left as it is: FXServer starts all the same, and the pages say why nothing is counted. The lines only count in a server Observer started: the same build run by hand counts nothing.

What is counted. Names, counts and sizes, sent once a second by each resource on the channel the bridge already reports on: an event's name with how often it passed, its size and the time its handlers took; a state bag key with how often it was written and whether the write goes out to clients; a request's method and host. Never an event's payload, a state bag's value, or the path, query, body or answer of a request. The counts stay in memory for fifteen minutes and go with the server's run; only the hosts each resource called are stored, with the time of the first call.

• Events: the traffic with clients over the last 1, 5 or 15 minutes, and each event by name with its way (from clients, to one client, to everyone, between resources, from the engine), its rate, its size and what it weighs. An event opens on the time its handlers take and the clients that send it most, each a click away from their profile. Refused lists what clients sent to an event a resource handles without having opened it to the network: FXServer drops those, and the list says who tried.
• State bags: each key with the kind of bag it is written in, the resource that writes it, its rate, its size and the share of its writes that go out to clients, and the largest single writes.
• HTTP requests: each host with the method, the resource, the rate and the size sent. A host a resource calls for the first time is marked new until the owner says they know it, which the audit log records. For the first 24 hours Debug counts, every host is taken as known.

The pages are read with View diagnostics; turning Debug on or off and saying a host is known are the owner's.

Limits.

• Server-side Lua only. A resource counts where it runs Lua on the server. Maps, clothes and client-only resources have nothing to count there; resources written in JavaScript or C# are not counted yet, nor is anything that happens on the clients. Each page says how many started resources report, and names the others.
• A cost. Each resource does a little more work on every event it sends or receives. Observer has not measured it on a busy server: turn Debug on to answer a question, then off again.
• Sizes are what the resources hand over, before FXServer packs and sends them: they compare events with one another, they are not the server's bandwidth. What a state bag sends to clients is worked out from the share of its writes that are replicated.
• Latent events count with the events sent to clients.

Turning features off

File access over SFTP (sftp), file changes (changes), the server tools (tools), the diagnostics (diagnostics), the statistics (metrics), the players (players), the player drops (drops), the update checks (updates), Debug (debug) and the entities (entities) can each be turned off; Debug alone starts off. A feature that is off runs none of its code: Observer does not open its port, start its tasks or open its database, and its API routes answer feature_disabled. What it recorded stays on disk, and its retention waits until it runs again. Supervision, the console, accounts, the audit log, the restart schedule and FXServer's builds are always on.

GET /api/features (View settings) says for each feature whether it runs, what holds it off, and what it opens: its port, its tasks, its files, its routes. Only the owner turns one on or off, with PUT /api/features/{feature} and the password again; each change is in the audit log.

• Off applies at once. By the time the API answers, the port is closed, the SFTP sessions have ended and the feature's tasks have stopped. Turning the players or the statistics off turns the drops off too: they are drawn from the players' sessions and from the runs the statistics record.
• On applies at the next start of Observer, which restarts FXServer, with three exceptions: the server tools and Debug apply from the next start of FXServer, and SFTP opens its port at once.
• What a feature left behind is still finished. With SFTP off, the failed logins it counted still reach the audit log, and the uploads a crash cut short still leave the server folder at Observer's start. While a fix of the server tools is applied, the work before each start still puts back what it took out once the copy it kept is gone, even with the tools off: undo the fixes first to turn the tools off entirely. An update that restarted Observer is settled at its start, kept or undone, even with updates turned off since.

The host can keep features off whatever the panel says, the owner's account included, in observer.toml; "off" is the only value:

[features]
sftp = "off"
diagnostics = "off"

Database backups

Observer can back up the game server's database, the one oxmysql or mysql-async read through mysql_connection_string, and send each backup to one or more destinations. Observer's own observer.sqlite3 has its own backups, in the data folder's backups/.

• The dump is made by the database's own tool: mariadb-dump for MariaDB, mysqldump for MySQL, with --single-transaction, so tables in InnoDB are as they were at one instant without locking them. Install the client tools on the host (apt install mariadb-client, or the client of the MariaDB or MySQL installer on Windows). Observer finds them on the PATH or in the usual install folders, or in the folder the settings name. The password goes to the tool in a temporary option file of Observer's data folder, never on its command line.
• The connection is the cfg's mysql_connection_string, read again at each backup, or one typed in the settings, for an account that only reads: it needs SELECT, SHOW VIEW, TRIGGER, EVENT, LOCK TABLES and the routines (SHOW_ROUTINE on MySQL 8, SELECT on mysql.proc on MariaDB).
• The file is gzipped, <database>-<UTC time>-<run>.sql.gz. A dump is kept only when the tool succeeded, the file starts like a dump and ends with its last line, and the file read back is the one written. Tables whose engine has no transactions (MyISAM, Aria, MEMORY) are named in the run: they are copied as the dump reaches them.
• When: before the schedule's restarts and the single restarts someone planned (the default), or before every restart; at fixed times, in the restart schedule's zone; and on request. The backup before a restart is dumped once FXServer has stopped, when resources have saved what they held and released the stop if they hold it, and FXServer waits for it one minute at most by default (up to ten): past that it starts anyway, the dump goes on, and the run says so. Copies go out while FXServer runs. A crash restart never waits for a backup.
• Destinations: a folder of the host or a share mounted on it, never inside the server folder; an SFTP server, logging in with a key Observer makes for it (its public half goes in the remote account's authorized_keys) or a password, to a host whose key you accept after a test; an S3-compatible store (AWS, Cloudflare R2, Backblaze B2, MinIO, Garage...), over HTTPS. Each keeps the last copies or those of the last days, as you set it, and never loses its newest: older copies go only after a new one arrived. Observer deletes only the files it wrote.
• Restoring is by hand: download a copy from the panel, then gunzip -c <file>.sql.gz | mariadb -u <user> -p <database>, into an empty database or one you mean to overwrite. Stop FXServer first.

Seeing the backups, starting one and downloading one are three permissions; the settings and the destinations are the owner's, with the password asked again. Passwords and keys are kept in observer.sqlite3, apart, and never shown again.

Integrations

resource/CONTRACT.md is the integration contract: the server events Observer raises before a restart or a stop (observer:restartScheduled, observer:shuttingDown, ...), what they carry, in which order, and what your resources can rely on. The bridge that raises them, a server-side resource, is written into the build before every start of FXServer under a name drawn for that start, which the observer_bridge server convar gives: do not ensure it yourself. Scripts that target a fixed name, as they do txAdmin's monitor, find nothing. resource/observer_example listens to each event and marks where a team plugs in its notifications and its saves.

The panel's Documentation page shows that contract as the build that runs ships it: each event with when it comes, its fields and a handler to copy, each export where the contract shows it called, and the files of the example resource. Every section has a link of its own, and the panel's search (Ctrl K) finds an event, an export or one of txAdmin's events by its name. Any account reads it. To the accounts that see the dashboard, it also says what this server does now: the contract's version, the name the bridge took for this start, and whether txAdmin's events are raised, now and from the next start.

A resource that saves when the server stops can hold the stop until it has saved, through the bridge's holdShutdown and releaseShutdown exports: see Stops that wait for your resources.

Your resources can also give each connected player a label, a roleplay character's name or an ID of your own, through the bridge's setPlayerLabel export. The panel shows it beside the player's FiveM name, marked as your resources' word, keeps each label on the player's profile as long as their names, and finds players by them. Who a player is stays their licence: a label never links or renames anyone.

Resources written for txAdmin keep working with Send txAdmin's events too, in the launch settings and the setup's last step: from the next start of FXServer, the bridge also raises txAdmin's events, such as txAdmin:events:scheduledRestart, txAdmin:events:serverShuttingDown and txAdmin:events:announcement, right after Observer's own. The contract maps each one. txAdmin showed its announcements and direct messages to players itself; Observer does not, so a server that counted on txAdmin for that needs a resource that shows them.

Bans

The staff bans a player from their profile on the Players page, or bans identifiers typed by hand on the Bans page, with a reason and a length: an hour, a day, a week, a month, any number of minutes, or for good. A ban holds the player's licence and every other identifier and hardware token their profile links to, so a second account that shares their tokens is refused too. The Bans page lists active, ended and lifted bans, and finds them by name, licence, reason or ban ID; finding them by another identifier takes View player identifiers. Lifting a ban takes a permission of its own and keeps the ban in the list, with who lifted it and why. Every ban, every edit and every lift is in the audit log.

The settings' Bans and kicks category keeps ban templates: the reasons you ban for most, each with its usual length, in the order you put them. The ban dialog offers them, and picking one fills in the reason and the length, which the staff can still change before banning. The list is saved with the refusal message, by who may change settings.

An active ban can be edited: its reason, its end, or both. A new length counts from the edit, as a new ban's counts from the ban, and a ban can be made permanent. Editing takes a permission of its own, Edit bans: a ban made to end sooner lets its player back in sooner, which banning does not allow. The ban keeps who made it and when. Each edit is kept with it, with who made it, when, and the reason or the end it replaced: the Bans page and the player's profile show them under the ban's reason. From then on a connecting player is checked against the new end and reads the new reason.

The bridge holds each connecting player on playerConnecting while Observer checks them, which takes milliseconds. A banned player is refused with the refusal message of the settings' Bans and kicks category: it can name the reason, when the ban ends and the ban ID to quote in an appeal. If Observer has not answered within 5 seconds, the player gets in, and Observer drops them once they joined if a ban holds them. A player banned while playing is dropped at once. By default one hardware token in common is enough to refuse a player; the settings can ask for more, or for none. Addresses are never matched: players behind one address would be refused together.

Your resources hear of both through observer:playerRefused and observer:playerBanned, in the contract. With Send txAdmin's events too, each ban made while the server runs also raises txAdmin:events:playerBanned, whether or not the player is connected, and each ban lifted raises txAdmin:events:actionRevoked.

Kicks, announcements and messages

The staff kicks a connected player from their row in Online now, from their profile or from the Live map, with a reason. The server drops them with the kick message of the settings' Bans and kicks category, which can name the reason and who kicked. A kick keeps no one out: the player can connect again at once. Kick everyone, in the menu beside Restart and Stop, drops every connected player with the same message. It empties the server as a restart does, so it takes Start and stop the server rather than Kick players.

Send an announcement, in the same menu, writes to the whole server, and Message, where a player is kicked from, to one connected player. Observer shows players nothing of either: the bridge raises observer:announcement or observer:directMessage with the text, and your resources show it, in the chat or the way your server speaks to its players. Until a resource handles the event, nobody reads the text: resource/observer_example marks where to plug in. Sending announcements and messaging players are two permissions.

Your resources hear of a kick through observer:playerKicked, raised while the player is still connected. Every kick, announcement and message is in the audit log, with its text.

Allowlist

With Approved players only on, in the settings' Allowlist category, the server lets in only the players the staff approved. It goes by the licence alone, and bans come first: an approved player who is banned stays out.

A player who is not approved is refused with the allowlist's refusal message, which you write: it names the ID of their request, such as 7KQ-2XP, for them to give your staff. The request shows on the Allowlist page with the player's name, their licence and how often they tried. The staff approves it, and the player gets in from their next connection, or denies it. A player who tries again keeps their request and its ID; a denied one who comes back makes a new request, under a new ID. Approve a licence lets a player in ahead of their first visit, with a note that says who it is. Taking an approval back keeps the player out again.

Reading the page takes View players; answering requests, approving a licence and taking an approval back take Manage the allowlist. The switch and the message are settings, changed with the password asked again, like the bans' message. Every answer, approval and change is in the audit log.

What it does not do: a player who shows no licence, as on a server with sv_lan, cannot be approved, and makes no request. Turning the allowlist on, or taking an approval back, drops no one: the players connected stay until they leave. If Observer has not answered a connection within 5 seconds, the player gets in. Observer keeps the 2,000 requests tried last. Your resources hear of each refusal through observer:playerRefused, with cause = 'allowlist', in the contract.

Maintenance mode

Maintenance, beside Restart and Stop above every page, closes the server to players while you work on it. The panel shows first what a refused player will read and how many players still get in, then asks. From then on the server refuses every connection with your maintenance message, and a banner on every page says since when and by whom, with End maintenance to open the server again. It holds through restarts of FXServer and of Observer.

The settings' Maintenance mode category holds the message and Who still gets in: the players maintenance lets through, by licence. Type a name to pick a player Observer knows, or paste a licence, with a note that says who it is. A panel account has no licence, so list the player each member of your staff plays as. A player on the list gets in whether the allowlist has them or not; a ban still refuses them.

Name in the server list, in the same category, is the name the server takes in FiveM's server list while maintenance is on; leave it empty and the server keeps its own. Observer's bridge sets sv_hostname to it, and sv_projectName when your cfg gives one, then gives both back when maintenance ends. FXServer sends its name to the list with each heartbeat, every three minutes or 15 seconds after a player connects or leaves; when the list shows it is up to Cfx.re. A restart of FXServer during maintenance reads the names of your cfg again, and the bridge sets the maintenance name once the server is online.

Turning maintenance on or off takes Start and stop the server. The message, the list and the name are settings, changed with the password asked again. Each switch and each change is in the audit log. Your resources hear observer:maintenance at each switch, and once more at each start of FXServer while it is on; each refusal raises observer:playerRefused with cause = 'maintenance'.

What it does not do: turning maintenance on drops no one, the players connected stay until they leave. If Observer has not answered a connection within 5 seconds, the player gets in. A player who shows no licence cannot be on the list.

Live map

The Live map page shows where the connected players stand, as the server sees them, so that the staff can follow the server without joining it. Each player is a mark coloured by play time: new under 10 hours, regular, veteran from 100 hours, not confirmed yet before their identity is; the marks differ in shape too, and the settings' Live map category moves the thresholds. A click on a player gives their play time, session, ping and position, a link to their profile and, each with its permission, a message, a kick and a ban. Seeing the map takes See the live map.

On the map lists the players placed, by name, with their ping: a click centres the map on one, and the search narrows the list by name or ID. Follow, on the selected player, keeps the map on them at each new position. Trails, 5 min draws where each player went over the last five minutes: the places this page received since it was opened, kept in the browser and gone with the page. Observer stores no position, so a trail starts when the page opens.

Under the players, the page draws Rockstar's own map of GTA V, as the game draws it (Atlas), seen from the sky (Satellite) or as a road map (Roads): the settings' Live map category chooses the one a browser opens on, and each viewer switches between the three on the map itself, a choice their browser remembers. The switch shows only while the owner's choice is one of Rockstar's maps. Observer ships no map and keeps none, GTA V's belongs to Rockstar Games: each browser that opens the page loads the map's tiles from Rockstar's servers (s.rsg.sc), which then see that browser's address, and nothing else of it: no referrer, no cookie of Observer's. It is the one thing the panel loads from elsewhere than Observer; the two other choices load nothing from outside.

My image is for another map, a custom island or RedM's for one: the owner gives the path of an image on the host they have the right to use, PNG, JPEG or WebP, up to 16,384 pixels a side and 64 MB. Observer keeps a copy in its data folder (map/) and serves it only to accounts that see the map. To place it in the game, stand at two spots you know on the image, far apart, pick yourself in the list and click where you stand, or type the game coordinates of the image's four edges. Grid draws a grid of game coordinates, which also stands in for an image not placed yet and for tiles Rockstar's servers did not send.

Positions come from the bridge, which reads them from OneSync: without OneSync there are none. The server sends them every 1, 2 or 5 seconds, only while someone has the page open, and stops within 30 seconds after the last one leaves. Observer keeps them in memory only: nothing of them is written.

Entities

The Entities page lists what the game holds on the server: its vehicles, its peds and its objects, as OneSync knows them. Each has its type, its model, the resource that created it, the player whose game simulates it, its routing bucket and its position. The players' own peds are counted and never listed. The list is filtered by type, vehicle class, creating resource, owner and bucket, searched by net ID, model, resource or owner's name, sorted by any column, and read a hundred at a time. Seeing it takes See the entities.

A model shows as its hash, which is all the server knows of it: Observer ships no table of GTA V's names. The creating resource is the one the server knew when the entity was first seen: none for the game's own traffic, and for what a client's script created. Who owns an entity, and whether a player sits in a vehicle, is told only to accounts that also hold See the live map: both say where a player is.

On the live map, Entities draws them under the players, a shape for each type. Past 5,000 in the part of the map in view, the map counts them by square instead, and zooming in shows each one again. A click on an entity gives what the list says of it.

Delete entities is a permission of its own. A deletion takes up to 100 entities, asks for a word typed from 20, and never takes a player's own ped, a vehicle a player sits in or a train. The server checks each entity again before it deletes it, so one that changed since the list was read stays, and the panel says what became of each. Nothing is deleted from a list more than three sweeps old, nor from one read before the server or its bridge restarted. Every deletion is in the audit log, with what was deleted and by whom.

The entities come from the bridge, which reads them from OneSync: without OneSync there are none, and only GTA V servers are read. The server lists them every 15 seconds, or as often as the settings' Entities category says, only while someone has the page or the map's layer open, and stops within 30 seconds after the last one leaves. A sweep works a millisecond at a time and gives the server its frame back in between, and stops at 50 ms of work, its list then partial. When a sweep costs more than half of that, the server spaces the next ones out by itself, up to once a minute, and comes back to its cadence once they are cheap again. Observer keeps the list in memory only: nothing of it is written.

Updates

Every 6 hours, Observer asks GitHub for the newest stable release. When it is newer, the owner sees New release at the foot of the rail, and the Updates card on the About page (the account menu) shows it with its notes. Install downloads it, verifies it, then restarts Observer into it. Only the owner installs a release, with the password asked again, and each step is in the audit log.

• What is verified, before anything runs. A release carries SHA256SUMS and SHA256SUMS.sig, the Ed25519 signature of the checksums by the release key, whose public half every Observer is built with. The archive must match the signed checksum and the digest GitHub computed when it was uploaded; then the new executable is run once, as --version, and must name the release. On Windows, a signed observer.exe only gives way to one signed by the same signer. A release that fails any of these is not installed, and the panel says which.
• FXServer. It stops whenever Observer does. Now stops it with the usual notice to your resources (observer:shuttingDown), and it starts again under the new release. At the next restart waits for the schedule's next restart, a single one included: your resources hear its warnings as usual, observer:shuttingDown says scheduled, and the restart takes a few seconds longer. Plan a single restart on the Restarts page to update in a few minutes with notice. FXServer comes back if it ran, whatever Start FXServer with Observer says; a stopped server stays stopped.
• The swap. Observer keeps the previous executable beside the new one, as observer.previous (observer.previous.exe), and backs up its database (pre-update). The account running Observer must be able to write next to its executable: under systemd, keep it out of /usr and /etc.
• The restart. Under systemd, Observer exits with 75 and the unit starts it again. On Linux otherwise, the new executable takes the place of the old in the same process. On Windows, the new executable starts in the same console; a terminal Observer was started from gets its prompt back while Observer goes on in it.
• When the new release does not start, Observer puts the previous executable back, and its database too if the new release had changed the database's schema, then starts it again. The Updates card says so, with the reason. Once a new release has started, it stays: going back is done by hand, as below.

Updates never install by themselves. A build from source is not updated: it says so. Turn the checks off on the Settings page (Features), or in observer.toml: Observer then never contacts GitHub.

To verify a release by hand, with OpenSSL 3 and the release key's public half in PEM:

sha256sum -c --ignore-missing SHA256SUMS
openssl pkeyutl -verify -rawin -pubin -inkey release-key.pem -in SHA256SUMS -sigfile SHA256SUMS.sig

Upgrading

Observer installs its releases itself (see Updates). By hand:

1. Stop Observer: Quit Observer… in the tray, Ctrl+C in its console, or sudo systemctl stop observer. FXServer stops with it.
2. Replace observer.exe or observer with the new version's. Keep observer.toml and the data folder. Under systemd, keep the binary's path, or print and install the unit again.
3. Start it. A version whose database schema is newer first writes a pre-migration backup to data/backups/, then migrates; its log names the backup.

An older version refuses a database a newer one has migrated: the database was written by a newer Observer (schema 8, this build knows 7); use that version or restore a backup. To go back, restore the pre-migration backup the newer version wrote, as below, then start the older version. It sets aside a statistics database it cannot read, as metrics.sqlite3.set-aside-<time>, and starts a new one.

Restoring a backup

1. Stop Observer, and make sure it has stopped: observer db check refuses to run while it does.
2. In the data folder, move observer.sqlite3 aside, with observer.sqlite3-wal and observer.sqlite3-shm when they are there. A crash leaves them behind; they belong to the database they sit next to and must not meet the backup.
3. Copy the backup from backups/ to observer.sqlite3.
4. Run observer db check. It prints integrity: ok and the backup's schema next to this build's. An older schema is migrated at the next start, after a pre-migration backup.
5. Start Observer.

Everything written since the backup is lost, and what was undone since comes back: a session ended since is valid again until it expires, and a password changed since works again. The statistics, the console logs and the builds are not part of the backup and stay as they are.

Building from source

You need Rust through rustup (rust-toolchain.toml pins the version, which installs itself), Node.js 24 with npm, and a C compiler: the Visual Studio Build Tools on Windows, gcc or clang on Linux.

git config core.hooksPath .githooks   # once per clone: main only moves through pull requests
cargo xtask web                       # build the panel into web/dist
cargo run -p observer                 # Observer, with the panel on http://localhost:40125
cargo xtask check                     # everything CI checks
cargo xtask dist                      # the release archive and its checksum, in target/dist

cargo xtask dist builds for the host; on Linux, cargo xtask dist --target x86_64-unknown-linux-gnu --max-glibc 2.35 also refuses a binary that needs a newer glibc. While working on the panel, npm run dev in web/ serves it on http://localhost:41820 with hot reload and forwards /api to a running Observer (OBSERVER_URL names another one). cargo xtask bindings regenerates the TypeScript types of the API from the Rust ones. cargo xtask e2e runs the browser tests against a debug Observer built with a test catalog, from which the flows install a build; a release never has that catalog, and cargo xtask dist refuses a binary that does. cargo xtask convars rewrites crates/observer-domain/convars.json, the catalog of FXServer's convars: it fetches the sources of citizenfx/fivem at a pinned commit with git (a shallow, sparse checkout under target/fivem/) and reads every convar the server components declare, with its type, default, values and flags. What the sources do not say stays empty. --commit <sha> --build <n> reads another build, --source <checkout> a checkout already there.

Known limits

• One operating system account for Observer and FXServer. Any resource FXServer runs can read and change Observer's files, the database included, and so take the owner account. Run only resources you trust.
• Signing out of Windows stops everything. Observer runs in the Windows session it was started from: signing out, over RDP too, stops FXServer and Observer. Disconnecting from RDP, or closing the RDP window, leaves them running.
• No Windows service. Observer starts when someone starts it, in a signed-in session.
• The status window's memory. Once opened, the window keeps its graphics memory until Observer exits, even after it is closed. --no-window avoids it.
• Colour codes in the classic console. The classic console window (conhost), still the default on Windows 10 and Windows Server 2016 to 2022, prints the log's colour codes as text. Windows Terminal renders them, and NO_COLOR=1 turns them off.
• English only. The panel, the command line and the logs are in English.
• SFTP: folders are not renamed or moved, files are; every *.cfg file is refused until Observer reads which ones the server executes; a file FXServer holds open on Windows cannot be replaced until it lets go of it.
• Database backups: MariaDB and MySQL only; one database per server; no encryption by Observer (use the destination's); an S3 copy is one upload of 5 GiB at most; an SFTP host must offer an Ed25519 or ECDSA host key, as every OpenSSH since 6.5 does; no restore from the panel.
• Changes: who changed a file is told only for SFTP writes, by path and time: a hint, since a file can be written again after. Changes made while the server runs show at the next start or with Check the disk, not as they happen.

Licence

No licence has been chosen yet. Until one is, all rights are reserved. The third-party software Observer ships is listed with its licences in THIRD-PARTY-LICENCES.txt and on the panel's About page.