diff --git a/README.md b/README.md index 28b94ec1..356c2471 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,11 @@ what to do with them. probation, and a return to the last healthy build. - **Statistics and diagnostics.** Players, tick times, CPU and memory of FXServer and the host, FXServer's threads and connections, crash dumps and hangs. -- **Live map.** Where the connected players stand, over a map image you add, coloured by play time, - a click away from their profile and a ban. +- **Live map.** Where the connected players stand, over Rockstar's map of GTA V or an image you add, + coloured by play time, a click away from their profile and a ban. +- **Debug.** What each resource sends and receives, writes into state bags and calls over HTTP, + counted inside the resources: the event that floods the clients, the client that sends far more + than the others, the resource that calls a host it never called. Off until the owner turns it on. - **Bans.** A ban holds a player's licence, other identifiers and hardware tokens, for a time or for good. The server refuses a banned player at connection with your message, and drops one banned while playing. @@ -110,6 +113,12 @@ http://localhost:40125/setup# Once the owner exists, the panel opens the server setup. Other accounts are added on the **Users** page, each with a temporary password that works for 72 hours and is changed at the first sign-in. +The panel's pages are grouped by what one comes for: **Players** (the players, the live map, the +bans), **Server** (restarts, the cfg, builds, backups, file changes), **Insights** (statistics, drops, +diagnostics) and **Maintenance** (tools, the audit log). The search button beside the logo, or Ctrl K +(⌘ K on a Mac) from any page, opens a search of the pages, of the settings by their name, and of the +connected players by name or ID. Each account finds only what it may open. + From the host, even while Observer runs: ```sh @@ -126,9 +135,10 @@ configured, and again from **Server → Run the setup**. It has four steps: 1. **Server folder.** The full path on the host of the folder that holds `resources` and the cfg, such as `C:\FXServer\server-data` or `/srv/fxserver/server-data`. Network paths are not supported. FXServer runs from this folder. Observer copies nothing into it, and writes there only what people - upload over [SFTP](#file-access-over-sftp) and the fixes of the [server tools](#server-tools) the - owner runs or turns on: the **Tools** page lists each fix with its backup. -2. **Config file.** The cfg FXServer executes, relative to the folder, such as `server.cfg`. Observer + upload over [SFTP](#file-access-over-sftp), the fixes of the [server tools](#server-tools) the + owner runs or turns on, which the **Tools** page lists each with its backup, and the cfg when the + owner saves it from the [server.cfg page](#editing-servercfg). +2. **Config file.** The cfg FXServer executes, relative to the folder, such as `server.cfg`. The setup reads it and the cfgs it runs in turn, never changes them, and lists what it finds with file and line: blocking problems, warnings and notes, and what the cfg sets (game, endpoints, slots, OneSync, hostname, whether a licence key is set). Fix your files, then **Check again**; only a blocking problem @@ -157,16 +167,37 @@ server whose heartbeat or HTTP is late when those minutes end passes only once i hang that began within them fails the probation however long the [hang delay](#hang-detection). With hang detection off, only a crash fails it, and a silent server still waits to answer before it passes. +### Following a start + +While FXServer starts, the Overview shows how far it has come: how many resources are up, the +command of the cfg it is at (`ensure [clothes]`, `server.cfg` line 226, 12 of its 150 resources +up), the last resource started and how long ago, the resources that did not start, and the five +slowest so far. The status bar above every page counts them too. + +Observer reads this from FXServer's own console: FXServer prints `Started resource ` once a +resource is up, and nothing while it starts one. The command it is at is worked out from the +resources each `ensure` and `start` of the cfg starts, a category being every resource under the +folder of that name. When Observer cannot read every command of the cfg, as when it executes a +file inside a resource, the page says so rather than guess. + +Once no resource has started for half a minute, the card says how long Observer still waits +before it takes the start for stuck (see hang detection below). Whoever may start and stop the +server can give that start 15 minutes more, as often as needed; the audit log keeps who did. The +hang delay itself is not changed. + ### Hang detection Observer watches two signals while FXServer runs: the heartbeat of its bridge resource, which only comes while the main loop turns, and FXServer's HTTP endpoints. Once the server is up, one of them silent for 10 seconds marks it degraded. Silent for the delay under **Server → Hang detection**, 6 -minutes by default and counted from the start until the first answer, the server is hung: Observer -writes a dump of it (a report of its threads on Linux), kills its process tree and starts it again. - -A big server's first start can keep its main loop busy for minutes while resources build their caches of -clothing and vehicles, which reads as a hang. Raise the delay, from 1 to 60 minutes, or turn hang detection +minutes by default, the server is hung: Observer writes a dump of it (a report of its threads on Linux), +kills its process tree and starts it again. + +A start gives no heartbeat: FXServer runs its cfg before any thread of a resource, the bridge's included. +Until the cfg has run, each resource FXServer starts counts in its place, so a start is hung once it has +started no resource for the delay, however long it takes in all, and the console names the last resource +it started. One resource alone can take minutes on a big server's first start, such as a clothing pack +whose files are read for the first time. Raise the delay, from 1 to 60 minutes, or turn hang detection off: FXServer is then left running however long it stays silent, until someone restarts it. Either change applies at once, to a server already starting too, and the delay then counts from the change: turning detection back on never kills a server that was silent before. A crash is another matter: whenever @@ -240,7 +271,8 @@ trusted_proxies = [] # A tracing filter, such as "debug" or "info,observer_web=debug". level = "info" -# File access over SFTP stays off until this section exists (see "File access over SFTP"). +# Where SFTP listens. Without this section the owner gives the address in the panel instead; with +# it, the panel cannot change it (see "File access over SFTP"). # [sftp] # listen = "0.0.0.0:40126" @@ -299,7 +331,7 @@ The backups are complete snapshots of the state database: | Port | What | |---|---| | 40125/TCP (`http.listen`) | The panel, its API and its websocket (`/api/ws`). | -| 40126/TCP (`sftp.listen`), when set | File access over SFTP. Off unless `[sftp]` is in `observer.toml` and SFTP is not [turned off](#turning-features-off). | +| 40126/TCP, or the port the owner gives | File access over SFTP. Closed until the owner gives its address in the panel or `[sftp]` names it in `observer.toml`, and while SFTP is [turned off](#turning-features-off). | | The cfg's endpoints, such as 30120 TCP and UDP | FXServer's own. Observer reads FXServer's HTTP endpoint (`/dynamic.json`, `/info.json`, `/perf/`, `/players.json`) on the cfg's first `endpoint_add_tcp`, through loopback when that one listens on every interface. | ### Outbound connections @@ -319,6 +351,10 @@ nothing else. | `api.github.com` | Observer's latest release, without an account: 60 checks an hour per address | | `github.com`, `release-assets.githubusercontent.com`, `objects.githubusercontent.com` | a release's archive, its checksums and their signature | +The panel, in the browser, loads everything from Observer, with one exception: the [live map](#live-map) +loads Rockstar's map of GTA V from `s.rsg.sc` unless its settings say **My image** or **Grid**. Observer +itself never talks to that host. + Cfx.re's [Platform License Agreement](https://fivem.net/terms) covers the server builds you download. ### Command line @@ -441,21 +477,68 @@ When the public address carries a port, such as `https://panel.example.com:8443` Observer pings the websocket every 20 seconds, well within nginx's default 60-second `proxy_read_timeout`. The panel opens `wss://` by itself when the page is HTTPS. +## Editing server.cfg + +**Server → server.cfg**, for the owner alone, edits the cfg of the launch settings in two views of the +same text. + +- **Guided** gives a field to the lines most servers change: the server name, the project name and + description, the tags, the language, the slots, the game build, the game address, the licence key + and the Steam Web API key. Each field names the line it is written on, or says that a save adds + one. Below them, every line that starts a resource or a category has a switch: off comments the + line out, on puts it back, and the order stays the order of the file. A line that starts something + neither the server folder nor the build holds is flagged. +- **File** is the whole text, with line numbers, and the checks of the [server setup](#the-server-setup-page) + read again as you type: what blocks a start, what FXServer will skip, with the file and the line. + +A save rewrites only what you changed: comments, blank lines, order, line ends and a byte order mark +stay as they were. It asks for your password, applies at the next start of FXServer, and is refused +when the file on disk is no longer the one you started from. **See the changes** shows the lines a +save would change before you save. + +Secrets never reach the browser. The values of `sv_licenseKey`, `mysql_connection_string`, +`rcon_password`, `sv_tebexSecret`, `steam_webApiKey`, a Discord webhook's token and anything else the +[file changes](#what-changed-since-the-last-start) hide stand in the text as `<>`, +`<>`. A save puts each one back where its mark still is; typing over a mark replaces the +secret. The comparisons show them as `[REDACTED]`. + +Before each save, Observer keeps the file as it was in its own database: the latest 20 versions, +with who replaced each and when, secrets included as the file held them. **Earlier versions** compares +any of them with the file and puts one back, which keeps the current file as a version too. The audit +log records each save and each version put back with the number of lines added and removed, never +their text. + +The page edits a plain UTF-8 file of at most 512 KiB. It refuses a link, a read-only file, a file in +another encoding, which a save would damage, and the files the cfg runs with `exec`: those are edited on the host or +over SFTP, and the checks still read them. + ## File access over SFTP Observer can let people and CI pipelines read and change chosen parts of the server folder over SFTP, -the file transfer of SSH. It is off until the owner adds this to `observer.toml` and restarts Observer: +the file transfer of SSH. Its port stays closed until it has an address to listen on, which the owner +gives in the panel, under **Settings → File access**, with the password again: every address of the +host, the host itself only (for a tunnel), or one of its addresses, and a port. The port opens, moves +and closes at once, without a restart of Observer or of FXServer, and each change is in the audit log. +A port that does not open, because another program holds it for one, never stops Observer: the panel +says what the host answered, and SFTP waits until the address is given again or changed. + +A host that wants the address out of the panel's reach names it in `observer.toml`: ```toml [sftp] listen = "0.0.0.0:40126" ``` -At that first start Observer makes the server's SSH host key and logs its fingerprint -(`SFTP listening ... fingerprint=SHA256:...`); clients compare it on their first connection. Expose the -port directly: behind a TCP proxy every client would share the proxy's address, which the lockouts and -the pipelines' allowed networks rely on. Allow TCP 40126 in the firewall only for the networks that -need it. +The panel then shows that address and cannot change it; it is read at Observer's start, which stops if +the port is taken. `GET /api/settings/sftp` (**View settings**) says where SFTP listens, whether the +file names the address, whether the port is open and the fingerprint of its host key; +`PUT /api/settings/sftp` is the owner's. + +The first time the port opens, Observer makes the server's SSH host key. Its fingerprint is beside the +address in the panel and in the log (`SFTP listening ... fingerprint=SHA256:...`); clients compare it on +their first connection. Expose the port directly: behind a TCP proxy every client would share the +proxy's address, which the lockouts and the pipelines' allowed networks rely on. Allow the port in the +firewall only for the networks that need it. In the panel, under **Files**: @@ -514,11 +597,35 @@ files changed while its version did not. Text files (Lua, JS, JSON, cfg, YAML an ## Server tools -Server tools look in the server's resources for problems FXServer does not report, and repair those -they can without changing what players see. The first one finds duplicate archetypes. A tool runs when -the owner asks, on the **Tools** page or from the [command line](#command-line), and before every -start of FXServer once the owner turns that on. Nothing is written into the server folder until the -owner runs a fix or turns fixes on. +Server tools look in the server's resources for problems FXServer does not report, or reports where +nobody reads, and repair those they can without changing what players see. A tool runs when the +owner asks, on the **Tools** page or from the [command line](#command-line), and before every start +of FXServer once the owner turns that on. Nothing is written into the server folder until the owner +runs a fix, stages one or turns fixes on. + +The **Tools** page, the owner's, gives each tool a card: what it found last, when, and the way in. +**Needs attention** and **Never run** filter them. Beside the cards: + +- **Staged for the next restart**: the fix that waits for the next start of FXServer, with the way + to drop it. +- **Before each start**: what each tool does before FXServer starts, changed in place. +- **Latest runs**: the last run of each tool; every run that wrote is in the audit log. + +Four tools are listed: + +- **Duplicate archetypes**, below, which scans, fixes and undoes. +- **cfg check**: the checks of the [server.cfg page](#editing-servercfg), counted, with the way to + the editor. +- **Heavy assets**: the streamed models and textures FXServer warned about as it last started their + resource, heaviest first, with the size it measured and whether it called them oversized. + Observer does not rewrite textures: the list is what to work from in a texture editor. +- **Missing files**: the files a manifest names that its resource does not hold, with the manifest + and the line that name each. + +The last two are read from FXServer's console as it starts its resources, and kept in memory +until the next start: they are empty until Observer has started FXServer once. FXServer prints +these warnings before the line that says a resource started, which is how each is given its +resource. ### Duplicate archetypes @@ -623,6 +730,17 @@ back by hand. **Fix**, **Undo**, **Undo all** and the settings ask for the owner and the three buttons run only while FXServer is stopped. Writes and settings changes are in the audit log, those of a run that Observer's stop cut short at its next start; report runs are not. +While FXServer runs, the plan on screen can be **staged for the next restart** instead. Before the +next start that is not a recovery from a crash, Observer scans the folder again: it writes the fix +if the scan plans exactly what was staged, and writes nothing if it plans anything else, saying so +in the console. Either way the staged fix is gone after that start, and so it is once any run has +changed files. Staging and dropping a staged fix ask for the password and are in the audit log. +With **At start** on **Fix** there is nothing to stage: every start fixes. + +The tool's page counts the duplicates it leaves by reason, with a switch for the two the owner +decides: **Change escrowed resources** and **Fix them too, keeping the latest copy** are the +settings below, saved and followed by a new scan. + Before a start, FXServer waits for the tool. Past 60 seconds, the tool is told to stop between two files, and FXServer starts once it has; a stop in the meantime cancels the start. FXServer never starts while a tool writes: a start asked during a fix or an undo waits for it to end, even past 60 @@ -690,14 +808,70 @@ use the **Tools** page. holds two stream files of the same name, which one FXServer sends may differ from the prediction. The report notes it, and such files are never changed. +## Debug + +The **Debug** group of the panel counts what the server's resources do, inside the resources +themselves: the events they send and receive, the state bags they write and the HTTP requests they +send out. It answers which event floods the clients, which client sends far more than the others, +which key is written sixty times a second, and which resource started calling a host it never +called. It is off until the owner turns it on, on any of its pages or on the Settings page +(**Features**), and it counts from the next start of FXServer. + +**What turning it on changes.** At each start of FXServer, Observer adds one line at the top and one +at the end of the build's `citizen/scripting/lua/scheduler.lua`, the file every server Lua runtime +runs first, and keeps the file as it was beside it, as `scheduler.lua.original`. The first line +loads `hook.lua` from the bridge's folder into the resource; the last hands it the scheduler's event +handlers. Nothing is added to the server's resources. Once Debug is off, Observer stops reading the +counts at once, and the next start takes the two lines out: the file is byte for byte what it was. +Until that start, the resources of the server that runs still count. A build whose scheduler +Observer does not recognise is left as it is: FXServer starts all the same, and the pages say why +nothing is counted. The lines only count in a server Observer started: the same build run by hand +counts nothing. + +**What is counted.** Names, counts and sizes, sent once a second by each resource on the channel +the bridge already reports on: an event's name with how often it passed, its size and the time its +handlers took; a state bag key with how often it was written and whether the write goes out to +clients; a request's method and host. Never an event's payload, a state bag's value, or the path, +query, body or answer of a request. The counts stay in memory for fifteen minutes and go with the +server's run; only the hosts each resource called are stored, with the time of the first call. + +- **Events**: the traffic with clients over the last 1, 5 or 15 minutes, and each event by name + with its way (from clients, to one client, to everyone, between resources, from the engine), its + rate, its size and what it weighs. An event opens on the time its handlers take and the clients + that send it most, each a click away from their profile. **Refused** lists what clients sent to + an event a resource handles without having opened it to the network: FXServer drops those, and + the list says who tried. +- **State bags**: each key with the kind of bag it is written in, the resource that writes it, its + rate, its size and the share of its writes that go out to clients, and the largest single writes. +- **HTTP requests**: each host with the method, the resource, the rate and the size sent. A host a + resource calls for the first time is marked new until the owner says they know it, which the + audit log records. For the first 24 hours Debug counts, every host is taken as known. + +The pages are read with **View diagnostics**; turning Debug on or off and saying a host is known +are the owner's. + +**Limits.** + +- **Server-side Lua only.** A resource counts where it runs Lua on the server. Maps, clothes and + client-only resources have nothing to count there; resources written in JavaScript or C# are not + counted yet, nor is anything that happens on the clients. Each page says how many started + resources report, and names the others. +- **A cost.** Each resource does a little more work on every event it sends or receives. Observer + has not measured it on a busy server: turn Debug on to answer a question, then off again. +- **Sizes are what the resources hand over,** before FXServer packs and sends them: they compare + events with one another, they are not the server's bandwidth. What a state bag sends to clients + is worked out from the share of its writes that are replicated. +- **Latent events** count with the events sent to clients. + ## Turning features off File access over SFTP (`sftp`), file changes (`changes`), the server tools (`tools`), the diagnostics -(`diagnostics`), the statistics (`metrics`), the players (`players`), the player drops (`drops`) and -the update checks (`updates`) can each be turned off. A feature that is off runs none of its code: Observer does not open its port, start -its tasks or open its database, and its API routes answer `feature_disabled`. What it recorded stays on -disk, and its retention waits until it runs again. Supervision, the console, accounts, the audit log, -the restart schedule and FXServer's builds are always on. +(`diagnostics`), the statistics (`metrics`), the players (`players`), the player drops (`drops`), the +update checks (`updates`) and Debug (`debug`) can each be turned off; Debug alone starts off. A +feature that is off runs none of its code: Observer does not open its port, start its tasks or open +its database, and its API routes answer `feature_disabled`. What it recorded stays on disk, and its +retention waits until it runs again. Supervision, the console, accounts, the audit log, the restart +schedule and FXServer's builds are always on. `GET /api/features` (**View settings**) says for each feature whether it runs, what holds it off, and what it opens: its port, its tasks, its files, its routes. Only the owner turns one on or off, with @@ -706,8 +880,8 @@ what it opens: its port, its tasks, its files, its routes. Only the owner turns - **Off applies at once.** By the time the API answers, the port is closed, the SFTP sessions have ended and the feature's tasks have stopped. Turning the players or the statistics off turns the drops off too: they are drawn from the players' sessions and from the runs the statistics record. -- **On applies at the next start of Observer,** which restarts FXServer, except for the server tools, - which apply from the next start of FXServer. A port only opens when the host restarts Observer. +- **On applies at the next start of Observer,** which restarts FXServer, with three exceptions: the + server tools and Debug apply from the next start of FXServer, and SFTP opens its port at once. - **What a feature left behind is still finished.** With SFTP off, the failed logins it counted still reach the audit log, and the uploads a crash cut short still leave the server folder at Observer's start. While a fix of the server tools is applied, the work before each start still puts back what it @@ -812,12 +986,28 @@ shape too, and the settings' Live map category moves the thresholds. A click on play time, session, ping and position, a link to their profile and, with **Ban players**, a ban. Seeing the map takes **See the live map**. -Observer ships no map: GTA V's belongs to Rockstar Games. The owner gives, in the settings, the path of -an image on the host they have the right to use, such as GTA V's atlas from their own copy of the game: -PNG, JPEG or WebP, up to 16,384 pixels a side and 64 MB. Observer keeps a copy in its data folder -(`map/`) and serves it only to accounts that see the map. To place it in the game, stand at two spots -you know on the image, far apart, pick yourself in the list and click where you stand, or type the game -coordinates of the image's four edges. Without an image, the page shows a grid of game coordinates. +**On the map** lists the players placed, by name, with their ping: a click centres the map on one, and +the search narrows the list by name or ID. **Follow**, on the selected player, keeps the map on them at +each new position. **Trails, 5 min** draws where each player went over the last five minutes: the +places this page received since it was opened, kept in the browser and gone with the page. Observer +stores no position, so a trail starts when the page opens. + +Under the players, the page draws Rockstar's own map of GTA V, as the game draws it (**Atlas**), seen +from the sky (**Satellite**) or as a road map (**Roads**): the settings' Live map category chooses +the one a browser opens on, and each viewer switches between the three on the map itself, a choice +their browser remembers. The switch shows only while the owner's choice is one of Rockstar's maps. +Observer ships no map and keeps none, GTA V's belongs to Rockstar Games: each browser that opens the +page loads the map's tiles from Rockstar's servers (`s.rsg.sc`), which then see that browser's address, +and nothing else of it: no referrer, no cookie of Observer's. It is the one thing the panel loads from +elsewhere than Observer; the two other choices load nothing from outside. + +**My image** is for another map, a custom island or RedM's for one: the owner gives the path of an image +on the host they have the right to use, PNG, JPEG or WebP, up to 16,384 pixels a side and 64 MB. +Observer keeps a copy in its data folder (`map/`) and serves it only to accounts that see the map. To +place it in the game, stand at two spots you know on the image, far apart, pick yourself in the list and +click where you stand, or type the game coordinates of the image's four edges. **Grid** draws a grid of +game coordinates, which also stands in for an image not placed yet and for tiles Rockstar's servers did +not send. Positions come from the bridge, which reads them from OneSync: without OneSync there are none. The server sends them every 1, 2 or 5 seconds, only while someone has the page open, and stops within 30 diff --git a/crates/fake-fxserver/src/main.rs b/crates/fake-fxserver/src/main.rs index 09af6683..f007d8f7 100644 --- a/crates/fake-fxserver/src/main.rs +++ b/crates/fake-fxserver/src/main.rs @@ -13,8 +13,22 @@ //! - `fake_quit_delay_ms` (500): time between `quit` and the end, which is what FXServer 35245 does: //! exit code 1 on Windows, SIGSEGV on Linux; //! - `fake_ignore_quit 1`: `quit` does nothing; +//! - `fake_start_ms ` (0): how long each resource takes to start, and `fake_start_ms:` +//! for one of them. FXServer 38435 prints `Started resource` once a resource is up and nothing +//! while it reads its stream files, which took a clothing pack minutes on a first start; +//! - `fake_fail_start `: that resource does not start, as one whose asset is disabled; +//! - `fake_heavy `, `fake_missing `, each a list separated by commas: as it starts +//! one of them, FXServer warns of two heavy streamed files, one of them oversized, or of a script +//! its manifest names and it does not hold, on the channel `resources::stream` cut at +//! twenty characters, and counts the warnings on the line that says the resource started; //! - `fake_heartbeat_ms` (3000): the bridge heartbeat period; //! - `fake_heap_ms` (10000): how often the bridge reports the JS heap; +//! - a build whose scheduler loads Observer's hook, which is its first line naming the bridge and +//! `hook.lua` in the bridge's folder: every started resource but the bridge prints the hook's +//! first report, as each server Lua runtime would, save those of `fake_debug_silent `, +//! which run no Lua on the server; with `fake_debug_demo 1`, the resources named `pma-voice`, +//! `chat`, `baseevents`, `garage` and `logs` then count each second what a small busy server +//! would; //! - `fake_players `: `/dynamic.json` counts `n` clients and `/players.json` lists ids 1 to `n` //! with a ping of `20 + 5 × id`, or the anonymised public list without the right token; //! - `fake_tick_ms ` (0): how long each svMain tick is observed to take in `/perf/`; @@ -46,6 +60,8 @@ //! - on stdin: `fake_move `, which puts a player at a position the bridge reports //! while Observer asks for positions; the others go round a circle. With `onesync off` the bridge //! reports no position, as it reads them from OneSync; +//! - on stdin: `fake_debug `, one row of a hook's report (`ev`, `np`, +//! `un`, `sb`, `sbl` or `http`) printed as that resource; //! - on stdin: `fake_crash [code]`, `fake_hang`, `fake_spin` (a busy loop), `fake_print `, //! `fake_trace `, `fake_stall_input `, which stops reading stdin for a while, as //! FXServer does while its main loop is stuck; `fake_spawn [NAME=value...] `, @@ -71,7 +87,7 @@ mod traces; use std::collections::HashMap; use std::io::{self, BufRead}; use std::net::{TcpListener, TcpStream, UdpSocket}; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::process::{self, Child, Command, Stdio}; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Mutex, PoisonError, mpsc}; @@ -188,6 +204,15 @@ struct Bridge { contract: u64, } +/// While the build's scheduler loads Observer's hook: every resource counts and reports. +struct Counting { + next: Instant, + /// The seconds counted so far, which the demo varies with. + second: u64, + /// How many of the started resources sent their first report. + greeted: usize, +} + /// While Observer asks for positions: how often the bridge reports them, until when. struct Following { every: Duration, @@ -211,6 +236,7 @@ struct Fake { traces: Traces, endpoint: Option, bridge: Option, + counting: Option, /// Every resource started, in order. started: Vec, quit_at: Option, @@ -220,6 +246,84 @@ struct Fake { datagrams: Vec, } +/// What a resource of the demo server counts in its `second`, by its name: a voice resource that +/// talks to clients all the time, with one client far above the others, a chat, a garage that +/// writes state bags and refuses an event, and a logger that calls a webhook. +fn demo_counts(resource: &str, second: u64) -> Option { + // A slow wave, so that the traffic moves on the page. + let wave = (second % 60).abs_diff(30); + Some(match resource { + "pma-voice" => { + let out = 150 + wave; + let talkers = [(1, 19), (2, 3), (3, 2)]; + let sent: u64 = talkers.iter().map(|(_, count)| count).sum(); + let mut events = vec![ + json!(["toC", "pma-voice:setTalkingOnRadio", out, out * 41, 44, 0, 0]), + json!([ + "inN", + "pma-voice:setTalkingOnRadio", + sent, + sent * 38, + 40, + sent * 20, + 60 + ]), + ]; + if second.is_multiple_of(3) { + events.push(json!(["toC", "pma-voice:syncRadioData", 1, 1200, 3800, 0, 0])); + } + if second.is_multiple_of(5) { + events.push(json!(["inN", "pma-voice:setPlayerRadio", 1, 36, 36, 40, 40])); + } + json!({ + "ev": events, + "np": talkers + .iter() + .map(|(player, count)| json!(["pma-voice:setTalkingOnRadio", player, count, count * 38])) + .collect::>(), + "sb": [ + ["player", "proximity", 28, 252, 9, 28], + ["player", "voiceIntent", 9, 72, 8, 9], + ["player", "radioChannel", 3, 15, 5, 3], + ], + "sbl": [["player:1", "proximity", 9, 1]], + }) + } + "chat" if second.is_multiple_of(4) => json!({ + "ev": [ + ["toA", "chat:addMessage", 1, 310, 310, 0, 0], + ["inN", "_chat:messageEntered", 1, 120, 120, 90, 90], + ], + "np": [["_chat:messageEntered", 2, 1, 120]], + }), + "baseevents" if second.is_multiple_of(10) => json!({ + "ev": [ + ["inN", "baseevents:enteredVehicle", 1, 84, 84, 30, 30], + ["inI", "playerJoining", 1, 12, 12, 15, 15], + ["in", "playerDropped", 1, 48, 48, 20, 20], + ], + "np": [["baseevents:enteredVehicle", 3, 1, 84]], + }), + "garage" => { + let mut report = json!({ + "sb": [["entity", "fuel", 4, 1240, 2100, 4], ["global", "weather", 1, 64, 64, 0]], + "sbl": [["entity:1042", "fuel", 2100, 1]], + }); + if second.is_multiple_of(20) { + report["un"] = json!([["garage:giveVehicle", 1, 1]]); + report["ev"] = json!([["inN", "garage:giveVehicle", 1, 60, 60, 0, 0]]); + report["np"] = json!([["garage:giveVehicle", 1, 1, 60]]); + } + if second == 30 { + report["http"] = json!([["POST", "paste.example", 1, 2300]]); + } + report + } + "logs" if second % 5 < 3 => json!({ "http": [["POST", "discord.com", 1, 400]] }), + _ => return None, + }) +} + fn main() { let args: Vec = std::env::args().skip(1).collect(); if args.first().map(String::as_str) == Some(HOLD_PIPES) @@ -246,6 +350,7 @@ fn main() { last_request: 0, endpoint: None, bridge: None, + counting: None, started: Vec::new(), quit_at: None, children: Vec::new(), @@ -262,6 +367,33 @@ fn main() { process::exit(101); } +/// The resources under the folders named `category` of `folder`, in name order, as FXServer +/// starts every one with that folder on its path. +fn category_members(folder: &Path, category: &str) -> Vec { + let mut members = Vec::new(); + let mut folders = vec![(folder.to_path_buf(), false)]; + while let Some((folder, inside)) = folders.pop() { + let Ok(entries) = std::fs::read_dir(folder) else { + continue; + }; + for entry in entries.flatten() { + let Some(name) = entry.file_name().to_str().map(str::to_owned) else { + continue; + }; + if !entry.path().is_dir() { + continue; + } + if name.starts_with('[') && name.ends_with(']') { + folders.push((entry.path(), inside || name.eq_ignore_ascii_case(category))); + } else if inside { + members.push(name); + } + } + } + members.sort_unstable(); + members +} + impl Fake { fn boot(&mut self, commands: &[Vec]) { // FXServer applies every `+set` of its command line before it runs anything else, the cfg @@ -331,6 +463,7 @@ impl Fake { self.admit_unanswered(now); self.report_positions(now); self.beat(now); + self.count(now); self.server.tick(); thread::sleep(TICK); } @@ -348,15 +481,10 @@ impl Fake { ("exec", Some(file)) => self.exec(file), ("endpoint_add_tcp", Some(address)) => self.endpoint = Some(address.to_owned()), ("endpoint_add_udp", _) => {} - ("ensure" | "start", Some(resource)) => { - let restarted = self.started.iter().any(|started| started == resource); - if self.bridge.as_ref().is_some_and(|bridge| bridge.name == resource) { - stop(resource); - } - self.start(resource); - let events: &[&str] = if restarted { &["stop", "start"] } else { &["start"] }; - self.resources_changed(resource, events); + ("ensure" | "start", Some(category)) if category.starts_with('[') && category.ends_with(']') => { + self.ensure_category(category); } + ("ensure" | "start", Some(resource)) => self.ensure(resource), ("restart", Some(resource)) => { stop(resource); self.start(resource); @@ -405,6 +533,7 @@ impl Fake { ); } ("fake_trace", Some(kind)) => self.trace(kind), + ("fake_debug", Some(resource)) => self.counted(resource, &args[1..]), ("fake_join", _) => self.join(args.first(), args.get(1), args.get(2)), ("fake_connecting", _) => self.connecting(args.first(), args.get(1), args.get(2)), ("fake_move", Some(id)) => self.place(id, &args[1..]), @@ -454,14 +583,78 @@ impl Fake { } } + fn ensure_category(&mut self, category: &str) { + let members = category_members(Path::new("resources"), category); + if members.is_empty() { + console::line(SERVER, &format!("Couldn't find resource category {category}.")); + } + for member in members { + self.ensure(&member); + } + } + + fn ensure(&mut self, resource: &str) { + if self.server.convar("fake_fail_start").as_deref() == Some(resource) { + console::line(SERVER, &format!("Couldn't start resource {resource}.")); + return; + } + let restarted = self.started.iter().any(|started| started == resource); + if self.bridge.as_ref().is_some_and(|bridge| bridge.name == resource) { + stop(resource); + } + self.start(resource); + let events: &[&str] = if restarted { &["stop", "start"] } else { &["start"] }; + self.resources_changed(resource, events); + } + fn start(&mut self, resource: &str) { console::line(SCRIPTING, &format!("Creating script environments for {resource}")); - console::line(RESOURCES, &format!("Started resource {resource}")); + let own = self.server.number(&format!("fake_start_ms:{resource}")); + if let Some(ms) = own.or_else(|| self.server.number("fake_start_ms")) { + thread::sleep(Duration::from_millis(ms)); + } + let listed = |convar: &str| { + self.server + .convar(convar) + .is_some_and(|names| names.split(',').any(|name| name == resource)) + }; + let stream = format!("resources:{resource}:stream"); + let channel = (stream.get(..20).unwrap_or(&stream), 83); + let mut warnings = Vec::new(); + if listed("fake_heavy") { + warnings.push(format!( + "Asset {resource}/{resource}_hi.ydr uses 49.5 MiB of physical memory. Oversized assets \ + can and WILL lead to streaming issues (such as models not loading/rendering)." + )); + warnings.push(format!( + "Asset {resource}/{resource}.ytd uses 16.2 MiB of physical memory." + )); + } + if listed("fake_missing") { + warnings.push( + "Warning: could not find client_script `client/gone.lua` (defined in fxmanifest.lua:9)" + .to_owned(), + ); + } + for warning in &warnings { + console::line(channel, warning); + } + let counted = match warnings.len() { + 0 => String::new(), + 1 => " (1 warning)".to_owned(), + several => format!(" ({several} warnings)"), + }; + console::line(RESOURCES, &format!("Started resource {resource}{counted}")); if !self.started.iter().any(|started| started == resource) { self.started.push(resource.to_owned()); } if let Some(contract) = self.bridge_script(resource).as_deref().and_then(contract_version) { let heap_period = Duration::from_millis(self.server.number("fake_heap_ms").unwrap_or(10_000)); + self.counting = self.hooked(resource).then(|| Counting { + next: Instant::now(), + second: 0, + greeted: 0, + }); self.bridge = Some(Bridge { name: resource.to_owned(), next_heartbeat: Instant::now(), @@ -699,11 +892,91 @@ impl Fake { /// The script of `resource` if it is Observer's bridge: a folder of the build's /// `system_resources` whose script registers `observer_cmd`. Its name changes with each start. - fn bridge_script(&self, resource: &str) -> Option { - let citizen = self.server.convar("citizen_dir").map(PathBuf::from).or_else(|| { + fn citizen(&self) -> Option { + self.server.convar("citizen_dir").map(PathBuf::from).or_else(|| { let program = std::env::current_exe().ok()?; Some(program.parent()?.join("citizen")) - })?; + }) + } + + /// Whether the build's scheduler loads the hook of `bridge`, as the real FXServer would find + /// when it runs the scheduler's first line in each resource. + fn hooked(&self, bridge: &str) -> bool { + let Some(citizen) = self.citizen() else { + return false; + }; + let scheduler = citizen.join("scripting").join("lua").join("scheduler.lua"); + let loads = std::fs::read_to_string(scheduler).is_ok_and(|scheduler| { + scheduler + .trim_start_matches('\u{feff}') + .starts_with("--[[obs:hook:head ") + && scheduler.contains(&format!("LoadResourceFile('{bridge}', 'hook.lua')")) + }); + loads + && citizen + .join("system_resources") + .join(bridge) + .join("hook.lua") + .is_file() + } + + /// What the hook prints from every resource: a first report from each, then, with + /// `fake_debug_demo`, what a small busy server counts each second. + fn count(&mut self, now: Instant) { + let Some(counting) = &mut self.counting else { + return; + }; + if now < counting.next { + return; + } + counting.next = now + Duration::from_secs(1); + counting.second += 1; + let second = counting.second; + let greeted = std::mem::replace(&mut counting.greeted, self.started.len()); + let demo = self.server.flag("fake_debug_demo"); + let silent = self.server.convar("fake_debug_silent").unwrap_or_default(); + let bridge = self.bridge.as_ref().map(|bridge| bridge.name.as_str()); + for (index, resource) in self.started.iter().enumerate() { + if Some(resource.as_str()) == bridge || silent.split(',').any(|name| name == resource) { + continue; + } + let counted = if demo { demo_counts(resource, second) } else { None }; + let mut report = match (counted, index >= greeted) { + (Some(counted), _) => counted, + (None, true) => json!({}), + (None, false) => continue, + }; + if index >= greeted { + report["hello"] = json!({ "hook": "1", "timer": "nanotime" }); + } + report["type"] = json!("observer_debug"); + report["v"] = json!(1); + report["seq"] = json!(second); + report["dur"] = json!(1000); + self.traces.script(resource, &report); + } + } + + /// `fake_debug `: one row of the hook's report, as `resource`. + fn counted(&self, resource: &str, row: &[String]) { + let Some((list, fields)) = row.split_first() else { + return; + }; + let fields: Vec = fields + .iter() + .map(|field| { + field + .parse::() + .map_or_else(|_| json!(field), |number| json!(number)) + }) + .collect(); + let mut report = json!({ "type": "observer_debug", "v": 1, "seq": 1, "dur": 1000 }); + report[list.as_str()] = json!([fields]); + self.traces.script(resource, &report); + } + + fn bridge_script(&self, resource: &str) -> Option { + let citizen = self.citizen()?; let script = citizen.join("system_resources").join(resource).join("server.lua"); std::fs::read_to_string(script) .ok() diff --git a/crates/fake-fxserver/tests/supervisor.rs b/crates/fake-fxserver/tests/supervisor.rs index 5a187fb7..fc6155ca 100644 --- a/crates/fake-fxserver/tests/supervisor.rs +++ b/crates/fake-fxserver/tests/supervisor.rs @@ -6,6 +6,7 @@ use std::fs; use std::net::{SocketAddr, TcpListener, UdpSocket}; use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, LazyLock, Mutex, PoisonError}; use std::time::{Duration, Instant}; @@ -20,12 +21,15 @@ use observer_fxserver::console::{ConsoleHub, LineKind, plain}; use observer_fxserver::crashes::analysis; use observer_fxserver::crashes::{HangSnapshotLimits, SnapshotFailure}; use observer_fxserver::health::{HangCause, HealthLimits, Problem}; +use observer_fxserver::hook::{self, Way}; use observer_fxserver::launch::ServerLaunch; +use observer_fxserver::layout::{Flavour, Layout}; use observer_fxserver::perf::Thread; use observer_fxserver::recovery::RecoveryPolicy; use observer_fxserver::stdin::{StdinError, UserCommand}; use observer_fxserver::supervisor::{ - Cause, ControlError, Event, EventKind, ProcessChange, Settings, State, StopTimings, Supervisor, TreeEvent, + Cause, ControlError, Counting, Event, EventKind, Hooked, ProcessChange, START_EXTENSION, Settings, + StartStatus, State, StopTimings, Supervisor, TreeEvent, }; use observer_platform::{ProcessInfo, Protocol, Socket, SocketState}; use tempfile::TempDir; @@ -247,6 +251,27 @@ impl Lab { self.supervisor.status().borrow().generation } + /// The start under way once it is as `wanted`, as the status publishes it. + async fn start_until(&self, wanted: impl Fn(&StartStatus) -> bool) -> StartStatus { + let mut status = self.supervisor.status(); + let found = timeout(WAIT, async { + loop { + let start = status.borrow_and_update().start.clone(); + if let Some(start) = start.filter(&wanted) { + return start; + } + status.changed().await.unwrap(); + } + }) + .await; + found.unwrap_or_else(|_| { + panic!( + "the start never came to that: {:#?}", + self.supervisor.status().borrow().start + ) + }) + } + /// The bridge's name in the server that runs now. fn bridge(&self) -> String { self.supervisor.status().borrow().bridge.clone().unwrap() @@ -856,6 +881,261 @@ async fn a_hung_main_loop_is_detected_and_the_server_restarted() { lab.shut_down(false).await; } +/// Adds `lines` to the lab's cfg, with a folder for each resource they start. +fn extend_cfg(lab: &Lab, resources: &[&str], lines: &[&str]) { + let data = lab.dir.path().join("server-data"); + for resource in resources { + let folder = data.join("resources").join(resource); + fs::create_dir_all(&folder).unwrap(); + fs::write( + folder.join("fxmanifest.lua"), + "fx_version 'cerulean'\ngame 'gta5'\n", + ) + .unwrap(); + } + let cfg = data.join("server.cfg"); + let added: String = resources + .iter() + .map(|resource| format!("ensure {resource}\n")) + .chain(lines.iter().map(|line| format!("{line}\n"))) + .collect(); + fs::write(&cfg, fs::read_to_string(&cfg).unwrap() + &added).unwrap(); +} + +#[tokio::test] +async fn a_start_longer_than_the_hang_delay_goes_online_while_it_keeps_starting_resources() { + // Eight resources of 600 ms each: a start of five seconds, under limits that take a server + // silent for two for hung. + let mut lab = Lab::new(&[("fake_start_ms", "600")]).await; + extend_cfg( + &lab, + &[ + "maps", "clothes", "vehicles", "weapons", "props", "peds", "sounds", "core", + ], + &[], + ); + + lab.supervisor.start("test").await.unwrap(); + + assert_eq!(lab.state(State::Online).await, Cause::Ready); + assert!( + !lab.seen.iter().any(hung_or_exited), + "the start was taken for a hang: {:#?}", + lab.seen + ); + assert_eq!(lab.generation(), 1); + lab.console_until("Started resource core").await; + lab.shut_down(false).await; +} + +#[tokio::test] +async fn a_start_that_stops_starting_resources_is_hung_and_says_where_it_stopped() { + let mut lab = Lab::new(&[("fake_start_ms", "300")]).await; + // The cfg sticks after its second resource, before any thread of the bridge runs. + extend_cfg(&lab, &["maps", "clothes"], &["fake_hang"]); + + lab.supervisor.start("test").await.unwrap(); + + assert_eq!( + lab.state(State::Hung).await, + Cause::Hang { + cause: HangCause::NoHeartbeat + } + ); + let stopped = lab.console_until("Started resource clothes").await; + let note = lab + .console_until("its start is stuck: no resource started since clothes, ") + .await; + assert!(stopped < note); + assert_eq!( + lab.state(State::Crashed).await.clone(), + Cause::Hang { + cause: HangCause::NoHeartbeat + } + ); + lab.supervisor.stop("test").await.ok(); + lab.shut_down(false).await; +} + +#[tokio::test] +async fn a_start_is_followed_through_the_commands_of_its_cfg() { + // Each resource outlasts a tick, so that every command shows in a status. + let mut lab = Lab::tuned(&[("fake_start_ms", "1800")], |settings| { + settings.health.heartbeat_fatal = Duration::from_secs(30); + }) + .await; + extend_cfg(&lab, &["maps", "clothes", "core"], &[]); + assert_eq!(lab.supervisor.status().borrow().start, None); + + lab.supervisor.start("test").await.unwrap(); + + let start = lab + .start_until(|start| { + start + .progress + .at + .as_ref() + .is_some_and(|at| at.target == "clothes") + }) + .await; + let progress = start.progress; + let at = progress.at.unwrap(); + assert_eq!( + (at.file.as_str(), at.line, at.done, at.of), + ("server.cfg", 6, 0, 1) + ); + let last = progress.last.unwrap(); + assert_eq!(last.name, "maps"); + assert_eq!(at.since_ms, last.at_ms, "FXServer came to clothes as maps was up"); + assert_eq!( + ( + progress.started, + progress.failed, + progress.planned, + progress.found + ), + (2, 0, Some(3), Some(4)), + "the bridge and maps, of the three the cfg starts" + ); + assert!( + progress.slowest.iter().all(|slow| slow.ms >= 1800), + "{:?}", + progress.slowest + ); + assert!(start.stuck_at_ms.is_some_and(|stuck| stuck > last.at_ms)); + + let start = lab + .start_until(|start| start.progress.at.as_ref().is_some_and(|at| at.target == "core")) + .await; + assert_eq!(start.progress.started, 3); + assert_eq!(lab.state(State::Online).await, Cause::Ready); + assert_eq!( + lab.supervisor.status().borrow().start, + None, + "a start is over once online" + ); + lab.shut_down(false).await; +} + +#[tokio::test] +async fn a_category_of_the_cfg_is_followed_as_the_resources_under_it() { + let mut lab = Lab::tuned(&[("fake_start_ms", "1800")], |settings| { + settings.health.heartbeat_fatal = Duration::from_secs(30); + }) + .await; + // Created as `resources/[maps]/docks` and `resources/[maps]/pier`, started by one command. + extend_cfg(&lab, &[], &["set fake_fail_start pier"]); + extend_cfg(&lab, &["[maps]/docks", "[maps]/pier"], &[]); + let cfg = lab.dir.path().join("server-data/server.cfg"); + let text = fs::read_to_string(&cfg) + .unwrap() + .replace("ensure [maps]/docks\nensure [maps]/pier\n", "ensure [maps]\n"); + fs::write(&cfg, text).unwrap(); + + lab.supervisor.start("test").await.unwrap(); + + let start = lab + .start_until(|start| start.progress.at.as_ref().is_some_and(|at| at.target == "[maps]")) + .await; + let at = start.progress.at.unwrap(); + assert_eq!((at.line, at.of), (6, 2)); + assert_eq!(start.progress.planned, Some(2)); + assert_eq!(lab.state(State::Online).await, Cause::Ready); + lab.console_until("Couldn't start resource pier.").await; + lab.shut_down(false).await; +} + +#[tokio::test] +async fn what_fxserver_warns_of_as_it_starts_is_kept_with_the_resource_it_was_starting() { + let mut lab = Lab::new(&[]).await; + assert_eq!(lab.supervisor.start_warnings(), None, "before the first start"); + extend_cfg( + &lab, + &[], + &[ + "set fake_heavy maps,a_long_resource_name", + "set fake_missing maps", + ], + ); + extend_cfg(&lab, &["chat", "maps", "a_long_resource_name"], &[]); + + lab.supervisor.start("test").await.unwrap(); + assert_eq!(lab.state(State::Online).await, Cause::Ready); + + lab.console_until("Started resource maps (3 warnings)").await; + let (_, warnings) = lab.supervisor.start_warnings().unwrap(); + let heavy: Vec<(&str, &str, u32, bool)> = warnings + .heavy + .iter() + .map(|asset| { + ( + asset.resource.as_str(), + asset.file.as_str(), + asset.tenths, + asset.oversized, + ) + }) + .collect(); + assert_eq!( + heavy, + [ + ("maps", "maps_hi.ydr", 495, true), + ("maps", "maps.ytd", 162, false), + ("a_long_resource_name", "a_long_resource_name_hi.ydr", 495, true), + ("a_long_resource_name", "a_long_resource_name.ytd", 162, false), + ] + ); + let missing: Vec<(&str, &str, &str, u32)> = warnings + .missing + .iter() + .map(|file| { + ( + file.resource.as_str(), + file.kind.as_str(), + file.file.as_str(), + file.line, + ) + }) + .collect(); + assert_eq!(missing, [("maps", "client_script", "client/gone.lua", 9)]); + lab.shut_down(false).await; +} + +#[tokio::test] +async fn a_start_given_more_time_is_not_taken_for_stuck() { + // The bridge alone takes longer to start than the two seconds a start may say nothing. + let mut lab = Lab::new(&[("fake_start_ms", "3500")]).await; + assert_eq!( + lab.supervisor.extend_start("test").await, + Err(ControlError::NotStarting) + ); + lab.supervisor.start("test").await.unwrap(); + let before = lab.start_until(|start| start.stuck_at_ms.is_some()).await; + + lab.supervisor.extend_start("ada").await.unwrap(); + + let extended = lab + .start_until(|start| start.stuck_at_ms != before.stuck_at_ms) + .await; + let given = extended.stuck_at_ms.unwrap() - before.stuck_at_ms.unwrap(); + let extension = i64::try_from(START_EXTENSION.as_millis()).unwrap(); + assert!((extension - 1000..=extension + 1000).contains(&given), "{given}"); + assert_eq!(lab.state(State::Online).await, Cause::Ready); + assert!( + !lab.seen.iter().any(hung_or_exited), + "the start was taken for a hang: {:#?}", + lab.seen + ); + lab.console_until("ada gave this start 15 more minutes before it is taken for stuck") + .await; + assert_eq!( + lab.supervisor.extend_start("test").await, + Err(ControlError::NotStarting), + "its cfg has run" + ); + lab.shut_down(false).await; +} + /// Launch settings with hang detection off; the lab's own launch keeps it on. fn without_hang_detection(settings: &mut Settings) { if let Some(launch) = &mut settings.launch { @@ -1736,3 +2016,91 @@ async fn a_detached_child_with_an_environment_of_its_own_is_not_ended_while_it_r "{process:?} was reported ended while it ran" ); } + +/// A start that counts puts the hook and its two lines into the build and reads what each +/// resource reports under the name FXServer gives it; the next start that does not count takes +/// the lines out again. +#[tokio::test] +async fn a_start_that_counts_hooks_the_build_and_the_next_that_does_not_restores_it() { + let counting = Arc::new(AtomicBool::new(true)); + let asked = Arc::clone(&counting); + let mut lab = Lab::tuned(&[], |settings| { + settings.counting = Some(Counting::new(move || asked.load(Ordering::Relaxed))); + }) + .await; + let scheduler = hook::scheduler(&Layout::new(&lab.launch.build, Flavour::HOST)); + let original = "local yield = coroutine.yield +Citizen.SetEventRoutine(function() end) +"; + fs::create_dir_all(scheduler.parent().unwrap()).unwrap(); + fs::write(&scheduler, original).unwrap(); + let mut reports = lab.supervisor.hook_reports(); + + lab.supervisor.start("test").await.unwrap(); + lab.state(State::Online).await; + + let counted = fs::read_to_string(&scheduler).unwrap(); + let bridge = lab.bridge(); + assert!(counted.starts_with("--[[obs:hook:head "), "{counted}"); + assert!(counted.contains(&format!("LoadResourceFile('{bridge}', 'hook.lua')"))); + assert_eq!(counted.lines().count(), original.lines().count() + 1); + assert!(lab.system_resources().join(&bridge).join("hook.lua").is_file()); + assert_eq!( + lab.supervisor.hooked(), + Some(Hooked { + generation: lab.generation(), + refused: None + }) + ); + // What the bridge itself counts is Observer's, not the server's: it is left out. + lab.type_command( + &format!("fake_debug {bridge} ev inI playerJoining 1 12 12 15"), + "test", + ) + .await; + lab.type_command("fake_debug chat ev toA chat:addMessage 4 1240 310 0", "test") + .await; + let report = timeout(WAIT, reports.recv()).await.unwrap().unwrap(); + assert_eq!(report.generation, lab.generation()); + assert_eq!(report.report.resource, "chat"); + let event = &report.report.events[0]; + assert_eq!( + ( + event.way, + event.name.as_str(), + event.count, + event.bytes, + event.largest + ), + (Way::ToAll, "chat:addMessage", 4, 1240, 310) + ); + + counting.store(false, Ordering::Relaxed); + lab.supervisor.restart("test").await.unwrap(); + lab.state(State::Starting).await; + lab.state(State::Online).await; + + assert_eq!(fs::read_to_string(&scheduler).unwrap(), original); + assert_eq!(lab.supervisor.hooked(), None); + lab.shut_down(false).await; +} + +/// A build whose scheduler Observer cannot change starts all the same, counts nothing, and the +/// console says why. +#[tokio::test] +async fn a_build_whose_scheduler_cannot_be_hooked_starts_all_the_same() { + let mut lab = Lab::tuned(&[], |settings| { + settings.counting = Some(Counting::new(|| true)); + }) + .await; + + lab.supervisor.start("test").await.unwrap(); + lab.state(State::Online).await; + + let hooked = lab.supervisor.hooked().unwrap(); + assert_eq!(hooked.generation, lab.generation()); + let reason = hooked.refused.unwrap(); + assert!(reason.contains("scheduler.lua"), "{reason}"); + lab.console_until("Debug counts nothing in this start").await; + lab.shut_down(false).await; +} diff --git a/crates/observer-app/src/changes/job.rs b/crates/observer-app/src/changes/job.rs index 5e0fe08e..5d44e2d0 100644 --- a/crates/observer-app/src/changes/job.rs +++ b/crates/observer-app/src/changes/job.rs @@ -31,8 +31,9 @@ use super::plan::{self, Chain, ListingInput, ListingPass, Planned, Read, Slot}; use super::{SERVER, Shared}; use crate::Error; -/// Hashing waits for the server to be online, or this long after the spawn. -const ONLINE_WAIT: Duration = Duration::from_mins(3); +/// Hashing waits for the server to be online, or this long after the spawn: a start reads the same +/// files, and a big server's first one takes minutes that hashing beside it only adds to. +const ONLINE_WAIT: Duration = Duration::from_mins(30); /// Below this much free space on the data folder's volume, digests only are kept. const DISK_FLOOR: u64 = 2_000_000_000; /// Files read in one go of the hashing pass, between which progress is written. @@ -60,7 +61,7 @@ pub(crate) enum Ended { } /// Takes the snapshot of a start: the listing at once, the hashing once `status` says the run is -/// online, or three minutes later. +/// online, or half an hour later. pub(crate) async fn start( shared: Arc, settings: ChangesSettings, @@ -517,7 +518,7 @@ impl Work { Ok(()) } - /// Waits for the run to be online, for three minutes at most. A run that ended frees the disk, + /// Waits for the run to be online, for half an hour at most. A run that ended frees the disk, /// so hashing starts at once. async fn wait_online( &self, diff --git a/crates/observer-app/src/debug.rs b/crates/observer-app/src/debug.rs new file mode 100644 index 00000000..3ab2a58e --- /dev/null +++ b/crates/observer-app/src/debug.rs @@ -0,0 +1,1348 @@ +//! What the server's resources count of themselves while Debug is on: events, state bag writes +//! and outgoing HTTP requests, totalled here over the last quarter of an hour and kept in memory. +//! A start of FXServer empties them. The hosts each resource called are the one thing written +//! down, so that a host called for the first time stands out. + +use std::cmp::Reverse; +use std::collections::{BTreeSet, HashMap, HashSet, VecDeque}; +use std::sync::{Arc, Mutex, PoisonError}; + +use jiff::{SignedDuration, Timestamp}; +use observer_domain::audit::AuditAction; +use observer_domain::debug::{ + BagKind, BagResourceRow, BagRow, BagTotals, DebugBags, DebugEvents, DebugHttp, DebugState, DebugStatus, + DebugWindow, EventPlayerRow, EventResourceRow, EventRow, EventSender, EventTotals, EventWay, Flow, + HttpRow, HttpTotals, LargestWrite, RefusedRow, TrafficPoint, +}; +use observer_domain::error::ErrorCode; +use observer_domain::features::Feature; +use observer_fxserver::hook::{self, Report, Way}; +use observer_fxserver::supervisor::{Event, EventKind, HookReport, Supervisor}; +use observer_store::Database; +use observer_store::debug::{self as store, StoredHost}; +use serde_json::json; +use tokio::sync::broadcast; +use tokio::sync::broadcast::error::RecvError; +use tokio::task::JoinHandle; +use tokio_util::sync::CancellationToken; +use tracing::warn; + +use crate::audit::Record; +use crate::{Caller, Clock, Error, Features, Supervision}; + +/// The counts are kept in slices of this many seconds. +const SLICE_SECONDS: u32 = 5; +/// A quarter of an hour of them. +const SLICES: i64 = 180; +/// The rows one slice keeps apart of each list. Past it the rest is left out, and the page says +/// that it shows less than there was. +const SLICE_ROWS: usize = 4000; +/// The rows a page is given of each list. +const PAGE_ROWS: usize = 200; +const SENDERS: usize = 5; +const LARGEST: usize = 10; +const SILENT_NAMES: usize = 100; +const REPORTING: usize = 4000; +/// The hosts remembered, over every resource. +const MAX_HOSTS: usize = 5000; +/// How long every host is taken as known, from the first report Debug ever read. +const LEARNING: SignedDuration = SignedDuration::from_hours(24); + +/// What [`DebugCounts::start`] works with. +pub struct DebugSetup { + pub db: Database, + pub clock: Clock, + pub supervision: Supervision, + pub features: Features, +} + +/// The Debug service. Clones share it. +#[derive(Clone)] +pub struct DebugCounts { + inner: Arc, +} + +struct Inner { + db: Database, + clock: Clock, + supervisor: Supervisor, + features: Features, + counts: Mutex, + hosts: Mutex, +} + +/// The task that reads the reports, until [`DebugTask::close`]. +pub struct DebugTask { + stop: CancellationToken, + task: JoinHandle<()>, +} + +impl DebugTask { + pub async fn close(self) { + self.stop.cancel(); + if let Err(error) = self.task.await { + warn!(%error, "the Debug task did not end cleanly"); + } + } +} + +#[derive(Debug, Clone, Copy, Default)] +struct EventSum { + count: u64, + bytes: u64, + largest: u64, + handler_us: u64, +} + +#[derive(Debug, Clone, Copy, Default)] +struct BagSum { + count: u64, + bytes: u64, + largest: u64, + replicated: u64, + replicated_bytes: u64, +} + +type Name = Arc; + +/// What was counted in one slice of the window. +#[derive(Debug, Default)] +struct Slice { + /// The slice's place in time: its seconds since the epoch over [`SLICE_SECONDS`]. + index: i64, + /// By resource, way and event. + events: HashMap<(Name, EventWay, Name), EventSum>, + /// By event and client, for what clients sent. + senders: HashMap<(Name, u32), Flow>, + /// By resource, event and client. + refused: HashMap<(Name, Name, Option), u64>, + /// By resource, bag kind and key. + bags: HashMap<(Name, BagKind, Name), BagSum>, + largest: Vec, + /// By resource, method and host. + hosts: HashMap<(Name, Name, Name), Flow>, + to_clients: u64, + from_clients: u64, + /// Less was kept than there was. + cut: bool, +} + +/// The counts of the start that runs. +#[derive(Debug, Default)] +struct Counts { + generation: u32, + since: Option, + reporting: BTreeSet, + /// What the bridge said is started, and at which start. + started: (u32, Vec), + slices: VecDeque, + /// Reports were lost between FXServer and here. + lagged: bool, +} + +const fn way_of(way: Way) -> Option { + match way { + Way::Received => Some(EventWay::Local), + Way::FromClient => Some(EventWay::FromClients), + Way::FromEngine => Some(EventWay::FromEngine), + Way::ToClient => Some(EventWay::ToClient), + Way::ToAll => Some(EventWay::ToAll), + // The resources that received it count it: counted here too, it would show twice. + Way::Triggered => None, + } +} + +const fn kind_of(kind: hook::BagKind) -> BagKind { + match kind { + hook::BagKind::Global => BagKind::Global, + hook::BagKind::Player => BagKind::Player, + hook::BagKind::Entity => BagKind::Entity, + hook::BagKind::LocalEntity => BagKind::LocalEntity, + hook::BagKind::Other => BagKind::Other, + } +} + +/// The entry of `key`, unless the map is full and does not hold it. +fn row<'a, K: std::hash::Hash + Eq, V: Default>( + map: &'a mut HashMap, + key: K, + cut: &mut bool, +) -> Option<&'a mut V> { + if map.len() >= SLICE_ROWS && !map.contains_key(&key) { + *cut = true; + return None; + } + Some(map.entry(key).or_default()) +} + +fn slice_of(at: Timestamp) -> i64 { + at.as_second().div_euclid(i64::from(SLICE_SECONDS)) +} + +/// A count of rows, for the wire. +fn small(count: usize) -> u32 { + u32::try_from(count).unwrap_or(u32::MAX) +} + +impl Slice { + /// Adds the events one resource counted, with who sent them and what was refused. + fn events(&mut self, resource: &Name, report: &Report) { + for event in &report.events { + let Some(way) = way_of(event.way) else { + continue; + }; + match way { + EventWay::ToClient | EventWay::ToAll => self.to_clients += event.bytes, + EventWay::FromClients => self.from_clients += event.bytes, + EventWay::Local | EventWay::FromEngine => {} + } + let key = (Arc::clone(resource), way, event.name.as_str().into()); + if let Some(sum) = row(&mut self.events, key, &mut self.cut) { + sum.count += event.count; + sum.bytes += event.bytes; + sum.largest = sum.largest.max(event.largest); + sum.handler_us += event.handler_us; + } + } + for sender in &report.senders { + let key = (sender.name.as_str().into(), sender.player); + if let Some(flow) = row(&mut self.senders, key, &mut self.cut) { + flow.count += sender.count; + flow.bytes += sender.bytes; + } + } + for refused in &report.refused { + let key = (Arc::clone(resource), refused.name.as_str().into(), refused.player); + if let Some(count) = row(&mut self.refused, key, &mut self.cut) { + *count += refused.count; + } + } + } + + /// Adds the state bag writes and the requests one resource counted. + fn writes(&mut self, resource: &Name, report: Report) { + for bag in report.bags { + let key = (Arc::clone(resource), kind_of(bag.kind), bag.key.as_str().into()); + if let Some(sum) = row(&mut self.bags, key, &mut self.cut) { + sum.count += bag.count; + sum.bytes += bag.bytes; + sum.largest = sum.largest.max(bag.largest); + sum.replicated += bag.replicated; + // A key goes out to clients or stays on the server: its writes share the bytes. + sum.replicated_bytes += bag + .bytes + .saturating_mul(bag.replicated.min(bag.count)) + .checked_div(bag.count) + .unwrap_or(0); + } + } + self.largest + .extend(report.largest.into_iter().map(|write| LargestWrite { + bag: write.bag, + key: write.key, + resource: resource.to_string(), + bytes: write.bytes, + replicated: write.replicated, + })); + self.largest.sort_by_key(|write| Reverse(write.bytes)); + self.largest.truncate(LARGEST); + for host in report.hosts { + let key = ( + Arc::clone(resource), + host.method.as_str().into(), + host.host.as_str().into(), + ); + if let Some(flow) = row(&mut self.hosts, key, &mut self.cut) { + flow.count += host.count; + flow.bytes += host.bytes; + } + } + } +} + +/// An event in one way, while its rows are added up. +#[derive(Default)] +struct EventSums { + sum: EventSum, + resources: HashMap, +} + +/// Who sent each event, by client. +type Senders = HashMap>; + +fn event_rows(rows: HashMap<(Name, EventWay), EventSums>, senders: &Senders) -> Vec { + let mut events: Vec = rows + .into_iter() + .map(|((name, way), row)| { + let mut by: Vec<(Name, Flow)> = row.resources.into_iter().collect(); + by.sort_by(|a, b| (b.1.bytes, b.1.count, &a.0).cmp(&(a.1.bytes, a.1.count, &b.0))); + let mut sent: Vec = senders + .get(&name) + .filter(|_| way == EventWay::FromClients) + .into_iter() + .flatten() + .map(|(&player, flow)| EventSender { + player, + count: flow.count, + bytes: flow.bytes, + }) + .collect(); + sent.sort_by_key(|by| (Reverse(by.count), by.player)); + sent.truncate(SENDERS); + EventRow { + name: name.to_string(), + way, + resource_count: small(by.len()), + resources: by.into_iter().take(3).map(|(name, _)| name.to_string()).collect(), + count: row.sum.count, + bytes: row.sum.bytes, + largest: row.sum.largest, + handler_us: row.sum.handler_us, + senders: sent, + } + }) + .collect(); + events.sort_by(|a, b| (b.bytes, b.count, &a.name, a.way).cmp(&(a.bytes, a.count, &b.name, b.way))); + events.truncate(PAGE_ROWS); + events +} + +/// What each client sent in all, with the event it sent most. +fn player_rows(senders: &Senders) -> Vec { + let mut players: HashMap = HashMap::new(); + for (name, by_player) in senders { + for (player, flow) in by_player { + let (total, top) = players.entry(*player).or_insert((Flow::default(), (0, name))); + total.count += flow.count; + total.bytes += flow.bytes; + if (flow.count, name) > *top { + *top = (flow.count, name); + } + } + } + let mut rows: Vec = players + .into_iter() + .map(|(player, (flow, (_, top)))| EventPlayerRow { + player, + count: flow.count, + bytes: flow.bytes, + top: top.to_string(), + }) + .collect(); + rows.sort_by_key(|row| (Reverse(row.count), row.player)); + rows.truncate(PAGE_ROWS); + rows +} + +impl Counts { + /// Adds what one resource counted, at `now`. The counts of an earlier start are dropped at + /// the first report of a new one. + fn add(&mut self, now: Timestamp, generation: u32, report: Report) { + if generation != self.generation { + let started = std::mem::take(&mut self.started); + *self = Self { + generation, + started, + ..Self::default() + }; + } + self.since.get_or_insert(now); + if self.reporting.len() < REPORTING { + self.reporting.insert(report.resource.clone()); + } + let resource: Name = report.resource.as_str().into(); + let slice = self.slice(now); + slice.cut |= report.overflow > 0; + slice.events(&resource, &report); + slice.writes(&resource, report); + } + + /// Drops what was counted, as the feature is turned off: counting starts afresh when it is + /// turned on again, and no window holds the gap in between. Which resources report is a fact + /// of the start, and stays: one that has nothing to count says so once and no more. + fn forget(&mut self) { + self.since = None; + self.slices.clear(); + self.lagged = false; + } + + /// The slice `now` falls in, the last one kept. Those older than a quarter of an hour go. + fn slice(&mut self, now: Timestamp) -> &mut Slice { + let index = slice_of(now).max(self.slices.back().map_or(i64::MIN, |last| last.index)); + if self.slices.back().is_none_or(|last| last.index != index) { + self.slices.push_back(Slice { + index, + ..Slice::default() + }); + } + while self + .slices + .front() + .is_some_and(|first| first.index <= index - SLICES) + { + self.slices.pop_front(); + } + let last = self.slices.len() - 1; + &mut self.slices[last] + } + + /// The slices of `window`, as of `now`. + fn window(&self, now: Timestamp, window: DebugWindow) -> impl Iterator { + let last = slice_of(now); + let first = last - i64::from(window.seconds() / SLICE_SECONDS); + self.slices + .iter() + .filter(move |slice| slice.index > first && slice.index <= last) + } + + /// The seconds of `window` that were counted as of `now`, so that a rate is not spread over + /// time nothing was counted in: the start may be younger than the window, and the window opens + /// with a slice. The whole window where nothing was counted at all. + fn covered(&self, now: Timestamp, window: DebugWindow) -> u32 { + let Some(since) = self.since else { + return window.seconds(); + }; + let slices = i64::from(window.seconds() / SLICE_SECONDS); + let opens = (slice_of(now) - slices + 1) * i64::from(SLICE_SECONDS); + let counted = now.as_second() - since.as_second().max(opens); + u32::try_from(counted.max(1)).unwrap_or(window.seconds()) + } + + /// Whether the whole of `slice` was counted as of `now`. The one this start began in and the + /// one that is not over hold less, and would draw as a dip in the traffic. + fn whole(&self, slice: &Slice, now: Timestamp) -> bool { + let opens = slice.index * i64::from(SLICE_SECONDS); + let over = opens + i64::from(SLICE_SECONDS) <= now.as_second(); + over && self.since.is_some_and(|since| since.as_second() <= opens) + } + + fn cut(&self, now: Timestamp, window: DebugWindow) -> bool { + self.lagged || self.window(now, window).any(|slice| slice.cut) + } + + /// Who sent each event from a client, over `window`. + fn senders(&self, now: Timestamp, window: DebugWindow) -> Senders { + let mut senders = Senders::new(); + for slice in self.window(now, window) { + for ((name, player), flow) in &slice.senders { + let by = senders + .entry(Arc::clone(name)) + .or_default() + .entry(*player) + .or_default(); + by.count += flow.count; + by.bytes += flow.bytes; + } + } + senders + } + + /// The events clients sent that no resource opened to them, with how many there were and + /// from how many clients. + fn refused(&self, now: Timestamp, window: DebugWindow) -> (Vec, u64, u32) { + let mut refused: HashMap<&(Name, Name, Option), u64> = HashMap::new(); + for slice in self.window(now, window) { + for (key, count) in &slice.refused { + *refused.entry(key).or_default() += count; + } + } + let total = refused.values().sum(); + let players: HashSet = refused.keys().filter_map(|(_, _, player)| *player).collect(); + let mut rows: Vec = refused + .into_iter() + .map(|((resource, name, player), count)| RefusedRow { + name: name.to_string(), + player: *player, + resource: resource.to_string(), + count, + }) + .collect(); + rows.sort_by(|a, b| (b.count, &a.name, a.player).cmp(&(a.count, &b.name, b.player))); + rows.truncate(PAGE_ROWS); + (rows, total, small(players.len())) + } + + fn events(&self, now: Timestamp, window: DebugWindow) -> Events { + let mut rows: HashMap<(Name, EventWay), EventSums> = HashMap::new(); + let mut resources: HashMap = HashMap::new(); + let mut totals = EventTotals::default(); + let mut series = Vec::new(); + for slice in self.window(now, window) { + if self.whole(slice, now) { + series.push(TrafficPoint { + at: Timestamp::from_second(slice.index * i64::from(SLICE_SECONDS)).unwrap_or(now), + seconds: SLICE_SECONDS, + to_clients: slice.to_clients, + from_clients: slice.from_clients, + }); + } + for ((resource, way, name), sum) in &slice.events { + let row = rows.entry((Arc::clone(name), *way)).or_default(); + row.sum.count += sum.count; + row.sum.bytes += sum.bytes; + row.sum.largest = row.sum.largest.max(sum.largest); + row.sum.handler_us += sum.handler_us; + let by = row.resources.entry(Arc::clone(resource)).or_default(); + by.count += sum.count; + by.bytes += sum.bytes; + let of = resources + .entry(Arc::clone(resource)) + .or_insert_with(|| EventResourceRow { + resource: resource.to_string(), + to_clients: Flow::default(), + from_clients: Flow::default(), + local: 0, + handler_us: 0, + }); + of.handler_us += sum.handler_us; + totals.handler_us += sum.handler_us; + let flows = match way { + EventWay::ToClient | EventWay::ToAll => [&mut of.to_clients, &mut totals.to_clients], + EventWay::FromClients => [&mut of.from_clients, &mut totals.from_clients], + EventWay::Local | EventWay::FromEngine => { + of.local += sum.count; + totals.local += if *way == EventWay::Local { sum.count } else { 0 }; + continue; + } + }; + for flow in flows { + flow.count += sum.count; + flow.bytes += sum.bytes; + } + } + } + let senders = self.senders(now, window); + let (refused, refused_count, refused_players) = self.refused(now, window); + totals.refused = refused_count; + totals.refused_players = refused_players; + + let mut resources: Vec = resources.into_values().collect(); + resources.sort_by(|a, b| { + let weight = |row: &EventResourceRow| row.to_clients.bytes + row.from_clients.bytes; + (weight(b), b.handler_us, &a.resource).cmp(&(weight(a), a.handler_us, &b.resource)) + }); + resources.truncate(PAGE_ROWS); + + Events { + totals, + series, + players: player_rows(&senders), + rows: event_rows(rows, &senders), + resources, + refused, + } + } + + fn bags(&self, now: Timestamp, window: DebugWindow) -> Bags { + #[derive(Default)] + struct Row { + sum: BagSum, + resources: HashMap, + } + let mut rows: HashMap<(Name, BagKind), Row> = HashMap::new(); + let mut resources: HashMap)> = HashMap::new(); + let mut largest: Vec = Vec::new(); + let mut totals = BagTotals::default(); + for slice in self.window(now, window) { + for ((resource, kind, key), sum) in &slice.bags { + let row = rows.entry((Arc::clone(key), *kind)).or_default(); + let (of, keys) = resources.entry(Arc::clone(resource)).or_default(); + for into in [&mut row.sum, of] { + into.count += sum.count; + into.bytes += sum.bytes; + into.largest = into.largest.max(sum.largest); + into.replicated += sum.replicated; + into.replicated_bytes += sum.replicated_bytes; + } + *row.resources.entry(Arc::clone(resource)).or_default() += sum.bytes; + keys.insert((Arc::clone(key), *kind)); + totals.writes += sum.count; + totals.bytes += sum.bytes; + totals.replicated_bytes += sum.replicated_bytes; + } + largest.extend(slice.largest.iter().cloned()); + } + totals.keys = small(rows.len()); + totals.resources = small(resources.len()); + largest.sort_by(|a, b| (b.bytes, &a.bag, &a.key).cmp(&(a.bytes, &b.bag, &b.key))); + largest.dedup_by(|a, b| a.bag == b.bag && a.key == b.key && a.resource == b.resource); + largest.truncate(LARGEST); + + let mut keys: Vec = rows + .into_iter() + .map(|((key, kind), row)| { + let mut by: Vec<(Name, u64)> = row.resources.into_iter().collect(); + by.sort_by(|a, b| (b.1, &a.0).cmp(&(a.1, &b.0))); + BagRow { + key: key.to_string(), + kind, + resource_count: small(by.len()), + resources: by.into_iter().take(3).map(|(name, _)| name.to_string()).collect(), + count: row.sum.count, + bytes: row.sum.bytes, + largest: row.sum.largest, + replicated: row.sum.replicated, + } + }) + .collect(); + keys.sort_by(|a, b| (b.bytes, b.count, &a.key, a.kind).cmp(&(a.bytes, a.count, &b.key, b.kind))); + keys.truncate(PAGE_ROWS); + + let mut resources: Vec = resources + .into_iter() + .map(|(resource, (sum, keys))| BagResourceRow { + resource: resource.to_string(), + count: sum.count, + bytes: sum.bytes, + replicated_bytes: sum.replicated_bytes, + keys: small(keys.len()), + }) + .collect(); + resources.sort_by(|a, b| (b.bytes, b.count, &a.resource).cmp(&(a.bytes, a.count, &b.resource))); + resources.truncate(PAGE_ROWS); + + Bags { + totals, + keys, + resources, + largest, + } + } + + /// The requests of `window`, by resource, method and host. + fn requests(&self, now: Timestamp, window: DebugWindow) -> Vec<(Name, Name, Name, Flow)> { + let mut rows: HashMap<&(Name, Name, Name), Flow> = HashMap::new(); + for slice in self.window(now, window) { + for (key, flow) in &slice.hosts { + let row = rows.entry(key).or_default(); + row.count += flow.count; + row.bytes += flow.bytes; + } + } + rows.into_iter() + .map(|((resource, method, host), flow)| { + (Arc::clone(resource), Arc::clone(method), Arc::clone(host), flow) + }) + .collect() + } +} + +struct Events { + totals: EventTotals, + series: Vec, + rows: Vec, + resources: Vec, + players: Vec, + refused: Vec, +} + +struct Bags { + totals: BagTotals, + keys: Vec, + resources: Vec, + largest: Vec, +} + +#[derive(Debug, Clone, Copy)] +struct Seen { + first: Timestamp, + known: bool, +} + +/// The hosts each resource called, as stored. +#[derive(Debug, Default)] +struct Hosts { + seen: HashMap<(String, String), Seen>, + learning_until: Option, +} + +impl Hosts { + /// Takes note of the hosts of a report at `now`, and answers what has to be stored: the hosts + /// seen for the first time, and the end of the day of learning when this report opens it. + fn note(&mut self, now: Timestamp, report: &Report) -> (Vec, Option) { + let opened = self + .learning_until + .is_none() + .then(|| now.saturating_add(LEARNING).unwrap_or(now)); + self.learning_until = self.learning_until.or(opened); + let learning = self.learning_until.is_some_and(|until| now < until); + let mut fresh = Vec::new(); + for host in &report.hosts { + let key = (report.resource.clone(), host.host.clone()); + if self.seen.len() >= MAX_HOSTS || self.seen.contains_key(&key) { + continue; + } + self.seen.insert( + key, + Seen { + first: now, + known: learning, + }, + ); + fresh.push(StoredHost { + resource: report.resource.clone(), + host: host.host.clone(), + first_seen: now, + known: learning, + }); + } + (fresh, opened) + } +} + +impl DebugCounts { + /// Reads the hosts stored and starts following the reports. Nothing is counted while the + /// feature is off: the reports of a server started when it was on are dropped. + pub async fn start(setup: DebugSetup) -> Result<(Self, DebugTask), Error> { + let supervisor = setup.supervision.supervisor().clone(); + let (stored, learning_until) = setup + .db + .read(|conn| Ok::<_, Error>((store::hosts(conn)?, store::learning_until(conn)?))) + .await?; + let hosts = Hosts { + seen: stored + .into_iter() + .map(|host| { + let seen = Seen { + first: host.first_seen, + known: host.known, + }; + ((host.resource, host.host), seen) + }) + .collect(), + learning_until, + }; + let reports = supervisor.hook_reports(); + let events = supervisor.events(); + let counts = Self { + inner: Arc::new(Inner { + db: setup.db, + clock: setup.clock, + supervisor, + features: setup.features, + counts: Mutex::new(Counts::default()), + hosts: Mutex::new(hosts), + }), + }; + let stop = CancellationToken::new(); + let task = tokio::spawn(counts.clone().run(stop.clone(), reports, events)); + Ok((counts, DebugTask { stop, task })) + } + + async fn run( + self, + stop: CancellationToken, + mut reports: broadcast::Receiver, + mut events: broadcast::Receiver, + ) { + loop { + tokio::select! { + () = stop.cancelled() => return, + report = reports.recv() => match report { + Ok(report) => self.take(report).await, + Err(RecvError::Lagged(_)) => self.counts().lagged = true, + Err(RecvError::Closed) => return, + }, + event = events.recv() => match event { + Ok(Event { generation, kind: EventKind::Resources { started, .. }, .. }) => { + self.counts().started = (generation, started); + } + Ok(_) | Err(RecvError::Lagged(_)) => {} + Err(RecvError::Closed) => return, + }, + } + } + } + + fn counts(&self) -> std::sync::MutexGuard<'_, Counts> { + self.inner.counts.lock().unwrap_or_else(PoisonError::into_inner) + } + + fn hosts(&self) -> std::sync::MutexGuard<'_, Hosts> { + self.inner.hosts.lock().unwrap_or_else(PoisonError::into_inner) + } + + async fn take(&self, report: HookReport) { + if !self.inner.features.runs(Feature::Debug) { + self.counts().forget(); + return; + } + let now = self.inner.clock.now(); + let (fresh, opened) = self.hosts().note(now, &report.report); + self.counts().add(now, report.generation, report.report); + if fresh.is_empty() && opened.is_none() { + return; + } + let stored = self + .inner + .db + .write(move |tx| -> Result<(), Error> { + if let Some(until) = opened { + store::set_learning_until(tx, until)?; + } + for host in &fresh { + store::add(tx, host)?; + } + Ok(()) + }) + .await; + if let Err(error) = stored { + warn!(%error, "the hosts the resources called could not be stored"); + } + } + + /// Where the counting stands, for the line under every Debug page's title. + pub fn status(&self) -> DebugStatus { + let (generation, running, bridge) = { + let server = self.inner.supervisor.status(); + let server = server.borrow(); + (server.generation, server.pid.is_some(), server.bridge.clone()) + }; + let hooked = self + .inner + .supervisor + .hooked() + .filter(|hooked| running && hooked.generation == generation); + let counts = self.counts(); + let of_this_start = counts.generation == generation; + let silent: Vec = match &counts.started { + (started, names) if *started == generation && hooked.is_some() => names + .iter() + // The bridge is Observer's own, and counts nothing. + .filter(|name| bridge.as_ref() != Some(*name)) + .filter(|name| !(of_this_start && counts.reporting.contains(*name))) + .cloned() + .collect(), + _ => Vec::new(), + }; + DebugStatus { + state: match &hooked { + Some(hooked) if hooked.refused.is_some() => DebugState::Refused, + Some(_) => DebugState::Counting, + None => DebugState::Waiting, + }, + running, + since: counts.since.filter(|_| of_this_start && hooked.is_some()), + reporting: if of_this_start && hooked.is_some() { + small(counts.reporting.len()) + } else { + 0 + }, + silent_count: small(silent.len()), + silent: silent.into_iter().take(SILENT_NAMES).collect(), + refused: hooked.and_then(|hooked| hooked.refused), + } + } + + pub fn events(&self, window: DebugWindow) -> DebugEvents { + let status = self.status(); + let now = self.inner.clock.now(); + let counts = self.counts(); + let events = counts.events(now, window); + DebugEvents { + status, + seconds: counts.covered(now, window), + totals: events.totals, + series: events.series, + events: events.rows, + resources: events.resources, + players: events.players, + refused: events.refused, + truncated: counts.cut(now, window), + } + } + + pub fn bags(&self, window: DebugWindow) -> DebugBags { + let status = self.status(); + let now = self.inner.clock.now(); + let counts = self.counts(); + let bags = counts.bags(now, window); + DebugBags { + status, + seconds: counts.covered(now, window), + totals: bags.totals, + keys: bags.keys, + resources: bags.resources, + largest: bags.largest, + truncated: counts.cut(now, window), + } + } + + pub fn http(&self, window: DebugWindow) -> DebugHttp { + let status = self.status(); + let now = self.inner.clock.now(); + let (requests, truncated, seconds) = { + let counts = self.counts(); + ( + counts.requests(now, window), + counts.cut(now, window), + counts.covered(now, window), + ) + }; + let hosts = self.hosts(); + let mut totals = HttpTotals::default(); + let (mut called, mut callers, mut fresh) = (HashSet::new(), HashSet::new(), HashSet::new()); + let mut rows: Vec = requests + .into_iter() + .map(|(resource, method, host, flow)| { + let seen = hosts.seen.get(&(resource.to_string(), host.to_string())); + let new = seen.is_some_and(|seen| !seen.known); + totals.requests += flow.count; + totals.bytes += flow.bytes; + called.insert(Arc::clone(&host)); + callers.insert(Arc::clone(&resource)); + if new { + fresh.insert((Arc::clone(&resource), Arc::clone(&host))); + } + HttpRow { + host: host.to_string(), + method: method.to_string(), + resource: resource.to_string(), + count: flow.count, + bytes: flow.bytes, + first_called: seen.map(|seen| seen.first), + new, + } + }) + .collect(); + totals.hosts = small(called.len()); + totals.resources = small(callers.len()); + totals.new_hosts = small(fresh.len()); + rows.sort_by(|a, b| { + (b.new, b.count, &a.host, &a.resource, &a.method).cmp(&( + a.new, + a.count, + &b.host, + &b.resource, + &b.method, + )) + }); + rows.truncate(PAGE_ROWS); + DebugHttp { + status, + seconds, + totals, + requests: rows, + learning_until: hosts.learning_until, + truncated, + } + } + + /// The owner knows why `resource` calls `host`: it stops showing as new, and that is audited. + pub async fn know_host(&self, caller: &Caller, resource: String, host: String) -> Result<(), Error> { + let now = self.inner.clock.now(); + let record = Record::by(caller, AuditAction::DebugHostKnown) + .target(resource.as_str()) + .details(json!({ "host": host })); + let key = (resource, host); + let stored = key.clone(); + let known = self + .inner + .db + .write(move |tx| -> Result { + if !store::know(tx, &stored.0, &stored.1)? { + return Ok(false); + } + record.write(tx, now)?; + Ok(true) + }) + .await?; + if !known { + return Err(ErrorCode::NotFound.into()); + } + if let Some(seen) = self.hosts().seen.get_mut(&key) { + seen.known = true; + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use observer_fxserver::hook::{BagCount, EventCount, HostCount, RefusedCount, SenderCount}; + + use super::*; + + fn at(time: &str) -> Timestamp { + format!("2026-11-10T{time}Z").parse().unwrap() + } + + fn event(way: Way, name: &str, count: u64, bytes: u64, largest: u64, handler_us: u64) -> EventCount { + EventCount { + way, + name: name.to_owned(), + count, + bytes, + largest, + handler_us, + } + } + + fn voice() -> Report { + Report { + resource: "pma-voice".to_owned(), + events: vec![ + event(Way::ToClient, "voice:talking", 186, 7626, 44, 0), + event(Way::FromClient, "voice:talking", 24, 912, 40, 480), + event(Way::Received, "playerDropped", 1, 48, 48, 30), + event(Way::Triggered, "voice:internal", 5, 50, 10, 0), + ], + senders: vec![ + SenderCount { + name: "voice:talking".to_owned(), + player: 41, + count: 19, + bytes: 722, + }, + SenderCount { + name: "voice:talking".to_owned(), + player: 7, + count: 5, + bytes: 190, + }, + ], + refused: vec![RefusedCount { + name: "bank:give".to_owned(), + player: Some(41), + count: 3, + }], + bags: vec![BagCount { + kind: hook::BagKind::Player, + key: "proximity".to_owned(), + count: 28, + bytes: 252, + largest: 9, + replicated: 28, + }], + largest: vec![hook::LargestWrite { + bag: "player:41".to_owned(), + key: "proximity".to_owned(), + bytes: 9, + replicated: true, + }], + hosts: vec![HostCount { + method: "POST".to_owned(), + host: "discord.com".to_owned(), + count: 2, + bytes: 800, + }], + overflow: 0, + } + } + + #[test] + fn a_rate_is_taken_over_the_seconds_counted_rather_than_over_the_whole_window() { + let mut counts = Counts::default(); + assert_eq!(counts.covered(at("21:50:10"), DebugWindow::FiveMinutes), 300); + counts.add(at("21:50:02"), 3, voice()); + counts.add(at("21:50:07"), 3, voice()); + counts.add(at("21:50:08"), 3, voice()); + + // Eight seconds after the first report, five minutes hold eight seconds of counts. + assert_eq!(counts.covered(at("21:50:10"), DebugWindow::FiveMinutes), 8); + // The traffic is drawn slice by slice, once a slice is over, and never the one this start + // began in: neither holds five seconds of counts. + let drawn = |now| counts.events(at(now), DebugWindow::FiveMinutes).series; + assert_eq!(drawn("21:50:09"), []); + assert_eq!( + drawn("21:50:10"), + [TrafficPoint { + at: at("21:50:05"), + seconds: 5, + to_clients: 15_252, + from_clients: 1824, + }] + ); + // Later, a window is whole but for the part of its last slice still to come. + assert_eq!(counts.covered(at("21:58:03"), DebugWindow::Minute), 58); + assert_eq!(counts.covered(at("21:58:05"), DebugWindow::Minute), 55); + // Eight minutes and seven seconds in, a quarter of an hour holds no more than that. + assert_eq!(counts.covered(at("21:58:09"), DebugWindow::QuarterHour), 487); + } + + #[test] + fn turned_off_and_on_again_the_counting_starts_afresh_for_the_resources_already_known() { + let mut counts = Counts::default(); + counts.add(at("21:50:01"), 3, voice()); + let idle = Report { + resource: "hud".to_owned(), + ..Report::default() + }; + counts.add(at("21:50:01"), 3, idle); + + counts.forget(); + counts.add(at("21:52:00"), 3, voice()); + + assert_eq!(counts.since, Some(at("21:52:00"))); + assert_eq!(counts.covered(at("21:52:30"), DebugWindow::FiveMinutes), 30); + let events = counts.events(at("21:52:30"), DebugWindow::FiveMinutes); + assert_eq!( + events.totals.to_clients.count, 186, + "what was counted before is gone" + ); + // A resource with nothing to count reports once, at its start: it is still known. + assert!(counts.reporting.contains("hud")); + } + + #[test] + fn events_are_totalled_by_way_with_who_sent_them_and_what_was_refused() { + let mut counts = Counts::default(); + counts.add(at("21:50:01"), 3, voice()); + counts.add(at("21:50:02"), 3, voice()); + counts.add( + at("21:50:02"), + 3, + Report { + resource: "hud".to_owned(), + events: vec![event(Way::Received, "playerDropped", 1, 48, 96, 10)], + ..Report::default() + }, + ); + + let events = counts.events(at("21:50:03"), DebugWindow::Minute); + + assert_eq!( + (events.totals.to_clients.count, events.totals.to_clients.bytes), + (372, 15_252) + ); + assert_eq!( + (events.totals.from_clients.count, events.totals.from_clients.bytes), + (48, 1824) + ); + assert_eq!( + events.totals.local, 3, + "what a resource triggers is counted where it lands" + ); + assert_eq!(events.totals.handler_us, 2 * 510 + 10); + assert_eq!((events.totals.refused, events.totals.refused_players), (6, 1)); + let names: Vec<_> = events + .rows + .iter() + .map(|row| (row.name.as_str(), row.way, row.count)) + .collect(); + assert_eq!( + names, + [ + ("voice:talking", EventWay::ToClient, 372), + ("voice:talking", EventWay::FromClients, 48), + ("playerDropped", EventWay::Local, 3), + ] + ); + let from_clients = &events.rows[1]; + assert_eq!(from_clients.resources, ["pma-voice"]); + assert_eq!( + from_clients + .senders + .iter() + .map(|by| (by.player, by.count)) + .collect::>(), + [(41, 38), (7, 10)] + ); + assert_eq!(events.rows[0].senders, Vec::new()); + let dropped = &events.rows[2]; + assert_eq!((dropped.resource_count, dropped.largest), (2, 96)); + assert_eq!(dropped.resources, ["pma-voice", "hud"], "by what they carried"); + assert_eq!( + events.players[0], + EventPlayerRow { + player: 41, + count: 38, + bytes: 1444, + top: "voice:talking".to_owned(), + } + ); + assert_eq!( + events.refused, + [RefusedRow { + name: "bank:give".to_owned(), + player: Some(41), + resource: "pma-voice".to_owned(), + count: 6, + }] + ); + assert_eq!(events.resources[0].resource, "pma-voice"); + assert_eq!(events.resources[1].local, 1); + } + + #[test] + fn a_window_holds_its_last_minutes_and_a_new_start_empties_the_counts() { + let mut counts = Counts::default(); + counts.add(at("21:40:00"), 3, voice()); + counts.add(at("21:52:00"), 3, voice()); + counts.add(at("21:54:30"), 3, voice()); + let now = at("21:54:58"); + + let sent = |window| counts.events(now, window).totals.to_clients.count; + + assert_eq!(sent(DebugWindow::Minute), 186); + assert_eq!(sent(DebugWindow::FiveMinutes), 372); + assert_eq!(sent(DebugWindow::QuarterHour), 558); + assert_eq!(counts.since, Some(at("21:40:00"))); + // Sixteen minutes on, the first report is past every window, and dropped. + counts.add(at("21:56:00"), 3, voice()); + assert_eq!(counts.slices.front().unwrap().index, slice_of(at("21:52:00"))); + + counts.started = (4, vec!["pma-voice".to_owned()]); + counts.add(at("21:57:00"), 4, voice()); + + assert_eq!(counts.slices.len(), 1); + assert_eq!(counts.since, Some(at("21:57:00"))); + assert_eq!( + counts.started.0, 4, + "what the bridge said of the new start is kept" + ); + } + + #[test] + fn state_bags_are_totalled_by_key_with_what_went_out_to_clients() { + let mut counts = Counts::default(); + counts.add(at("21:50:01"), 3, voice()); + counts.add( + at("21:50:01"), + 3, + Report { + resource: "garage".to_owned(), + bags: vec![ + BagCount { + kind: hook::BagKind::Entity, + key: "fuel".to_owned(), + count: 4, + bytes: 1240, + largest: 2100, + replicated: 4, + }, + BagCount { + kind: hook::BagKind::Global, + key: "weather".to_owned(), + count: 2, + bytes: 128, + largest: 64, + replicated: 0, + }, + ], + largest: vec![hook::LargestWrite { + bag: "entity:1042".to_owned(), + key: "fuel".to_owned(), + bytes: 2100, + replicated: true, + }], + ..Report::default() + }, + ); + + let bags = counts.bags(at("21:50:03"), DebugWindow::FiveMinutes); + + assert_eq!( + bags.totals, + BagTotals { + writes: 34, + bytes: 1620, + replicated_bytes: 1492, + keys: 3, + resources: 2, + } + ); + assert_eq!( + bags.keys + .iter() + .map(|row| (row.key.as_str(), row.kind, row.replicated)) + .collect::>(), + [ + ("fuel", BagKind::Entity, 4), + ("proximity", BagKind::Player, 28), + ("weather", BagKind::Global, 0), + ] + ); + assert_eq!(bags.resources[0].resource, "garage"); + assert_eq!( + (bags.resources[0].keys, bags.resources[0].replicated_bytes), + (2, 1240) + ); + assert_eq!(bags.largest[0].bag, "entity:1042"); + assert_eq!(bags.largest[0].resource, "garage"); + assert_eq!(bags.largest.len(), 2); + } + + #[test] + fn a_slice_keeps_so_many_rows_and_says_when_there_were_more() { + let mut counts = Counts::default(); + let many = (0..SLICE_ROWS + 10) + .map(|index| event(Way::Received, &format!("event-{index}"), 1, 1, 1, 0)) + .collect(); + let now = at("21:50:01"); + + counts.add( + now, + 3, + Report { + resource: "noisy".to_owned(), + events: many, + ..Report::default() + }, + ); + + assert_eq!(counts.slices[0].events.len(), SLICE_ROWS); + assert!(counts.cut(now, DebugWindow::Minute)); + assert_eq!(counts.events(now, DebugWindow::Minute).rows.len(), PAGE_ROWS); + let mut whole = Counts::default(); + whole.add(now, 3, voice()); + assert!(!whole.cut(now, DebugWindow::Minute)); + whole.add( + now, + 3, + Report { + overflow: 4, + ..voice() + }, + ); + assert!( + whole.cut(now, DebugWindow::Minute), + "the resource itself kept less" + ); + } + + #[test] + fn a_host_is_known_while_observer_learns_and_new_once_the_day_is_over() { + let mut hosts = Hosts::default(); + let first = at("21:35:00"); + + let (stored, opened) = hosts.note(first, &voice()); + + let until = first.saturating_add(LEARNING).unwrap(); + assert_eq!(opened, Some(until)); + assert_eq!( + stored, + [StoredHost { + resource: "pma-voice".to_owned(), + host: "discord.com".to_owned(), + first_seen: first, + known: true, + }] + ); + assert_eq!( + hosts.note(at("21:36:00"), &voice()), + (Vec::new(), None), + "seen already" + ); + + let later = until.saturating_add(SignedDuration::from_mins(5)).unwrap(); + let elsewhere = Report { + hosts: vec![HostCount { + method: "POST".to_owned(), + host: "paste.example".to_owned(), + count: 1, + bytes: 2300, + }], + ..voice() + }; + let (stored, opened) = hosts.note(later, &elsewhere); + + assert_eq!(opened, None); + assert_eq!( + (stored[0].host.as_str(), stored[0].known), + ("paste.example", false) + ); + let key = ("pma-voice".to_owned(), "paste.example".to_owned()); + assert_eq!(hosts.seen[&key].first, later); + } +} diff --git a/crates/observer-app/src/features.rs b/crates/observer-app/src/features.rs index ba5fe0cc..88b9ae62 100644 --- a/crates/observer-app/src/features.rs +++ b/crates/observer-app/src/features.rs @@ -2,6 +2,8 @@ //! off, and what runs in this Observer. A feature that is off when Observer starts is never built. //! Turning one off stops it at once: its routes and topics refuse, and the tasks handed to //! [`Features::switched`] close. Turning one on takes effect when [`Feature::turn_on`] says. +//! SFTP alone also takes an address, from `observer.toml` or from the owner in the panel, and its +//! port follows both the switch and the address while Observer runs. use std::collections::btree_map::Entry; use std::collections::{BTreeMap, BTreeSet}; @@ -15,7 +17,9 @@ use std::time::Duration; use jiff::Timestamp; use observer_domain::audit::{AuditAction, Outcome}; use observer_domain::error::{ApiError, ErrorCode}; -use observer_domain::features::{Applies, Feature, FeatureHold, FeatureList, FeatureReach, FeatureView}; +use observer_domain::features::{ + Applies, Feature, FeatureHold, FeatureList, FeatureReach, FeatureView, SftpListen, +}; use observer_store::Database; use observer_store::features as store; use serde_json::json; @@ -29,6 +33,8 @@ use crate::{Caller, Clock, Error}; /// How long turning a feature off waits for its tasks to close before it answers all the same. const CLOSE_WAIT: Duration = Duration::from_secs(20); +/// How long a change waits to learn what became of SFTP's port before it answers all the same. +const PORT_WAIT: Duration = Duration::from_secs(10); /// What [`Features::load`] works with. pub struct FeaturesSetup { @@ -36,12 +42,38 @@ pub struct FeaturesSetup { pub clock: Clock, /// What `observer.toml` turns off: the panel cannot turn these on. pub host_off: BTreeSet, - /// Where SFTP listens, when `observer.toml` says. + /// Where SFTP listens, when `observer.toml` says: the panel then cannot change it. pub sftp: Option, /// Observer's data folder, which holds the features' files. pub data_dir: PathBuf, } +/// SFTP's address, and what became of its port. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +struct Sftp { + /// What `observer.toml` names. + pinned: Option, + /// What the owner gave in the panel. + chosen: Option, + /// The address the port is open on. + open: Option, + /// The fingerprint of the host key served on the open port. + fingerprint: Option, + /// Why the port did not open on the address in force. + refused: Option, +} + +impl Sftp { + fn listen(&self) -> Option { + self.pinned.or(self.chosen) + } + + /// Whether the port is where the address in force wants it, or was refused there. + fn settled(&self) -> bool { + self.refused.is_some() || self.open == self.listen() + } +} + /// The feature switches. Clones share them. #[derive(Clone)] pub struct Features { @@ -52,9 +84,11 @@ struct Inner { db: Database, clock: Clock, host_off: BTreeSet, - sftp: Option, + sftp: watch::Sender, + /// The address in force as it changes, for whoever keeps the port. + sftp_listen: watch::Sender>, data_dir: PathBuf, - /// What the owner chose and when; a feature missing keeps its default, on. + /// What the owner chose and when; a feature missing keeps its default. chosen: Mutex>, /// What runs in this Observer: decided at its start, then narrowed by every feature turned /// off, and widened only by those whose switch applies at once. @@ -75,11 +109,17 @@ impl Features { .into_iter() .map(|switch| (switch.feature, (switch.enabled, switch.changed_at))) .collect(); + let sftp = Sftp { + pinned: setup.sftp, + chosen: setup.db.read(store::sftp_listen).await?, + ..Sftp::default() + }; let inner = Inner { db: setup.db, clock: setup.clock, host_off: setup.host_off, - sftp: setup.sftp, + sftp_listen: watch::Sender::new(sftp.listen()), + sftp: watch::Sender::new(sftp), data_dir: setup.data_dir, chosen: Mutex::new(chosen), running: watch::Sender::new(BTreeSet::new()), @@ -133,17 +173,20 @@ impl Features { .await .unwrap_or_default(); let running = self.inner.running.borrow().clone(); + let listen = self.inner.sftp.borrow().listen(); let chosen = self.inner.chosen.lock().unwrap_or_else(PoisonError::into_inner); let features = Feature::ALL .into_iter() .map(|feature| { - let listens = match (feature, self.inner.sftp) { + let listens = match (feature, listen) { (Feature::Sftp, Some(listen)) => vec![listen.to_string()], _ => Vec::new(), }; FeatureView { feature, - enabled: chosen.get(&feature).is_none_or(|&(enabled, _)| enabled), + enabled: chosen + .get(&feature) + .map_or(feature.on_by_default(), |&(enabled, _)| enabled), running: running.contains(&feature), held: self.inner.hold(feature, &running), requires: feature.requires().to_vec(), @@ -196,6 +239,12 @@ impl Features { .lock() .unwrap_or_else(PoisonError::into_inner) .insert(feature, (enabled, now)); + if feature == Feature::Sftp { + // The switch turned on again gives a refused port another try. + self.inner + .sftp + .send_if_modified(|sftp| sftp.refused.take().is_some()); + } let mut stopped = Vec::new(); self.inner.running.send_if_modified(|running| { if enabled { @@ -223,9 +272,150 @@ impl Features { "turned off, but its tasks have not stopped yet" ); } + if enabled && feature == Feature::Sftp && self.runs(Feature::Sftp) { + self.port_settled().await; + } Ok(self.view().await) } + /// Where SFTP listens, who says so, and whether its port is open. + pub fn sftp(&self) -> SftpListen { + let sftp = self.inner.sftp.borrow(); + let listen = sftp.listen(); + let open = listen.is_some() && sftp.open == listen; + SftpListen { + listen: listen.map(|listen| listen.to_string()), + pinned: sftp.pinned.is_some(), + open, + fingerprint: sftp.fingerprint.clone().filter(|_| open), + refused: sftp.refused.clone(), + } + } + + /// Gives SFTP its address for the owner, or takes it back, and audits it. The port follows + /// at once while the feature is on, and the answer says what became of it. The same address + /// given again tries a refused port anew. `observer.toml` refuses when it names the address. + pub async fn set_sftp_listen( + &self, + caller: &Caller, + listen: Option, + ) -> Result { + let _changing = self.inner.changing.lock().await; + let (previous, refused) = { + let sftp = self.inner.sftp.borrow(); + if sftp.pinned.is_some() { + return Err(ApiError::new(ErrorCode::SftpListenPinned).into()); + } + (sftp.chosen, sftp.refused.is_some()) + }; + if listen.is_some_and(|listen| listen.port() == 0) { + return Err(ApiError::new(ErrorCode::InvalidSftpListen) + .with("reason", "port") + .into()); + } + if previous == listen && !refused { + return Ok(self.sftp()); + } + if previous != listen { + let now = self.inner.clock.now(); + let record = Record::by(caller, AuditAction::SftpListenChanged) + .target(Feature::Sftp.as_str()) + .details(json!({ + "listen": listen.map(|listen| listen.to_string()), + "previous": previous.map(|previous| previous.to_string()), + })); + self.inner + .db + .write(move |tx| -> Result<(), Error> { + store::set_sftp_listen(tx, listen)?; + record.write(tx, now)?; + Ok(()) + }) + .await?; + } + self.inner.sftp.send_modify(|sftp| { + sftp.chosen = listen; + sftp.refused = None; + }); + let mut closing = false; + self.inner.running.send_if_modified(|running| { + if self.inner.enabled(Feature::Sftp) && self.inner.hold(Feature::Sftp, running).is_none() { + running.insert(Feature::Sftp) + } else { + closing = running.remove(&Feature::Sftp); + closing + } + }); + self.inner.sftp_listen.send_replace(listen); + if closing { + if tokio::time::timeout(CLOSE_WAIT, closed(self.inner.open.subscribe(), &[Feature::Sftp])) + .await + .is_err() + { + warn!("SFTP lost its address, but its port has not closed yet"); + } + } else if self.runs(Feature::Sftp) { + self.port_settled().await; + } + Ok(self.sftp()) + } + + /// The address SFTP listens on, as it changes. Whoever keeps the port follows it together + /// with [`Features::running`], and says what became of the port. + pub fn sftp_listen(&self) -> watch::Receiver> { + self.inner.sftp_listen.subscribe() + } + + /// The port is open on `listen`, where clients are shown the host key of `fingerprint`. + pub fn sftp_opened(&self, listen: SocketAddr, fingerprint: String) { + self.inner.sftp.send_modify(|sftp| { + sftp.open = Some(listen); + sftp.fingerprint = Some(fingerprint); + }); + } + + /// The port is closed. + pub fn sftp_closed(&self) { + self.inner.sftp.send_if_modified(|sftp| { + sftp.fingerprint = None; + sftp.open.take().is_some() + }); + } + + /// `listen` did not open. SFTP stops running until its address is given again or its switch + /// is turned on again, which is when the port is tried anew. + pub fn sftp_refused(&self, listen: SocketAddr, error: String) { + let in_force = self.inner.sftp.send_if_modified(|sftp| { + if sftp.listen() != Some(listen) { + return false; + } + sftp.open = None; + sftp.fingerprint = None; + sftp.refused = Some(error); + true + }); + if in_force { + self.inner + .running + .send_if_modified(|running| running.remove(&Feature::Sftp)); + } + } + + /// Waits a while for the port to open or be refused on the address in force. Nobody keeps + /// the port in a build without SFTP's server, and then there is nothing to wait for. + async fn port_settled(&self) { + if self.inner.sftp_listen.receiver_count() == 0 { + return; + } + let mut sftp = self.inner.sftp.subscribe(); + if tokio::time::timeout(PORT_WAIT, sftp.wait_for(Sftp::settled)) + .await + .is_err() + { + warn!("SFTP's port has neither opened nor been refused yet"); + } + } + /// Hands over a feature's task: `close` runs as soon as the feature stops running, or when /// [`Switched::close`] says Observer stops, whichever comes first. pub fn switched(&self, feature: Feature, close: impl FnOnce() -> F + Send + 'static) -> Switched @@ -240,6 +430,9 @@ impl Features { let stopping = stopping.clone(); async move { let turned_off = tokio::select! { + // Whoever keeps the task may close it as it sees the feature stop running: + // it is still the switch that ended it, and the log says so. + biased; () = turned_off(running, feature) => true, () = stopping.cancelled() => false, }; @@ -267,24 +460,28 @@ impl Inner { .lock() .unwrap_or_else(PoisonError::into_inner) .get(&feature) - .is_none_or(|&(enabled, _)| enabled) + .map_or(feature.on_by_default(), |&(enabled, _)| enabled) } /// Why `feature` cannot run beside `running`, whatever the owner chose. fn hold(&self, feature: Feature, running: &BTreeSet) -> Option { if self.host_off.contains(&feature) { - Some(FeatureHold::Host) - } else if feature == Feature::Sftp && self.sftp.is_none() { - Some(FeatureHold::NotConfigured) - } else if feature + return Some(FeatureHold::Host); + } + if feature == Feature::Sftp { + let sftp = self.sftp.borrow(); + if sftp.listen().is_none() { + return Some(FeatureHold::NotConfigured); + } + if sftp.refused.is_some() { + return Some(FeatureHold::PortRefused); + } + } + feature .requires() .iter() .any(|required| !running.contains(required)) - { - Some(FeatureHold::Requires) - } else { - None - } + .then_some(FeatureHold::Requires) } } @@ -446,7 +643,17 @@ mod tests { async fn every_feature_runs_by_default_and_sftp_waits_for_its_address() { let lab = Lab::new().await; let features = lab.features(&[]).await; - assert!(Feature::ALL.into_iter().all(|feature| features.runs(feature))); + let running: Vec = Feature::ALL + .into_iter() + .filter(|&feature| features.runs(feature)) + .collect(); + let default_on: Vec = Feature::ALL + .into_iter() + .filter(|feature| feature.on_by_default()) + .collect(); + assert_eq!(running, default_on); + assert!(!features.runs(Feature::Debug), "Debug waits to be asked for"); + assert!(!view(&features.view().await, Feature::Debug).enabled); let unconfigured = Features::load(FeaturesSetup { db: lab.db.clone(), @@ -648,4 +855,237 @@ mod tests { assert_eq!(view(&features.view().await, Feature::Metrics).changed_at, None); assert_eq!(lab.audited().await, []); } + + /// The switches as Observer loads them when `observer.toml` leaves SFTP's address to the panel. + async fn unpinned(lab: &Lab) -> Features { + Features::load(FeaturesSetup { + db: lab.db.clone(), + clock: lab.clock.clone(), + host_off: BTreeSet::new(), + sftp: None, + data_dir: lab.dir.path().join("data"), + }) + .await + .unwrap() + } + + /// Stands for whoever keeps the port: each address it is asked to open, `answer` opens or + /// refuses. + fn keeper( + features: &Features, + answer: impl Fn(SocketAddr) -> Result<(), String> + Send + 'static, + ) -> JoinHandle<()> { + let features = features.clone(); + let mut running = features.running(); + let mut listen = features.sftp_listen(); + tokio::spawn(async move { + let mut open = None; + loop { + let on = running.borrow_and_update().contains(&Feature::Sftp); + let at = *listen.borrow_and_update(); + let wanted = if on { at } else { None }; + if open != wanted { + if open.take().is_some() { + features.sftp_closed(); + } + match wanted.map(|at| (at, answer(at))) { + Some((at, Ok(()))) => { + open = Some(at); + features.sftp_opened(at, "SHA256:host-key".to_owned()); + } + Some((at, Err(error))) => features.sftp_refused(at, error), + None => {} + } + } + tokio::select! { + changed = running.changed() => if changed.is_err() { return }, + changed = listen.changed() => if changed.is_err() { return }, + } + } + }) + } + + fn address(listen: &str) -> SocketAddr { + listen.parse().unwrap() + } + + #[tokio::test] + async fn the_owner_gives_sftp_its_address_which_outlives_observer_and_takes_it_back() { + let lab = Lab::new().await; + let features = unpinned(&lab).await; + let _keeper = keeper(&features, |_| Ok(())); + + let given = features + .set_sftp_listen(&owner(), Some(address("127.0.0.1:40126"))) + .await + .unwrap(); + + assert_eq!( + given, + SftpListen { + listen: Some("127.0.0.1:40126".to_owned()), + pinned: false, + open: true, + fingerprint: Some("SHA256:host-key".to_owned()), + refused: None, + } + ); + assert!(features.runs(Feature::Sftp)); + let sftp = view(&features.view().await, Feature::Sftp); + assert_eq!(sftp.turn_on, Applies::Now); + assert_eq!(sftp.reach.listens, ["127.0.0.1:40126"]); + let restarted = unpinned(&lab).await; + assert!(restarted.runs(Feature::Sftp)); + assert_eq!(restarted.sftp().listen.as_deref(), Some("127.0.0.1:40126")); + + let taken_back = features.set_sftp_listen(&owner(), None).await.unwrap(); + + assert_eq!((taken_back.listen, taken_back.open), (None, false)); + assert_eq!(taken_back.fingerprint, None); + assert!(!features.runs(Feature::Sftp)); + let sftp = view(&features.view().await, Feature::Sftp); + assert_eq!(sftp.held, Some(FeatureHold::NotConfigured)); + assert!(!unpinned(&lab).await.runs(Feature::Sftp)); + } + + #[tokio::test] + async fn a_new_address_moves_the_port_and_the_same_one_changes_nothing() { + let lab = Lab::new().await; + let features = unpinned(&lab).await; + let _keeper = keeper(&features, |_| Ok(())); + let listened = |features: &Features| features.sftp().listen; + features + .set_sftp_listen(&owner(), Some(address("127.0.0.1:40126"))) + .await + .unwrap(); + + let same = features + .set_sftp_listen(&owner(), Some(address("127.0.0.1:40126"))) + .await + .unwrap(); + let moved = features + .set_sftp_listen(&owner(), Some(address("0.0.0.0:2022"))) + .await + .unwrap(); + + assert!(same.open); + assert!( + moved.open, + "the answer waits for the port to open on the new address" + ); + assert_eq!(listened(&features).as_deref(), Some("0.0.0.0:2022")); + let changes = lab + .db + .read(|conn| -> Result { + Ok(conn.query_row( + "SELECT count(*) FROM audit_log WHERE action = 'sftp_listen_changed'", + [], + |row| row.get(0), + )?) + }) + .await + .unwrap(); + assert_eq!(changes, 2, "the same address given again is no change"); + } + + #[tokio::test] + async fn a_port_that_does_not_open_holds_sftp_until_its_address_is_given_again() { + let lab = Lab::new().await; + let features = unpinned(&lab).await; + let tries = Arc::new(Mutex::new(Vec::new())); + let _keeper = keeper(&features, { + let tries = Arc::clone(&tries); + move |at| { + tries.lock().unwrap().push(at.port()); + if at.port() == 40126 { + Err("the port is taken".to_owned()) + } else { + Ok(()) + } + } + }); + + let refused = features + .set_sftp_listen(&owner(), Some(address("127.0.0.1:40126"))) + .await + .unwrap(); + + assert_eq!( + refused.listen.as_deref(), + Some("127.0.0.1:40126"), + "the address is kept" + ); + assert_eq!( + (refused.open, refused.refused.as_deref()), + (false, Some("the port is taken")) + ); + assert!(!features.runs(Feature::Sftp)); + let sftp = view(&features.view().await, Feature::Sftp); + assert_eq!(sftp.held, Some(FeatureHold::PortRefused)); + + let again = features + .set_sftp_listen(&owner(), Some(address("127.0.0.1:40126"))) + .await + .unwrap(); + let elsewhere = features + .set_sftp_listen(&owner(), Some(address("127.0.0.1:40127"))) + .await + .unwrap(); + + assert_eq!(again.refused.as_deref(), Some("the port is taken")); + assert!(elsewhere.open && elsewhere.refused.is_none()); + assert!(features.runs(Feature::Sftp)); + assert_eq!(*tries.lock().unwrap(), [40126, 40126, 40127]); + } + + #[tokio::test] + async fn the_switch_of_sftp_opens_and_closes_its_port_while_observer_runs() { + let lab = Lab::new().await; + let features = lab.features(&[]).await; + let _keeper = keeper(&features, |_| Ok(())); + // The keeper opens the port of a feature that runs from Observer's start. + tokio::time::timeout(Duration::from_secs(5), async { + while !features.sftp().open { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + + features.set(&owner(), Feature::Sftp, false).await.unwrap(); + tokio::task::yield_now().await; + let off = features.sftp(); + let list = features.set(&owner(), Feature::Sftp, true).await.unwrap(); + + assert!(!off.open); + assert!(view(&list, Feature::Sftp).running); + assert!( + features.sftp().open, + "turned on, it listens without a restart of Observer" + ); + } + + #[tokio::test] + async fn the_address_observer_toml_names_is_not_the_panels_and_a_port_takes_a_number() { + let lab = Lab::new().await; + let pinned = lab.features(&[]).await; + + let refused = pinned + .set_sftp_listen(&owner(), Some(address("127.0.0.1:2022"))) + .await + .unwrap_err(); + let no_port = unpinned(&lab) + .await + .set_sftp_listen(&owner(), Some(address("127.0.0.1:0"))) + .await + .unwrap_err(); + + assert_eq!(refused.code(), Some(ErrorCode::SftpListenPinned)); + let shown = pinned.sftp(); + assert_eq!( + (shown.listen.as_deref(), shown.pinned), + (Some("0.0.0.0:40126"), true) + ); + assert_eq!(no_port.code(), Some(ErrorCode::InvalidSftpListen)); + } } diff --git a/crates/observer-app/src/lib.rs b/crates/observer-app/src/lib.rs index 2def56ec..6d766a71 100644 --- a/crates/observer-app/src/lib.rs +++ b/crates/observer-app/src/lib.rs @@ -10,6 +10,7 @@ mod backups; mod bans; mod changes; mod clock; +mod debug; mod diagnostics; mod drops; mod error; @@ -46,6 +47,7 @@ pub use backups::{ pub use bans::{Bans, BansSetup, BansTask}; pub use changes::{Changes, ChangesSetup, ChangesTask, ComparedTo, DiffOptions}; pub use clock::Clock; +pub use debug::{DebugCounts, DebugSetup, DebugTask}; pub use diagnostics::{Diagnostics, DiagnosticsSetup, DiagnosticsTask, MAX_DOWNLOAD}; pub use drops::{DayRates, Drops, DropsSetup, DropsTask, LogQuery}; pub use error::Error; diff --git a/crates/observer-app/src/map/tests.rs b/crates/observer-app/src/map/tests.rs index 3f09e8f0..1fe77de4 100644 --- a/crates/observer-app/src/map/tests.rs +++ b/crates/observer-app/src/map/tests.rs @@ -1,6 +1,6 @@ use std::net::{IpAddr, SocketAddr}; -use observer_domain::map::MapBounds; +use observer_domain::map::{MapBounds, MapImagery}; use observer_domain::permissions::Permissions; use observer_domain::schedule::{ScheduleSettings, ScheduleStatus}; use tempfile::TempDir; @@ -194,6 +194,7 @@ async fn settings_are_checked_before_they_are_kept() { let lab = Lab::new().await; let (map, task) = lab.map(); let placed = MapSettings { + imagery: MapImagery::Image, bounds: Some(MapBounds { west: -5640.0, east: 6470.0, diff --git a/crates/observer-app/src/supervision.rs b/crates/observer-app/src/supervision.rs index f3268f8c..70e5e42f 100644 --- a/crates/observer-app/src/supervision.rs +++ b/crates/observer-app/src/supervision.rs @@ -2,6 +2,7 @@ //! selected build, the status the panel shows of it, the console it fills, the commands operators //! send it, the console log files, and the server setup's checks of a folder and its cfg. +mod cfg; mod setup; use std::collections::{HashMap, VecDeque}; @@ -18,14 +19,16 @@ use observer_domain::launch::{Platform, StoredLaunch}; use observer_domain::permissions::Permissions; use observer_domain::schedule::RestartSource; use observer_domain::server::{ - CommandAck, CommandRefusal, ConsoleLine, ConsoleSnapshot, Delivery, HangCause, HealthProblem, - LINE_MASKED, LINE_NO_EOL, LINE_TRUNCATED, LaunchReadiness, LineKind, LogFile, PlayerCount, ServerState, - ServerStatus, StateCause, + AssetMemory, CommandAck, CommandRefusal, ConsoleLine, ConsoleSnapshot, Delivery, HangCause, + HealthProblem, HeavyAsset, LINE_MASKED, LINE_NO_EOL, LINE_TRUNCATED, LaunchReadiness, LineKind, LogFile, + MissingFile, PlayerCount, ServerState, ServerStatus, SlowResource, StartCommand, StartProgress, + StartWarnings, StartedResource, StateCause, }; use observer_fxserver::bridge::BridgeEvent; use observer_fxserver::console::{self as fx_console, ConsoleHub, ConsoleLog, LineFlags}; use observer_fxserver::health; use observer_fxserver::launch::ServerLaunch; +use observer_fxserver::start as fx_start; use observer_fxserver::stdin::{self, StdinError, UserCommand}; use observer_fxserver::supervisor::{ self as fx, Cause, ControlError, EventKind, Settings, State, Supervisor, @@ -49,6 +52,9 @@ use crate::{Accounts, Caller, Clock, Error, Servers}; const OBSERVER: &str = "observer"; /// Who they name for someone at the host's own window, who signs in to nothing. const LOCAL: &str = "local"; +/// What a start is given when someone says it needs longer. +#[expect(clippy::cast_possible_truncation, reason = "a quarter of an hour")] +const EXTENSION_MINUTES: u32 = (fx::START_EXTENSION.as_secs() / 60) as u32; /// Who the console names for the schedule's notes. const SCHEDULE: &str = "schedule"; /// Commands one user may send within a window. @@ -386,6 +392,57 @@ impl Supervision { .await } + /// What FXServer warned about as it started its resources, the latest time it was started. + pub fn start_warnings(&self) -> StartWarnings { + let Some((spawned_ms, warnings)) = self.inner.supervisor.start_warnings() else { + return StartWarnings { + started_at: None, + heavy: Vec::new(), + missing: Vec::new(), + truncated: false, + }; + }; + StartWarnings { + started_at: Timestamp::from_millisecond(spawned_ms).ok(), + heavy: warnings + .heavy + .into_iter() + .map(|asset| HeavyAsset { + resource: asset.resource, + file: asset.file, + tenths: asset.tenths, + memory: match asset.memory { + fx_start::Memory::Physical => AssetMemory::Physical, + fx_start::Memory::Virtual => AssetMemory::Virtual, + }, + oversized: asset.oversized, + }) + .collect(), + missing: warnings + .missing + .into_iter() + .map(|file| MissingFile { + resource: file.resource, + kind: file.kind, + file: file.file, + manifest: file.manifest, + line: file.line, + }) + .collect(), + truncated: warnings.truncated, + } + } + + /// Gives the start under way more time before it is taken for stuck: a first start, or one + /// with a resource that takes longer than the hang delay, without changing the delay itself. + pub async fn extend_start(&self, caller: &Caller) -> Result<(), Error> { + let extended = self.inner.supervisor.extend_start(&caller.username).await; + let record = Record::by(caller, AuditAction::ServerStartExtended) + .details(json!({ "minutes": EXTENSION_MINUTES })); + self.record_control(record, &extended).await?; + extended.map_err(control_error) + } + /// Stops FXServer gracefully and starts it again with the stored settings, which must be fit /// to start before anything stops. pub async fn restart_server(&self, caller: &Caller) -> Result<(), Error> { @@ -890,6 +947,39 @@ fn server_status( launch, name: status.name.clone(), port: status.port, + start: status.start.as_ref().map(|start| { + let progress = &start.progress; + StartProgress { + started: progress.started, + failed: progress.failed, + found: progress.found, + planned: progress.planned, + last: progress.last.as_ref().map(|last| StartedResource { + name: last.name.clone(), + at: at(last.at_ms), + warnings: last.warnings, + }), + at: progress.at.as_ref().map(|command| StartCommand { + file: command.file.clone(), + line: command.line, + target: command.target.clone(), + since: at(command.since_ms), + done: command.done, + of: command.of, + resource: command.resource.clone(), + }), + slowest: progress + .slowest + .iter() + .map(|slow| SlowResource { + name: slow.name.clone(), + ms: u32::try_from(slow.ms).unwrap_or(u32::MAX), + }) + .collect(), + stuck_at: start.stuck_at_ms.map(at), + extension_minutes: EXTENSION_MINUTES, + } + }), } } @@ -993,6 +1083,7 @@ fn control_error(error: ControlError) -> Error { ErrorCode::ServerNotRunning.into() } ControlError::Stopping => ErrorCode::ServerStopping.into(), + ControlError::NotStarting => ErrorCode::ServerNotStarting.into(), ControlError::ShuttingDown => ErrorCode::ShuttingDown.into(), ControlError::Launch(error) => ApiError::new(ErrorCode::LaunchFailed) .with("reason", serde_json::to_value(error.failure).unwrap_or_default()) diff --git a/crates/observer-app/src/supervision/cfg.rs b/crates/observer-app/src/supervision/cfg.rs new file mode 100644 index 00000000..9282861b --- /dev/null +++ b/crates/observer-app/src/supervision/cfg.rs @@ -0,0 +1,361 @@ +//! The launch's cfg edited from the panel: read with a mark in place of each secret, checked as it +//! is typed, and written back whole once the file as it was is kept. With the SFTP uploads and the +//! server tools' fixes, the only writes Observer makes in the server folder; each is the owner's, +//! password given again. + +use std::fs::{self, File}; +use std::io::{self, Read as _, Write as _}; +use std::path::{Path, PathBuf}; +use std::thread; +use std::time::Duration; + +use jiff::Timestamp; +use observer_domain::audit::AuditAction; +use observer_domain::cfg::{ + CfgDocument, CfgDraft, CfgRefusal, CfgRestore, CfgReview, CfgSave, CfgVersion, MAX_TEXT_BYTES, + VERSIONS_KEPT, +}; +use observer_domain::changes::TextDiff; +use observer_domain::error::{ApiError, ErrorCode}; +use observer_domain::launch::{self, LaunchSettings, OneSync, Platform}; +use observer_fxserver::cfg_edit::{self, Editable, Refused, Unreadable}; +use observer_fxserver::changes::{diff, mask}; +use observer_fxserver::server_data; +use observer_store::cfg_versions::{self, Replaced}; +use serde_json::json; + +use super::setup::blocking; +use super::{Supervision, read_launch}; +use crate::audit::Record; +use crate::{Caller, Error, Servers, secrets}; + +/// Antivirus scanners hold a freshly written file for a moment on Windows, which makes the rename +/// over the cfg fail; the same call a little later succeeds. +const BUSY_DELAYS_MS: [u64; 4] = [50, 100, 200, 400]; +const SAVING: &str = ".observer-saving"; + +/// The cfg of the saved launch, as its file holds it. +struct Read { + data: PathBuf, + /// Named from the server folder. + file: String, + onesync: OneSync, + bytes: Vec, + modified: Option, + editable: Editable, +} + +impl Read { + fn path(&self) -> PathBuf { + self.data.join(&self.file) + } + + fn base(&self) -> String { + base(&self.bytes) + } + + /// The bytes a save of `text` writes. + fn written(&self, text: &str) -> Result, Error> { + self.editable.write(text).map_err(|refused| { + let reason = match refused { + Refused::UnknownMark => "unknownMark", + Refused::Control => "control", + Refused::TooLarge => "tooLarge", + }; + ApiError::new(ErrorCode::InvalidCfgText) + .with("reason", json!(reason)) + .into() + }) + } +} + +fn base(bytes: &[u8]) -> String { + secrets::hex(&secrets::sha256(bytes)) +} + +fn refusal(reason: CfgRefusal) -> Error { + ApiError::new(ErrorCode::CfgNotEditable) + .with("reason", serde_json::to_value(reason).unwrap_or_default()) + .into() +} + +fn lines(bytes: &[u8]) -> u32 { + u32::try_from(String::from_utf8_lossy(bytes).lines().count()).unwrap_or(u32::MAX) +} + +impl Supervision { + /// The launch's cfg, for the owner to edit. It changes nothing. + pub async fn cfg_document(&self) -> Result { + let read = self.read_cfg().await?; + self.document(read).await + } + + /// What a text of the cfg sets and starts, and what the scan finds in it and in the files it + /// runs. It changes nothing. + pub async fn check_cfg_draft(&self, draft: CfgDraft) -> Result { + if draft.text.len() > MAX_TEXT_BYTES { + return Err(ApiError::new(ErrorCode::InvalidCfgText) + .with("reason", json!("tooLarge")) + .into()); + } + let read = self.read_cfg().await?; + let whole = read.editable.revealed(&draft.text); + self.check(read, draft.text, whole).await + } + + /// The lines a save of the text would change in the file, secrets hidden. It changes nothing. + pub async fn diff_cfg_draft(&self, draft: CfgDraft) -> Result { + let read = self.read_cfg().await?; + let written = read.written(&draft.text)?; + blocking(move || changed(&read.file, &read.bytes, &written)).await? + } + + /// Writes the text as the cfg, once the file as it was is kept. Refused when the file is no + /// longer the one the edit started from. + pub async fn save_cfg(&self, caller: &Caller, save: CfgSave) -> Result { + let read = self.read_cfg().await?; + if read.base() != save.base { + return Err(ErrorCode::CfgChanged.into()); + } + let written = read.written(&save.text)?; + let record = Record::by(caller, AuditAction::ServerCfgChanged); + self.replace_cfg(caller, read, written, record, json!({})).await?; + self.cfg_document().await + } + + /// The versions kept of the cfg, the latest first. + pub async fn cfg_versions(&self) -> Result, Error> { + let file = self.cfg_file().await?; + self.inner + .db + .read(move |conn| Ok::<_, Error>(cfg_versions::list(conn, Servers::DEFAULT, &file)?)) + .await + } + + /// The lines putting the version back would change in the file, secrets hidden. + pub async fn cfg_version_diff(&self, id: i64) -> Result { + let read = self.read_cfg().await?; + let version = self.version(&read.file, id).await?; + blocking(move || changed(&read.file, &read.bytes, &version)).await? + } + + /// Puts a version back as the cfg, the file as it is kept first. + pub async fn restore_cfg( + &self, + caller: &Caller, + id: i64, + restore: CfgRestore, + ) -> Result { + let read = self.read_cfg().await?; + if read.base() != restore.base { + return Err(ErrorCode::CfgChanged.into()); + } + let version = self.version(&read.file, id).await?; + let record = Record::by(caller, AuditAction::ServerCfgRestored); + self.replace_cfg(caller, read, version, record, json!({ "version": id })) + .await?; + self.cfg_document().await + } + + async fn cfg_file(&self) -> Result { + let (stored, _) = read_launch(&self.inner.db).await?; + stored + .map(|stored| stored.settings.cfg_file) + .ok_or_else(|| refusal(CfgRefusal::NotConfigured)) + } + + async fn version(&self, file: &str, id: i64) -> Result, Error> { + let file = file.to_owned(); + self.inner + .db + .read(move |conn| Ok::<_, Error>(cfg_versions::content(conn, Servers::DEFAULT, &file, id)?)) + .await? + .ok_or_else(|| ErrorCode::CfgVersionNotFound.into()) + } + + async fn read_cfg(&self) -> Result { + let (stored, _) = read_launch(&self.inner.db).await?; + let Some(stored) = stored else { + return Err(refusal(CfgRefusal::NotConfigured)); + }; + blocking(move || read(&stored.settings)).await? + } + + async fn document(&self, read: Read) -> Result { + let file = read.file.clone(); + let versions = self + .inner + .db + .read(move |conn| Ok::<_, Error>(cfg_versions::count(conn, Servers::DEFAULT, &file)?)) + .await?; + let shown = read.editable.text.clone(); + let whole = read.editable.revealed(&shown); + let document = CfgDocument { + file: read.file.clone(), + text: shown.clone(), + base: read.base(), + bytes: u32::try_from(read.bytes.len()).unwrap_or(u32::MAX), + modified_at: read.modified, + secrets: u32::try_from(read.editable.secrets()).unwrap_or(u32::MAX), + check: CfgReview::default(), + versions, + }; + let check = self.check(read, shown, whole).await?; + Ok(CfgDocument { check, ..document }) + } + + async fn check(&self, read: Read, shown: String, whole: String) -> Result { + let context = self.cfg_context(read.onesync).await?; + blocking(move || { + let resources = server_data::walk(&read.data); + let mut review = cfg_edit::check(&read.data, &read.file, &shown, &whole, &resources, &context); + let key = secrets::random::<32>()?; + let file = read.editable.revealed(&read.editable.text); + let old = mask::mask(&file, &read.file, true, &key); + let new = mask::mask(&whole, &read.file, true, &key); + (review.added, review.removed) = diff::counts(Some(&old), Some(&new)); + Ok(review) + }) + .await? + } + + /// Keeps the cfg as it is, writes `written` in its place and records it. Nothing happens when + /// the file already holds those bytes. + async fn replace_cfg( + &self, + caller: &Caller, + read: Read, + written: Vec, + record: Record, + mut details: serde_json::Value, + ) -> Result<(), Error> { + if written == read.bytes { + return Ok(()); + } + let now = self.inner.clock.now(); + let (file, by, kept) = (read.file.clone(), caller.username.clone(), read.bytes.clone()); + self.inner + .db + .write(move |tx| { + let replaced = Replaced { + server: Servers::DEFAULT, + file: &file, + content: &kept, + lines: lines(&kept), + by: &by, + at: now, + }; + cfg_versions::keep(tx, replaced, VERSIONS_KEPT)?; + Ok::<_, Error>(()) + }) + .await?; + let file = read.file.clone(); + let (added, removed) = blocking(move || { + replace(&read.path(), &read.bytes, &written)?; + let key = secrets::random::<32>()?; + let old = mask::mask(&String::from_utf8_lossy(&read.bytes), &read.file, true, &key); + let new = mask::mask(&String::from_utf8_lossy(&written), &read.file, true, &key); + Ok::<_, Error>(diff::counts(Some(&old), Some(&new))) + }) + .await??; + if let Some(details) = details.as_object_mut() { + details.insert("added".to_owned(), json!(added)); + details.insert("removed".to_owned(), json!(removed)); + } + self.audit(record.target(file).details(details)).await + } +} + +/// Reads the launch's cfg for editing: a plain file of the server folder, never a link. +fn read(settings: &LaunchSettings) -> Result { + let platform = Platform::host(); + if launch::folder_problem(&settings.data_path, platform).is_some() + || launch::cfg_problem(&settings.cfg_file, platform).is_some() + { + return Err(refusal(CfgRefusal::NotConfigured)); + } + let data = PathBuf::from(&settings.data_path); + let path = data.join(&settings.cfg_file); + let unreadable = |error: io::Error| { + refusal(match error.kind() { + io::ErrorKind::NotFound => CfgRefusal::Missing, + _ => CfgRefusal::Unreadable, + }) + }; + let metadata = fs::symlink_metadata(&path).map_err(unreadable)?; + if !metadata.is_file() { + return Err(refusal(CfgRefusal::NotAFile)); + } + if metadata.permissions().readonly() { + return Err(refusal(CfgRefusal::ReadOnly)); + } + let limit = u64::try_from(MAX_TEXT_BYTES).unwrap_or(u64::MAX); + let mut bytes = Vec::new(); + // One byte past the limit tells a file that grew since its size was read. + File::open(&path) + .and_then(|file| file.take(limit + 1).read_to_end(&mut bytes)) + .map_err(unreadable)?; + if metadata.len() > limit || bytes.len() > MAX_TEXT_BYTES { + return Err(refusal(CfgRefusal::TooLarge)); + } + let editable = Editable::read(&bytes, &settings.cfg_file).map_err(|unreadable| { + refusal(match unreadable { + Unreadable::NotUtf8 => CfgRefusal::NotUtf8, + Unreadable::HoldsMarks => CfgRefusal::HoldsMarks, + }) + })?; + Ok(Read { + data, + file: settings.cfg_file.clone(), + onesync: settings.onesync, + modified: metadata + .modified() + .ok() + .and_then(|modified| Timestamp::try_from(modified).ok()), + bytes, + editable, + }) +} + +/// What changes between two versions of the cfg `file`, their secrets hidden. +fn changed(file: &str, old: &[u8], new: &[u8]) -> Result { + let key = secrets::random::<32>()?; + let old = mask::mask(&String::from_utf8_lossy(old), file, true, &key); + let new = mask::mask(&String::from_utf8_lossy(new), file, true, &key); + Ok(diff::diff(Some(&old), Some(&new), diff::DiffOptions::default())) +} + +/// Writes `written` over the cfg at `path`, which must still hold `expected`: beside it under a +/// temporary name, flushed, then renamed over it, so that FXServer never reads half a file. +fn replace(path: &Path, expected: &[u8], written: &[u8]) -> Result<(), Error> { + let metadata = fs::symlink_metadata(path)?; + if !metadata.is_file() || fs::read(path)? != expected { + return Err(ErrorCode::CfgChanged.into()); + } + let name = path.file_name().and_then(|name| name.to_str()).unwrap_or("cfg"); + let temporary = path.with_file_name(format!(".{name}{SAVING}")); + let result = (|| { + let mut file = File::create(&temporary)?; + file.write_all(written)?; + file.sync_all()?; + drop(file); + fs::set_permissions(&temporary, metadata.permissions())?; + retry(|| fs::rename(&temporary, path)) + })(); + if result.is_err() { + let _ = fs::remove_file(&temporary); + } + Ok(result?) +} + +fn retry(mut call: impl FnMut() -> io::Result<()>) -> io::Result<()> { + for delay in BUSY_DELAYS_MS { + match call() { + Err(error) if error.kind() == io::ErrorKind::PermissionDenied => { + thread::sleep(Duration::from_millis(delay)); + } + result => return result, + } + } + call() +} diff --git a/crates/observer-app/src/supervision/setup.rs b/crates/observer-app/src/supervision/setup.rs index b072be84..41c76899 100644 --- a/crates/observer-app/src/supervision/setup.rs +++ b/crates/observer-app/src/supervision/setup.rs @@ -1,7 +1,7 @@ //! The server setup: checks of an existing server's folder and cfg, read where they are, and the //! save that follows them. The setup never writes into that folder; the owner fixes what blocks. -//! Observer writes there only SFTP uploads and the fixes of the server tools the owner runs or -//! turns on. +//! Observer writes there only SFTP uploads, the fixes of the server tools the owner runs or turns +//! on, and the cfg the owner saves from its page. use std::io; use std::time::Duration; @@ -165,7 +165,7 @@ impl Supervision { .await } - async fn cfg_context(&self, onesync: OneSync) -> Result { + pub(super) async fn cfg_context(&self, onesync: OneSync) -> Result { let (_, state) = read_launch(&self.inner.db).await?; let system_resources = state.selected.map(|build| { Layout::new(self.inner.artifacts.join(build.to_string()), Flavour::HOST).system_resources() @@ -181,7 +181,9 @@ impl Supervision { } } -async fn blocking(work: impl FnOnce() -> T + Send + 'static) -> Result { +pub(super) async fn blocking( + work: impl FnOnce() -> T + Send + 'static, +) -> Result { Ok(tokio::task::spawn_blocking(work) .await .map_err(io::Error::other)?) diff --git a/crates/observer-app/src/tools.rs b/crates/observer-app/src/tools.rs index d22e50dc..1e88e757 100644 --- a/crates/observer-app/src/tools.rs +++ b/crates/observer-app/src/tools.rs @@ -44,8 +44,8 @@ use observer_domain::features::Feature; use observer_domain::server::{ServerState, ServerStatus}; use observer_domain::server_data::Game; use observer_domain::tools::{ - ToolChangeState, ToolCloseReason, ToolKind, ToolMode, ToolRun, ToolRunMode, ToolRunOutcome, ToolSettings, - ToolSettingsChange, ToolTrigger, ToolUndoRequest, ToolUndoScope, ToolView, ToolsView, + StagedFix, ToolChangeState, ToolCloseReason, ToolKind, ToolMode, ToolRun, ToolRunMode, ToolRunOutcome, + ToolSettings, ToolSettingsChange, ToolTrigger, ToolUndoRequest, ToolUndoScope, ToolView, ToolsView, }; use observer_fxserver::layout::{Flavour, Layout}; use observer_fxserver::pre_launch::{PreLaunch, PreLaunchFuture, PreLaunchRequest, PreLaunchResult}; @@ -207,11 +207,12 @@ impl Tools { /// what their backups take. pub async fn view(&self) -> Result { let since = self.inner.clock.now() - CLOSED_KEPT; - let (settings, version, last_run, last_report, changes) = self + let (settings, version, last_run, last_report, changes, staged) = self .inner .db .read(move |conn| -> Result<_, StoreError> { let (settings, version) = stored::settings(conn, Servers::DEFAULT, TOOL)?; + let staged = stored::staged(conn, Servers::DEFAULT, TOOL)?; let last_run = match stored::runs(conn, Servers::DEFAULT, TOOL, 1)?.first() { Some(latest) => stored::run(conn, latest.id)?, None => None, @@ -219,7 +220,7 @@ impl Tools { let last_report = stored::last_report(conn, Servers::DEFAULT, TOOL)? .filter(|report| last_run.as_ref().is_none_or(|last| last.id != report.id)); let changes = stored::changes(conn, Servers::DEFAULT, TOOL, since, CLOSED_LISTED)?; - Ok((settings, version, last_run, last_report, changes)) + Ok((settings, version, last_run, last_report, changes, staged)) }) .await?; let originals: BTreeSet<[u8; 32]> = changes @@ -250,6 +251,7 @@ impl Tools { unreachable, closed: changes.closed.iter().map(ToolChangeRow::view).collect(), backup_bytes, + staged, }], }) } @@ -345,6 +347,57 @@ impl Tools { .await } + /// Keeps the plan of the latest report for the next start of FXServer: its fix is written + /// then, before FXServer starts, if a fresh scan plans exactly that. Refused with + /// `plan_changed` when `plan` is not the latest report's, or fixes nothing. + pub async fn stage(&self, caller: &Caller, plan: &str) -> Result { + observer_domain::tools::sha256_from_hex(plan).ok_or(ErrorCode::InvalidRequest)?; + let record = Asker::Owner(caller.clone()).record(AuditAction::ToolFixStaged); + let (plan, by, now) = (plan.to_owned(), caller.username.clone(), self.inner.clock.now()); + self.inner + .db + .write(move |tx| -> Result<(), Error> { + let report = stored::last_report(tx, Servers::DEFAULT, TOOL)?; + let fixes = report + .as_ref() + .filter(|report| report.plan.as_deref() == Some(plan.as_str())) + .and_then(|report| report.summary.as_ref()) + .map_or(0, |summary| summary.fixable); + if fixes == 0 { + return Err(ApiError::new(ErrorCode::PlanChanged) + .with("run", serde_json::to_value(&report).unwrap_or_default()) + .into()); + } + let fix = StagedFix { + plan, + fixes, + staged_at: now, + staged_by: by, + }; + stored::stage(tx, Servers::DEFAULT, TOOL, &fix)?; + record.details(json!({ "fixes": fixes })).write(tx, now)?; + Ok(()) + }) + .await?; + self.view().await + } + + /// Drops the fix staged for the next start, if there is one. + pub async fn unstage(&self, caller: &Caller) -> Result { + let record = Asker::Owner(caller.clone()).record(AuditAction::ToolFixUnstaged); + let now = self.inner.clock.now(); + self.inner + .db + .write(move |tx| -> Result<(), Error> { + if stored::unstage(tx, Servers::DEFAULT, TOOL)? { + record.write(tx, now)?; + } + Ok(()) + }) + .await?; + self.view().await + } + /// Scans the server folder as the launch settings give it, and records the report. Writes /// nothing; FXServer may run. pub async fn scan(&self, caller: &Caller) -> Result { @@ -506,6 +559,15 @@ impl Tools { } else { settings.at_start }; + // A fix at every start already writes whatever the fresh scan plans. + let staged = if request.cause == Cause::Recovery || !on || mode == ToolMode::Fix { + None + } else { + self.inner + .db + .read(|conn| stored::staged(conn, Servers::DEFAULT, TOOL)) + .await? + }; let target = Target::Scanned { folder: PathBuf::from(&request.launch.settings.data_path), system_resources: Some(request.layout.system_resources()), @@ -513,7 +575,12 @@ impl Tools { }; let prepared = self.prepare(target.clone()).await?; let open = self.open_count(&prepared.folder).await?; + let confirmed = staged + .as_ref() + .filter(|_| prepared.supported()) + .and_then(|staged| observer_domain::tools::sha256_from_hex(&staged.plan)); let (steps, run_mode) = match mode { + _ if confirmed.is_some() => (Steps::Fix(confirmed), ToolRunMode::Fix), _ if open == 0 && (mode == ToolMode::Off || !prepared.supported()) => return Ok(Vec::new()), ToolMode::Off => (Steps::Heal, ToolRunMode::Heal), ToolMode::Report => (Steps::HealAndReport, ToolRunMode::Report), @@ -529,7 +596,8 @@ impl Tools { }; let run = self.insert_run(&Asker::Start, run_mode).await?; // A report alone writes nothing: SFTP goes on meanwhile. - let writes = open > 0 || mode == ToolMode::Fix; + let writes = open > 0 || mode == ToolMode::Fix || confirmed.is_some(); + let mode = if confirmed.is_some() { ToolMode::Fix } else { mode }; let gate = if writes { Some(self.inner.gate.exclusive().await) } else { @@ -538,11 +606,40 @@ impl Tools { let job = self.job(&target, prepared, settings, run, request.cancel, open); let finished = blocking(move || work::run(job, &steps)).await?; drop(gate); - let lines = lines::start(&finished, mode, previous.as_ref()); + let mut lines = lines::start(&finished, mode, previous.as_ref()); + let read = finished.outcome() == ToolRunOutcome::Done; + let changed = finished.plan_changed; self.finish(run, &Asker::Start, finished).await?; + // Read once: written, or another plan by now, which the owner reviews before staging again. + if let Some(staged) = staged.filter(|_| confirmed.is_some() && read) { + if changed { + lines.push(format!( + "the fix {} staged was not written: the scan plans another one by now, to review on the Tools page", + staged.staged_by + )); + } + self.drop_staged(changed).await?; + } Ok(lines) } + /// Drops the staged fix once a start has read it, recorded when it was not written. + async fn drop_staged(&self, plan_changed: bool) -> Result<(), Error> { + let record = Asker::Start + .record(AuditAction::ToolFixUnstaged) + .details(json!({ "planChanged": true })); + let now = self.inner.clock.now(); + self.inner + .db + .write(move |tx| -> Result<(), Error> { + if stored::unstage(tx, Servers::DEFAULT, TOOL)? && plan_changed { + record.write(tx, now)?; + } + Ok(()) + }) + .await + } + /// At Observer's start, before FXServer's: settles the writes an earlier Observer was /// interrupted in, in the folder the launch settings give, then ends the runs it left /// unfinished, audited with the files the journal says they changed or put back. @@ -894,6 +991,10 @@ impl Tools { for record in records { record.write(tx, now)?; } + // Files changed: a plan staged before is no longer the one a scan would make. + if wrote { + stored::unstage(tx, Servers::DEFAULT, TOOL)?; + } Ok(()) }) .await?; diff --git a/crates/observer-app/src/tools/tests.rs b/crates/observer-app/src/tools/tests.rs index 56ce0588..5f5c93d2 100644 --- a/crates/observer-app/src/tools/tests.rs +++ b/crates/observer-app/src/tools/tests.rs @@ -91,6 +91,7 @@ impl Lab { launch: LaunchReadiness::Ready, name: None, port: None, + start: None, }); let tools = Tools::new(ToolsSetup { db: db.clone(), @@ -409,6 +410,119 @@ async fn a_fix_of_a_plan_that_changed_since_it_was_shown_writes_nothing_and_hand ); } +#[tokio::test] +async fn a_fix_staged_while_fxserver_runs_is_written_before_its_next_start() { + let lab = Lab::new().await; + let camp = lab.read(CAMP); + lab.set_state(ServerState::Online); + let shown = lab.tools.scan(&owner()).await.unwrap(); + + let view = lab + .tools + .stage(&owner(), shown.plan.as_deref().unwrap()) + .await + .unwrap(); + + let staged = view.tools[0].staged.clone().unwrap(); + assert_eq!( + (staged.plan.as_str(), staged.fixes), + (shown.plan.as_deref().unwrap(), 2) + ); + assert_eq!(staged.staged_by, "ada"); + assert_eq!(lab.read(CAMP), camp, "nothing is written while FXServer runs"); + let audited = lab.audited(AuditAction::ToolFixStaged).await; + assert_eq!(audited, [("user".to_owned(), json!({ "fixes": 2 }))]); + + // A recovery from a crash starts the server as it was: the fix keeps waiting. + assert_eq!(lab.start(Cause::Recovery).await, Vec::::new()); + assert_eq!(lab.read(CAMP), camp); + assert!(lab.tools.view().await.unwrap().tools[0].staged.is_some()); + + let lines = lab.start(Cause::Requested { by: "ada".to_owned() }).await; + + assert_eq!(lab.archetypes(CAMP), hashes(&["stick", "marshmallow"])); + assert!(lines.iter().any(|line| line.contains("2 file")), "{lines:?}"); + let view = lab.tools.view().await.unwrap(); + assert_eq!(view.tools[0].staged, None, "read once"); + let run = view.tools[0].last_run.clone().unwrap(); + assert_eq!((run.mode, run.trigger), (ToolRunMode::Fix, ToolTrigger::Start)); + assert_eq!(run.summary.unwrap().files_changed, 2); + assert_eq!(lab.audited(AuditAction::ToolFixUnstaged).await, Vec::new()); +} + +#[tokio::test] +async fn a_staged_fix_is_not_written_when_the_scan_before_the_start_plans_another_one() { + let lab = Lab::new().await; + let camp = lab.read(CAMP); + lab.set_state(ServerState::Online); + let shown = lab.tools.scan(&owner()).await.unwrap(); + lab.tools + .stage(&owner(), shown.plan.as_deref().unwrap()) + .await + .unwrap(); + ytyp(&lab.folder(), TENT, &["tent", "tent"]); + + let lines = lab.start(Cause::Requested { by: "ada".to_owned() }).await; + + assert_eq!(lab.read(CAMP), camp); + assert!( + lines + .iter() + .any(|line| line.contains("the fix ada staged was not written")), + "{lines:?}" + ); + let view = lab.tools.view().await.unwrap(); + assert_eq!(view.tools[0].staged, None, "to review and stage again"); + assert_eq!( + lab.audited(AuditAction::ToolFixUnstaged).await, + [("system".to_owned(), json!({ "planChanged": true }))] + ); + assert_eq!(lab.audited(AuditAction::ToolFilesChanged).await, Vec::new()); +} + +#[tokio::test] +async fn only_the_plan_of_the_latest_report_is_staged_and_a_run_that_writes_drops_it() { + let lab = Lab::new().await; + let refused = lab.tools.stage(&owner(), &"0".repeat(64)).await.unwrap_err(); + assert_eq!(code(&refused), Some(ErrorCode::PlanChanged), "no report yet"); + let shown = lab.tools.scan(&owner()).await.unwrap(); + let plan = shown.plan.as_deref().unwrap(); + assert_eq!( + code(&lab.tools.stage(&owner(), "not a digest").await.unwrap_err()), + Some(ErrorCode::InvalidRequest) + ); + assert_eq!( + code(&lab.tools.stage(&owner(), &"0".repeat(64)).await.unwrap_err()), + Some(ErrorCode::PlanChanged) + ); + + lab.tools.stage(&owner(), plan).await.unwrap(); + let dropped = lab.tools.unstage(&owner()).await.unwrap(); + assert_eq!(dropped.tools[0].staged, None); + assert_eq!(lab.audited(AuditAction::ToolFixUnstaged).await.len(), 1); + lab.tools.unstage(&owner()).await.unwrap(); + assert_eq!( + lab.audited(AuditAction::ToolFixUnstaged).await.len(), + 1, + "nothing to drop" + ); + + lab.tools.stage(&owner(), plan).await.unwrap(); + lab.tools.fix(&owner(), plan).await.unwrap(); + assert_eq!(lab.tools.view().await.unwrap().tools[0].staged, None); + let fixed = lab.tools.scan(&owner()).await.unwrap(); + assert_eq!( + code( + &lab.tools + .stage(&owner(), fixed.plan.as_deref().unwrap()) + .await + .unwrap_err() + ), + Some(ErrorCode::PlanChanged), + "a plan that fixes nothing" + ); +} + #[tokio::test] async fn an_undo_gives_every_file_its_original_back_and_is_audited() { let lab = Lab::new().await; diff --git a/crates/observer-domain/src/audit.rs b/crates/observer-domain/src/audit.rs index 20377450..1dc32d68 100644 --- a/crates/observer-domain/src/audit.rs +++ b/crates/observer-domain/src/audit.rs @@ -37,6 +37,15 @@ pub enum AuditAction { ServerStarted, ServerStopped, ServerRestarted, + /// Someone gave a start more time before Observer takes it for stuck. `details` holds the + /// `minutes` given. + ServerStartExtended, + /// The owner saved the launch's cfg from the panel. The target is the file; `details` holds + /// the lines `added` and `removed`, never their text. + ServerCfgChanged, + /// The owner put an earlier version of the cfg back. `details` holds the `version` and the + /// lines `added` and `removed`. + ServerCfgRestored, /// Written before the command reaches FXServer, with the `command` as the console shows it, /// secrets masked. `observer_cmd` is denied, with its `refusal`. ConsoleCommandSent, @@ -173,10 +182,22 @@ pub enum AuditAction { /// tool; `details` holds the `run`, the `reason` (`heal` or `undo`), the `files` (a capped /// list) and their `count`. ToolFilesRestored, + /// The owner staged a tool's fix for the next start of FXServer. The target is the tool; + /// `details` holds the `fixes` it plans. + ToolFixStaged, + /// A staged fix was dropped: by the owner, or by Observer when the scan before the start + /// planned another fix than the one staged (`planChanged`). + ToolFixUnstaged, /// The owner turned an optional feature on or off. The target is the feature; `details` /// holds `enabled` and when it `applies` (`now` or `nextStart`). A refusal by the host's /// lock in `observer.toml` is `denied`. FeatureChanged, + /// The owner said where SFTP listens, or took its address back. `details` holds `listen` and + /// `previous`, each an address or null. + SftpListenChanged, + /// The owner said they know why a resource calls a host, which Debug then no longer shows as + /// new. The target is the resource; `details` holds the `host`. + DebugHostKnown, /// `details` holds each changed setting with its value before and after, and /// `passwordChanged` when the manual source's password was replaced, never the password. BackupSettingsChanged, @@ -220,7 +241,7 @@ pub enum AuditAction { } impl AuditAction { - pub const ALL: [Self; 104] = [ + pub const ALL: [Self; 111] = [ Self::SetupCompleted, Self::SetupLinkCreated, Self::LoginSucceeded, @@ -240,6 +261,9 @@ impl AuditAction { Self::ServerStarted, Self::ServerStopped, Self::ServerRestarted, + Self::ServerStartExtended, + Self::ServerCfgChanged, + Self::ServerCfgRestored, Self::ConsoleCommandSent, Self::ConsoleLogDownloaded, Self::ServerCrashLoop, @@ -310,7 +334,11 @@ impl AuditAction { Self::ToolSettingsChanged, Self::ToolFilesChanged, Self::ToolFilesRestored, + Self::ToolFixStaged, + Self::ToolFixUnstaged, Self::FeatureChanged, + Self::SftpListenChanged, + Self::DebugHostKnown, Self::BackupSettingsChanged, Self::BackupDestinationCreated, Self::BackupDestinationChanged, @@ -349,6 +377,9 @@ impl AuditAction { Self::ServerStarted => "server_started", Self::ServerStopped => "server_stopped", Self::ServerRestarted => "server_restarted", + Self::ServerStartExtended => "server_start_extended", + Self::ServerCfgChanged => "server_cfg_changed", + Self::ServerCfgRestored => "server_cfg_restored", Self::ConsoleCommandSent => "console_command_sent", Self::ConsoleLogDownloaded => "console_log_downloaded", Self::ServerCrashLoop => "server_crash_loop", @@ -419,7 +450,11 @@ impl AuditAction { Self::ToolSettingsChanged => "tool_settings_changed", Self::ToolFilesChanged => "tool_files_changed", Self::ToolFilesRestored => "tool_files_restored", + Self::ToolFixStaged => "tool_fix_staged", + Self::ToolFixUnstaged => "tool_fix_unstaged", Self::FeatureChanged => "feature_changed", + Self::SftpListenChanged => "sftp_listen_changed", + Self::DebugHostKnown => "debug_host_known", Self::BackupSettingsChanged => "backup_settings_changed", Self::BackupDestinationCreated => "backup_destination_created", Self::BackupDestinationChanged => "backup_destination_changed", diff --git a/crates/observer-domain/src/cfg.rs b/crates/observer-domain/src/cfg.rs new file mode 100644 index 00000000..52576901 --- /dev/null +++ b/crates/observer-domain/src/cfg.rs @@ -0,0 +1,173 @@ +//! The launch's cfg as the panel edits it: its text with every secret replaced by a mark, what its +//! lines set and start, and what a save sends back. + +use jiff::Timestamp; +use serde::{Deserialize, Serialize}; +use ts_rs::TS; + +use crate::server_data::{CfgFinding, EndpointProtocol}; + +/// A secret of the file stands in the text as `<>`, numbered from 1 in the order of the +/// file. A save puts the secret back where its mark still is. +pub const SECRET_OPEN: &str = "<, + /// How many secrets the text hides. + pub secrets: u32, + pub check: CfgReview, + /// How many earlier versions are kept. + pub versions: u32, +} + +/// What a text of the cfg sets and starts, and what the scan of it and of the files it runs finds. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct CfgReview { + pub outline: CfgOutline, + pub findings: Vec, + /// The lines a save of the text would add to the file and take from it: a line changed + /// counts once in each. + pub added: u32, + pub removed: u32, +} + +/// The lines of a cfg the panel edits by field. Lines count from 1; `start` and `end` say where a +/// value lies in its line, in UTF-16 code units, its quotes included. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct CfgOutline { + /// Every `set` and its kin, and the convars the panel has a field for set by their name alone, + /// in the order of the file. The last one of a name is the one FXServer keeps. + pub settings: Vec, + pub endpoints: Vec, + /// Every line that starts a resource or a category, and those commented out that would start + /// one the folder holds. + pub resources: Vec, + /// The file has more of them than are listed. + pub truncated: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct CfgSetting { + pub line: u32, + /// `set`, `sets`, `setr` or `seta`; null when the convar's name is the command. + pub command: Option, + /// As written. + pub name: String, + /// Null for a secret. + pub value: Option, + pub start: u32, + pub end: u32, + /// The line holds nothing else that FXServer runs. + pub alone: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct CfgEndpointLine { + pub line: u32, + pub protocol: EndpointProtocol, + pub address: String, + pub start: u32, + pub end: u32, + pub alone: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct CfgResourceLine { + pub line: u32, + /// The resource, or the category in brackets, as written. + pub name: String, + /// False for a line commented out. + pub enabled: bool, + /// The line holds nothing else: it can be commented out, or put back, as a whole. + pub alone: bool, + /// Neither the server folder nor the build holds it. + pub missing: bool, +} + +/// `POST /api/servers/default/cfg/check` and `/cfg/diff`: the text as edited, marks included. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize, TS)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +#[ts(export)] +pub struct CfgDraft { + pub text: String, +} + +/// `PUT /api/servers/default/cfg`. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize, TS)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +#[ts(export)] +pub struct CfgSave { + pub text: String, + /// [`CfgDocument::base`] of the text the edit started from. + pub base: String, +} + +/// `POST /api/servers/default/cfg/versions/{id}/restore`. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize, TS)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +#[ts(export)] +pub struct CfgRestore { + /// [`CfgDocument::base`] of the file the version replaces. + pub base: String, +} + +/// The cfg as it was before a save replaced it. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct CfgVersion { + pub id: i64, + /// When it was replaced, and by whom. + pub replaced_at: Timestamp, + pub replaced_by: String, + pub bytes: u32, + pub lines: u32, +} + +/// Why the cfg cannot be edited from the panel, as the `reason` of `cfg_not_editable`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub enum CfgRefusal { + /// No launch settings are saved yet. + NotConfigured, + Missing, + /// A folder, a link, or anything but a plain file. + NotAFile, + Unreadable, + /// Marked read-only on the host, which a save could not write over. + ReadOnly, + /// Over 512 KiB. + TooLarge, + /// Its bytes are not UTF-8: saving it would change every accented letter. + NotUtf8, + /// It already holds text written like a secret's mark. + HoldsMarks, +} diff --git a/crates/observer-domain/src/debug.rs b/crates/observer-domain/src/debug.rs new file mode 100644 index 00000000..75a078ab --- /dev/null +++ b/crates/observer-domain/src/debug.rs @@ -0,0 +1,323 @@ +//! What the Debug pages read: the events, the state bag writes and the outgoing HTTP requests +//! the server's resources count themselves, once the owner turned Debug on. Names, counts and +//! sizes over the last minutes, kept in memory: never what an event, a bag or a request carries. + +use jiff::Timestamp; +use serde::{Deserialize, Serialize}; +use ts_rs::TS; + +/// How far back a page looks. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize, TS)] +#[ts(export)] +pub enum DebugWindow { + #[serde(rename = "1m")] + Minute, + #[default] + #[serde(rename = "5m")] + FiveMinutes, + #[serde(rename = "15m")] + QuarterHour, +} + +impl DebugWindow { + pub const fn seconds(self) -> u32 { + match self { + Self::Minute => 60, + Self::FiveMinutes => 300, + Self::QuarterHour => 900, + } + } +} + +/// `?window=` of the Debug routes. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, TS)] +#[serde(deny_unknown_fields)] +#[ts(export)] +pub struct DebugQuery { + #[serde(default)] + pub window: DebugWindow, +} + +/// Where the counting stands. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub enum DebugState { + /// On, but FXServer started before it was, or does not run: it counts from the next start. + Waiting, + /// The server that runs was started to count. + Counting, + /// On, but the build's scheduler was left as it was at this start: nothing is counted. + Refused, +} + +/// What every Debug page says under its title. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct DebugStatus { + pub state: DebugState, + /// Whether FXServer runs now. + pub running: bool, + /// When the first report of this start came in. + pub since: Option, + /// How many resources report. + pub reporting: u32, + /// The started resources that do not: in JavaScript or C#, or with no script. The first + /// hundred by name. + pub silent: Vec, + pub silent_count: u32, + /// Why the scheduler was left as it was, in the host's words. + pub refused: Option, +} + +/// Which way an event went. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub enum EventWay { + FromClients, + ToClient, + ToAll, + /// Between the server's resources. + Local, + /// From the engine, such as `playerJoining`. + FromEngine, +} + +/// A count of events with the bytes they carried. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, TS)] +#[ts(export)] +pub struct Flow { + pub count: u64, + pub bytes: u64, +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct EventTotals { + pub to_clients: Flow, + pub from_clients: Flow, + /// Events received from the server's own resources. + pub local: u64, + /// In the handlers of the events received, in microseconds. + pub handler_us: u64, + /// Events clients sent that no resource opened to them. + pub refused: u64, + pub refused_players: u32, +} + +/// The bytes that went to and came from clients in one slice of the window. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct TrafficPoint { + pub at: Timestamp, + pub seconds: u32, + pub to_clients: u64, + pub from_clients: u64, +} + +/// A client that sent an event. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, TS)] +#[ts(export)] +pub struct EventSender { + pub player: u32, + pub count: u64, + pub bytes: u64, +} + +/// An event in one way, over the resources that sent or received it. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct EventRow { + pub name: String, + pub way: EventWay, + /// The first three by traffic. + pub resources: Vec, + pub resource_count: u32, + pub count: u64, + pub bytes: u64, + /// The largest single payload. + pub largest: u64, + pub handler_us: u64, + /// Who sent it most, for an event from clients: five at most. + pub senders: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct EventResourceRow { + pub resource: String, + pub to_clients: Flow, + pub from_clients: Flow, + /// Events received from the server's resources and from the engine. + pub local: u64, + pub handler_us: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[ts(export)] +pub struct EventPlayerRow { + pub player: u32, + pub count: u64, + pub bytes: u64, + /// The event they sent most. + pub top: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[ts(export)] +pub struct RefusedRow { + pub name: String, + /// Null when the scheduler did not say who. + pub player: Option, + pub resource: String, + pub count: u64, +} + +/// `GET /api/debug/events`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[ts(export)] +pub struct DebugEvents { + pub status: DebugStatus, + /// The window's length. + pub seconds: u32, + pub totals: EventTotals, + pub series: Vec, + /// By traffic, the first two hundred. + pub events: Vec, + pub resources: Vec, + pub players: Vec, + pub refused: Vec, + /// A resource or Observer kept less than there was. + pub truncated: bool, +} + +/// The bag a write went to. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub enum BagKind { + Global, + Player, + Entity, + LocalEntity, + Other, +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct BagTotals { + pub writes: u64, + pub bytes: u64, + /// Of the writes that went out to clients. + pub replicated_bytes: u64, + pub keys: u32, + pub resources: u32, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct BagRow { + pub key: String, + pub kind: BagKind, + pub resources: Vec, + pub resource_count: u32, + pub count: u64, + pub bytes: u64, + pub largest: u64, + /// How many of the writes went out to clients. + pub replicated: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct BagResourceRow { + pub resource: String, + pub count: u64, + pub bytes: u64, + pub replicated_bytes: u64, + pub keys: u32, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[ts(export)] +pub struct LargestWrite { + /// The bag's own name, such as `player:41`. + pub bag: String, + pub key: String, + pub resource: String, + pub bytes: u64, + pub replicated: bool, +} + +/// `GET /api/debug/bags`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[ts(export)] +pub struct DebugBags { + pub status: DebugStatus, + pub seconds: u32, + pub totals: BagTotals, + pub keys: Vec, + pub resources: Vec, + pub largest: Vec, + pub truncated: bool, +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct HttpTotals { + pub requests: u64, + /// Of the request bodies. + pub bytes: u64, + pub hosts: u32, + pub resources: u32, + /// The hosts no resource had called before, and nobody said they know. + pub new_hosts: u32, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct HttpRow { + pub host: String, + pub method: String, + pub resource: String, + pub count: u64, + pub bytes: u64, + /// When this resource first called this host, as far as Observer saw. + pub first_called: Option, + /// First called after Observer's day of learning, and not marked as known since. + pub new: bool, +} + +/// `GET /api/debug/http`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct DebugHttp { + pub status: DebugStatus, + pub seconds: u32, + pub totals: HttpTotals, + pub requests: Vec, + /// Until when every host is taken as known: the first day Debug counted. + pub learning_until: Option, + pub truncated: bool, +} + +/// `POST /api/debug/http/known`: the owner knows why this resource calls this host. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize, TS)] +#[serde(deny_unknown_fields)] +#[ts(export)] +pub struct KnownHost { + pub resource: String, + pub host: String, +} diff --git a/crates/observer-domain/src/error.rs b/crates/observer-domain/src/error.rs index 58d1e17e..8d74da76 100644 --- a/crates/observer-domain/src/error.rs +++ b/crates/observer-domain/src/error.rs @@ -56,6 +56,17 @@ pub enum ErrorCode { ServerNotRunning, /// FXServer is stopping; it can be started again once it has. ServerStopping, + /// Only a start whose cfg still runs can be given more time. + ServerNotStarting, + /// The launch's cfg is not edited from the panel. Params: `reason`, a + /// [`CfgRefusal`](crate::cfg::CfgRefusal). + CfgNotEditable, + /// The cfg is no longer the file the edit started from: it is read again before a save. + CfgChanged, + /// The text is not written as a cfg. Params: `reason`, `unknownMark` for a secret's mark that + /// stands for nothing, `control` for a NUL, `tooLarge`. + InvalidCfgText, + CfgVersionNotFound, /// Observer is stopping, and FXServer with it. ShuttingDown, /// Params: `reason`, a `CommandRefusal`. @@ -201,6 +212,11 @@ pub enum ErrorCode { FeatureDisabled, /// `observer.toml` turns the feature off, which only the host can undo. Params: `feature`. FeatureLocked, + /// `observer.toml` names where SFTP listens, which only the host changes. + SftpListenPinned, + /// Not an address SFTP can listen on. Params: `reason`, `address` when it is not an IP address + /// with a port, `port` when the port is 0. + InvalidSftpListen, /// A backup of the server's database is under way: one at a time. BackupRunning, /// Params: `issues`, a list of `BackupSettingsIssue`. diff --git a/crates/observer-domain/src/features.rs b/crates/observer-domain/src/features.rs index ab31346c..ed0de0d5 100644 --- a/crates/observer-domain/src/features.rs +++ b/crates/observer-domain/src/features.rs @@ -13,7 +13,7 @@ use crate::live::Topic; #[serde(rename_all = "camelCase")] #[ts(export)] pub enum Feature { - /// File access over SFTP, on the port `[sftp] listen` of `observer.toml` names. + /// File access over SFTP, on the address the owner gives in the panel or `observer.toml` pins. Sftp, /// What changed in the server folder from one start of FXServer to the next. Changes, @@ -29,6 +29,10 @@ pub enum Feature { Drops, /// Checks GitHub for a newer release of Observer, which the owner installs from the panel. Updates, + /// What the server's resources send and receive, counted by the resources themselves: events, + /// state bag writes and outgoing HTTP requests. Off until the owner turns it on, since it adds + /// two lines to the build's Lua scheduler. + Debug, } /// When a change of a switch takes effect. @@ -50,8 +54,10 @@ pub enum Applies { pub enum FeatureHold { /// `observer.toml` turns it off, which the panel cannot undo. Host, - /// It needs a setting `observer.toml` does not give: SFTP's `listen`. + /// It needs a setting nobody gave: the address SFTP listens on. NotConfigured, + /// Its port did not open: another program holds it, or the host refuses the address. + PortRefused, /// It works on what a feature in `requires` collects, which does not run. Requires, } @@ -98,6 +104,11 @@ pub enum FeatureTask { UpdateCheck, /// After an install, settles whether the new version started, or puts the previous one back. UpdateProbation, + /// The two lines in the build's Lua scheduler: put in while Debug is on, taken out while it + /// is off. + HookLines, + /// Reads and totals what the resources count. + HookCounts, } /// When a task of a feature runs. @@ -138,7 +149,7 @@ pub enum ServerFolder { } impl Feature { - pub const ALL: [Self; 8] = [ + pub const ALL: [Self; 9] = [ Self::Sftp, Self::Changes, Self::Tools, @@ -147,6 +158,7 @@ impl Feature { Self::Players, Self::Drops, Self::Updates, + Self::Debug, ]; /// The name on the wire, in the store and in `observer.toml`. @@ -160,9 +172,16 @@ impl Feature { Self::Players => "players", Self::Drops => "drops", Self::Updates => "updates", + Self::Debug => "debug", } } + /// Whether it is on until the owner says otherwise. Debug alone waits to be asked for: it + /// changes a file of the build. + pub const fn on_by_default(self) -> bool { + !matches!(self, Self::Debug) + } + pub fn from_name(name: &str) -> Option { Self::ALL.into_iter().find(|feature| feature.as_str() == name) } @@ -176,12 +195,13 @@ impl Feature { } } - /// When turning it on takes effect. The tools' hook reads the switch before every start; - /// the others follow FXServer from Observer's start, and a port opens only when the host - /// restarts Observer. + /// When turning it on takes effect. SFTP's port opens and closes while Observer runs; the + /// tools' hook reads the switch before every start; the others follow FXServer from + /// Observer's start. pub const fn turn_on(self) -> Applies { match self { - Self::Tools => Applies::NextServerStart, + Self::Sftp => Applies::Now, + Self::Tools | Self::Debug => Applies::NextServerStart, _ => Applies::ObserverRestart, } } @@ -227,6 +247,10 @@ impl Feature { run(T::UpdateCheck, C::Every { seconds: 6 * 3600 }), run(T::UpdateProbation, C::ObserverStart), ]; + const DEBUG: &[FeatureTaskRun] = &[ + run(T::HookLines, C::ServerStart), + run(T::HookCounts, C::Continuous), + ]; match self { Self::Sftp => SFTP, Self::Changes => CHANGES, @@ -236,6 +260,7 @@ impl Feature { Self::Players => PLAYERS, Self::Drops => DROPS, Self::Updates => UPDATES, + Self::Debug => DEBUG, } } @@ -266,6 +291,11 @@ impl Feature { task: FeatureTask::UpdateProbation, cadence: Cadence::ObserverStart, }], + // The two lines a start put into the scheduler come out at the next one. + Self::Debug => &[FeatureTaskRun { + task: FeatureTask::HookLines, + cadence: Cadence::ServerStart, + }], _ => &[], } } @@ -273,7 +303,7 @@ impl Feature { /// The files and folders it keeps, under Observer's data folder. pub const fn files(self) -> &'static [&'static str] { match self { - Self::Sftp => &[], + Self::Sftp | Self::Debug => &[], Self::Changes => &["changes.sqlite3"], Self::Tools => &["tools/backups"], Self::Diagnostics => &["crashes"], @@ -286,7 +316,7 @@ impl Feature { match self { Self::Sftp | Self::Tools => ServerFolder::Writes, Self::Changes => ServerFolder::Reads, - Self::Diagnostics | Self::Metrics | Self::Players | Self::Drops | Self::Updates => { + Self::Diagnostics | Self::Metrics | Self::Players | Self::Drops | Self::Updates | Self::Debug => { ServerFolder::Untouched } } @@ -303,6 +333,7 @@ impl Feature { Self::Players => &["/servers/default/players"], Self::Drops => &["/servers/default/drops"], Self::Updates => &["/updates"], + Self::Debug => &["/debug"], } } @@ -312,7 +343,7 @@ impl Feature { Self::Diagnostics => &[Topic::ServerDiagnostics, Topic::ServerAlerts], Self::Metrics => &[Topic::MetricsLive], Self::Players => &[Topic::ServerMap], - Self::Sftp | Self::Tools | Self::Drops | Self::Updates => &[], + Self::Sftp | Self::Tools | Self::Drops | Self::Updates | Self::Debug => &[], } } @@ -364,7 +395,7 @@ pub struct FeatureView { #[serde(rename_all = "camelCase")] #[ts(export)] pub struct FeatureReach { - /// The addresses it listens on, as `observer.toml` gives them. + /// The addresses it listens on. pub listens: Vec, pub tasks: Vec, pub while_off: Vec, @@ -415,6 +446,31 @@ pub struct FeatureSwitch { pub enabled: bool, } +/// `GET /api/settings/sftp`, and the answer to a change: where SFTP listens, and who says so. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct SftpListen { + /// The address in force, such as `0.0.0.0:40126`; `None` until someone gives one. + pub listen: Option, + /// Whether `observer.toml` names it, which the panel cannot change. + pub pinned: bool, + /// Whether the port is open on that address now. + pub open: bool, + /// The fingerprint of the host key a client is shown on the open port. + pub fingerprint: Option, + /// Why the port did not open, in the host's words. + pub refused: Option, +} + +/// `PUT /api/settings/sftp`. No address closes the port and forgets the one there was. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize, TS)] +#[serde(deny_unknown_fields)] +#[ts(export)] +pub struct SftpListenChange { + pub listen: Option, +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/observer-domain/src/lib.rs b/crates/observer-domain/src/lib.rs index 74e5a70a..d81de765 100644 --- a/crates/observer-domain/src/lib.rs +++ b/crates/observer-domain/src/lib.rs @@ -11,8 +11,10 @@ pub mod artifacts; pub mod audit; pub mod backups; pub mod bans; +pub mod cfg; pub mod changes; pub mod convars; +pub mod debug; pub mod delegation; pub mod diagnostics; pub mod drops; diff --git a/crates/observer-domain/src/map.rs b/crates/observer-domain/src/map.rs index c709ff5c..0835e738 100644 --- a/crates/observer-domain/src/map.rs +++ b/crates/observer-domain/src/map.rs @@ -1,5 +1,6 @@ -//! The live map: where connected players stand, as the server sees them, drawn over a map image -//! the owner adds. Observer ships no map: GTA V's belongs to Rockstar Games. +//! The live map: where connected players stand, as the server sees them, drawn over Rockstar's map +//! of GTA V, which each browser loads from Rockstar's servers, or over an image the owner adds. +//! Observer ships and keeps no map of its own: GTA V's belongs to Rockstar Games. use jiff::Timestamp; use serde::{Deserialize, Serialize}; @@ -38,11 +39,33 @@ impl MapBounds { } } +/// What the map draws under the players. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub enum MapImagery { + /// Rockstar's map of GTA V as the game draws it. Like the two after it, each browser that + /// opens the map loads it from Rockstar's servers. + #[default] + Atlas, + /// The same map, seen from the sky. + Satellite, + /// The same map, as a road map. + Roads, + /// The image the owner added, once it is placed in the game. + Image, + /// A grid of game coordinates. + Grid, +} + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)] #[serde(rename_all = "camelCase", deny_unknown_fields)] #[ts(export)] pub struct MapSettings { - /// `None` until the image is calibrated: the map then shows a grid of game coordinates. + #[serde(default)] + pub imagery: MapImagery, + /// Where the owner's image sits; `None` until it is calibrated: the map then shows a grid of + /// game coordinates in its place. pub bounds: Option, /// A player is new under this many hours of play time. pub new_hours: u32, @@ -55,6 +78,7 @@ pub struct MapSettings { impl Default for MapSettings { fn default() -> Self { Self { + imagery: MapImagery::default(), bounds: None, new_hours: 10, veteran_hours: 100, @@ -157,6 +181,20 @@ mod tests { assert!(MapSettings::default().is_valid()); assert!(valid.is_valid()); + assert_eq!(MapSettings::default().imagery, MapImagery::Atlas); + // Settings sent without it, as a panel loaded before it existed sends them. + let sent: MapSettings = + serde_json::from_str(r#"{"bounds":null,"newHours":10,"veteranHours":100,"everySeconds":2}"#) + .unwrap(); + assert_eq!(sent, MapSettings::default()); + assert_eq!( + serde_json::to_value(MapSettings { + imagery: MapImagery::Satellite, + ..MapSettings::default() + }) + .unwrap()["imagery"], + "satellite" + ); for invalid in [ MapSettings { new_hours: 100, diff --git a/crates/observer-domain/src/server.rs b/crates/observer-domain/src/server.rs index 0b257c45..32614258 100644 --- a/crates/observer-domain/src/server.rs +++ b/crates/observer-domain/src/server.rs @@ -150,6 +150,125 @@ pub struct PlayerCount { pub max: Option, } +/// The last resource FXServer said it started. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct StartedResource { + pub name: String, + pub at: Timestamp, + /// What FXServer counted after its name as it started. + pub warnings: u32, +} + +/// The command of the cfg a start is at. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct StartCommand { + /// The cfg file the command is written in, named from the server folder. + pub file: String, + pub line: u32, + /// Its argument as written: a resource, or a category in brackets. + pub target: String, + /// When FXServer came to it. + pub since: Timestamp, + /// How many of the resources it starts are up or refused to start, out of `of`. + pub done: u32, + pub of: u32, + /// The resource whose server scripts FXServer is loading, when it said so. + pub resource: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct SlowResource { + pub name: String, + /// From the resource started before it to this one. + pub ms: u32, +} + +/// How far the start under way has come, read from what FXServer prints while it runs its cfg: +/// each resource once it is up, and nothing of the one it is starting. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct StartProgress { + /// Resources FXServer said it started. + pub started: u32, + /// Resources it could not start. + pub failed: u32, + /// How many the server folder holds, once FXServer has said. + pub found: Option, + /// How many the cfg's commands start. Null when Observer could not read them all. + pub planned: Option, + pub last: Option, + /// Worked out from the resources each command starts. Null before the first command and + /// after the last, and when Observer could not read every command. + pub at: Option, + /// The five slowest, the slowest first. + pub slowest: Vec, + /// When Observer takes the start for stuck and restarts it, should FXServer start no resource + /// until then. Null with hang detection off, and once the cfg has run. + pub stuck_at: Option, + /// What `POST /servers/default/start/extend` adds to `stuckAt`, in minutes. + pub extension_minutes: u32, +} + +/// The memory FXServer measures a streamed file by. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub enum AssetMemory { + Physical, + Virtual, +} + +/// A streamed file FXServer said is heavy as it started the file's resource. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct HeavyAsset { + pub resource: String, + /// Within the resource, as FXServer names it. + pub file: String, + /// In tenths of a MiB, as FXServer prints it. + pub tenths: u32, + pub memory: AssetMemory, + /// FXServer says it can keep models from loading. + pub oversized: bool, +} + +/// A file a manifest names that its resource does not hold, as FXServer said when it started the +/// resource. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct MissingFile { + pub resource: String, + /// What the manifest asks for it as: `client_script`, `file`. + pub kind: String, + pub file: String, + /// The manifest and the line that name it. + pub manifest: String, + pub line: u32, +} + +/// `GET /api/servers/default/start/warnings`: what FXServer warned about as it started its +/// resources, the latest time Observer started it. Read from its console, kept in memory. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct StartWarnings { + /// Null when Observer has not started FXServer since it started itself. + pub started_at: Option, + pub heavy: Vec, + pub missing: Vec, + /// FXServer warned of more than Observer keeps. + pub truncated: bool, +} + /// The `server.status` topic. #[derive(Debug, Clone, PartialEq, Eq, Serialize, TS)] #[serde(rename_all = "camelCase")] @@ -174,6 +293,8 @@ pub struct ServerStatus { pub name: Option, /// The game port, as of the last run. pub port: Option, + /// Only while FXServer is starting, from its first second. + pub start: Option, } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, TS)] diff --git a/crates/observer-domain/src/tools.rs b/crates/observer-domain/src/tools.rs index 71c53db3..678f9bdc 100644 --- a/crates/observer-domain/src/tools.rs +++ b/crates/observer-domain/src/tools.rs @@ -827,6 +827,23 @@ pub struct ToolView { pub closed: Vec, /// What the backups of these changes take on the disk. pub backup_bytes: u64, + /// The fix that waits for the next start of FXServer. + pub staged: Option, +} + +/// A fix the owner prepared while FXServer ran. Before the next start that is not a recovery from +/// a crash, a fresh scan runs: the fix is written if it plans exactly what the owner saw, and +/// dropped either way. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export)] +pub struct StagedFix { + /// The `plan` of the report the owner saw. + pub plan: String, + /// The groups it fixes. + pub fixes: u32, + pub staged_at: Timestamp, + pub staged_by: String, } /// The owner's change of a tool's settings. diff --git a/crates/observer-fxserver/src/cfg.rs b/crates/observer-fxserver/src/cfg.rs index 2b0b9fe7..75440a63 100644 --- a/crates/observer-fxserver/src/cfg.rs +++ b/crates/observer-fxserver/src/cfg.rs @@ -857,7 +857,7 @@ impl Scanner<'_> { } /// The names of a build's system resources, in lower case. -fn system_resources(folder: &Path) -> HashSet { +pub(crate) fn system_resources(folder: &Path) -> HashSet { let Ok(entries) = fs::read_dir(folder) else { return HashSet::new(); }; diff --git a/crates/observer-fxserver/src/cfg_edit.rs b/crates/observer-fxserver/src/cfg_edit.rs new file mode 100644 index 00000000..63081690 --- /dev/null +++ b/crates/observer-fxserver/src/cfg_edit.rs @@ -0,0 +1,569 @@ +//! The launch's cfg as the panel edits it: its text with each secret replaced by a mark, what its +//! lines set and start, and the bytes a save writes. Nothing here opens a file but the scan. + +use std::ops::Range; +use std::path::Path; + +use observer_domain::cfg::{ + CfgEndpointLine, CfgOutline, CfgResourceLine, CfgReview, CfgSetting, MAX_TEXT_BYTES, SECRET_CLOSE, + SECRET_OPEN, +}; +use observer_domain::server_data::EndpointProtocol; + +use crate::cfg::{self, Context}; +use crate::changes::mask; +use crate::command; +use crate::server_data::Resources; + +/// Lines of each kind an outline lists. +const MAX_LISTED: usize = 5000; +const BOM: char = '\u{feff}'; +const SET_COMMANDS: [&str; 4] = ["set", "sets", "setr", "seta"]; +/// The convars the panel has a field for: a cfg may set them by their name alone, which FXServer +/// reads as a command of any other name. +const FIELDS: [&str; 9] = [ + "sv_hostname", + "sv_projectName", + "sv_projectDesc", + "tags", + "locale", + "sv_maxclients", + "sv_enforceGameBuild", + "sv_licenseKey", + "steam_webApiKey", +]; + +/// Why a file is not edited from the panel. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Unreadable { + /// Saving it would rewrite every byte that is not UTF-8. + NotUtf8, + /// It holds text written like a secret's mark, which a save could not tell from one. + HoldsMarks, +} + +/// Why an edited text is not written. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Refused { + /// A mark that stands for no secret of the file. + UnknownMark, + /// A NUL, which no cfg holds. + Control, + TooLarge, +} + +/// A cfg read for editing. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Editable { + /// What the panel shows and edits: every line ended by `\n`, each secret as its mark. + pub text: String, + /// What the marks stand for, in the order of the file. + secrets: Vec, + crlf: bool, + bom: bool, +} + +impl Editable { + /// Reads the bytes of `file`, a cfg of the exec chain named from the server folder. + pub fn read(bytes: &[u8], file: &str) -> Result { + let text = std::str::from_utf8(bytes).map_err(|_| Unreadable::NotUtf8)?; + if text.contains(SECRET_OPEN) { + return Err(Unreadable::HoldsMarks); + } + let bom = text.starts_with(BOM); + let text = text.strip_prefix(BOM).unwrap_or(text); + let crlf = text.matches("\r\n").count() * 2 > text.matches('\n').count(); + let text = unix_lines(text); + let mut shown = String::with_capacity(text.len()); + let mut secrets = Vec::new(); + let mut copied = 0; + for secret in line_bound(&text, merged(mask::secrets(&text, file, true))) { + shown.push_str(&text[copied..secret.start]); + secrets.push(text[secret.clone()].to_owned()); + shown.push_str(&mark(secrets.len())); + copied = secret.end; + } + shown.push_str(&text[copied..]); + Ok(Self { + text: shown, + secrets, + crlf, + bom, + }) + } + + pub fn secrets(&self) -> usize { + self.secrets.len() + } + + /// `edited` with each mark of the file back to its secret, lines ended by `\n`. A mark that + /// stands for nothing stays as written: this is for reading the text, not for writing it. + pub fn revealed(&self, edited: &str) -> String { + self.reveal(&unix_lines(edited)).0 + } + + /// The bytes to write for `edited`: each mark back to its secret, the file's line ends and + /// byte order mark as they were. + pub fn write(&self, edited: &str) -> Result, Refused> { + if edited.len() > MAX_TEXT_BYTES { + return Err(Refused::TooLarge); + } + if edited.contains('\0') { + return Err(Refused::Control); + } + let (text, unknown) = self.reveal(&unix_lines(edited)); + if unknown { + return Err(Refused::UnknownMark); + } + let text = if self.crlf { + text.replace('\n', "\r\n") + } else { + text + }; + let mut bytes = Vec::with_capacity(text.len() + BOM.len_utf8()); + if self.bom { + bytes.extend_from_slice(BOM.encode_utf8(&mut [0; 4]).as_bytes()); + } + bytes.extend_from_slice(text.as_bytes()); + if bytes.len() > MAX_TEXT_BYTES { + return Err(Refused::TooLarge); + } + Ok(bytes) + } + + /// `text` with its marks replaced, and whether one of them stands for no secret. + fn reveal(&self, text: &str) -> (String, bool) { + let mut revealed = String::with_capacity(text.len()); + let mut unknown = false; + let mut rest = text; + while let Some(at) = rest.find(SECRET_OPEN) { + let after = &rest[at + SECRET_OPEN.len()..]; + let secret = after.split_once(SECRET_CLOSE).and_then(|(number, _)| { + let number: usize = number.parse().ok()?; + Some(( + self.secrets.get(number.checked_sub(1)?)?, + number.to_string().len(), + )) + }); + revealed.push_str(&rest[..at]); + if let Some((secret, digits)) = secret { + revealed.push_str(secret); + rest = &after[digits + SECRET_CLOSE.len()..]; + } else { + unknown = true; + revealed.push_str(SECRET_OPEN); + rest = after; + } + } + revealed.push_str(rest); + (revealed, unknown) + } +} + +fn mark(number: usize) -> String { + format!("{SECRET_OPEN}{number}{SECRET_CLOSE}") +} + +/// `text` with every line ended by `\n`, as a browser's text field holds it. +fn unix_lines(text: &str) -> String { + text.replace("\r\n", "\n").replace('\r', "\n") +} + +/// The secrets in order, those that touch or overlap made one. +pub(crate) fn merged(mut secrets: Vec>) -> Vec> { + secrets.sort_by_key(|secret| secret.start); + let mut out: Vec> = Vec::with_capacity(secrets.len()); + for secret in secrets { + match out.last_mut() { + Some(last) if secret.start <= last.end => last.end = last.end.max(secret.end), + _ => out.push(secret), + } + } + out +} + +/// Each secret cut at the line ends it runs over, so that a mark never takes a line away. +fn line_bound(text: &str, secrets: Vec>) -> Vec> { + let mut out = Vec::with_capacity(secrets.len()); + for secret in secrets { + let mut start = secret.start; + for (offset, _) in text[secret.clone()].match_indices('\n') { + out.push(start..secret.start + offset); + start = secret.start + offset + 1; + } + out.push(start..secret.end); + } + out.retain(|secret| !secret.is_empty()); + out +} + +/// What the cfg `cfg_file` of `data` sets and starts as `shown`, the text the panel holds, and +/// what the scan finds in `real`, the same text with its secrets, and in the files it runs. +pub fn check( + data: &Path, + cfg_file: &str, + shown: &str, + real: &str, + resources: &Resources, + context: &Context, +) -> CfgReview { + let scan = cfg::scan(data, cfg_file, real, resources, context); + let system = context + .system_resources + .as_deref() + .map(cfg::system_resources) + .unwrap_or_default(); + let holds = |name: &str| resources.holds(name) || system.contains(&name.to_ascii_lowercase()); + CfgReview { + outline: outline(shown, holds), + findings: scan.findings, + added: 0, + removed: 0, + } +} + +/// The lines of `text` the panel edits by field, read the way FXServer reads them. `holds` says +/// whether the server has a resource or a category. +pub fn outline(text: &str, holds: impl Fn(&str) -> bool) -> CfgOutline { + let mut outline = CfgOutline::default(); + for (index, line) in text.split('\n').enumerate() { + let number = u32::try_from(index + 1).unwrap_or(u32::MAX); + let commands = command::spans(line); + if commands.is_empty() { + if let Some(name) = commented_resource(line).filter(|name| holds(name)) { + outline.resources.push(CfgResourceLine { + line: number, + name, + enabled: false, + alone: true, + missing: false, + }); + } + continue; + } + let alone = commands.len() == 1; + for span in commands { + let words = command::word_spans(&line[span.clone()]); + let Some((_, first)) = words.first() else { + continue; + }; + let lower = first.to_ascii_lowercase(); + let valued = |values: &[(Range, String)]| value(line, span.start, values); + match lower.as_str() { + "ensure" | "start" if words.len() == 2 => outline.resources.push(CfgResourceLine { + line: number, + missing: !holds(&words[1].1), + name: words[1].1.clone(), + enabled: true, + alone, + }), + "endpoint_add_tcp" | "endpoint_add_udp" if words.len() >= 2 => { + let (start, end, _) = valued(&words[1..2]); + outline.endpoints.push(CfgEndpointLine { + line: number, + protocol: if lower.ends_with("tcp") { + EndpointProtocol::Tcp + } else { + EndpointProtocol::Udp + }, + address: words[1].1.clone(), + start, + end, + alone, + }); + } + command if SET_COMMANDS.contains(&command) && words.len() >= 3 => { + let (start, end, value) = valued(&words[2..]); + outline.settings.push(CfgSetting { + line: number, + command: Some(lower.clone()), + name: words[1].1.clone(), + value, + start, + end, + alone, + }); + } + _ if words.len() >= 2 && FIELDS.iter().any(|field| field.eq_ignore_ascii_case(first)) => { + let (start, end, value) = valued(&words[1..]); + outline.settings.push(CfgSetting { + line: number, + command: None, + name: first.clone(), + value, + start, + end, + alone, + }); + } + _ => {} + } + } + } + for listed in [ + outline.settings.len(), + outline.endpoints.len(), + outline.resources.len(), + ] { + outline.truncated |= listed > MAX_LISTED; + } + outline.settings.truncate(MAX_LISTED); + outline.endpoints.truncate(MAX_LISTED); + outline.resources.truncate(MAX_LISTED); + outline +} + +/// Where the words `values` of the command at `offset` of `line` lie, in UTF-16 code units, and +/// what they give: the one word as FXServer reads it, or several as they are written. Nothing for +/// a value that holds a secret's mark. +fn value(line: &str, offset: usize, values: &[(Range, String)]) -> (u32, u32, Option) { + let (Some((first, word)), Some((last, _))) = (values.first(), values.last()) else { + return (0, 0, None); + }; + let span = offset + first.start..offset + last.end; + let written = line.get(span.clone()).unwrap_or_default(); + let units = |byte: usize| { + let before = line.get(..byte).unwrap_or(line); + u32::try_from(before.encode_utf16().count()).unwrap_or(u32::MAX) + }; + let value = if written.contains(SECRET_OPEN) { + None + } else if values.len() == 1 { + Some(word.clone()) + } else { + Some(written.to_owned()) + }; + (units(span.start), units(span.end), value) +} + +/// The resource a line commented out as a whole would start: `#ensure maps`, `// start maps`. +fn commented_resource(line: &str) -> Option { + let line = line.trim_start(); + let rest = line.strip_prefix('#').or_else(|| line.strip_prefix("//"))?; + let [command] = command::spans(rest).try_into().ok()?; + let words = command::words(&rest[command]); + match words.as_slice() { + [verb, name] if verb.eq_ignore_ascii_case("ensure") || verb.eq_ignore_ascii_case("start") => { + Some(name.clone()) + } + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const CFG: &str = "endpoint_add_tcp \"0.0.0.0:30120\"\r\nendpoint_add_udp \"0.0.0.0:30120\"\r\n\ + sv_hostname \"Caf\u{e9} \u{1f600} RP\"\r\nsets tags \"roleplay, fr\"\r\n\ + sv_licenseKey cfxk_1a2b3c4d5e6f7g8h9i0j_AbC123\r\n\ + set mysql_connection_string \"mysql://observer:hunter2@localhost/lab\"\r\n\ + ensure chat\r\n#ensure [xmas]\r\n"; + + fn editable(text: &str) -> Editable { + Editable::read(text.as_bytes(), "server.cfg").unwrap() + } + + #[test] + fn a_cfg_is_shown_with_a_mark_in_place_of_each_secret() { + let cfg = editable(CFG); + + assert_eq!(cfg.secrets(), 2); + assert!(!cfg.text.contains('\r')); + assert!( + !cfg.text.contains("hunter2") && !cfg.text.contains("1a2b3c"), + "{}", + cfg.text + ); + assert_eq!(cfg.text.lines().count(), CFG.lines().count()); + let marks: Vec<&str> = cfg + .text + .lines() + .filter(|line| line.contains(SECRET_OPEN)) + .collect(); + assert_eq!(marks.len(), 2, "{}", cfg.text); + assert!(marks[0].starts_with("sv_licenseKey ") && marks[0].contains("<>")); + assert!(marks[1].starts_with("set mysql_connection_string ") && marks[1].contains("<>")); + } + + #[test] + fn a_text_left_as_shown_is_written_as_the_file_was_byte_for_byte() { + for text in [ + CFG.to_owned(), + CFG.replace("\r\n", "\n"), + format!("\u{feff}{CFG}"), + "sv_hostname \"no line end\"".to_owned(), + String::new(), + ] { + let cfg = editable(&text); + assert_eq!(cfg.write(&cfg.text).unwrap(), text.as_bytes(), "{text:?}"); + } + } + + #[test] + fn a_save_keeps_the_secrets_whose_marks_stay_and_takes_what_is_typed_over_one() { + let cfg = editable(CFG); + let edited = cfg + .text + .replace("roleplay, fr", "roleplay, fr, serious") + .replace("<>", "\"mysql://observer:another@localhost/lab\""); + + let written = String::from_utf8(cfg.write(&edited).unwrap()).unwrap(); + + assert!(written.contains("sv_licenseKey cfxk_1a2b3c4d5e6f7g8h9i0j_AbC123\r\n")); + assert!(written.contains("\"mysql://observer:another@localhost/lab\"\r\n")); + assert!(written.contains("sets tags \"roleplay, fr, serious\"\r\n")); + assert!(!written.contains("hunter2") && !written.contains(SECRET_OPEN)); + assert_eq!(cfg.revealed(&edited), written.replace("\r\n", "\n")); + } + + #[test] + fn a_mark_that_stands_for_no_secret_is_refused_rather_than_written() { + let cfg = editable(CFG); + for mark in ["<>", "<>", "<>", "<>\"\n", "server.cfg"), + Err(Unreadable::HoldsMarks) + ); + } + + #[test] + fn the_line_ends_of_a_file_are_those_most_of_its_lines_have() { + let mostly_unix = editable("a 1\nb 2\nc 3\r\n"); + let mostly_windows = editable("a 1\r\nb 2\r\nc 3\n"); + let old_mac = editable("a 1\rb 2\r"); + + assert_eq!(mostly_unix.write("a 1\nb 2\n").unwrap(), b"a 1\nb 2\n"); + assert_eq!(mostly_windows.write("a 1\nb 2\n").unwrap(), b"a 1\r\nb 2\r\n"); + assert_eq!(old_mac.text, "a 1\nb 2\n"); + } + + #[test] + fn the_outline_places_each_value_in_its_line_as_a_browser_counts() { + let cfg = editable(CFG); + let outline = outline(&cfg.text, |name| name == "chat" || name == "[xmas]"); + let line = |number: u32| cfg.text.lines().nth(number as usize - 1).unwrap(); + let written = |number: u32, start: u32, end: u32| { + let units: Vec = line(number).encode_utf16().collect(); + String::from_utf16(&units[start as usize..end as usize]).unwrap() + }; + + let settings: Vec<(u32, Option<&str>, &str, Option<&str>)> = outline + .settings + .iter() + .map(|setting| { + ( + setting.line, + setting.command.as_deref(), + setting.name.as_str(), + setting.value.as_deref(), + ) + }) + .collect(); + assert_eq!( + settings, + [ + (3, None, "sv_hostname", Some("Caf\u{e9} \u{1f600} RP")), + (4, Some("sets"), "tags", Some("roleplay, fr")), + (5, None, "sv_licenseKey", None), + (6, Some("set"), "mysql_connection_string", None), + ] + ); + let hostname = &outline.settings[0]; + assert_eq!( + written(3, hostname.start, hostname.end), + "\"Caf\u{e9} \u{1f600} RP\"", + "an emoji counts for two" + ); + let endpoints: Vec<(u32, EndpointProtocol, &str)> = outline + .endpoints + .iter() + .map(|endpoint| (endpoint.line, endpoint.protocol, endpoint.address.as_str())) + .collect(); + assert_eq!( + endpoints, + [ + (1, EndpointProtocol::Tcp, "0.0.0.0:30120"), + (2, EndpointProtocol::Udp, "0.0.0.0:30120") + ] + ); + assert_eq!( + written(1, outline.endpoints[0].start, outline.endpoints[0].end), + "\"0.0.0.0:30120\"" + ); + assert!(!outline.truncated); + } + + #[test] + fn the_outline_lists_what_a_cfg_starts_and_what_it_would_were_a_line_not_commented_out() { + let text = "ensure chat\nstart [Maps] # the island\nensure gone; ensure chat\n\ + #ensure [xmas]\n // start chat\n# ensure you have set the key\n# start here\n\ + ensure\nensure a b\nrestart chat\n"; + + let outline = outline(text, |name| { + ["chat", "[maps]", "[xmas]"].contains(&name.to_ascii_lowercase().as_str()) + }); + + let resources: Vec<(u32, &str, bool, bool, bool)> = outline + .resources + .iter() + .map(|resource| { + ( + resource.line, + resource.name.as_str(), + resource.enabled, + resource.alone, + resource.missing, + ) + }) + .collect(); + assert_eq!( + resources, + [ + (1, "chat", true, true, false), + (2, "[Maps]", true, true, false), + (3, "gone", true, false, true), + (3, "chat", true, false, false), + (4, "[xmas]", false, true, false), + (5, "chat", false, true, false), + ], + "a comment that only reads like a command names nothing the folder holds" + ); + } + + #[test] + fn a_value_of_several_words_is_given_as_written_and_a_bare_name_only_for_the_fields() { + let outline = outline( + "sv_hostname My Server // public\nset sv_lan 1\nsv_lan 1\nSV_MAXCLIENTS 64\nsets tags\n", + |_| false, + ); + + let settings: Vec<(&str, Option<&str>)> = outline + .settings + .iter() + .map(|setting| (setting.name.as_str(), setting.value.as_deref())) + .collect(); + assert_eq!( + settings, + [ + ("sv_hostname", Some("My Server")), + ("sv_lan", Some("1")), + ("SV_MAXCLIENTS", Some("64")) + ] + ); + } +} diff --git a/crates/observer-fxserver/src/console/mod.rs b/crates/observer-fxserver/src/console/mod.rs index 63e250c9..e84c3c3f 100644 --- a/crates/observer-fxserver/src/console/mod.rs +++ b/crates/observer-fxserver/src/console/mod.rs @@ -28,6 +28,18 @@ const LIVE_CAPACITY: usize = 1024; /// Numbers available to each millisecond Observer runs, far more than it ever prints. const SEQS_PER_MS: u64 = 1000; +/// The channel FXServer printed a line on, whose name it right-aligns in brackets and cuts at +/// twenty characters, and the line's text. +pub(crate) fn channel(line: &str) -> Option<(&str, &str)> { + let (name, text) = line.strip_prefix('[')?.split_once("] ")?; + Some((name.trim_start(), text)) +} + +/// The text of a line FXServer printed on `channel`. +pub(crate) fn on_channel<'a>(line: &'a str, name: &str) -> Option<&'a str> { + channel(line).and_then(|(on, text)| (on == name).then_some(text)) +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize)] pub struct ConsoleLine { /// Numbers every line Observer saw since it started, across server restarts. Counted from the diff --git a/crates/observer-fxserver/src/health.rs b/crates/observer-fxserver/src/health.rs index c45fde8f..6c062cc1 100644 --- a/crates/observer-fxserver/src/health.rs +++ b/crates/observer-fxserver/src/health.rs @@ -1,7 +1,8 @@ //! How Observer tells a running FXServer from a stuck one: its HTTP endpoints, answered off the //! main thread, the main thread's tick count in `/perf/`, and the bridge's heartbeat, which only -//! comes while the main loop runs. The same probes feed the statistics: `/perf/` and -//! `/players.json` are read in the middle of each five-second slot, once for both. +//! comes while the main loop runs. Until the cfg has run, the resources FXServer starts stand in for +//! the heartbeat. The same probes feed the statistics: `/perf/` and `/players.json` are read in the +//! middle of each five-second slot, once for both. use std::net::SocketAddr; use std::time::{Duration, Instant}; @@ -127,6 +128,10 @@ pub enum Verdict { pub struct Health { spawned: Instant, heartbeat: Option, + /// When the bridge first reported in, which it does once the cfg has run. + cfg_ran: Option, + /// When FXServer last started a resource of its cfg. + progress: Option, http: Option, /// The last `svMain` tick count, and whether it moved since the read before. ticks: Option, @@ -138,6 +143,8 @@ pub struct Health { bark: bool, /// When hang detection was last turned on, or given another delay, during this run. armed: Option, + /// Until when a start is borne with however silent: someone said it needs longer. + extended: Option, } impl Health { @@ -145,6 +152,8 @@ impl Health { Self { spawned, heartbeat: None, + cfg_ran: None, + progress: None, http: None, ticks: None, stalled: false, @@ -152,6 +161,7 @@ impl Health { refusals: 0, bark: false, armed: None, + extended: None, } } @@ -165,10 +175,43 @@ impl Health { } pub fn heartbeat(&mut self, now: Instant) { + self.cfg_ran.get_or_insert(now); self.heartbeat = Some(now); self.bark = false; } + /// FXServer started a resource at `at`. It runs its cfg before any thread of the bridge, so no + /// heartbeat comes during a start however long it takes: a resource started is what shows the + /// start going on. + pub fn progressed(&mut self, at: Instant) { + self.progress = self.progress.max(Some(at)); + } + + /// Whether the bridge has reported in since the spawn: the cfg has run. + pub fn cfg_ran(&self) -> bool { + self.cfg_ran.is_some() + } + + /// The start is not taken for stuck before `until`, whatever its silence. + pub fn extend(&mut self, until: Instant) { + self.extended = self.extended.max(Some(until)); + } + + /// When the start is taken for stuck, should FXServer start no resource until then. None once + /// the cfg has run, which leaves the watch to the heartbeat, and with no limit to a silence. + pub fn stuck_at(&self, limits: &HealthLimits) -> Option { + if self.cfg_ran.is_some() { + return None; + } + let silent_since = self + .progress + .max(self.armed) + .map_or(self.spawned, |last| last.max(self.spawned)); + let stuck = silent_since.checked_add(limits.heartbeat_fatal)?; + let grace = self.spawned.checked_add(limits.boot_grace)?; + Some(stuck.max(grace).max(self.extended.unwrap_or(stuck))) + } + pub fn bark(&mut self) { self.bark = true; } @@ -210,10 +253,19 @@ impl Health { .armed .map_or(Duration::MAX, |armed| now.saturating_duration_since(armed)); if now.saturating_duration_since(self.spawned) >= limits.boot_grace { - if heartbeat.min(armed) >= limits.heartbeat_fatal { + // A start is stuck once it starts no resource for that long, not once it took that long. + let stuck = match self.cfg_ran { + Some(_) => heartbeat.min(armed) >= limits.heartbeat_fatal, + None => self.stuck_at(limits).is_some_and(|at| now >= at), + }; + if stuck { return Verdict::Hung(HangCause::NoHeartbeat); } - if http.min(armed) >= limits.http_fatal { + // The cfg opens the HTTP endpoint when it comes to that line: HTTP is owed once it ran. + let unanswered = self + .cfg_ran + .map(|ran| silent(Some(self.http.map_or(ran, |http| http.max(ran))))); + if unanswered.is_some_and(|unanswered| unanswered.min(armed) >= limits.http_fatal) { return Verdict::Hung(HangCause::NoHttp); } } @@ -699,6 +751,126 @@ mod tests { ); } + #[test] + fn a_start_is_hung_once_it_starts_no_resource_for_the_delay_however_long_it_takes() { + let limits = HealthLimits::default().under(&launch(true, 360)); + let start = Instant::now(); + let mut health = Health::new(start); + // Twenty minutes of resources, the slowest one five minutes on a cold cache. + for second in [15, 120, 420, 700, 1000, 1200] { + health.progressed(start + secs(second)); + assert_eq!( + health.verdict(start + secs(second + 299), &limits), + Verdict::Starting, + "{second}" + ); + } + + // Told late, an earlier one takes nothing back: the latest counts. + health.progressed(start + secs(900)); + assert_eq!(health.verdict(start + secs(1559), &limits), Verdict::Starting); + assert_eq!( + health.verdict(start + secs(1560), &limits), + Verdict::Hung(HangCause::NoHeartbeat) + ); + } + + #[test] + fn a_start_given_more_time_is_not_hung_before_that_runs_out() { + let limits = HealthLimits::default().under(&launch(true, 360)); + let start = Instant::now(); + let mut health = Health::new(start); + health.progressed(start + secs(100)); + assert_eq!(health.stuck_at(&limits), Some(start + secs(460))); + + health.extend(start + secs(1360)); + health.extend(start + secs(900)); + assert_eq!(health.stuck_at(&limits), Some(start + secs(1360))); + assert_eq!(health.verdict(start + secs(1359), &limits), Verdict::Starting); + assert_eq!( + health.verdict(start + secs(1360), &limits), + Verdict::Hung(HangCause::NoHeartbeat) + ); + + // A resource started late in the time given: its own silence counts from there. + health.progressed(start + secs(1300)); + assert_eq!(health.stuck_at(&limits), Some(start + secs(1660))); + assert_eq!(health.verdict(start + secs(1659), &limits), Verdict::Starting); + } + + #[test] + fn when_a_start_is_stuck_is_said_only_while_its_cfg_runs_under_hang_detection() { + let start = Instant::now(); + let mut health = Health::new(start); + let off = HealthLimits::default().under(&launch(false, 360)); + let short = HealthLimits::default().under(&launch(true, 60)); + let limits = HealthLimits::default().under(&launch(true, 360)); + + assert_eq!(health.stuck_at(&off), None); + assert_eq!(health.stuck_at(&limits), Some(start + secs(360))); + assert_eq!( + health.stuck_at(&HealthLimits { + boot_grace: secs(90), + ..short + }), + Some(start + secs(90)), + "never during the grace of a start" + ); + health.arm(start + secs(200)); + assert_eq!(health.stuck_at(&limits), Some(start + secs(560))); + + health.heartbeat(start + secs(210)); + assert_eq!(health.stuck_at(&limits), None, "the heartbeat watches from here"); + } + + #[test] + fn once_the_cfg_has_run_only_the_heartbeat_tells_the_main_loop_runs() { + let limits = HealthLimits::default().under(&launch(true, 360)); + let start = Instant::now(); + let mut health = Health::new(start); + health.probed(&http_ok(), start + secs(10)); + health.progressed(start + secs(300)); + health.heartbeat(start + secs(600)); + assert!(health.cfg_ran()); + for second in (600..1200).step_by(3) { + health.probed(&http_ok(), start + secs(second)); + } + // A resource restarted from the console, which the main loop runs before it sticks. + health.progressed(start + secs(900)); + + assert_eq!( + health.verdict(start + secs(959), &limits), + Verdict::Degraded(vec![Problem::HeartbeatLate]) + ); + assert_eq!( + health.verdict(start + secs(960), &limits), + Verdict::Hung(HangCause::NoHeartbeat) + ); + } + + #[test] + fn http_is_owed_from_the_end_of_the_cfg_which_opens_its_endpoint() { + let limits = HealthLimits::default().under(&launch(true, 360)); + let start = Instant::now(); + let mut health = Health::new(start); + // A long start whose cfg adds the endpoint last: HTTP never answered meanwhile. + for second in (60..=1200).step_by(60) { + health.progressed(start + secs(second)); + assert_eq!(health.verdict(start + secs(second), &limits), Verdict::Starting); + } + health.heartbeat(start + secs(1200)); + for second in (1200..1600).step_by(3) { + health.heartbeat(start + secs(second)); + } + + assert_eq!(health.verdict(start + secs(1201), &limits), Verdict::Starting); + assert_eq!(health.verdict(start + secs(1559), &limits), Verdict::Starting); + assert_eq!( + health.verdict(start + secs(1560), &limits), + Verdict::Hung(HangCause::NoHttp) + ); + } + #[test] fn a_silent_heartbeat_degrades_then_hangs_the_server() { let start = Instant::now(); diff --git a/crates/observer-fxserver/src/hook.rs b/crates/observer-fxserver/src/hook.rs new file mode 100644 index 00000000..b7caed4a --- /dev/null +++ b/crates/observer-fxserver/src/hook.rs @@ -0,0 +1,650 @@ +//! What the Debug feature takes from a build and reads from its server: the two lines it adds to +//! the build's Lua scheduler, which load the bridge's hook into every resource, and the reports +//! the hook prints each second through the structured trace channel. Counts, names and sizes: +//! the hook never sends what an event, a state bag or a request carries. + +use std::fs; +use std::io; +use std::path::{Path, PathBuf}; + +use serde_json::Value; + +use crate::layout::Layout; +use crate::trace::TraceKind; + +/// The hook's file in the bridge's folder. The bridge's manifest does not list it: it is no +/// script of the bridge, the scheduler's first line loads it. +pub const FILE: &str = "hook.lua"; +const SOURCE: &str = include_str!("../../../resource/observer/hook.lua"); +/// How the hook shows in a Lua traceback. +const CHUNK: &str = "=observer-hook"; +const HEAD_OPEN: &str = "--[[obs:hook:head v1]]"; +const HEAD_PREFIX: &str = "--[[obs:hook:head "; +const HEAD_CLOSE: &str = "--[[obs:hook:headend]]"; +const TAIL_PREFIX: &str = "--[[obs:hook:tail "; +const TAIL_OPEN: &str = "--[[obs:hook:tail v1]]"; +const TAIL_CLOSE: &str = "--[[obs:hook:tailend]]"; +/// The call the hook wraps to count what a resource receives: a scheduler without it is not one +/// the hook was written for. +const KNOWN_BY: &str = "SetEventRoutine"; +const BOM: &[u8] = b"\xEF\xBB\xBF"; +/// The scheduler as the build shipped it, kept beside it while the two lines are in. +const ORIGINAL_SUFFIX: &str = ".original"; + +const WIRE_TYPE: &str = "observer_debug"; +const WIRE_VERSION: u64 = 1; +/// What one report may hold of each list, and how long a name may be: a resource writes its own +/// report, and nothing says it writes it honestly. +const MAX_ROWS: usize = 512; +const MAX_LARGEST: usize = 10; +const MAX_NAME: usize = 128; + +/// Why the two lines did not go in or come out. +#[derive(Debug, thiserror::Error)] +pub enum Refusal { + #[error("{} could not be read or written: {source}", path.display())] + File { + path: PathBuf, + #[source] + source: io::Error, + }, + #[error("{} is not a scheduler Observer knows", .0.display())] + Unknown(PathBuf), + #[error("{} holds lines of Observer that were changed by hand", .0.display())] + Damaged(PathBuf), +} + +/// Where the build keeps the scheduler every server Lua runtime runs first. +pub fn scheduler(layout: &Layout) -> PathBuf { + layout + .citizen_root() + .join("citizen") + .join("scripting") + .join("lua") + .join("scheduler.lua") +} + +fn head(bridge: &str) -> String { + format!( + "{HEAD_OPEN} local __obsHook; do local ok, api = pcall(function() local src = \ + LoadResourceFile('{bridge}', '{FILE}'); if not src then return nil end; local fn = \ + load(src, '{CHUNK}'); return fn and fn('{bridge}') end); if ok and type(api) == 'table' \ + then __obsHook = api end end {HEAD_CLOSE}" + ) +} + +fn tail() -> String { + format!( + "{TAIL_OPEN} if __obsHook and __obsHook.init then pcall(__obsHook.init, {{ eventHandlers = \ + eventHandlers }}) end {TAIL_CLOSE}" + ) +} + +fn find(haystack: &[u8], needle: &[u8]) -> Option { + memchr::memmem::find(haystack, needle) +} + +/// The scheduler without Observer's two lines: byte for byte what [`with_lines`] was given. +/// `None` when a line of Observer's is there in part, which only a hand could have left. +fn without_lines(content: &[u8]) -> Option> { + let (bom, mut body) = match content.strip_prefix(BOM) { + Some(body) => (BOM, body), + None => (&[][..], content), + }; + if body.starts_with(HEAD_PREFIX.as_bytes()) { + let line = memchr::memchr(b'\n', body).unwrap_or(body.len()); + let close = find(&body[..line], HEAD_CLOSE.as_bytes())?; + body = &body[close + HEAD_CLOSE.len()..]; + body = body.strip_prefix(b" ").unwrap_or(body); + } + // The last line, whether or not the file ends with a line break. + let (text, ending) = match body { + [text @ .., b'\r', b'\n'] => (text, &b"\r\n"[..]), + [text @ .., b'\n'] => (text, &b"\n"[..]), + text => (text, &[][..]), + }; + let start = memchr::memrchr(b'\n', text).map_or(0, |at| at + 1); + let last = &text[start..]; + let kept = if last.starts_with(TAIL_PREFIX.as_bytes()) { + if !last.ends_with(TAIL_CLOSE.as_bytes()) { + return None; + } + if ending.is_empty() { + // The file ended without a line break: the one before the line came with it. + let before = &text[..start]; + before + .strip_suffix(b"\r\n") + .or_else(|| before.strip_suffix(b"\n")) + .unwrap_or(before) + } else { + &text[..start] + } + } else { + body + }; + Some([bom, kept].concat()) +} + +/// The scheduler with Observer's two lines: the first before its first line, on that line, so +/// that no line of the scheduler changes its number, and the second as a last line of its own. +fn with_lines(original: &[u8], bridge: &str) -> Vec { + let (bom, body) = match original.strip_prefix(BOM) { + Some(body) => (BOM, body), + None => (&[][..], original), + }; + let eol: &[u8] = if find(body, b"\r\n").is_some() { + b"\r\n" + } else { + b"\n" + }; + let (head, tail) = (head(bridge), tail()); + let end: [&[u8]; 2] = if body.ends_with(b"\n") { + [tail.as_bytes(), eol] + } else { + [eol, tail.as_bytes()] + }; + [bom, head.as_bytes(), b" ", body, end[0], end[1]].concat() +} + +fn replace(path: &Path, bytes: &[u8]) -> io::Result<()> { + let mut temporary = path.as_os_str().to_owned(); + temporary.push(".observer-tmp"); + let temporary = PathBuf::from(temporary); + fs::write(&temporary, bytes)?; + fs::rename(&temporary, path).inspect_err(|_| { + let _ = fs::remove_file(&temporary); + }) +} + +fn original_beside(scheduler: &Path) -> PathBuf { + let mut path = scheduler.as_os_str().to_owned(); + path.push(ORIGINAL_SUFFIX); + PathBuf::from(path) +} + +/// Puts the two lines into the build's scheduler and the hook into the bridge's folder, for a +/// start that counts; takes the lines out for one that does not. The bridge is installed first. +/// A refusal leaves the scheduler as it was. +pub fn install(layout: &Layout, bridge: &str, counting: bool) -> Result<(), Refusal> { + let path = scheduler(layout); + let at = |path: &Path| { + let path = path.to_path_buf(); + move |source| Refusal::File { path, source } + }; + let current = match fs::read(&path) { + Ok(current) => current, + // Nothing to take out of a build that has no scheduler. + Err(error) if !counting && error.kind() == io::ErrorKind::NotFound => return Ok(()), + Err(error) => return Err(at(&path)(error)), + }; + let original = without_lines(¤t).ok_or_else(|| Refusal::Damaged(path.clone()))?; + let beside = original_beside(&path); + if !counting { + if original != current { + replace(&path, &original).map_err(at(&path))?; + } + let _ = fs::remove_file(beside); + return Ok(()); + } + // The name is drawn by Observer from letters and digits; anything else would be Lua source. + let named = bridge + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || byte == b'_'); + if !named || find(&original, KNOWN_BY.as_bytes()).is_none() { + return Err(Refusal::Unknown(path)); + } + let hook = layout.system_resources().join(bridge).join(FILE); + fs::write(&hook, SOURCE).map_err(at(&hook))?; + fs::write(&beside, &original).map_err(at(&beside))?; + let counted = with_lines(&original, bridge); + if counted != current { + replace(&path, &counted).map_err(at(&path))?; + } + Ok(()) +} + +/// Which way an event went, as the resource that reports it saw it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum Way { + /// Received from a resource of the server, itself included. + Received, + /// Received from a client. + FromClient, + /// Received from the engine, such as `playerJoining`. + FromEngine, + /// Triggered for the server's resources. + Triggered, + /// Sent to one client. + ToClient, + /// Sent to every client. + ToAll, +} + +impl Way { + fn of(code: &str) -> Option { + Some(match code { + "in" => Self::Received, + "inN" => Self::FromClient, + "inI" => Self::FromEngine, + "loc" => Self::Triggered, + "toC" | "lat" => Self::ToClient, + "toA" | "latA" => Self::ToAll, + _ => return None, + }) + } +} + +/// The bag a state bag write went to, from the bag's name. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum BagKind { + Global, + Player, + Entity, + LocalEntity, + Other, +} + +impl BagKind { + fn of(code: &str) -> Self { + match code { + "global" => Self::Global, + "player" => Self::Player, + "entity" => Self::Entity, + "localEntity" => Self::LocalEntity, + _ => Self::Other, + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct EventCount { + pub way: Way, + pub name: String, + pub count: u64, + pub bytes: u64, + pub largest: u64, + /// In the resource's handlers, for an event received: microseconds. + pub handler_us: u64, +} + +/// An event received from one client. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SenderCount { + pub name: String, + pub player: u32, + pub count: u64, + pub bytes: u64, +} + +/// An event a client sent that the resource never opened to clients. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RefusedCount { + pub name: String, + /// `None` when the scheduler did not say who. + pub player: Option, + pub count: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct BagCount { + pub kind: BagKind, + pub key: String, + pub count: u64, + pub bytes: u64, + pub largest: u64, + /// How many of the writes went out to clients. + pub replicated: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LargestWrite { + pub bag: String, + pub key: String, + pub bytes: u64, + pub replicated: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct HostCount { + pub method: String, + pub host: String, + pub count: u64, + /// Of the request bodies. + pub bytes: u64, +} + +/// What one resource counted since its report before. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Report { + /// FXServer's word for who printed the report, not the report's own. + pub resource: String, + pub events: Vec, + pub senders: Vec, + pub refused: Vec, + pub bags: Vec, + pub largest: Vec, + pub hosts: Vec, + /// What the hook could not keep apart, past its limits. + pub overflow: u64, +} + +fn name(value: &Value) -> Option { + let text = value.as_str()?; + (!text.is_empty()).then(|| text.chars().take(MAX_NAME).collect()) +} + +fn rows<'a>(payload: &'a Value, key: &str, limit: usize) -> impl Iterator> { + payload + .get(key) + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter_map(Value::as_array) + .take(limit) +} + +fn number(row: &[Value], at: usize) -> u64 { + row.get(at).and_then(Value::as_u64).unwrap_or(0) +} + +impl Report { + /// The hook's report in a trace, if the trace is one. Every resource sends its own, and + /// FXServer names the resource: none can count for another. + pub fn from_trace(trace: &TraceKind) -> Option { + let TraceKind::Script { resource, payload } = trace else { + return None; + }; + if payload.get("type").and_then(Value::as_str) != Some(WIRE_TYPE) + || payload.get("v").and_then(Value::as_u64) != Some(WIRE_VERSION) + { + return None; + } + let events = rows(payload, "ev", MAX_ROWS) + .filter_map(|row| { + Some(EventCount { + way: Way::of(row.first()?.as_str()?)?, + name: name(row.get(1)?)?, + count: number(row, 2), + bytes: number(row, 3), + largest: number(row, 4), + handler_us: number(row, 5), + }) + }) + .collect(); + let player = |value: Option<&Value>| value?.as_u64().and_then(|id| u32::try_from(id).ok()); + let senders = rows(payload, "np", MAX_ROWS) + .filter_map(|row| { + Some(SenderCount { + name: name(row.first()?)?, + player: player(row.get(1))?, + count: number(row, 2), + bytes: number(row, 3), + }) + }) + .collect(); + let refused = rows(payload, "un", MAX_ROWS) + .filter_map(|row| { + Some(RefusedCount { + name: name(row.first()?)?, + player: player(row.get(1)), + count: number(row, 2), + }) + }) + .collect(); + let bags = rows(payload, "sb", MAX_ROWS) + .filter_map(|row| { + Some(BagCount { + kind: BagKind::of(row.first()?.as_str()?), + key: name(row.get(1)?)?, + count: number(row, 2), + bytes: number(row, 3), + largest: number(row, 4), + replicated: number(row, 5), + }) + }) + .collect(); + let largest = rows(payload, "sbl", MAX_LARGEST) + .filter_map(|row| { + Some(LargestWrite { + bag: name(row.first()?)?, + key: name(row.get(1)?)?, + bytes: number(row, 2), + replicated: number(row, 3) != 0, + }) + }) + .collect(); + let hosts = rows(payload, "http", MAX_ROWS) + .filter_map(|row| { + Some(HostCount { + method: name(row.first()?)?, + host: name(row.get(1)?)?.to_ascii_lowercase(), + count: number(row, 2), + bytes: number(row, 3), + }) + }) + .collect(); + Some(Self { + resource: resource.clone(), + events, + senders, + refused, + bags, + largest, + hosts, + overflow: payload.get("ovf").and_then(Value::as_u64).unwrap_or(0), + }) + } +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::*; + use crate::layout::Flavour; + + const BRIDGE: &str = "sys_abcdefgh2345"; + + fn scheduler_of(body: &str) -> Vec { + format!("local coroutine_yield = coroutine.yield\n{body}Citizen.SetEventRoutine(function() end)") + .into_bytes() + } + + #[test] + fn the_two_lines_go_in_without_moving_a_line_and_come_out_byte_for_byte() { + for original in [ + scheduler_of("local a = 1\n"), + [scheduler_of("local a = 1\n"), b"\n".to_vec()].concat(), + b"local y = coroutine.yield\r\nCitizen.SetEventRoutine(f)\r\nreturn y\r\n".to_vec(), + b"local y = coroutine.yield\r\nCitizen.SetEventRoutine(f)".to_vec(), + [BOM, &scheduler_of("-- with a mark\n"), b"\r\n"].concat(), + b"Citizen.SetEventRoutine(routine)".to_vec(), + ] { + let counted = with_lines(&original, BRIDGE); + + let text = String::from_utf8_lossy(&counted); + let lines: Vec<&str> = text.lines().collect(); + let unpatched = String::from_utf8_lossy(&original); + assert_eq!( + lines.len(), + unpatched.lines().count() + 1, + "one line more, the last" + ); + assert!(lines[0].trim_start_matches('\u{feff}').starts_with(HEAD_OPEN)); + let first = unpatched.lines().next().unwrap().trim_start_matches('\u{feff}'); + assert!(lines[0].ends_with(first)); + assert!(lines.last().unwrap().starts_with(TAIL_OPEN)); + assert!(text.contains(&format!("LoadResourceFile('{BRIDGE}', 'hook.lua')"))); + assert_eq!(counted.starts_with(BOM), original.starts_with(BOM)); + assert_eq!(counted.ends_with(b"\n"), original.ends_with(b"\n")); + assert_eq!(without_lines(&counted).unwrap(), original); + assert_eq!(without_lines(&original).unwrap(), original, "nothing to take out"); + // Another start draws another name: the lines are replaced, never stacked. + let again = with_lines(&without_lines(&counted).unwrap(), "sys_zyxwvuts7654"); + assert_eq!(without_lines(&again).unwrap(), original); + } + } + + #[test] + fn a_line_of_observers_left_in_part_is_not_taken_for_the_schedulers() { + let counted = with_lines(&scheduler_of("local a = 1\n"), BRIDGE); + let text = String::from_utf8(counted).unwrap(); + + let no_head_end = text.replacen(HEAD_CLOSE, "", 1); + let no_tail_end = text.replacen(TAIL_CLOSE, "", 1); + + assert_eq!(without_lines(no_head_end.as_bytes()), None); + assert_eq!(without_lines(no_tail_end.as_bytes()), None); + } + + fn build() -> (tempfile::TempDir, Layout) { + let dir = tempfile::tempdir().unwrap(); + let layout = Layout::new(dir.path(), Flavour::Windows); + fs::create_dir_all(scheduler(&layout).parent().unwrap()).unwrap(); + fs::create_dir_all(layout.system_resources().join(BRIDGE)).unwrap(); + (dir, layout) + } + + #[test] + fn a_start_that_counts_patches_the_build_and_the_next_one_that_does_not_restores_it() { + let (_dir, layout) = build(); + let path = scheduler(&layout); + let original = scheduler_of("local a = 1\n"); + fs::write(&path, &original).unwrap(); + let hook = layout.system_resources().join(BRIDGE).join(FILE); + let beside = original_beside(&path); + + install(&layout, BRIDGE, true).unwrap(); + + assert_eq!(fs::read(&path).unwrap(), with_lines(&original, BRIDGE)); + assert_eq!(fs::read(&beside).unwrap(), original); + assert_eq!(fs::read_to_string(&hook).unwrap(), SOURCE); + install(&layout, BRIDGE, true).unwrap(); + assert_eq!(fs::read(&path).unwrap(), with_lines(&original, BRIDGE)); + + install(&layout, BRIDGE, false).unwrap(); + + assert_eq!(fs::read(&path).unwrap(), original); + assert!(!beside.exists()); + install(&layout, BRIDGE, false).unwrap(); + assert_eq!(fs::read(&path).unwrap(), original); + } + + #[test] + fn a_scheduler_observer_does_not_know_is_left_as_it_is() { + let (_dir, layout) = build(); + let path = scheduler(&layout); + + let missing = install(&layout, BRIDGE, true).unwrap_err(); + install(&layout, BRIDGE, false).unwrap(); + fs::write(&path, "print('another runtime')\n").unwrap(); + let unknown = install(&layout, BRIDGE, true).unwrap_err(); + let odd_name = { + fs::write(&path, scheduler_of("")).unwrap(); + install(&layout, "sys'); os.exit() --", true).unwrap_err() + }; + let damaged = { + let counted = with_lines(&scheduler_of(""), BRIDGE); + let text = String::from_utf8(counted).unwrap().replacen(HEAD_CLOSE, "", 1); + fs::write(&path, &text).unwrap(); + (install(&layout, BRIDGE, false).unwrap_err(), text) + }; + + assert!(matches!(missing, Refusal::File { .. }), "{missing}"); + assert!(matches!(unknown, Refusal::Unknown(_)), "{unknown}"); + assert!(matches!(odd_name, Refusal::Unknown(_)), "{odd_name}"); + assert!(matches!(damaged.0, Refusal::Damaged(_)), "{}", damaged.0); + assert_eq!(fs::read_to_string(&path).unwrap(), damaged.1, "left as it was"); + } + + fn trace(resource: &str, payload: Value) -> TraceKind { + TraceKind::Script { + resource: resource.to_owned(), + payload, + } + } + + #[test] + fn a_report_is_read_row_by_row_under_the_name_fxserver_gives_its_resource() { + let payload = json!({ + "type": "observer_debug", "v": 1, "seq": 4, "dur": 1000, + "ev": [ + ["toC", "voice:talking", 186, 7626, 44, 0, 0], + ["latA", "map:sync", 1, 4096, 4096, 0, 0], + ["inN", "voice:talking", 24, 912, 40, 480, 60], + ["sideways", "ignored", 1, 1, 1, 0, 0], + ["in", "", 1, 1, 1, 0, 0], + "not a row", + ], + "np": [["voice:talking", 41, 19, 722], ["voice:talking", -1, 1, 1]], + "un": [["bank:give", 41, 3], ["bank:give", -1, 1]], + "sb": [["player", "proximity", 28, 252, 9, 28], ["somewhere", "odd", 1, 1, 1, 0]], + "sbl": [["entity:1042", "fuel", 2100, 1]], + "http": [["POST", "Discord.com", 36, 14400]], + "ovf": 2, + "resource": "someone-else", + }); + + let report = Report::from_trace(&trace("pma-voice", payload)).unwrap(); + + assert_eq!(report.resource, "pma-voice"); + assert_eq!( + report.events, + [ + EventCount { + way: Way::ToClient, + name: "voice:talking".to_owned(), + count: 186, + bytes: 7626, + largest: 44, + handler_us: 0, + }, + EventCount { + way: Way::ToAll, + name: "map:sync".to_owned(), + count: 1, + bytes: 4096, + largest: 4096, + handler_us: 0, + }, + EventCount { + way: Way::FromClient, + name: "voice:talking".to_owned(), + count: 24, + bytes: 912, + largest: 40, + handler_us: 480, + }, + ] + ); + assert_eq!(report.senders.len(), 1, "a sender has an id"); + assert_eq!((report.senders[0].player, report.senders[0].count), (41, 19)); + assert_eq!(report.refused[0].player, Some(41)); + assert_eq!(report.refused[1].player, None); + assert_eq!(report.bags[0].kind, BagKind::Player); + assert_eq!(report.bags[1].kind, BagKind::Other); + assert!(report.largest[0].replicated); + assert_eq!(report.hosts[0].host, "discord.com"); + assert_eq!(report.overflow, 2); + } + + #[test] + fn only_a_report_of_this_version_is_read_and_its_lists_are_cut_at_their_limits() { + let many: Vec = (0..MAX_ROWS + 50) + .map(|index| json!(["in", format!("event-{index}"), 1, 1, 1, 0, 0])) + .collect(); + let long = "n".repeat(MAX_NAME * 2); + let payload = + json!({ "type": "observer_debug", "v": 1, "ev": many, "sb": [["global", long, 1, 1, 1, 0]] }); + + let report = Report::from_trace(&trace("big", payload)).unwrap(); + + assert_eq!(report.events.len(), MAX_ROWS); + assert_eq!(report.bags[0].key.len(), MAX_NAME); + let other = json!({ "type": "observer", "kind": "hb" }); + assert_eq!(Report::from_trace(&trace("big", other)), None); + let later = json!({ "type": "observer_debug", "v": 2 }); + assert_eq!(Report::from_trace(&trace("big", later)), None); + assert_eq!(Report::from_trace(&TraceKind::Probe), None); + } +} diff --git a/crates/observer-fxserver/src/launch.rs b/crates/observer-fxserver/src/launch.rs index 49fd25ce..72a56515 100644 --- a/crates/observer-fxserver/src/launch.rs +++ b/crates/observer-fxserver/src/launch.rs @@ -14,7 +14,7 @@ use observer_domain::server_data::{CfgFinding, Severity}; use observer_platform::Launch; use crate::layout::{Flavour, Layout}; -use crate::{bridge, cfg, server_data}; +use crate::{bridge, cfg, server_data, start}; /// What a spawn starts: a build folder, and how FXServer runs from it. #[derive(Debug, Clone, PartialEq, Eq)] @@ -38,6 +38,8 @@ pub struct Prepared { /// The whole scan of the cfg, which the warnings and the endpoint come from: the pre-launch /// hook reads the commands that start resources in it, rather than scan the cfg again. pub scan: cfg::Scan, + /// The commands of the cfg that start resources, which the start is followed along. + pub plan: start::Plan, } #[derive(Debug, thiserror::Error)] @@ -188,6 +190,7 @@ pub fn prepare(launch: &ServerLaunch, flavour: Flavour, bridge: &str) -> Result< .cloned() .collect(); let endpoint = scan.endpoint; + let plan = start::Plan::new(&scan, &resources); Ok(Prepared { launch: Launch { @@ -200,6 +203,7 @@ pub fn prepare(launch: &ServerLaunch, flavour: Flavour, bridge: &str) -> Result< warnings, endpoint, scan, + plan, }) } diff --git a/crates/observer-fxserver/src/lib.rs b/crates/observer-fxserver/src/lib.rs index f7dd7851..599da905 100644 --- a/crates/observer-fxserver/src/lib.rs +++ b/crates/observer-fxserver/src/lib.rs @@ -2,6 +2,7 @@ pub mod bridge; pub mod cfg; +pub mod cfg_edit; pub mod changes; pub mod command; pub mod console; @@ -9,6 +10,7 @@ pub mod crashes; pub mod exit; pub mod folders; pub mod health; +pub mod hook; pub mod launch; pub mod layout; pub mod perf; @@ -17,6 +19,7 @@ pub mod pre_launch; pub mod recovery; pub mod server_data; pub mod server_files; +pub mod start; pub mod stdin; pub mod supervisor; pub mod trace; diff --git a/crates/observer-fxserver/src/server_data.rs b/crates/observer-fxserver/src/server_data.rs index 91f8bd28..285c8a37 100644 --- a/crates/observer-fxserver/src/server_data.rs +++ b/crates/observer-fxserver/src/server_data.rs @@ -2,7 +2,7 @@ //! setup's checks of a folder and of its cfg, which read and never write. Observer writes into the //! folder only SFTP uploads and the fixes of the server tools the owner runs or turns on. -use std::collections::{HashSet, VecDeque}; +use std::collections::{HashMap, HashSet, VecDeque}; use std::ffi::OsStr; use std::fs; use std::io; @@ -42,6 +42,9 @@ pub struct Resources { categories: HashSet, /// Resources with their manifest. manifested: u32, + /// The categories each resource with a manifest lies under, outermost first: FXServer starts + /// those and no other folder. Of two folders of one name, the first found. + under: HashMap>, pub summary: ResourceSummary, } @@ -52,6 +55,28 @@ impl Resources { self.names.contains(&name) || self.categories.contains(&name) } + /// The resources `ensure ` starts, in lower case and in name order: the one of that + /// name, or every one under the category `[target]`. None when the folder holds no such + /// resource, as for one of the build's own. + pub fn started_by(&self, target: &str) -> Vec { + let target = target.to_ascii_lowercase(); + let mut started: Vec = if target.starts_with('[') && target.ends_with(']') { + self.under + .iter() + .filter(|(_, categories)| categories.contains(&target)) + .map(|(name, _)| name.clone()) + .collect() + } else { + self.under + .get_key_value(&target) + .map(|(name, _)| name.clone()) + .into_iter() + .collect() + }; + started.sort_unstable(); + started + } + /// How many resources hold `fxmanifest.lua` or `__resource.lua`. pub fn manifested(&self) -> u32 { self.manifested @@ -100,6 +125,10 @@ pub fn walk(data: &Path) -> Resources { let manifest = MANIFESTS.iter().any(|manifest| path.join(manifest).is_file()); if manifest { resources.manifested += 1; + resources + .under + .entry(lower.clone()) + .or_insert_with(|| relative.split('/').skip(1).map(str::to_ascii_lowercase).collect()); } if !manifest && resources.summary.without_manifest.len() < MAX_UNMANIFESTED { resources.summary.without_manifest.push(name.clone()); @@ -470,6 +499,30 @@ mod tests { assert!(!resources.holds("README.md")); } + #[test] + fn a_command_starts_the_resource_of_its_name_or_every_one_under_its_category() { + let dir = data(&[ + "chat", + "[Maps]/docks", + "[Maps]/[interiors]/Bank", + "[Maps]/drafts!", + "[jobs]/police", + ]); + + let resources = walk(dir.path()); + + assert_eq!(resources.started_by("[maps]"), ["bank", "docks"]); + assert_eq!(resources.started_by("[Interiors]"), ["bank"]); + assert_eq!(resources.started_by("Chat"), ["chat"]); + assert_eq!( + resources.started_by("drafts"), + Vec::::new(), + "no manifest" + ); + assert_eq!(resources.started_by("sessionmanager"), Vec::::new()); + assert_eq!(resources.started_by("[gone]"), Vec::::new()); + } + #[test] fn caches_and_tool_folders_are_passed_over() { let dir = data(&[ diff --git a/crates/observer-fxserver/src/server_files.rs b/crates/observer-fxserver/src/server_files.rs index 7bc69a40..b62afeb9 100644 --- a/crates/observer-fxserver/src/server_files.rs +++ b/crates/observer-fxserver/src/server_files.rs @@ -14,6 +14,7 @@ use observer_domain::server_data::{ }; use crate::cfg::{self, ReadError}; +use crate::cfg_edit::merged; use crate::changes::mask; const MAX_ENTRIES: usize = 500; @@ -225,19 +226,6 @@ fn is_cfg(name: &str) -> bool { .is_some_and(|end| end.eq_ignore_ascii_case(".cfg")) } -/// The secrets in order, those that touch or overlap made one. -fn merged(mut secrets: Vec>) -> Vec> { - secrets.sort_by_key(|secret| secret.start); - let mut out: Vec> = Vec::with_capacity(secrets.len()); - for secret in secrets { - match out.last_mut() { - Some(last) if secret.start <= last.end => last.end = last.end.max(secret.end), - _ => out.push(secret), - } - } - out -} - /// The line `line` of `text`, cut at each secret: text between them, and an empty span where each /// one was. fn spans(text: &str, line: Range, secrets: &[Range]) -> Vec { diff --git a/crates/observer-fxserver/src/start.rs b/crates/observer-fxserver/src/start.rs new file mode 100644 index 00000000..e72b587a --- /dev/null +++ b/crates/observer-fxserver/src/start.rs @@ -0,0 +1,952 @@ +//! How far a start has come, read from what FXServer prints while it runs its cfg: each resource +//! once it is up, and nothing of the one it is starting. The command of the cfg it is at is worked +//! out from the resources each command starts. + +use std::collections::HashSet; +use std::sync::{Mutex, PoisonError}; +use std::time::{Duration, Instant}; + +use serde::Serialize; + +use crate::cfg::{ResourceVerb, Scan}; +use crate::console::channel; +use crate::server_data::Resources; + +/// The slowest resources kept of one start. +const SLOWEST: usize = 5; +/// Warnings of each kind kept of one run. +const MAX_WARNINGS: usize = 2000; +/// Resources of later commands started in a row, none of the command the start is taken to be at, +/// before it is taken to be past that command. FXServer starts a resource's dependencies ahead of +/// their own commands, a few in a row at most; a command whose resource never says how it ended +/// would hold the start there for ever. +const STRAYS: u32 = 3; + +/// A command of the cfg that starts resources, and the resources it starts, in lower case. +#[derive(Debug, Clone, PartialEq, Eq)] +struct Step { + file: String, + line: u32, + target: String, + members: Vec, +} + +impl Step { + fn starts(&self, resource: &str) -> bool { + self.members.iter().any(|member| member == resource) + } +} + +/// The commands of a cfg that start resources, in the order FXServer runs them. Empty when the +/// scan could not read them all: a start cannot be placed among commands that are missing. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Plan { + steps: Vec, + resources: Option, +} + +impl Plan { + pub fn new(scan: &Scan, resources: &Resources) -> Self { + if !scan.resource_commands_complete { + return Self::default(); + } + let steps: Vec = scan + .resource_commands + .iter() + .filter(|command| matches!(command.verb, ResourceVerb::Start | ResourceVerb::Ensure)) + .map(|command| { + let mut members = resources.started_by(&command.name); + // A resource the folder does not hold, as one of the build's own. + if members.is_empty() && !command.name.starts_with('[') { + members.push(command.name.to_ascii_lowercase()); + } + Step { + file: command.file.clone(), + line: command.line, + target: command.name.clone(), + members, + } + }) + .collect(); + let distinct: HashSet<&str> = steps + .iter() + .flat_map(|step| step.members.iter().map(String::as_str)) + .collect(); + let resources = u32::try_from(distinct.len()).ok(); + Self { steps, resources } + } +} + +/// What a line of FXServer's console says of its start. +#[derive(Debug, PartialEq, Eq)] +enum Said<'a> { + /// How many resources the folder holds, said once as the scan of it ends. + Found(u32), + /// A resource with server scripts begins to start. + Entering(&'a str), + Started { + resource: &'a str, + warnings: u32, + }, + /// A resource did not start, or no resource has that name. + Failed(&'a str), + /// A streamed file FXServer finds heavy, as `resource/file`, in tenths of a MiB. + Heavy { + path: &'a str, + tenths: u32, + memory: Memory, + oversized: bool, + }, + /// A file a manifest names that its resource does not hold. + Missing { + kind: &'a str, + file: &'a str, + manifest: &'a str, + line: u32, + }, +} + +/// The memory FXServer measures a streamed file by. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize)] +#[serde(rename_all = "camelCase")] +pub enum Memory { + Physical, + Virtual, +} + +impl<'a> Said<'a> { + fn read(line: &'a str) -> Option { + let (channel, text) = channel(line)?; + // FXServer names the channel `resources::stream`, cut at twenty characters. + if channel.starts_with("resources:") { + return Self::warning(text); + } + if channel == "resources" { + if let Some(rest) = text.strip_prefix("Started resource ") { + let (resource, counted) = rest.split_once(' ').unwrap_or((rest, "")); + let warnings = counted + .strip_prefix('(') + .and_then(|counted| counted.split_once(" warning")) + .and_then(|(count, _)| count.parse().ok()) + .unwrap_or(0); + return (!resource.is_empty()).then_some(Self::Started { resource, warnings }); + } + let found = text.strip_prefix("Found ")?.strip_suffix(" resources.")?; + return found.parse().ok().map(Self::Found); + } + if channel == "c-scripting-core" { + let resource = text.strip_prefix("Creating script environments for ")?; + return (!resource.is_empty()).then_some(Self::Entering(resource)); + } + if channel != "citizen-server-impl" { + return None; + } + let named = text + .strip_prefix("Couldn't start resource ") + .or_else(|| text.strip_prefix("Couldn't find resource "))?; + let resource = named.strip_suffix('.').unwrap_or(named); + // A category that is not there starts nothing: its command has no resource to wait for. + (!resource.is_empty() && !resource.starts_with("category ")).then_some(Self::Failed(resource)) + } + + /// What FXServer says of a resource's files as it starts it, before the line that says it + /// started: `Asset maps/dock.ydr uses 49.5 MiB of physical memory.`, and + /// ``Warning: could not find file `x.lua` (defined in fxmanifest.lua:12)``. + fn warning(text: &'a str) -> Option { + if let Some(rest) = text.strip_prefix("Asset ") { + let (path, said) = rest.split_once(" uses ")?; + let (size, of) = said.split_once(" MiB of ")?; + let memory = if of.starts_with("physical memory") { + Memory::Physical + } else if of.starts_with("virtual memory") { + Memory::Virtual + } else { + return None; + }; + let (whole, tenth) = size.split_once('.').unwrap_or((size, "0")); + let tenths = whole + .parse::() + .ok()? + .saturating_mul(10) + .saturating_add(tenth.get(..1)?.parse().ok()?); + return path.contains('/').then_some(Self::Heavy { + path, + tenths, + memory, + oversized: of.contains("Oversized assets"), + }); + } + let rest = text.strip_prefix("Warning: could not find ")?; + let (kind, rest) = rest.split_once(" `")?; + let (file, rest) = rest.split_once("` (defined in ")?; + let (manifest, line) = rest.strip_suffix(')')?.rsplit_once(':')?; + Some(Self::Missing { + kind, + file, + manifest, + line: line.parse().ok()?, + }) + } +} + +/// A streamed file FXServer finds heavy. +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize)] +pub struct HeavyAsset { + pub resource: String, + /// Within the resource, as FXServer names it. + pub file: String, + /// In tenths of a MiB. + pub tenths: u32, + pub memory: Memory, + /// FXServer says it can keep models from loading. + pub oversized: bool, +} + +/// A file a manifest names that its resource does not hold. +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize)] +pub struct MissingFile { + pub resource: String, + /// What the manifest asks for it as: `client_script`, `file`. + pub kind: String, + pub file: String, + pub manifest: String, + pub line: u32, +} + +/// What FXServer warned about as it started its resources, each once. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)] +pub struct Warnings { + pub heavy: Vec, + pub missing: Vec, + /// FXServer warned of more than are kept. + pub truncated: bool, +} + +impl Warnings { + fn heavy(&mut self, asset: HeavyAsset) { + if self.heavy.contains(&asset) { + return; + } + if self.heavy.len() < MAX_WARNINGS { + self.heavy.push(asset); + } else { + self.truncated = true; + } + } + + fn missing(&mut self, file: MissingFile) { + if self.missing.contains(&file) { + return; + } + if self.missing.len() < MAX_WARNINGS { + self.missing.push(file); + } else { + self.truncated = true; + } + } +} + +/// The last resource FXServer said it started. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct Started { + /// As FXServer names it. + pub name: String, + pub at_ms: i64, + /// What FXServer counted after its name. + pub warnings: u32, +} + +/// The command of the cfg a start is at. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct At { + pub file: String, + pub line: u32, + /// The command's argument as written: a resource, or a category in brackets. + pub target: String, + /// When FXServer came to it. + pub since_ms: i64, + /// How many of the resources it starts are up, failed ones counted with them. + pub done: u32, + pub of: u32, + /// The resource whose server scripts FXServer is loading, when it said so. + pub resource: Option, +} + +/// A resource and the time FXServer took from the resource before it to this one. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct Slow { + pub name: String, + pub ms: u64, +} + +/// A start as it stands. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)] +pub struct Snapshot { + /// Resources FXServer said it started. + pub started: u32, + /// Resources it could not start. + pub failed: u32, + /// How many the folder holds, once FXServer has said. + pub found: Option, + /// How many the cfg's commands start, when Observer read them all. + pub planned: Option, + pub last: Option, + /// None before the cfg's first command and after its last, or when Observer could not read + /// every command. + pub at: Option, + /// The slowest first. + pub slowest: Vec, +} + +#[derive(Debug)] +struct State { + plan: Plan, + found: Option, + started: HashSet, + failed: HashSet, + last: Option<(Instant, Started)>, + /// When FXServer last said anything of its start, which the next resource's time counts from. + mark: Instant, + moved: Option, + /// The step the start is at: every step before it is over. + step: usize, + step_since_ms: i64, + /// The step the start was at before it was taken to be past it without its resources saying + /// so: one of them starting proves it was not. + left: Option, + strays: u32, + entering: Option, + slowest: Vec, + warnings: Warnings, + /// The files FXServer could not find since the last resource started: it says them before + /// the resource they belong to, which the line does not name. + unplaced: Vec, +} + +impl State { + /// Gives the files not found since the last resource to `resource`, which they are said before. + fn place_missing(&mut self, resource: &str) { + for mut file in std::mem::take(&mut self.unplaced) { + resource.clone_into(&mut file.resource); + self.warnings.missing(file); + } + } + + fn settled(&self, resource: &str) -> bool { + self.started.contains(resource) || self.failed.contains(resource) + } + + fn settle(&mut self, at_ms: i64) { + let before = self.step; + while self + .plan + .steps + .get(self.step) + .is_some_and(|step| step.members.iter().all(|member| self.settled(member))) + { + self.step += 1; + } + if self.step != before { + self.step_since_ms = at_ms; + self.strays = 0; + } + } + + /// Places the start after `resource` started: at its command, or still where it was when the + /// resource is a dependency started ahead of its own command. + fn place(&mut self, resource: &str, at_ms: i64) { + let steps = &self.plan.steps; + let skipped = self.left.unwrap_or(self.step)..self.step; + if let Some(back) = skipped.clone().find(|&index| steps[index].starts(resource)) { + self.step = back; + self.left = None; + self.strays = 0; + } else if steps.get(self.step).is_some_and(|step| step.starts(resource)) { + self.strays = 0; + } else if let Some(later) = (self.step + 1..steps.len()).find(|&index| steps[index].starts(resource)) + { + self.strays += 1; + if self.strays >= STRAYS { + self.left.get_or_insert(self.step); + self.step = later; + self.step_since_ms = at_ms; + self.strays = 0; + } + } + } + + fn timed(&mut self, name: &str, took: Duration) { + let ms = u64::try_from(took.as_millis()).unwrap_or(u64::MAX); + let rank = self.slowest.partition_point(|slow| slow.ms >= ms); + if rank < SLOWEST { + self.slowest.insert( + rank, + Slow { + name: name.to_owned(), + ms, + }, + ); + self.slowest.truncate(SLOWEST); + } + } +} + +/// One start, fed by the console's reader and read by the supervisor. +#[derive(Debug)] +pub struct Progress { + state: Mutex, +} + +impl Progress { + pub fn new(plan: Plan, spawned: Instant, spawned_ms: i64) -> Self { + Self { + state: Mutex::new(State { + plan, + found: None, + started: HashSet::new(), + failed: HashSet::new(), + last: None, + mark: spawned, + moved: None, + step: 0, + step_since_ms: spawned_ms, + left: None, + strays: 0, + entering: None, + slowest: Vec::new(), + warnings: Warnings::default(), + unplaced: Vec::new(), + }), + } + } + + fn state(&self) -> std::sync::MutexGuard<'_, State> { + self.state.lock().unwrap_or_else(PoisonError::into_inner) + } + + /// Takes a line FXServer printed, colours removed, at `at`. + pub fn read(&self, line: &str, at: Instant, at_ms: i64) { + let Some(said) = Said::read(line) else { + return; + }; + let mut state = self.state(); + match said { + Said::Found(found) => state.found = Some(found), + Said::Entering(resource) => { + state.entering = Some(resource.to_owned()); + return; + } + Said::Heavy { + path, + tenths, + memory, + oversized, + } => { + let (resource, file) = path.split_once('/').unwrap_or((path, "")); + state.warnings.heavy(HeavyAsset { + resource: resource.to_owned(), + file: file.to_owned(), + tenths, + memory, + oversized, + }); + return; + } + Said::Missing { + kind, + file, + manifest, + line, + } => { + if state.unplaced.len() < MAX_WARNINGS { + state.unplaced.push(MissingFile { + resource: String::new(), + kind: kind.to_owned(), + file: file.to_owned(), + manifest: manifest.to_owned(), + line, + }); + } + return; + } + Said::Started { resource, warnings } => { + let lower = resource.to_ascii_lowercase(); + let took = at.saturating_duration_since(state.mark); + if state.started.insert(lower.clone()) { + state.timed(resource, took); + } + state.failed.remove(&lower); + state.last = Some(( + at, + Started { + name: resource.to_owned(), + at_ms, + warnings, + }, + )); + state.entering = None; + state.place_missing(resource); + state.place(&lower, at_ms); + } + Said::Failed(resource) => { + state.place_missing(resource); + let lower = resource.to_ascii_lowercase(); + if !state.started.contains(&lower) { + state.failed.insert(lower); + } + state.entering = None; + } + } + state.mark = at; + state.moved = Some(at); + state.settle(at_ms); + } + + /// When FXServer last said its start went on. + pub fn moved(&self) -> Option { + self.state().moved + } + + /// The last resource FXServer said it started, and when. + pub fn last(&self) -> Option<(Instant, String)> { + let state = self.state(); + state + .last + .as_ref() + .map(|(at, started)| (*at, started.name.clone())) + } + + /// What FXServer warned about so far, in this run. + pub fn warnings(&self) -> Warnings { + self.state().warnings.clone() + } + + pub fn snapshot(&self) -> Snapshot { + let state = self.state(); + let count = |set: &HashSet| u32::try_from(set.len()).unwrap_or(u32::MAX); + let at = state.plan.steps.get(state.step).map(|step| At { + file: step.file.clone(), + line: step.line, + target: step.target.clone(), + since_ms: state.step_since_ms, + done: u32::try_from(step.members.iter().filter(|member| state.settled(member)).count()) + .unwrap_or(u32::MAX), + of: u32::try_from(step.members.len()).unwrap_or(u32::MAX), + resource: state.entering.clone(), + }); + Snapshot { + started: count(&state.started), + failed: count(&state.failed), + found: state.found, + planned: state.plan.resources, + last: state.last.as_ref().map(|(_, started)| started.clone()), + at, + slowest: state.slowest.clone(), + } + } +} + +#[cfg(test)] +mod tests { + use std::fs; + + use observer_domain::launch::OneSync; + + use super::*; + use crate::{cfg, server_data}; + + fn scanned(cfg_text: &str, folders: &[&str]) -> (Scan, Resources) { + let dir = tempfile::tempdir().unwrap(); + for folder in folders { + let path = dir.path().join("resources").join(folder); + fs::create_dir_all(&path).unwrap(); + fs::write(path.join("fxmanifest.lua"), "fx_version 'cerulean'\n").unwrap(); + } + fs::write(dir.path().join("server.cfg"), cfg_text).unwrap(); + let resources = server_data::walk(dir.path()); + let context = cfg::Context { + panel: None, + onesync: OneSync::On, + probe_ports: false, + system_resources: None, + follow_outside: false, + convars: None, + }; + let scan = cfg::scan(dir.path(), "server.cfg", cfg_text, &resources, &context); + (scan, resources) + } + + #[test] + fn a_plan_holds_the_commands_that_start_resources_with_those_each_one_starts() { + let (scan, resources) = scanned( + "ensure chat\nstop chat\nensure [Maps]\nstart sessionmanager\nensure [gone]\nrestart chat\n", + &["chat", "[maps]/docks", "[maps]/[interiors]/bank"], + ); + + let plan = Plan::new(&scan, &resources); + + assert_eq!( + plan.steps, + [ + step(1, "chat", &["chat"]), + step(3, "[Maps]", &["bank", "docks"]), + step(4, "sessionmanager", &["sessionmanager"]), + step(5, "[gone]", &[]), + ] + ); + assert_eq!(plan.resources, Some(4)); + } + + #[test] + fn a_cfg_whose_commands_could_not_all_be_read_gives_no_plan() { + let (scan, resources) = scanned("ensure chat\nexec @chat/more.cfg\n", &["chat"]); + assert!(!scan.resource_commands_complete); + + assert_eq!(Plan::new(&scan, &resources), Plan::default()); + } + + fn step(line: u32, target: &str, members: &[&str]) -> Step { + Step { + file: "server.cfg".to_owned(), + line, + target: target.to_owned(), + members: members.iter().map(|&member| member.to_owned()).collect(), + } + } + + fn plan(steps: Vec) -> Plan { + let resources = steps.iter().map(|step| step.members.len()).sum::(); + Plan { + steps, + resources: u32::try_from(resources).ok(), + } + } + + struct Start { + progress: Progress, + spawned: Instant, + } + + impl Start { + fn new(plan: Plan) -> Self { + let spawned = Instant::now(); + Self { + progress: Progress::new(plan, spawned, 0), + spawned, + } + } + + fn says(&self, seconds: u64, line: &str) -> &Self { + let at = self.spawned + Duration::from_secs(seconds); + self.progress + .read(line, at, i64::try_from(seconds * 1000).unwrap_or(i64::MAX)); + self + } + + fn starts(&self, seconds: u64, resource: &str) -> &Self { + self.says( + seconds, + &format!("[ resources] Started resource {resource}"), + ) + } + + fn at(&self) -> Option<(u32, String)> { + self.progress.snapshot().at.map(|at| (at.line, at.target)) + } + } + + #[test] + fn the_lines_of_a_start_are_read_from_fxservers_own_channels_only() { + assert_eq!( + Said::read("[ resources] Started resource Docks_Props"), + Some(Said::Started { + resource: "Docks_Props", + warnings: 0 + }) + ); + assert_eq!( + Said::read("[ resources] Started resource maps (1 warning)"), + Some(Said::Started { + resource: "maps", + warnings: 1 + }) + ); + assert_eq!( + Said::read("[ resources] Started resource maps (12 warnings)"), + Some(Said::Started { + resource: "maps", + warnings: 12 + }) + ); + assert_eq!( + Said::read("[ resources] Found 391 resources."), + Some(Said::Found(391)) + ); + assert_eq!( + Said::read("[ c-scripting-core] Creating script environments for chat"), + Some(Said::Entering("chat")) + ); + assert_eq!( + Said::read("[ citizen-server-impl] Couldn't start resource nursery."), + Some(Said::Failed("nursery")) + ); + assert_eq!( + Said::read("[ citizen-server-impl] Couldn't find resource typo."), + Some(Said::Failed("typo")) + ); + for line in [ + "[ citizen-server-impl] Couldn't find resource category [hotel].", + "[ script:chat] Started resource maps", + "[ script:chat] Couldn't start resource maps.", + "Started resource maps", + "[ resources] Scanning resources.", + "[ resources] Stopping resource maps", + ] { + assert_eq!(Said::read(line), None, "{line}"); + } + } + + #[test] + fn what_fxserver_warns_of_is_kept_once_with_the_resource_it_was_starting() { + let start = Start::new(plan(Vec::new())); + let heavy = "[resources:maps:stream] Asset maps/dock.ydr uses 49.5 MiB of physical memory. \ + Oversized assets can and WILL lead to streaming issues (such as models not loading/rendering)."; + start + .says(1, heavy) + .says(1, "[ resources:maps:str] Asset maps/pier.ytd uses 16 MiB of virtual memory.") + .says(1, "[resources:Long_Name] Warning: could not find file `data/**/x.meta` (defined in fxmanifest.lua:12)") + .says(1, "[resources:Long_Name] Warning: could not find client_script `cl.lua` (defined in __resource.lua:3)") + .says(1, "[resources:Long_Name] Warning: Long_Name1 does not have a resource manifest (fxmanifest.lua)") + .starts(2, "Long_Name_Of_A_Resource (2 warnings)") + .says(3, heavy) + .says(3, "[ script:maps] Asset maps/fake.ydr uses 99.0 MiB of physical memory.") + .says(3, "[resources:gone:strea] Warning: could not find file `a.lua` (defined in fxmanifest.lua:1)") + .says(4, "[ citizen-server-impl] Couldn't start resource gone."); + + assert_eq!( + start.progress.warnings(), + Warnings { + heavy: vec![ + HeavyAsset { + resource: "maps".to_owned(), + file: "dock.ydr".to_owned(), + tenths: 495, + memory: Memory::Physical, + oversized: true, + }, + HeavyAsset { + resource: "maps".to_owned(), + file: "pier.ytd".to_owned(), + tenths: 160, + memory: Memory::Virtual, + oversized: false, + }, + ], + missing: vec![ + MissingFile { + resource: "Long_Name_Of_A_Resource".to_owned(), + kind: "file".to_owned(), + file: "data/**/x.meta".to_owned(), + manifest: "fxmanifest.lua".to_owned(), + line: 12, + }, + MissingFile { + resource: "Long_Name_Of_A_Resource".to_owned(), + kind: "client_script".to_owned(), + file: "cl.lua".to_owned(), + manifest: "__resource.lua".to_owned(), + line: 3, + }, + MissingFile { + resource: "gone".to_owned(), + kind: "file".to_owned(), + file: "a.lua".to_owned(), + manifest: "fxmanifest.lua".to_owned(), + line: 1, + }, + ], + truncated: false, + } + ); + } + + #[test] + fn a_start_is_at_the_first_command_whose_resources_are_not_all_up() { + let start = Start::new(plan(vec![ + step(3, "chat", &["chat"]), + step(4, "[maps]", &["bridge", "docks", "pier"]), + step(9, "jobs", &["jobs"]), + ])); + assert_eq!(start.at(), Some((3, "chat".to_owned()))); + + start.starts(2, "chat"); + let snapshot = start.progress.snapshot(); + let at = snapshot.at.unwrap(); + assert_eq!((at.line, at.since_ms, at.done, at.of), (4, 2000, 0, 3)); + + start.starts(5, "Docks").starts(6, "pier"); + let at = start.progress.snapshot().at.unwrap(); + assert_eq!((at.line, at.since_ms, at.done, at.of), (4, 2000, 2, 3)); + + start.starts(9, "bridge"); + assert_eq!(start.at(), Some((9, "jobs".to_owned()))); + start.starts(10, "jobs"); + assert_eq!(start.at(), None); + let snapshot = start.progress.snapshot(); + assert_eq!((snapshot.started, snapshot.planned), (5, Some(5))); + assert_eq!(snapshot.last.unwrap().name, "jobs"); + } + + #[test] + fn a_resource_that_does_not_start_does_not_hold_the_start_at_its_command() { + let start = Start::new(plan(vec![ + step(3, "typo", &["typo"]), + step(4, "[props]", &["nursery", "toys"]), + step(5, "[hotel]", &[]), + step(6, "jobs", &["jobs"]), + ])); + + start.says(1, "[ citizen-server-impl] Couldn't find resource typo."); + assert_eq!(start.at(), Some((4, "[props]".to_owned()))); + start + .starts(2, "toys") + .says(3, "[ citizen-server-impl] Couldn't start resource nursery."); + + assert_eq!(start.at(), Some((6, "jobs".to_owned()))); + let snapshot = start.progress.snapshot(); + assert_eq!((snapshot.started, snapshot.failed), (1, 2)); + } + + #[test] + fn a_dependency_started_ahead_of_its_command_leaves_the_start_where_it_is() { + let start = Start::new(plan(vec![ + step(3, "framework", &["framework"]), + step(4, "database", &["database"]), + step(5, "jobs", &["jobs"]), + ])); + + start.starts(1, "database"); + assert_eq!(start.at(), Some((3, "framework".to_owned()))); + + start.starts(4, "framework"); + assert_eq!( + start.at(), + Some((5, "jobs".to_owned())), + "the database is up already" + ); + } + + #[test] + fn a_command_whose_resource_never_says_how_it_ended_is_left_after_three_later_ones() { + let start = Start::new(plan(vec![ + step(3, "[props]", &["silent", "toys"]), + step(4, "a", &["a"]), + step(5, "b", &["b"]), + step(6, "c", &["c"]), + step(7, "d", &["d"]), + ])); + + start.starts(1, "toys").starts(2, "a").starts(3, "b"); + assert_eq!(start.at(), Some((3, "[props]".to_owned()))); + start.starts(4, "c"); + assert_eq!(start.at(), Some((7, "d".to_owned()))); + + start.starts(9, "silent"); + assert_eq!( + start.at(), + Some((7, "d".to_owned())), + "what was left is over, and so are a, b and c" + ); + } + + #[test] + fn a_command_left_too_soon_is_come_back_to_when_its_resource_starts() { + let start = Start::new(plan(vec![ + step(3, "framework", &["framework"]), + step(4, "a", &["a"]), + step(5, "b", &["b"]), + step(6, "c", &["c"]), + step(7, "[maps]", &["docks", "pier"]), + ])); + + start.starts(1, "a").starts(2, "b").starts(3, "c"); + assert_eq!(start.at(), Some((7, "[maps]".to_owned()))); + + start.starts(60, "framework"); + assert_eq!( + start.at(), + Some((7, "[maps]".to_owned())), + "its three dependencies were up" + ); + let start = Start::new(plan(vec![ + step(3, "[core]", &["framework", "hud"]), + step(4, "a", &["a"]), + step(5, "b", &["b"]), + step(6, "c", &["c"]), + step(7, "d", &["d"]), + ])); + start.starts(1, "a").starts(2, "b").starts(3, "c"); + assert_eq!(start.at(), Some((7, "d".to_owned()))); + start.starts(60, "framework"); + assert_eq!( + start.at(), + Some((3, "[core]".to_owned())), + "hud is still to start" + ); + } + + #[test] + fn the_slowest_resources_are_timed_from_the_one_before_them() { + let start = Start::new(plan(Vec::new())); + start.says(4, "[ resources] Found 12 resources."); + for (seconds, resource) in [ + (5, "a"), + (11, "b"), + (12, "c"), + (20, "d"), + (22, "e"), + (25, "f"), + (29, "g"), + ] { + start.starts(seconds, resource); + } + start.starts(40, "b"); + + let snapshot = start.progress.snapshot(); + let slowest: Vec<(&str, u64)> = snapshot + .slowest + .iter() + .map(|slow| (slow.name.as_str(), slow.ms)) + .collect(); + assert_eq!( + slowest, + [("d", 8000), ("b", 6000), ("g", 4000), ("f", 3000), ("e", 2000)] + ); + assert_eq!( + (snapshot.found, snapshot.started, snapshot.at), + (Some(12), 7, None) + ); + } + + #[test] + fn the_resource_whose_scripts_load_is_named_until_it_is_up() { + let start = Start::new(plan(vec![step(3, "[core]", &["framework", "hud"])])); + + start.says( + 1, + "[ c-scripting-core] Creating script environments for framework", + ); + assert_eq!( + start.progress.snapshot().at.unwrap().resource.as_deref(), + Some("framework") + ); + assert_eq!(start.progress.moved(), None, "nothing is up yet"); + + start.starts(3, "framework"); + assert_eq!(start.progress.snapshot().at.unwrap().resource, None); + assert_eq!( + start.progress.moved(), + Some(start.spawned + Duration::from_secs(3)) + ); + } +} diff --git a/crates/observer-fxserver/src/supervisor.rs b/crates/observer-fxserver/src/supervisor.rs index 2f6b8b79..c8b1982c 100644 --- a/crates/observer-fxserver/src/supervisor.rs +++ b/crates/observer-fxserver/src/supervisor.rs @@ -6,10 +6,11 @@ //! once it has ended. A stop or a shutdown meanwhile tells it to end, waits, and spawns nothing. use std::collections::{HashMap, HashSet}; +use std::fmt; use std::net::{Ipv4Addr, SocketAddr}; use std::path::PathBuf; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Weak}; +use std::sync::{Arc, Mutex, PoisonError, Weak}; use std::time::{Duration, Instant}; use observer_domain::diagnostics::ProcessKind; @@ -25,16 +26,18 @@ use tracing::{error, info, warn}; use crate::bridge::{self, BridgeAction, BridgeEvent, Report, StopReason}; use crate::cfg; -use crate::console::{ConsoleHub, LineAssembler, LineFlags, LineKind, plain, sanitize}; +use crate::console::{ConsoleHub, LineAssembler, LineFlags, LineKind, on_channel, plain, sanitize}; use crate::crashes::{self, Collected, DumpLimits, HangSnapshotLimits, SnapshotFailure}; use crate::exit; use crate::health::{self, HangCause, Health, HealthLimits, PlayerList, Players, Probe, Problem, Verdict}; +use crate::hook; use crate::launch::{self, ServerLaunch}; use crate::layout::{Flavour, Layout}; use crate::perf::TickReport; use crate::players::{PlayerReport, PlayerReportKind}; use crate::pre_launch::{self, PreLaunch, PreLaunchRequest, PreLaunchResult, TOOLS}; use crate::recovery::{Next, Recovery, RecoveryPolicy}; +use crate::start::{self, Progress}; use crate::stdin::{CommandRefusal, Credentials, StdinError, StdinWriter, SystemLine, UserCommand, Write}; use crate::trace::{self, Trace, TraceDecoder, TraceKind}; use crate::tree::{self, TreeContext}; @@ -48,6 +51,8 @@ const TREE_KILL: Duration = Duration::from_secs(5); const TASK_JOIN: Duration = Duration::from_secs(2); /// A dump file touched this recently means the dump server is writing. const DUMP_ACTIVITY: Duration = Duration::from_secs(10); +/// What someone gives a start that needs longer than the hang delay between two resources. +pub const START_EXTENSION: Duration = Duration::from_mins(15); /// FXServer's own default, for a cfg that names no TCP endpoint. const DEFAULT_PORT: u16 = 30120; const EVENTS: usize = 256; @@ -55,6 +60,8 @@ const EVENTS: usize = 256; const PROCESS_EVENTS: usize = 1024; /// A restart of a full server reports a join or a drop for each of its players at once. const PLAYER_REPORTS: usize = 8192; +/// Every resource of a server that counts reports each second it had something to count. +const HOOK_REPORTS: usize = 8192; /// How often the tree's members are compared with what its notifications said. const RECONCILE_EVERY: Duration = Duration::from_secs(5); /// A member younger than this is the notifications' to report: a fork may not have run its program @@ -90,6 +97,53 @@ pub enum State { CrashLoop, } +/// Whether the next start counts what its resources send and receive, asked at each spawn. +#[derive(Clone)] +pub struct Counting(Arc bool + Send + Sync>); + +impl Counting { + pub fn new(asked: impl Fn() -> bool + Send + Sync + 'static) -> Self { + Self(Arc::new(asked)) + } + + fn asked(&self) -> bool { + (self.0)() + } +} + +impl fmt::Debug for Counting { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("Counting") + } +} + +/// What became of the hook at a start that was to count. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Hooked { + /// The start it was for. + pub generation: u32, + /// Why the build's scheduler was left as it was, and that start counts nothing. + pub refused: Option, +} + +type LatestHook = Arc>>; + +/// What a start did with the hook. +enum Hook { + /// The start does not count: the scheduler is the build's own. + Out, + In, + /// The start was to count, and the scheduler was left as it was. + Refused(String), +} + +/// What one resource counted, with the start it is of. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct HookReport { + pub generation: u32, + pub report: hook::Report, +} + #[derive(Debug, Clone)] pub struct Settings { /// What the next spawn launches, none until [`Supervisor::configure`] gives it. @@ -115,6 +169,8 @@ pub struct Settings { pub pre_launch: Option>, /// How long that work gets before it is told to end. The spawn still waits until it has. pub pre_launch_limit: Duration, + /// Whether a start counts what its resources send and receive; `None` never does. + pub counting: Option, } impl Settings { @@ -132,6 +188,7 @@ impl Settings { drop_first_tree_change: false, pre_launch: None, pre_launch_limit: pre_launch::DEFAULT_LIMIT, + counting: None, } } } @@ -363,6 +420,17 @@ pub struct Status { pub port: Option, /// The bridge's resource name in the server that runs now, drawn for its start. pub bridge: Option, + /// How far the start has come, while the server is starting. + pub start: Option, +} + +/// A start under way: what FXServer said of it, and how long Observer still waits for more. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct StartStatus { + pub progress: start::Snapshot, + /// When the start is taken for stuck, should FXServer start no resource until then. None with + /// hang detection off, and once the cfg has run. + pub stuck_at_ms: Option, } /// What the probes read last of the server that runs now, for the statistics. It changes every @@ -401,6 +469,8 @@ pub enum ControlError { NotRunning, #[error("the server is stopping")] Stopping, + #[error("the server is not starting")] + NotStarting, #[error("Observer is shutting down")] ShuttingDown, #[error("cannot start FXServer: {0}")] @@ -426,6 +496,10 @@ enum Request { by: String, reply: Reply<()>, }, + ExtendStart { + by: String, + reply: Reply<()>, + }, RestartScheduled { by: String, reply: Reply<()>, @@ -453,13 +527,20 @@ enum Request { SkipHangSnapshots, } +/// The latest spawn's progress and when it was spawned: what FXServer said as it started, kept +/// after the start and until the next one. +type LatestStart = Arc)>>>; + /// The handle to the supervisor task. Clones talk to the same task. #[derive(Clone)] pub struct Supervisor { + latest_start: LatestStart, + latest_hook: LatestHook, requests: mpsc::Sender, events: broadcast::Sender, processes: broadcast::Sender, players: broadcast::Sender, + counts: broadcast::Sender, status: watch::Receiver, readings: watch::Receiver, } @@ -472,6 +553,7 @@ impl Supervisor { let events = broadcast::channel(EVENTS).0; let processes = broadcast::channel(PROCESS_EVENTS).0; let players = broadcast::channel(PLAYER_REPORTS).0; + let counts = broadcast::channel(HOOK_REPORTS).0; let initial = Status { state: State::Stopped, generation: 0, @@ -485,13 +567,19 @@ impl Supervisor { name: None, port: None, bridge: None, + start: None, }; let (status_tx, status) = watch::channel(initial); let (readings_tx, readings) = watch::channel(Readings::default()); let (traces_tx, traces) = mpsc::channel(CHANNEL); let (signals_tx, signals) = mpsc::channel(CHANNEL); let (probes_tx, probes) = mpsc::channel(CHANNEL); + let latest_start = LatestStart::default(); + let latest_hook = LatestHook::default(); let actor = Actor { + latest_start: Arc::clone(&latest_start), + latest_hook: Arc::clone(&latest_hook), + counts: counts.clone(), recovery: Recovery::new(settings.recovery.clone()), settings, console, @@ -525,10 +613,13 @@ impl Supervisor { let task = tokio::spawn(actor.run()); ( Self { + latest_start, + latest_hook, requests, events, processes, players, + counts, status, readings, }, @@ -562,6 +653,25 @@ impl Supervisor { .await } + /// What FXServer warned about as it started its resources, in the latest start, with when + /// that start was spawned. None before the first start. + pub fn start_warnings(&self) -> Option<(i64, start::Warnings)> { + let latest = self.latest_start.lock().unwrap_or_else(PoisonError::into_inner); + latest + .as_ref() + .map(|(spawned_ms, progress)| (*spawned_ms, progress.warnings())) + } + + /// Gives the start under way [`START_EXTENSION`] more before it is taken for stuck, counted + /// from when it would have been. Only of a server whose cfg is still running. + pub async fn extend_start(&self, by: &str) -> Result<(), ControlError> { + self.ask(|reply| Request::ExtendStart { + by: by.to_owned(), + reply, + }) + .await + } + /// The schedule's restart: like [`Supervisor::restart`], with `scheduled` as the reason /// resources hear, but only of a server that runs. A stopped server, or one waiting to restart /// after a failure, is never started by it. @@ -647,6 +757,20 @@ impl Supervisor { self.players.subscribe() } + /// What the resources of a start that counts report, each second. Off the event bus too: + /// there are hundreds a second. + pub fn hook_reports(&self) -> broadcast::Receiver { + self.counts.subscribe() + } + + /// What became of the hook at the latest start, if that start was to count. + pub fn hooked(&self) -> Option { + self.latest_hook + .lock() + .unwrap_or_else(PoisonError::into_inner) + .clone() + } + pub fn status(&self) -> watch::Receiver { self.status.clone() } @@ -763,12 +887,6 @@ fn is_licence_rejection(message: &str) -> bool { .any(|rejection| message.starts_with(rejection)) } -/// The text of a line FXServer printed on `channel`, whose name it right-aligns in brackets. -fn on_channel<'a>(line: &'a str, channel: &str) -> Option<&'a str> { - let (name, text) = line.strip_prefix('[')?.split_once("] ")?; - (name.trim_start() == channel).then_some(text) -} - /// One spawn of FXServer, from its start until its tree is empty. struct Run { generation: u32, @@ -787,6 +905,10 @@ struct Run { cancel: CancellationToken, tasks: Vec>, health: Health, + /// What FXServer says of its start: what shows it going on before the bridge's first report. + progress: Arc, + /// The start as published last, and the instant its `stuck_at_ms` was worked out from. + start: Option<(StartStatus, Option)>, bridge_ready: bool, ready_at: Option, after: Option, @@ -821,6 +943,7 @@ struct Ready { endpoint: Option, /// The launch settings the spawn goes by, which the spawn's event reports. settings: LaunchSettings, + plan: start::Plan, } impl Ready { @@ -832,6 +955,7 @@ impl Ready { warnings, endpoint, scan, + plan, } = prepared; let ready = Self { launch, @@ -840,6 +964,7 @@ impl Ready { warnings, endpoint, settings, + plan, }; (ready, scan) } @@ -886,11 +1011,14 @@ enum Wake { } struct Actor { + latest_start: LatestStart, + latest_hook: LatestHook, settings: Settings, console: Arc, events: broadcast::Sender, processes: broadcast::Sender, players: broadcast::Sender, + counts: broadcast::Sender, status: watch::Sender, readings: watch::Sender, requests: mpsc::Receiver, @@ -985,6 +1113,9 @@ impl Actor { let result = self.restart(by).await; let _ = reply.send(result); } + Request::ExtendStart { by, reply } => { + let _ = reply.send(self.extend_start(&by)); + } Request::RestartScheduled { by, reply } => { let _ = reply.send(self.restart_scheduled(by)); } @@ -1491,16 +1622,75 @@ impl Actor { } } - async fn weigh_health(&mut self, now: Instant) { - // The launch settings given last, rather than the spawn's: turning hang detection off, or - // giving a big first start more time, must reach the server that is starting now. - let limits = match &self.settings.launch { + /// The launch settings given last, rather than the spawn's: turning hang detection off, or + /// giving a big first start more time, must reach the server that is starting now. + fn health_limits(&self) -> HealthLimits { + match &self.settings.launch { Some(launch) => self.settings.health.under(&launch.settings), None => self.settings.health, + } + } + + fn extend_start(&mut self, by: &str) -> Result<(), ControlError> { + let limits = self.health_limits(); + let now = Instant::now(); + match &mut self.run { + Some(run) if self.state == State::Starting && !run.health.cfg_ran() => { + let stuck = run.health.stuck_at(&limits).map_or(now, |at| at.max(now)); + run.health.extend(stuck + START_EXTENSION); + } + _ => return Err(ControlError::NotStarting), + } + self.note(&format!( + "{by} gave this start {} more minutes before it is taken for stuck", + START_EXTENSION.as_secs() / 60 + )); + self.follow_start(now, &limits); + Ok(()) + } + + /// Publishes how far the start has come, when that changed since it was last published. + fn follow_start(&mut self, now: Instant, limits: &HealthLimits) { + let Some(run) = &mut self.run else { + return; }; + if self.state != State::Starting { + return; + } + let stuck = run.health.stuck_at(limits); + let stuck_at_ms = match &run.start { + Some((status, from)) if *from == stuck => status.stuck_at_ms, + _ => stuck.map(|at| { + let ahead = i64::try_from(at.saturating_duration_since(now).as_millis()).unwrap_or(i64::MAX); + let behind = i64::try_from(now.saturating_duration_since(at).as_millis()).unwrap_or(i64::MAX); + now_ms().saturating_add(ahead).saturating_sub(behind) + }), + }; + let status = StartStatus { + progress: run.progress.snapshot(), + stuck_at_ms, + }; + if run + .start + .as_ref() + .is_some_and(|(published, _)| *published == status) + { + return; + } + run.start = Some((status, stuck)); + self.publish_status(); + } + + async fn weigh_health(&mut self, now: Instant) { + let limits = self.health_limits(); let Some(run) = &mut self.run else { return; }; + if !run.health.cfg_ran() + && let Some(at) = run.progress.moved() + { + run.health.progressed(at); + } let verdict = run.health.verdict(now, &limits); let newly_ready = !matches!(verdict, Verdict::Starting | Verdict::Hung(_)) && run.ready_at.is_none(); if newly_ready { @@ -1511,7 +1701,7 @@ impl Actor { self.recovery.ready(); } match verdict { - Verdict::Starting => {} + Verdict::Starting => self.follow_start(now, &limits), Verdict::Healthy => match self.state { State::Starting => self.set_state(State::Online, Cause::Ready), State::Degraded => self.set_state(State::Online, Cause::Recovered), @@ -1538,6 +1728,10 @@ impl Actor { if dumping { return; } + let why = match &self.run { + Some(run) if !run.health.cfg_ran() => stalled_text(run.progress.last(), now), + _ => hang_text(cause).to_owned(), + }; if let Some(run) = &mut self.run { run.end(Ending::Hang { cause }); } @@ -1545,15 +1739,13 @@ impl Actor { match self.settings.hang_snapshot { Some(limits) => { self.note(&format!( - "FXServer is not responding ({}); writing {SNAPSHOT} of the hung server before killing it", - hang_text(cause) + "FXServer is not responding ({why}); writing {SNAPSHOT} of the hung server before killing it" )); self.snapshot(limits).await; self.note("killing FXServer's process tree"); } None => self.note(&format!( - "FXServer is not responding ({}); killing its process tree", - hang_text(cause) + "FXServer is not responding ({why}); killing its process tree" )), } self.kill(); @@ -1565,6 +1757,16 @@ impl Actor { let Some(run) = self.run.as_mut().filter(|run| run.generation == trace.generation) else { return; }; + if let Some(report) = hook::Report::from_trace(&trace.kind) { + // The bridge is Observer's own: what it counts is not the server's. + if report.resource != run.bridge { + let _ = self.counts.send(HookReport { + generation: trace.generation, + report, + }); + } + return; + } if let Some(kind) = PlayerReportKind::from_trace(&trace.kind, &run.bridge) { let _ = self.players.send(PlayerReport { generation: trace.generation, @@ -1889,17 +2091,13 @@ impl Actor { taken: bool, ) -> Result<(Run, SocketAddr, Vec, LaunchSettings), SpawnError> { let layout = prepared.layout.clone(); - self.install(&layout, &prepared.bridge).await?; + let hook = self.install(&layout, &prepared.bridge).await?; let credentials = Credentials::generate().map_err(|error| { let message = format!("cannot generate the launch credentials: {error}"); SpawnError::new(SpawnFailure::Spawn, message) })?; - let generation = if taken { - self.generation - } else { - self.take_generation() - }; + let generation = self.generation_of(taken, hook); let spawned = observer_platform::spawn(&prepared.launch) .map_err(|error| SpawnError::new(SpawnFailure::Spawn, error.to_string()))?; let tree = Arc::new(spawned.tree); @@ -1919,6 +2117,7 @@ impl Actor { .unwrap_or_else(|| SocketAddr::from((Ipv4Addr::LOCALHOST, DEFAULT_PORT))); let cancel = CancellationToken::new(); let (stdin, stdin_task) = StdinWriter::start(spawned.stdin, generation, Arc::clone(&self.console)); + let progress = self.progress_of(prepared.plan); let mut tasks = vec![ tokio::spawn(read_console( spawned.stdout, @@ -1926,6 +2125,7 @@ impl Actor { generation, Arc::clone(&self.console), self.signals_tx.clone(), + Arc::clone(&progress), )), tokio::spawn(read_console( spawned.stderr, @@ -1933,6 +2133,7 @@ impl Actor { generation, Arc::clone(&self.console), self.signals_tx.clone(), + Arc::clone(&progress), )), tokio::spawn(trace::read( spawned.traces, @@ -1977,6 +2178,8 @@ impl Actor { cancel, tasks, health: Health::new(Instant::now()), + progress, + start: None, bridge_ready: false, ready_at: None, after: None, @@ -1986,9 +2189,46 @@ impl Actor { Ok((run, probe, prepared.warnings, prepared.settings)) } - /// Writes the bridge into the build as `bridge`, and moves away the dumps of a run Observer - /// did not see end, such as before Observer itself crashed. - async fn install(&mut self, layout: &Layout, bridge: &str) -> Result<(), SpawnError> { + /// What a spawn says of its start from now on, kept as the latest start's. + fn progress_of(&self, plan: start::Plan) -> Arc { + let spawned_ms = now_ms(); + let progress = Arc::new(Progress::new(plan, Instant::now(), spawned_ms)); + *self.latest_start.lock().unwrap_or_else(PoisonError::into_inner) = + Some((spawned_ms, Arc::clone(&progress))); + progress + } + + /// The generation of the spawn about to happen, taken now unless the pre-launch phase took + /// it as it began. What became of the hook is kept for it, and the console says why a start + /// that should count counts nothing. + fn generation_of(&mut self, taken: bool, hook: Hook) -> u32 { + let generation = if taken { + self.generation + } else { + self.take_generation() + }; + let hooked = match hook { + Hook::Out => None, + Hook::In => Some(None), + Hook::Refused(reason) => { + warn!(%reason, "this start counts nothing: the build's scheduler was left as it was"); + let note = format!("Debug counts nothing in this start: {reason}"); + self.note_as(SUPERVISOR, generation, ¬e); + Some(Some(reason)) + } + }; + *self.latest_hook.lock().unwrap_or_else(PoisonError::into_inner) = + hooked.map(|refused| Hooked { generation, refused }); + generation + } + + /// Writes the bridge into the build as `bridge`, with the hook and its two lines in the + /// scheduler when the start counts and without them when it does not, and moves away the + /// dumps of a run Observer did not see end, such as before Observer itself crashed. Answers, + /// for a start that counts, why the scheduler was left as it was: the server starts all the + /// same. + async fn install(&mut self, layout: &Layout, bridge: &str) -> Result { + let counting = self.settings.counting.as_ref().is_some_and(Counting::asked); let installing = layout.clone(); let bridge = bridge.to_owned(); let dumps = ( @@ -1997,17 +2237,22 @@ impl Actor { self.settings.dumps, ); let (installed, moved) = tokio::task::spawn_blocking(move || { - let installed = bridge::install(&installing, &bridge); + let installed = bridge::install(&installing, &bridge) + .map(|()| hook::install(&installing, &bridge, counting).err()); (installed, crashes::collect(&dumps.0, &dumps.1, dumps.2)) }) .await .map_err(|error| SpawnError::new(SpawnFailure::Spawn, error.to_string()))?; - if let Err(error) = installed { + let refused = installed.map_err(|error| { let message = format!("cannot install the observer bridge: {error}"); - return Err(SpawnError::new(SpawnFailure::Bridge, message)); - } + SpawnError::new(SpawnFailure::Bridge, message) + })?; self.report_dumps(moved); - Ok(()) + Ok(match refused { + _ if !counting => Hook::Out, + None => Hook::In, + Some(refusal) => Hook::Refused(refusal.to_string()), + }) } async fn exited(&mut self, exit: Exit) { @@ -2217,6 +2462,10 @@ impl Actor { name: self.name.clone(), port: self.port, bridge: run.map(|run| run.bridge.clone()), + start: run + .filter(|_| self.state == State::Starting) + .and_then(|run| run.start.as_ref()) + .map(|(status, _)| status.clone()), }); } @@ -2284,6 +2533,7 @@ async fn read_console( generation: u32, console: Arc, signals: mpsc::Sender<(u32, Signal)>, + progress: Arc, ) { let mut lines = LineAssembler::default(); let publish = |assembled: crate::console::Assembled| { @@ -2293,7 +2543,11 @@ async fn read_console( masked: false, }; let line = console.push(generation, kind, None, sanitize(&assembled.text), flags); - if let Some(signal) = Signal::read(kind, &plain(&line.text)) { + let text = plain(&line.text); + if kind == LineKind::Stdout { + progress.read(&text, Instant::now(), now_ms()); + } + if let Some(signal) = Signal::read(kind, &text) { let _ = signals.try_send((generation, signal)); } }; @@ -2431,6 +2685,23 @@ fn hang_text(cause: HangCause) -> &'static str { } } +/// Why a start whose cfg never finished is taken for stuck: the last resource it started, and how +/// long ago. +fn stalled_text(last: Option<(Instant, String)>, now: Instant) -> String { + match last { + Some((at, resource)) => { + let seconds = now.saturating_duration_since(at).as_secs(); + let ago = if seconds >= 120 { + format!("{} min", seconds / 60) + } else { + format!("{seconds} s") + }; + format!("its start is stuck: no resource started since {resource}, {ago} ago") + } + None => "its start is stuck: it started no resource".to_owned(), + } +} + fn now_ms() -> i64 { jiff::Timestamp::now().as_millisecond() } @@ -2439,6 +2710,26 @@ fn now_ms() -> i64 { mod tests { use super::*; + #[test] + fn a_stuck_start_names_the_last_resource_started_and_how_long_ago() { + let started = Instant::now(); + let last = || Some((started, "clothes".to_owned())); + let after = |seconds| started + Duration::from_secs(seconds); + + assert_eq!( + stalled_text(last(), after(371)), + "its start is stuck: no resource started since clothes, 6 min ago" + ); + assert_eq!( + stalled_text(last(), after(45)), + "its start is stuck: no resource started since clothes, 45 s ago" + ); + assert_eq!( + stalled_text(None, after(45)), + "its start is stuck: it started no resource" + ); + } + #[test] fn licence_rejections_are_recognised_on_either_stream() { let lines = [ diff --git a/crates/observer-store/migrations/0016_cfg_versions.sql b/crates/observer-store/migrations/0016_cfg_versions.sql new file mode 100644 index 00000000..f7b1a9ed --- /dev/null +++ b/crates/observer-store/migrations/0016_cfg_versions.sql @@ -0,0 +1,16 @@ +-- The launch's cfg as it was before each save from the panel: the latest few of each file, with +-- who replaced them and when. The content is the file's own bytes, its secrets included, as the +-- server folder held them. Times are RFC 3339 UTC with three fractional digits. + +CREATE TABLE cfg_version ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + server_id TEXT NOT NULL, + -- Relative to the server folder, with '/'. + file TEXT NOT NULL CHECK (length(file) BETWEEN 1 AND 1024), + replaced_at TEXT NOT NULL, + replaced_by TEXT NOT NULL, + lines INTEGER NOT NULL CHECK (lines >= 0), + content BLOB NOT NULL CHECK (length(content) <= 1048576) +) STRICT; + +CREATE INDEX cfg_version_by_file ON cfg_version (server_id, file, id); diff --git a/crates/observer-store/migrations/0017_tool_staged.sql b/crates/observer-store/migrations/0017_tool_staged.sql new file mode 100644 index 00000000..915f7ac6 --- /dev/null +++ b/crates/observer-store/migrations/0017_tool_staged.sql @@ -0,0 +1,14 @@ +-- A fix the owner prepared while FXServer ran: written before its next start, if a fresh scan then +-- plans exactly what the owner saw. One at most for each tool; it goes once that start has read it, +-- or once any run changed files, after which the plan is another. + +CREATE TABLE tool_staged ( + server_id TEXT NOT NULL, + tool TEXT NOT NULL CHECK (length(tool) BETWEEN 1 AND 64), + plan_sha256 TEXT NOT NULL CHECK (length(plan_sha256) = 64 AND plan_sha256 NOT GLOB '*[^0-9a-f]*'), + -- The groups the plan fixes, for the page. + fixes INTEGER NOT NULL CHECK (fixes > 0), + staged_at TEXT NOT NULL, + staged_by TEXT NOT NULL, + PRIMARY KEY (server_id, tool) +) STRICT, WITHOUT ROWID; diff --git a/crates/observer-store/migrations/0018_debug_hosts.sql b/crates/observer-store/migrations/0018_debug_hosts.sql new file mode 100644 index 00000000..93feac33 --- /dev/null +++ b/crates/observer-store/migrations/0018_debug_hosts.sql @@ -0,0 +1,11 @@ +-- The hosts each resource called over HTTP, as the Debug feature saw them: the host alone, never +-- a path, a query or a body. `known` once the owner said they know it, or when it was first seen +-- during the day Observer spends learning them, so that a host called for the first time later +-- stands out. +CREATE TABLE debug_hosts ( + resource TEXT NOT NULL CHECK (length(resource) BETWEEN 1 AND 128), + host TEXT NOT NULL CHECK (length(host) BETWEEN 1 AND 128), + first_seen TEXT NOT NULL, + known INTEGER NOT NULL CHECK (known IN (0, 1)), + PRIMARY KEY (resource, host) +) STRICT, WITHOUT ROWID; diff --git a/crates/observer-store/src/cfg_versions.rs b/crates/observer-store/src/cfg_versions.rs new file mode 100644 index 00000000..4a2f2ae8 --- /dev/null +++ b/crates/observer-store/src/cfg_versions.rs @@ -0,0 +1,140 @@ +//! The launch's cfg as it was before each save from the panel, the latest few of each file. + +use jiff::Timestamp; +use observer_domain::cfg::CfgVersion; +use rusqlite::{Connection, OptionalExtension, Transaction, params}; + +use crate::database::StoreError; +use crate::time::Utc; + +/// The file as it was, about to be replaced. +#[derive(Debug, Clone, Copy)] +pub struct Replaced<'a> { + pub server: &'a str, + /// Relative to the server folder, with `/`. + pub file: &'a str, + pub content: &'a [u8], + pub lines: u32, + pub by: &'a str, + pub at: Timestamp, +} + +/// Keeps `replaced`, and drops the oldest versions of its file past the `kept` latest. +pub fn keep(tx: &Transaction<'_>, replaced: Replaced<'_>, kept: u32) -> Result { + tx.execute( + "INSERT INTO cfg_version (server_id, file, replaced_at, replaced_by, lines, content) + VALUES (?1, ?2, ?3, ?4, ?5, ?6)", + params![ + replaced.server, + replaced.file, + Utc(replaced.at), + replaced.by, + replaced.lines, + replaced.content + ], + )?; + let id = tx.last_insert_rowid(); + tx.execute( + "DELETE FROM cfg_version WHERE server_id = ?1 AND file = ?2 AND id NOT IN ( + SELECT id FROM cfg_version WHERE server_id = ?1 AND file = ?2 ORDER BY id DESC LIMIT ?3)", + params![replaced.server, replaced.file, kept], + )?; + Ok(id) +} + +/// The versions kept of `file`, the latest first. +pub fn list(conn: &Connection, server: &str, file: &str) -> Result, StoreError> { + let mut statement = conn.prepare( + "SELECT id, replaced_at, replaced_by, length(content), lines FROM cfg_version + WHERE server_id = ?1 AND file = ?2 ORDER BY id DESC", + )?; + let rows = statement.query_map(params![server, file], |row| { + let replaced_at: Utc = row.get(1)?; + Ok(CfgVersion { + id: row.get(0)?, + replaced_at: replaced_at.0, + replaced_by: row.get(2)?, + bytes: row.get(3)?, + lines: row.get(4)?, + }) + })?; + Ok(rows.collect::>()?) +} + +pub fn count(conn: &Connection, server: &str, file: &str) -> Result { + Ok(conn.query_row( + "SELECT count(*) FROM cfg_version WHERE server_id = ?1 AND file = ?2", + params![server, file], + |row| row.get(0), + )?) +} + +/// The bytes of the version `id` of `file`, if it is still kept. +pub fn content(conn: &Connection, server: &str, file: &str, id: i64) -> Result>, StoreError> { + Ok(conn + .query_row( + "SELECT content FROM cfg_version WHERE server_id = ?1 AND file = ?2 AND id = ?3", + params![server, file, id], + |row| row.get(0), + ) + .optional()?) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::database::memory; + + fn replaced<'a>(file: &'a str, content: &'a [u8], by: &'a str, at: Timestamp) -> Replaced<'a> { + Replaced { + server: "default", + file, + content, + lines: 2, + by, + at, + } + } + + #[test] + fn the_latest_versions_of_each_file_are_kept_and_the_oldest_dropped() { + let mut conn = memory(); + let at: Timestamp = "2026-11-09T21:35:00.250Z".parse().unwrap(); + let tx = conn.transaction().unwrap(); + let first = keep(&tx, replaced("server.cfg", b"one\n", "ada", at), 2).unwrap(); + keep(&tx, replaced("resources.cfg", b"other\n", "ada", at), 2).unwrap(); + let second = keep(&tx, replaced("server.cfg", b"two\n", "grace", at), 2).unwrap(); + let third = keep(&tx, replaced("server.cfg", b"three!\n", "ada", at), 2).unwrap(); + tx.commit().unwrap(); + + let kept = list(&conn, "default", "server.cfg").unwrap(); + assert_eq!( + kept, + [ + CfgVersion { + id: third, + replaced_at: at, + replaced_by: "ada".to_owned(), + bytes: 7, + lines: 2, + }, + CfgVersion { + id: second, + replaced_at: at, + replaced_by: "grace".to_owned(), + bytes: 4, + lines: 2, + }, + ] + ); + assert_eq!(count(&conn, "default", "server.cfg").unwrap(), 2); + assert_eq!(count(&conn, "default", "resources.cfg").unwrap(), 1); + assert_eq!(content(&conn, "default", "server.cfg", first).unwrap(), None); + assert_eq!( + content(&conn, "default", "server.cfg", second).unwrap(), + Some(b"two\n".to_vec()) + ); + assert_eq!(content(&conn, "default", "resources.cfg", second).unwrap(), None); + assert_eq!(content(&conn, "other", "server.cfg", second).unwrap(), None); + } +} diff --git a/crates/observer-store/src/debug.rs b/crates/observer-store/src/debug.rs new file mode 100644 index 00000000..14df5170 --- /dev/null +++ b/crates/observer-store/src/debug.rs @@ -0,0 +1,106 @@ +//! The hosts each resource called over HTTP, as the Debug feature saw them, and until when it +//! takes every new one as known. + +use jiff::Timestamp; +use rusqlite::{Connection, params}; + +use crate::database::StoreError; +use crate::settings; +use crate::time::Utc; + +const LEARNING_UNTIL: &str = "debug.learning_until"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct StoredHost { + pub resource: String, + pub host: String, + pub first_seen: Timestamp, + pub known: bool, +} + +pub fn hosts(conn: &Connection) -> Result, StoreError> { + let mut statement = + conn.prepare("SELECT resource, host, first_seen, known FROM debug_hosts ORDER BY resource, host")?; + let rows = statement.query_map([], |row| { + let first_seen: Utc = row.get(2)?; + Ok(StoredHost { + resource: row.get(0)?, + host: row.get(1)?, + first_seen: first_seen.0, + known: row.get(3)?, + }) + })?; + Ok(rows.collect::>()?) +} + +/// Keeps a host seen for the first time. One already there keeps what it holds. +pub fn add(conn: &Connection, host: &StoredHost) -> Result<(), StoreError> { + conn.execute( + "INSERT INTO debug_hosts (resource, host, first_seen, known) VALUES (?1, ?2, ?3, ?4) + ON CONFLICT (resource, host) DO NOTHING", + params![host.resource, host.host, Utc(host.first_seen), host.known], + )?; + Ok(()) +} + +/// Marks a host as known. False when the resource never called it. +pub fn know(conn: &Connection, resource: &str, host: &str) -> Result { + let changed = conn.execute( + "UPDATE debug_hosts SET known = 1 WHERE resource = ?1 AND host = ?2", + params![resource, host], + )?; + Ok(changed > 0) +} + +pub fn learning_until(conn: &Connection) -> Result, StoreError> { + Ok(settings::get(conn, LEARNING_UNTIL)?.and_then(|until| until.parse().ok())) +} + +pub fn set_learning_until(conn: &Connection, until: Timestamp) -> Result<(), StoreError> { + settings::set(conn, LEARNING_UNTIL, &until.to_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::database::memory; + + #[test] + fn a_host_keeps_its_first_day_and_is_known_once_said_so() { + let conn = memory(); + let first: Timestamp = "2026-11-10T21:40:00Z".parse().unwrap(); + let later: Timestamp = "2026-11-11T08:00:00Z".parse().unwrap(); + let seen = |known| StoredHost { + resource: "logs".to_owned(), + host: "discord.com".to_owned(), + first_seen: first, + known, + }; + + add(&conn, &seen(false)).unwrap(); + add( + &conn, + &StoredHost { + first_seen: later, + ..seen(true) + }, + ) + .unwrap(); + + assert_eq!(hosts(&conn).unwrap(), [seen(false)], "the first sight is kept"); + assert!(know(&conn, "logs", "discord.com").unwrap()); + assert!(!know(&conn, "logs", "example.com").unwrap()); + assert_eq!(hosts(&conn).unwrap(), [seen(true)]); + } + + #[test] + fn the_day_of_learning_is_kept_once_set() { + let conn = memory(); + assert_eq!(learning_until(&conn).unwrap(), None); + + let until: Timestamp = "2026-11-11T21:35:00Z".parse().unwrap(); + set_learning_until(&conn, until).unwrap(); + + assert_eq!(learning_until(&conn).unwrap(), Some(until)); + } +} diff --git a/crates/observer-store/src/features.rs b/crates/observer-store/src/features.rs index 8e304213..569dd7ef 100644 --- a/crates/observer-store/src/features.rs +++ b/crates/observer-store/src/features.rs @@ -1,13 +1,19 @@ //! The switches of Observer's optional features, as the owner set them. A feature without a row //! keeps its default; a row of a feature this build does not know is left alone. +use std::net::SocketAddr; + use jiff::Timestamp; use observer_domain::features::Feature; use rusqlite::{Connection, params}; use crate::database::StoreError; +use crate::settings; use crate::time::Utc; +/// Where SFTP listens, once the owner said so in the panel. +const SFTP_LISTEN: &str = "sftp.listen"; + /// A switch the owner set, and when. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct StoredSwitch { @@ -46,11 +52,40 @@ pub fn set(conn: &Connection, feature: Feature, enabled: bool, now: Timestamp) - Ok(()) } +/// The address the owner gave SFTP. `None`: never given, or taken back. +pub fn sftp_listen(conn: &Connection) -> Result, StoreError> { + Ok(settings::get(conn, SFTP_LISTEN)?.and_then(|listen| listen.parse().ok())) +} + +pub fn set_sftp_listen(conn: &Connection, listen: Option) -> Result<(), StoreError> { + if let Some(listen) = listen { + return settings::set(conn, SFTP_LISTEN, &listen.to_string()); + } + conn.execute("DELETE FROM settings WHERE key = ?1", [SFTP_LISTEN])?; + Ok(()) +} + #[cfg(test)] mod tests { use super::*; use crate::database::memory; + #[test] + fn the_address_of_sftp_is_kept_until_it_is_taken_back() { + let conn = memory(); + assert_eq!(sftp_listen(&conn).unwrap(), None); + + let listen: SocketAddr = "0.0.0.0:40126".parse().unwrap(); + set_sftp_listen(&conn, Some(listen)).unwrap(); + assert_eq!(sftp_listen(&conn).unwrap(), Some(listen)); + let v6: SocketAddr = "[::1]:2022".parse().unwrap(); + set_sftp_listen(&conn, Some(v6)).unwrap(); + assert_eq!(sftp_listen(&conn).unwrap(), Some(v6)); + + set_sftp_listen(&conn, None).unwrap(); + assert_eq!(sftp_listen(&conn).unwrap(), None); + } + #[test] fn a_switch_is_replaced_on_save_and_unknown_keys_are_skipped() { let conn = memory(); diff --git a/crates/observer-store/src/lib.rs b/crates/observer-store/src/lib.rs index f13e9be7..46153aec 100644 --- a/crates/observer-store/src/lib.rs +++ b/crates/observer-store/src/lib.rs @@ -7,8 +7,10 @@ pub mod audit; mod backup; pub mod backups; pub mod bans; +pub mod cfg_versions; pub mod changes; mod database; +pub mod debug; pub mod diagnostics; pub mod features; pub mod files; diff --git a/crates/observer-store/src/map.rs b/crates/observer-store/src/map.rs index 2fb95719..dae046f6 100644 --- a/crates/observer-store/src/map.rs +++ b/crates/observer-store/src/map.rs @@ -11,12 +11,23 @@ const IMAGE: &str = "map.image"; pub fn settings(conn: &Connection) -> Result { Ok(settings::get(conn, SETTINGS)? - .and_then(|text| serde_json::from_str::(&text).ok()) + .and_then(|text| read(&text)) // A value edited by hand that the panel could not have saved falls back to the defaults. .filter(MapSettings::is_valid) .unwrap_or_default()) } +/// Settings saved before the map could draw Rockstar's name no imagery: an image their owner +/// placed stays what the map draws. +fn read(text: &str) -> Option { + let mut value: serde_json::Value = serde_json::from_str(text).ok()?; + let fields = value.as_object_mut()?; + if !fields.contains_key("imagery") && fields.get("bounds").is_some_and(|bounds| !bounds.is_null()) { + fields.insert("imagery".to_owned(), "image".into()); + } + serde_json::from_value(value).ok() +} + pub fn save_settings(conn: &Connection, saved: &MapSettings) -> Result<(), StoreError> { let text = serde_json::to_string(saved).expect("numbers always serialise"); settings::set(conn, SETTINGS, &text) @@ -52,15 +63,46 @@ struct Stored { #[cfg(test)] mod tests { use jiff::Timestamp; - use observer_domain::map::MapBounds; + use observer_domain::map::{MapBounds, MapImagery}; use super::*; use crate::database::memory; + #[test] + fn settings_saved_before_the_imagery_keep_a_placed_image_and_take_the_default_without_one() { + let conn = memory(); + settings::set( + &conn, + SETTINGS, + r#"{"bounds":{"west":-5640.0,"east":6470.0,"south":-4140.0,"north":7970.0},"newHours":5,"veteranHours":50,"everySeconds":1}"#, + ) + .unwrap(); + let placed = settings(&conn).unwrap(); + assert_eq!(placed.imagery, MapImagery::Image); + assert_eq!(placed.new_hours, 5); + + settings::set( + &conn, + SETTINGS, + r#"{"bounds":null,"newHours":5,"veteranHours":50,"everySeconds":1}"#, + ) + .unwrap(); + assert_eq!(settings(&conn).unwrap().imagery, MapImagery::Atlas); + + // What the owner chose since is kept as chosen, bounds or not. + let chosen = MapSettings { + imagery: MapImagery::Roads, + ..placed + }; + save_settings(&conn, &chosen).unwrap(); + assert_eq!(settings(&conn).unwrap(), chosen); + } + #[test] fn settings_and_the_image_round_trip_and_bad_ones_fall_back() { let conn = memory(); let saved = MapSettings { + imagery: MapImagery::Image, bounds: Some(MapBounds { west: -5640.0, east: 6470.0, diff --git a/crates/observer-store/src/migrations.rs b/crates/observer-store/src/migrations.rs index 5bf63f2c..87749239 100644 --- a/crates/observer-store/src/migrations.rs +++ b/crates/observer-store/src/migrations.rs @@ -24,6 +24,9 @@ const MIGRATIONS: &[M<'static>] = &[ M::up(include_str!("../migrations/0013_backups.sql")), M::up(include_str!("../migrations/0014_player_labels.sql")), M::up(include_str!("../migrations/0015_bans.sql")), + M::up(include_str!("../migrations/0016_cfg_versions.sql")), + M::up(include_str!("../migrations/0017_tool_staged.sql")), + M::up(include_str!("../migrations/0018_debug_hosts.sql")), ]; /// The schema version this build migrates to. diff --git a/crates/observer-store/src/tools.rs b/crates/observer-store/src/tools.rs index b0abae34..e3373df2 100644 --- a/crates/observer-store/src/tools.rs +++ b/crates/observer-store/src/tools.rs @@ -6,9 +6,9 @@ use jiff::Timestamp; use observer_domain::tools::{ - ArchetypeReport, Differences, EscrowPolicy, ToolChange, ToolChangeState, ToolCloseReason, ToolFileFormat, - ToolKind, ToolMode, ToolRemoval, ToolRun, ToolRunMode, ToolRunOutcome, ToolRunSummary, ToolSettings, - ToolTrigger, sha256_from_hex, sha256_hex, + ArchetypeReport, Differences, EscrowPolicy, StagedFix, ToolChange, ToolChangeState, ToolCloseReason, + ToolFileFormat, ToolKind, ToolMode, ToolRemoval, ToolRun, ToolRunMode, ToolRunOutcome, ToolRunSummary, + ToolSettings, ToolTrigger, sha256_from_hex, sha256_hex, }; use rusqlite::types::{FromSqlError, Type}; use rusqlite::{Connection, OptionalExtension, Row, Transaction, params}; @@ -57,6 +57,53 @@ pub fn settings( )) } +/// The fix staged for the next start of FXServer, if there is one. +pub fn staged(conn: &Connection, server_id: &str, tool: ToolKind) -> Result, StoreError> { + Ok(conn + .query_row( + "SELECT plan_sha256, fixes, staged_at, staged_by FROM tool_staged + WHERE server_id = ?1 AND tool = ?2", + params![server_id, tool.as_str()], + |row| { + let staged_at: Utc = row.get(2)?; + Ok(StagedFix { + plan: row.get(0)?, + fixes: row.get(1)?, + staged_at: staged_at.0, + staged_by: row.get(3)?, + }) + }, + ) + .optional()?) +} + +/// Stages `fix` for the next start, in the place of any staged before it. +pub fn stage(conn: &Connection, server_id: &str, tool: ToolKind, fix: &StagedFix) -> Result<(), StoreError> { + conn.execute( + "INSERT INTO tool_staged (server_id, tool, plan_sha256, fixes, staged_at, staged_by) + VALUES (?1, ?2, ?3, ?4, ?5, ?6) + ON CONFLICT (server_id, tool) DO UPDATE SET plan_sha256 = excluded.plan_sha256, + fixes = excluded.fixes, staged_at = excluded.staged_at, staged_by = excluded.staged_by", + params![ + server_id, + tool.as_str(), + fix.plan, + fix.fixes, + Utc(fix.staged_at), + fix.staged_by + ], + )?; + Ok(()) +} + +/// Drops the staged fix. Whether there was one. +pub fn unstage(conn: &Connection, server_id: &str, tool: ToolKind) -> Result { + Ok(conn.execute( + "DELETE FROM tool_staged WHERE server_id = ?1 AND tool = ?2", + params![server_id, tool.as_str()], + )? > 0) +} + /// Saves a tool's settings over the version they were read at, `None` for settings never saved. /// Returns the new version, or `None` when someone saved them since. pub fn save_settings( diff --git a/crates/observer-web/src/api.rs b/crates/observer-web/src/api.rs index 047ec8af..290a580f 100644 --- a/crates/observer-web/src/api.rs +++ b/crates/observer-web/src/api.rs @@ -9,7 +9,9 @@ mod audit; mod auth; mod backups; mod bans; +mod cfg; mod changes; +mod debug; mod diagnostics; mod drops; mod features; @@ -122,6 +124,8 @@ pub(crate) fn routes() -> Vec { // A feature turned on may open a port or read the host's folders: the owner's alone, as the // launch settings are. let read_features = Permission(P::SETTINGS_VIEW); + let read_debug = Permission(P::DIAGNOSTICS_VIEW); + let judge_hosts = Owner; let switch_features = ReauthenticatedOwner; // A backup is the whole game database. Seeing the backups and starting one are permissions; // taking one out takes the critical one and the password again. The settings and destinations @@ -181,7 +185,16 @@ pub(crate) fn routes() -> Vec { route(M::POST, "/servers/default/setup/cfg/text", read_launch, servers::cfg_text), route(M::POST, "/servers/default/setup/discover", read_launch, servers::discover), route(M::PUT, "/servers/default/setup", edit_launch, servers::save_setup), + route(M::GET, "/servers/default/cfg", read_launch, cfg::document), + route(M::PUT, "/servers/default/cfg", edit_launch, cfg::save), + route(M::POST, "/servers/default/cfg/check", read_launch, cfg::check), + route(M::POST, "/servers/default/cfg/diff", read_launch, cfg::diff), + route(M::GET, "/servers/default/cfg/versions", read_launch, cfg::versions), + route(M::GET, "/servers/default/cfg/versions/{id}", read_launch, cfg::version_diff), + route(M::POST, "/servers/default/cfg/versions/{id}/restore", edit_launch, cfg::restore), route(M::POST, "/servers/default/start", control, servers::start), + route(M::POST, "/servers/default/start/extend", control, servers::extend_start), + route(M::GET, "/servers/default/start/warnings", read_tools, servers::start_warnings), route(M::POST, "/servers/default/stop", control, servers::stop), route(M::POST, "/servers/default/restart", control, servers::restart), route(M::POST, "/servers/default/console", command, servers::command), @@ -265,10 +278,19 @@ pub(crate) fn routes() -> Vec { route(M::POST, "/tools/duplicate-archetypes/scan", read_tools, tools::scan), route(M::POST, "/tools/duplicate-archetypes/fix", change_tools, tools::fix), route(M::POST, "/tools/duplicate-archetypes/undo", change_tools, tools::undo), + route(M::POST, "/tools/duplicate-archetypes/stage", change_tools, tools::stage), + route(M::DELETE, "/tools/duplicate-archetypes/stage", change_tools, tools::unstage), route(M::PUT, "/tools/duplicate-archetypes/settings", change_tools, tools::change_settings), route(M::GET, "/settings/observer", read_bootstrap, settings::observer), route(M::GET, "/features", read_features, features::list), route(M::PUT, "/features/{feature}", switch_features, features::switch), + route(M::GET, "/settings/sftp", read_features, features::sftp_listen), + route(M::PUT, "/settings/sftp", switch_features, features::set_sftp_listen), + route(M::GET, "/debug/status", read_debug, debug::status), + route(M::GET, "/debug/events", read_debug, debug::events), + route(M::GET, "/debug/bags", read_debug, debug::bags), + route(M::GET, "/debug/http", read_debug, debug::http), + route(M::POST, "/debug/http/known", judge_hosts, debug::know_host), route(M::GET, "/servers/default/backups", read_backups, backups::view), route(M::PUT, "/servers/default/backups/settings", change_backups, backups::change_settings), route(M::POST, "/servers/default/backups/source/test", test_backups, backups::test_source), diff --git a/crates/observer-web/src/api/cfg.rs b/crates/observer-web/src/api/cfg.rs new file mode 100644 index 00000000..ad2235c6 --- /dev/null +++ b/crates/observer-web/src/api/cfg.rs @@ -0,0 +1,61 @@ +//! The launch's cfg, edited from the panel: the owner's to read, with its secrets as marks, and +//! theirs to write once the password is given again. + +use axum::extract::{Path, State}; +use axum::{Extension, Json}; +use observer_app::Caller; +use observer_domain::cfg::{CfgDocument, CfgDraft, CfgRestore, CfgReview, CfgSave, CfgVersion}; +use observer_domain::changes::TextDiff; + +use crate::AppState; +use crate::error::Failure; +use crate::extract::JsonBody; + +pub(crate) async fn document(State(state): State) -> Result, Failure> { + Ok(Json(state.supervision.cfg_document().await?)) +} + +/// What the text sets and starts and what the scan finds, as it is typed: nothing is written. +pub(crate) async fn check( + State(state): State, + JsonBody(draft): JsonBody, +) -> Result, Failure> { + Ok(Json(state.supervision.check_cfg_draft(draft).await?)) +} + +/// The lines a save of the text would change: nothing is written. +pub(crate) async fn diff( + State(state): State, + JsonBody(draft): JsonBody, +) -> Result, Failure> { + Ok(Json(state.supervision.diff_cfg_draft(draft).await?)) +} + +pub(crate) async fn save( + State(state): State, + Extension(caller): Extension, + JsonBody(save): JsonBody, +) -> Result, Failure> { + Ok(Json(state.supervision.save_cfg(&caller, save).await?)) +} + +pub(crate) async fn versions(State(state): State) -> Result>, Failure> { + Ok(Json(state.supervision.cfg_versions().await?)) +} + +/// The lines putting the version back would change. +pub(crate) async fn version_diff( + State(state): State, + Path(id): Path, +) -> Result, Failure> { + Ok(Json(state.supervision.cfg_version_diff(id).await?)) +} + +pub(crate) async fn restore( + State(state): State, + Extension(caller): Extension, + Path(id): Path, + JsonBody(restore): JsonBody, +) -> Result, Failure> { + Ok(Json(state.supervision.restore_cfg(&caller, id, restore).await?)) +} diff --git a/crates/observer-web/src/api/debug.rs b/crates/observer-web/src/api/debug.rs new file mode 100644 index 00000000..c5e45b0e --- /dev/null +++ b/crates/observer-web/src/api/debug.rs @@ -0,0 +1,46 @@ +//! What the resources of a server that counts send and receive. Reading it takes the diagnostics +//! permission; saying a host is known is the owner's. + +use axum::extract::State; +use axum::http::StatusCode; +use axum::{Extension, Json}; +use observer_app::Caller; +use observer_domain::debug::{DebugBags, DebugEvents, DebugHttp, DebugQuery, DebugStatus, KnownHost}; + +use crate::AppState; +use crate::error::Failure; +use crate::extract::{JsonBody, QueryParams}; + +pub(crate) async fn status(State(state): State) -> Json { + Json(state.debug.status()) +} + +pub(crate) async fn events( + State(state): State, + QueryParams(query): QueryParams, +) -> Json { + Json(state.debug.events(query.window)) +} + +pub(crate) async fn bags( + State(state): State, + QueryParams(query): QueryParams, +) -> Json { + Json(state.debug.bags(query.window)) +} + +pub(crate) async fn http( + State(state): State, + QueryParams(query): QueryParams, +) -> Json { + Json(state.debug.http(query.window)) +} + +pub(crate) async fn know_host( + State(state): State, + Extension(caller): Extension, + JsonBody(known): JsonBody, +) -> Result { + state.debug.know_host(&caller, known.resource, known.host).await?; + Ok(StatusCode::NO_CONTENT) +} diff --git a/crates/observer-web/src/api/features.rs b/crates/observer-web/src/api/features.rs index b078ae4f..6dcf8b0e 100644 --- a/crates/observer-web/src/api/features.rs +++ b/crates/observer-web/src/api/features.rs @@ -1,12 +1,15 @@ -//! The switches of Observer's optional features. Reading them takes the settings permission; -//! turning one on or off is the owner's, with the password again: it opens or closes a port, or -//! starts or stops reading the host's folders. +//! The switches of Observer's optional features, and the address SFTP listens on. Reading them +//! takes the settings permission; turning a feature on or off and saying where SFTP listens are +//! the owner's, with the password again: they open or close a port, or start or stop reading the +//! host's folders. + +use std::net::SocketAddr; use axum::extract::State; use axum::{Extension, Json}; use observer_app::Caller; -use observer_domain::error::ErrorCode; -use observer_domain::features::{Feature, FeatureList, FeatureSwitch}; +use observer_domain::error::{ApiError, ErrorCode}; +use observer_domain::features::{Feature, FeatureList, FeatureSwitch, SftpListen, SftpListenChange}; use crate::AppState; use crate::error::Failure; @@ -25,3 +28,21 @@ pub(crate) async fn switch( let feature = Feature::from_name(&name).ok_or(ErrorCode::NotFound)?; Ok(Json(state.features.set(&caller, feature, switch.enabled).await?)) } + +pub(crate) async fn sftp_listen(State(state): State) -> Json { + Json(state.features.sftp()) +} + +pub(crate) async fn set_sftp_listen( + State(state): State, + Extension(caller): Extension, + JsonBody(change): JsonBody, +) -> Result, Failure> { + let listen = match change.listen { + Some(listen) => Some(listen.trim().parse::().map_err(|_| { + observer_app::Error::from(ApiError::new(ErrorCode::InvalidSftpListen).with("reason", "address")) + })?), + None => None, + }; + Ok(Json(state.features.set_sftp_listen(&caller, listen).await?)) +} diff --git a/crates/observer-web/src/api/servers.rs b/crates/observer-web/src/api/servers.rs index 7674f2d4..85f0331b 100644 --- a/crates/observer-web/src/api/servers.rs +++ b/crates/observer-web/src/api/servers.rs @@ -6,7 +6,7 @@ use axum::{Extension, Json}; use observer_app::{Caller, Servers}; use observer_domain::convars::{CATALOG_JSON, CfgConvars}; use observer_domain::launch::{LaunchChange, StoredLaunch}; -use observer_domain::server::{CommandAck, ConsoleCommand, LogFile}; +use observer_domain::server::{CommandAck, ConsoleCommand, LogFile, StartWarnings}; use observer_domain::server_data::{ CfgCheck, CfgReport, CfgText, CfgTextQuery, Discovery, FileListing, FilesQuery, FolderCheck, FolderListing, FolderQuery, FolderReport, @@ -110,6 +110,20 @@ pub(crate) async fn start( Ok(StatusCode::NO_CONTENT) } +/// What FXServer warned about as it last started its resources, which the tools list. +pub(crate) async fn start_warnings(State(state): State) -> Json { + Json(state.supervision.start_warnings()) +} + +/// Gives the start under way more time before Observer takes it for stuck. +pub(crate) async fn extend_start( + State(state): State, + Extension(caller): Extension, +) -> Result { + state.supervision.extend_start(&caller).await?; + Ok(StatusCode::NO_CONTENT) +} + pub(crate) async fn stop( State(state): State, Extension(caller): Extension, diff --git a/crates/observer-web/src/api/tools.rs b/crates/observer-web/src/api/tools.rs index 1e12fbd7..a9b21eeb 100644 --- a/crates/observer-web/src/api/tools.rs +++ b/crates/observer-web/src/api/tools.rs @@ -15,6 +15,23 @@ pub(crate) async fn view(State(state): State) -> Result, + Extension(caller): Extension, + JsonBody(request): JsonBody, +) -> Result, Failure> { + Ok(Json(state.tools.stage(&caller, &request.plan).await?)) +} + +pub(crate) async fn unstage( + State(state): State, + Extension(caller): Extension, +) -> Result, Failure> { + Ok(Json(state.tools.unstage(&caller).await?)) +} + /// Scans the server folder and records the report: nothing is written. pub(crate) async fn scan( State(state): State, diff --git a/crates/observer-web/src/error.rs b/crates/observer-web/src/error.rs index 634469a8..b82d44f5 100644 --- a/crates/observer-web/src/error.rs +++ b/crates/observer-web/src/error.rs @@ -61,8 +61,10 @@ fn status(code: ErrorCode) -> StatusCode { | ErrorCode::InvalidSetupToken | ErrorCode::OriginMismatch | ErrorCode::ContentHidden - | ErrorCode::FeatureLocked => StatusCode::FORBIDDEN, + | ErrorCode::FeatureLocked + | ErrorCode::SftpListenPinned => StatusCode::FORBIDDEN, ErrorCode::NotFound + | ErrorCode::CfgVersionNotFound | ErrorCode::BuildNotListed | ErrorCode::CrashNotFound | ErrorCode::CrashFileNotFound @@ -79,6 +81,9 @@ fn status(code: ErrorCode) -> StatusCode { | ErrorCode::ServerRunning | ErrorCode::ServerNotRunning | ErrorCode::ServerStopping + | ErrorCode::ServerNotStarting + | ErrorCode::CfgNotEditable + | ErrorCode::CfgChanged | ErrorCode::NothingToSkip | ErrorCode::AlreadySkipped | ErrorCode::NothingToRestore @@ -135,6 +140,8 @@ fn status(code: ErrorCode) -> StatusCode { | ErrorCode::InvalidLaunchSettings | ErrorCode::CommandRefused | ErrorCode::InvalidRequest + | ErrorCode::InvalidCfgText + | ErrorCode::InvalidSftpListen | ErrorCode::InvalidSchedule | ErrorCode::InvalidArtifactSettings | ErrorCode::InvalidImportFolder diff --git a/crates/observer-web/src/lib.rs b/crates/observer-web/src/lib.rs index a13af006..f22a1c0c 100644 --- a/crates/observer-web/src/lib.rs +++ b/crates/observer-web/src/lib.rs @@ -20,8 +20,8 @@ use std::sync::Arc; use axum::extract::Request; use axum::{Router, middleware}; use observer_app::{ - Accounts, Artifacts, AuditLog, Backups, Bans, Changes, Diagnostics, Drops, Error, Features, Files, - LiveMap, Metrics, Players, Schedule, Servers, Statistics, Supervision, Tools, Updates, + Accounts, Artifacts, AuditLog, Backups, Bans, Changes, DebugCounts, Diagnostics, Drops, Error, Features, + Files, LiveMap, Metrics, Players, Schedule, Servers, Statistics, Supervision, Tools, Updates, }; use observer_domain::about::Build; use observer_domain::features::Feature; @@ -51,6 +51,7 @@ pub struct AppState { pub tools: Tools, pub statistics: Statistics, pub features: Features, + pub debug: DebugCounts, /// The services of the features Observer started without are absent. Their routes and topics /// are refused before anything reaches for them, through the accessors below. diagnostics: Option, @@ -91,6 +92,7 @@ pub struct Services { pub tools: Tools, pub statistics: Statistics, pub features: Features, + pub debug: DebugCounts, /// `None` for each feature that does not run at Observer's start. pub diagnostics: Option, pub metrics: Option, @@ -122,6 +124,7 @@ impl AppState { files, tools, features, + debug, diagnostics, metrics, players, @@ -144,6 +147,7 @@ impl AppState { files, tools, features, + debug, diagnostics, metrics, players, diff --git a/crates/observer-web/src/security.rs b/crates/observer-web/src/security.rs index 83c1a976..6956699d 100644 --- a/crates/observer-web/src/security.rs +++ b/crates/observer-web/src/security.rs @@ -12,11 +12,13 @@ use crate::AppState; use crate::error::Failure; use crate::proxy::Remote; -/// Everything the panel loads comes from Observer itself. `data:` images carry the grain texture; -/// `blob:` ones the live map's image, which the panel fetches with the session key. +/// Everything the panel loads comes from Observer itself, but the tiles of Rockstar's map of +/// GTA V, which the live map loads from Rockstar's servers when the owner picks it: images only, +/// from that one host, without a referrer. `data:` images carry the grain texture; `blob:` ones +/// the owner's map image, which the panel fetches with the session key. const CONTENT_SECURITY_POLICY: &str = "default-src 'self'; script-src 'self'; style-src 'self'; \ - img-src 'self' data: blob:; font-src 'self'; connect-src 'self'; object-src 'none'; base-uri 'none'; \ - form-action 'self'; frame-ancestors 'none'"; + img-src 'self' data: blob: https://s.rsg.sc; font-src 'self'; connect-src 'self'; object-src 'none'; \ + base-uri 'none'; form-action 'self'; frame-ancestors 'none'"; const PERMISSIONS_POLICY: &str = "camera=(), geolocation=(), microphone=(), payment=(), usb=()"; diff --git a/crates/observer-web/src/tests.rs b/crates/observer-web/src/tests.rs index b356698c..ad0b75d0 100644 --- a/crates/observer-web/src/tests.rs +++ b/crates/observer-web/src/tests.rs @@ -4,7 +4,9 @@ mod artifacts; mod backups; mod bans; +mod cfg; mod changes; +mod debug; mod diagnostics; mod drops; mod features; @@ -79,6 +81,7 @@ struct Server { _playing: PlayersTask, _dropping: DropsTask, _banning: BansTask, + _debugging: observer_app::DebugTask, _mapping: MapTask, _following: ChangesTask, _backing_up: BackupsTask, @@ -138,12 +141,23 @@ impl Server { Self::reached_from(SocketAddr::from(([10, 0, 0, 2], 50_000)), &proxies).await } - /// A server whose every connection comes from `peer`, believing the proxies listed. + /// A server whose every connection comes from `peer`, believing the proxies listed. SFTP's + /// address is `observer.toml`'s, though nothing listens on it here. + async fn reached_from(peer: SocketAddr, proxies: &[TrustedProxy]) -> Self { + let sftp = SocketAddr::from(([127, 0, 0, 1], 41976)); + Self::built(peer, proxies, Some(sftp)).await + } + + /// A server whose `observer.toml` leaves SFTP's address to the panel. + async fn unpinned() -> Self { + Self::built(SocketAddr::from(([203, 0, 113, 7], 50_000)), &[], None).await + } + #[expect( clippy::too_many_lines, reason = "every service the API stands on, started as Observer starts them" )] - async fn reached_from(peer: SocketAddr, proxies: &[TrustedProxy]) -> Self { + async fn built(peer: SocketAddr, proxies: &[TrustedProxy], sftp: Option) -> Self { let dir = tempfile::tempdir().unwrap(); let db = Database::open(dir.path().join("observer.sqlite3"), dir.path().join("backups")) .await @@ -255,16 +269,23 @@ impl Server { schedule: schedule.status(), }); let instance_id = db.instance_id().await.unwrap(); - // Every feature on, SFTP's address given though nothing listens on it here. let features = Features::load(FeaturesSetup { db: db.clone(), clock: clock.clone(), host_off: BTreeSet::new(), - sftp: Some(SocketAddr::from(([127, 0, 0, 1], 41976))), + sftp, data_dir: dir.path().join("data"), }) .await .unwrap(); + let (debug, debugging) = observer_app::DebugCounts::start(observer_app::DebugSetup { + db: db.clone(), + clock: clock.clone(), + supervision: supervision.clone(), + features: features.clone(), + }) + .await + .unwrap(); let files = observer_app::Files::new(db.clone(), clock.clone(), accounts.clone()); let tools = Tools::new(ToolsSetup { db: db.clone(), @@ -315,6 +336,7 @@ impl Server { files, tools, features, + debug, diagnostics: Some(diagnostics), metrics: Some(metrics.clone()), players: Some(players), @@ -357,6 +379,7 @@ impl Server { _playing: playing, _dropping: dropping, _banning: banning, + _debugging: debugging, _mapping: mapping, _following: following, _backing_up: backing_up, @@ -645,6 +668,9 @@ async fn every_response_carries_the_security_headers() { let csp = reply.headers[header::CONTENT_SECURITY_POLICY].to_str().unwrap(); assert!(csp.contains("script-src 'self'")); assert!(csp.contains("frame-ancestors 'none'")); + // Images alone come from elsewhere, and from one host: the live map's tiles. + assert!(csp.contains("img-src 'self' data: blob: https://s.rsg.sc;")); + assert_eq!(csp.matches("https://").count(), 1); assert_eq!(reply.headers[header::X_CONTENT_TYPE_OPTIONS], "nosniff"); assert_eq!(reply.headers[header::REFERRER_POLICY], "no-referrer"); assert_eq!(reply.headers[header::CACHE_CONTROL], "no-store"); diff --git a/crates/observer-web/src/tests/cfg.rs b/crates/observer-web/src/tests/cfg.rs new file mode 100644 index 00000000..2cad89c6 --- /dev/null +++ b/crates/observer-web/src/tests/cfg.rs @@ -0,0 +1,430 @@ +//! The launch's cfg over HTTP: who reads it and what they are shown of its secrets, what a draft +//! is told without anything written, the password again for a save, and the versions kept. + +use std::fs; +use std::path::PathBuf; + +use axum::http::{Method, StatusCode}; +use jiff::SignedDuration; +use observer_domain::launch::{LaunchSettings, OneSync, StoredLaunch}; +use observer_store::launch; +use serde_json::{Value, json}; + +use super::{OWNER_PASSWORD, Server, Session}; + +const CFG: &str = "/api/servers/default/cfg"; +const CHECK: &str = "/api/servers/default/cfg/check"; +const DIFF: &str = "/api/servers/default/cfg/diff"; +const VERSIONS: &str = "/api/servers/default/cfg/versions"; + +const KEY: &str = "cfxk_1a2b3c4d5e6f7g8h9i0j_AbC123"; +const DATABASE: &str = "mysql://observer:hunter2@localhost/lab"; + +/// A cfg with Windows line ends, a licence key and a database password. +fn cfg() -> String { + format!( + "endpoint_add_tcp \"0.0.0.0:30120\"\r\nendpoint_add_udp \"0.0.0.0:30120\"\r\n\ + sv_hostname \"Lab\"\r\nsv_maxclients 48\r\nsv_licenseKey {KEY}\r\n\ + set mysql_connection_string \"{DATABASE}\"\r\nensure chat\r\n#ensure maps\r\n" + ) +} + +impl Server { + /// A server folder with the resources `chat` and `maps` and the cfg above, saved as the + /// launch settings' folder. The cfg's path. + async fn cfg_folder(&self) -> PathBuf { + let folder = self.dir.path().join("server-data"); + for resource in ["chat", "maps"] { + let resource = folder.join("resources").join(resource); + fs::create_dir_all(&resource).unwrap(); + fs::write(resource.join("fxmanifest.lua"), "fx_version 'cerulean'\n").unwrap(); + } + fs::write(folder.join("server.cfg"), cfg()).unwrap(); + let stored = StoredLaunch { + settings: LaunchSettings { + data_path: folder.to_string_lossy().into_owned(), + cfg_file: "server.cfg".to_owned(), + onesync: OneSync::On, + auto_start: false, + txadmin_events: false, + extra_arguments: Vec::new(), + hang_detection: true, + hang_timeout_seconds: 360, + }, + version: 1, + legacy_build_path: None, + }; + let now = self.clock.now(); + self.db + .write(move |tx| launch::save(tx, "default", &stored, now)) + .await + .unwrap(); + folder.join("server.cfg") + } + + async fn password_again(&self, session: &Session) { + let reply = self + .call( + Method::POST, + "/api/auth/reauth", + Some(session), + Some(json!({ "password": OWNER_PASSWORD })), + ) + .await; + assert_eq!(reply.status, StatusCode::NO_CONTENT, "{}", reply.text); + } + + async fn cfg_document(&self, owner: &Session) -> Value { + let reply = self.call(Method::GET, CFG, Some(owner), None).await; + assert_eq!(reply.status, StatusCode::OK, "{}", reply.text); + reply.json() + } +} + +/// What a review found, as kind and line, but for the endpoints another program of this host +/// holds: whether their port is free is the machine's business, not the editor's. +fn found(review: &Value) -> Vec<(String, u64)> { + review["findings"] + .as_array() + .unwrap() + .iter() + .filter(|finding| finding["kind"] != "portInUse") + .map(|finding| { + ( + finding["kind"].as_str().unwrap().to_owned(), + finding["line"].as_u64().unwrap(), + ) + }) + .collect() +} + +fn holds_no_secret(text: &str) { + assert!(!text.contains("hunter2") && !text.contains("1a2b3c"), "{text}"); +} + +#[tokio::test] +async fn the_cfg_is_the_owners_to_read_with_a_mark_in_place_of_each_secret() { + let server = Server::new().await; + let owner = server.owner().await; + let manager = server.member(&owner, "grace", "ffff").await; + + let unset = server.call(Method::GET, CFG, Some(&owner), None).await; + assert_eq!( + ( + unset.status, + unset.code(), + unset.json()["params"]["reason"].clone() + ), + ( + StatusCode::CONFLICT, + json!("cfg_not_editable"), + json!("notConfigured") + ) + ); + server.cfg_folder().await; + for (method, path, body) in [ + (Method::GET, CFG, None), + (Method::POST, CHECK, Some(json!({ "text": "" }))), + (Method::POST, DIFF, Some(json!({ "text": "" }))), + (Method::PUT, CFG, Some(json!({ "text": "", "base": "" }))), + (Method::GET, VERSIONS, None), + ] { + let refused = server.call(method, path, Some(&manager), body).await; + assert_eq!(refused.status, StatusCode::FORBIDDEN, "{path}"); + assert_eq!(refused.code(), "owner_only", "{path}"); + } + + let reply = server.call(Method::GET, CFG, Some(&owner), None).await; + holds_no_secret(&reply.text); + let document = reply.json(); + assert_eq!(document["file"], "server.cfg"); + assert_eq!(document["secrets"], 2); + assert_eq!(document["versions"], 0); + assert_eq!(document["bytes"], cfg().len()); + assert_eq!(document["base"].as_str().unwrap().len(), 64); + let text = document["text"].as_str().unwrap(); + assert!( + text.contains("sv_licenseKey ") && text.contains("<>") && text.contains("<>") + ); + assert!(!text.contains('\r')); + let outline = &document["check"]["outline"]; + assert_eq!( + outline["resources"], + json!([ + { "line": 7, "name": "chat", "enabled": true, "alone": true, "missing": false }, + { "line": 8, "name": "maps", "enabled": false, "alone": true, "missing": false }, + ]) + ); + assert_eq!(outline["settings"][1]["name"], "sv_maxclients"); + assert_eq!(outline["settings"][1]["value"], "48"); + assert_eq!(outline["settings"][2]["value"], Value::Null, "the licence key"); + assert_eq!(outline["endpoints"][0]["address"], "0.0.0.0:30120"); + assert_eq!(found(&document["check"]), []); + assert_eq!( + (&document["check"]["added"], &document["check"]["removed"]), + (&json!(0), &json!(0)) + ); +} + +#[tokio::test] +async fn a_draft_is_checked_and_compared_with_the_file_without_anything_written() { + let server = Server::new().await; + let owner = server.owner().await; + let path = server.cfg_folder().await; + let document = server.cfg_document(&owner).await; + let draft = document["text"] + .as_str() + .unwrap() + .replace("sv_maxclients 48", "sv_maxclients 64") + + "ensure gone\n"; + + let check = server + .call(Method::POST, CHECK, Some(&owner), Some(json!({ "text": draft }))) + .await; + let diff = server + .call(Method::POST, DIFF, Some(&owner), Some(json!({ "text": draft }))) + .await; + + assert_eq!(check.status, StatusCode::OK, "{}", check.text); + let check = check.json(); + assert_eq!( + check["outline"]["resources"][2], + json!({ "line": 9, "name": "gone", "enabled": true, "alone": true, "missing": true }) + ); + assert_eq!( + found(&check), + [("resourceMissing".to_owned(), 9)], + "the licence key is read behind its mark" + ); + assert_eq!((&check["added"], &check["removed"]), (&json!(2), &json!(1))); + assert_eq!(diff.status, StatusCode::OK, "{}", diff.text); + holds_no_secret(&diff.text); + assert_eq!( + (&diff.json()["added"], &diff.json()["removed"]), + (&json!(2), &json!(1)) + ); + assert_eq!(fs::read_to_string(path).unwrap(), cfg(), "nothing is written"); + + let unknown = server + .call( + Method::POST, + DIFF, + Some(&owner), + Some(json!({ "text": "rcon_password <>\n" })), + ) + .await; + assert_eq!( + ( + unknown.status, + unknown.code(), + unknown.json()["params"]["reason"].clone() + ), + ( + StatusCode::BAD_REQUEST, + json!("invalid_cfg_text"), + json!("unknownMark") + ) + ); +} + +#[tokio::test] +async fn a_save_takes_the_password_again_keeps_the_file_as_it_was_and_is_audited() { + let server = Server::new().await; + let owner = server.owner().await; + let path = server.cfg_folder().await; + server.clock.advance(SignedDuration::from_mins(11)); + let document = server.cfg_document(&owner).await; + let edited = document["text"] + .as_str() + .unwrap() + .replace("sv_hostname \"Lab\"", "sv_hostname \"Caf\u{e9} RP\"") + .replace("#ensure maps", "ensure maps"); + let save = json!({ "text": edited, "base": document["base"] }); + + let without_password = server + .call(Method::PUT, CFG, Some(&owner), Some(save.clone())) + .await; + assert_eq!( + (without_password.status, without_password.code()), + (StatusCode::FORBIDDEN, json!("reauth_required")) + ); + assert_eq!(fs::read_to_string(&path).unwrap(), cfg()); + server.password_again(&owner).await; + let saved = server + .call(Method::PUT, CFG, Some(&owner), Some(save.clone())) + .await; + + assert_eq!(saved.status, StatusCode::OK, "{}", saved.text); + holds_no_secret(&saved.text); + let written = fs::read_to_string(&path).unwrap(); + assert_eq!( + written, + cfg() + .replace("sv_hostname \"Lab\"", "sv_hostname \"Caf\u{e9} RP\"") + .replace("#ensure maps", "ensure maps"), + "the secrets and the line ends are the file's own" + ); + let saved = saved.json(); + assert_eq!(saved["versions"], 1); + assert_ne!(saved["base"], document["base"]); + assert_eq!(saved["check"]["outline"]["resources"][1]["enabled"], true); + let leftovers: Vec<_> = fs::read_dir(path.parent().unwrap()) + .unwrap() + .flatten() + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name != "server.cfg" && name != "resources") + .collect(); + assert_eq!(leftovers, Vec::::new()); + + let audited = server.audited("server_cfg_changed").await; + assert_eq!(audited.len(), 1); + assert_eq!(audited[0]["target"], "server.cfg"); + assert_eq!(audited[0]["details"], json!({ "added": 2, "removed": 2 })); + assert_eq!(audited[0]["actorName"], "ada"); + + let stale = server.call(Method::PUT, CFG, Some(&owner), Some(save)).await; + assert_eq!( + (stale.status, stale.code()), + (StatusCode::CONFLICT, json!("cfg_changed")) + ); + let unknown = json!({ "text": "rcon_password <>\n", "base": saved["base"] }); + let unknown = server.call(Method::PUT, CFG, Some(&owner), Some(unknown)).await; + assert_eq!( + (unknown.status, unknown.code()), + (StatusCode::BAD_REQUEST, json!("invalid_cfg_text")) + ); + assert_eq!(fs::read_to_string(&path).unwrap(), written); + assert_eq!(server.audited("server_cfg_changed").await.len(), 1); +} + +#[tokio::test] +async fn an_earlier_version_is_put_back_and_the_file_it_replaces_is_kept_too() { + let server = Server::new().await; + let owner = server.owner().await; + let path = server.cfg_folder().await; + server.clock.advance(SignedDuration::from_mins(11)); + server.password_again(&owner).await; + let document = server.cfg_document(&owner).await; + let edited = document["text"].as_str().unwrap().replace("48", "64"); + let saved = server + .call( + Method::PUT, + CFG, + Some(&owner), + Some(json!({ "text": edited, "base": document["base"] })), + ) + .await + .json(); + + let versions = server + .call(Method::GET, VERSIONS, Some(&owner), None) + .await + .json(); + let version = versions[0]["id"].as_i64().unwrap(); + assert_eq!(versions.as_array().unwrap().len(), 1); + assert_eq!(versions[0]["replacedBy"], "ada"); + assert_eq!( + (&versions[0]["bytes"], &versions[0]["lines"]), + (&json!(cfg().len()), &json!(8)) + ); + let diff = server + .call(Method::GET, &format!("{VERSIONS}/{version}"), Some(&owner), None) + .await; + holds_no_secret(&diff.text); + assert_eq!( + (&diff.json()["added"], &diff.json()["removed"]), + (&json!(1), &json!(1)) + ); + let missing = server + .call(Method::GET, &format!("{VERSIONS}/999"), Some(&owner), None) + .await; + assert_eq!( + (missing.status, missing.code()), + (StatusCode::NOT_FOUND, json!("cfg_version_not_found")) + ); + + let stale = server + .call( + Method::POST, + &format!("{VERSIONS}/{version}/restore"), + Some(&owner), + Some(json!({ "base": document["base"] })), + ) + .await; + assert_eq!( + (stale.status, stale.code()), + (StatusCode::CONFLICT, json!("cfg_changed")) + ); + let restored = server + .call( + Method::POST, + &format!("{VERSIONS}/{version}/restore"), + Some(&owner), + Some(json!({ "base": saved["base"] })), + ) + .await; + + assert_eq!(restored.status, StatusCode::OK, "{}", restored.text); + assert_eq!(fs::read_to_string(&path).unwrap(), cfg()); + assert_eq!(restored.json()["versions"], 2); + assert_eq!(restored.json()["base"], document["base"]); + let audited = server.audited("server_cfg_restored").await; + assert_eq!( + audited[0]["details"], + json!({ "version": version, "added": 1, "removed": 1 }) + ); +} + +#[tokio::test] +async fn a_cfg_that_cannot_be_written_back_as_it_is_read_is_not_edited() { + let server = Server::new().await; + let owner = server.owner().await; + let path = server.cfg_folder().await; + let reason = |reply: super::Reply| { + ( + reply.status, + reply.code(), + reply.json()["params"]["reason"].clone(), + ) + }; + let refused = |reason: &str| (StatusCode::CONFLICT, json!("cfg_not_editable"), json!(reason)); + + fs::write(&path, b"sv_hostname \"Caf\xe9\"\n").unwrap(); + assert_eq!( + reason(server.call(Method::GET, CFG, Some(&owner), None).await), + refused("notUtf8") + ); + fs::write(&path, "sv_hostname \"<>\"\n").unwrap(); + assert_eq!( + reason(server.call(Method::GET, CFG, Some(&owner), None).await), + refused("holdsMarks") + ); + fs::write(&path, "#".repeat(512 * 1024 + 1)).unwrap(); + assert_eq!( + reason(server.call(Method::GET, CFG, Some(&owner), None).await), + refused("tooLarge") + ); + let mut permissions = fs::metadata(&path).unwrap().permissions(); + permissions.set_readonly(true); + fs::set_permissions(&path, permissions.clone()).unwrap(); + assert_eq!( + reason(server.call(Method::GET, CFG, Some(&owner), None).await), + refused("readOnly") + ); + #[expect( + clippy::permissions_set_readonly_false, + reason = "the test's own file, to delete it" + )] + permissions.set_readonly(false); + fs::set_permissions(&path, permissions).unwrap(); + fs::remove_file(&path).unwrap(); + assert_eq!( + reason(server.call(Method::GET, CFG, Some(&owner), None).await), + refused("missing") + ); + fs::create_dir(&path).unwrap(); + assert_eq!( + reason(server.call(Method::GET, CFG, Some(&owner), None).await), + refused("notAFile") + ); +} diff --git a/crates/observer-web/src/tests/debug.rs b/crates/observer-web/src/tests/debug.rs new file mode 100644 index 00000000..d1b55701 --- /dev/null +++ b/crates/observer-web/src/tests/debug.rs @@ -0,0 +1,142 @@ +//! The Debug routes: off until the owner turns the feature on, read with the diagnostics +//! permission, and empty while no server counts. + +use axum::http::{Method, StatusCode}; +use serde_json::json; + +use super::{Reply, Server, Session}; + +const STATUS: &str = "/api/debug/status"; + +impl Server { + async fn turn_debug(&self, owner: &Session, enabled: bool) -> Reply { + self.call( + Method::PUT, + "/api/features/debug", + Some(owner), + Some(json!({ "enabled": enabled })), + ) + .await + } +} + +#[tokio::test] +async fn debug_is_off_until_the_owner_turns_it_on_and_then_waits_for_a_start() { + let server = Server::new().await; + let owner = server.owner().await; + + let off = server.call(Method::GET, STATUS, Some(&owner), None).await; + let switches = server + .call(Method::GET, "/api/features", Some(&owner), None) + .await; + let turned_on = server.turn_debug(&owner, true).await; + let waiting = server.call(Method::GET, STATUS, Some(&owner), None).await; + + assert_eq!(off.status, StatusCode::NOT_FOUND); + assert_eq!(off.code(), "feature_disabled"); + assert_eq!(off.json()["params"]["feature"], "debug"); + let debug = &switches.json()["features"][8]; + assert_eq!(debug["feature"], "debug"); + assert_eq!( + (&debug["enabled"], &debug["running"]), + (&json!(false), &json!(false)) + ); + assert_eq!(debug["turnOn"], "nextServerStart"); + assert_eq!(debug["reach"]["routes"], json!(["/debug"])); + assert_eq!(turned_on.status, StatusCode::OK, "{}", turned_on.text); + let debug = &turned_on.json()["features"][8]; + assert_eq!( + (&debug["enabled"], &debug["running"]), + (&json!(true), &json!(true)) + ); + assert_eq!(waiting.status, StatusCode::OK, "{}", waiting.text); + assert_eq!( + waiting.json(), + json!({ + "state": "waiting", + "running": false, + "since": null, + "reporting": 0, + "silent": [], + "silentCount": 0, + "refused": null, + }) + ); +} + +#[tokio::test] +async fn the_debug_pages_are_read_with_the_diagnostics_permission_over_a_window() { + let server = Server::new().await; + let owner = server.owner().await; + server.turn_debug(&owner, true).await; + let reader = server.member(&owner, "grace", "1000").await; + let stranger = server.member(&owner, "kenji", "200").await; + + let events = server + .call(Method::GET, "/api/debug/events?window=1m", Some(&reader), None) + .await; + let bags = server + .call(Method::GET, "/api/debug/bags", Some(&reader), None) + .await; + let http = server + .call(Method::GET, "/api/debug/http?window=15m", Some(&reader), None) + .await; + let odd = server + .call(Method::GET, "/api/debug/events?window=2h", Some(&reader), None) + .await; + let refused = server + .call(Method::GET, "/api/debug/events", Some(&stranger), None) + .await; + + assert_eq!(events.status, StatusCode::OK, "{}", events.text); + let events = events.json(); + assert_eq!(events["seconds"], 60); + assert_eq!(events["status"]["state"], "waiting"); + assert_eq!( + events["totals"], + json!({ + "toClients": { "count": 0, "bytes": 0 }, + "fromClients": { "count": 0, "bytes": 0 }, + "local": 0, + "handlerUs": 0, + "refused": 0, + "refusedPlayers": 0, + }) + ); + assert_eq!(events["events"], json!([])); + assert_eq!(events["truncated"], false); + assert_eq!(bags.json()["seconds"], 300, "five minutes when nothing is asked"); + assert_eq!(bags.json()["totals"]["keys"], 0); + assert_eq!(http.json()["seconds"], 900); + assert_eq!(http.json()["learningUntil"], json!(null)); + assert_eq!(odd.status, StatusCode::BAD_REQUEST); + assert_eq!(odd.code(), "invalid_request"); + assert_eq!(refused.status, StatusCode::FORBIDDEN); +} + +#[tokio::test] +async fn only_the_owner_says_a_host_is_known_and_only_of_a_host_a_resource_called() { + let server = Server::new().await; + let owner = server.owner().await; + server.turn_debug(&owner, true).await; + let reader = server.member(&owner, "grace", "1000").await; + let known = json!({ "resource": "logs", "host": "discord.com" }); + + let by_reader = server + .call( + Method::POST, + "/api/debug/http/known", + Some(&reader), + Some(known.clone()), + ) + .await; + let never_called = server + .call(Method::POST, "/api/debug/http/known", Some(&owner), Some(known)) + .await; + server.turn_debug(&owner, false).await; + let off = server.call(Method::GET, STATUS, Some(&owner), None).await; + + assert_eq!(by_reader.code(), "owner_only"); + assert_eq!(never_called.status, StatusCode::NOT_FOUND); + assert_eq!(off.code(), "feature_disabled"); +} diff --git a/crates/observer-web/src/tests/features.rs b/crates/observer-web/src/tests/features.rs index 6f1c0ba2..919767a1 100644 --- a/crates/observer-web/src/tests/features.rs +++ b/crates/observer-web/src/tests/features.rs @@ -11,6 +11,8 @@ use super::{OWNER_PASSWORD, Reply, Server, Session}; use crate::api::routes; const FEATURES: &str = "/api/features"; +const SFTP: &str = "/api/settings/sftp"; +const GROUPS: &str = "/api/servers/default/files/groups"; impl Server { async fn switch(&self, session: &Session, feature: &str, enabled: bool) -> Reply { @@ -62,8 +64,9 @@ async fn the_switches_are_read_with_the_settings_permission_and_say_what_each_fe let sftp = &features[0]; assert_eq!(sftp["feature"], "sftp"); assert_eq!((&sftp["enabled"], &sftp["running"]), (&json!(true), &json!(true))); - assert_eq!(sftp["turnOn"], "observerRestart"); + assert_eq!(sftp["turnOn"], "now"); assert_eq!(sftp["turnOff"], "now"); + assert_eq!(features[3]["turnOn"], "observerRestart"); assert_eq!(sftp["reach"]["listens"], json!(["127.0.0.1:41976"])); assert_eq!(sftp["reach"]["serverFolder"], "writes"); assert_eq!( @@ -200,3 +203,114 @@ async fn the_topics_of_a_feature_turned_off_are_withdrawn_and_refused_while_the_ send(&mut socket, json!({ "type": "sub", "topic": "server.status" })).await; assert_eq!(receive(&mut socket).await["type"], "snapshot"); } + +#[tokio::test] +async fn the_owner_gives_sftp_its_address_from_the_panel_and_takes_it_back() { + let server = Server::unpinned().await; + let owner = server.owner().await; + let manager = server.member(&owner, "grace", "400").await; + let give = |listen: &str| Some(json!({ "listen": listen })); + + let before = server.call(Method::GET, SFTP, Some(&manager), None).await; + let waiting = server.call(Method::GET, GROUPS, Some(&owner), None).await; + let by_manager = server + .call(Method::PUT, SFTP, Some(&manager), give("127.0.0.1:41977")) + .await; + let a_name = server + .call(Method::PUT, SFTP, Some(&owner), give("sftp.example.com:22")) + .await; + let no_port = server + .call(Method::PUT, SFTP, Some(&owner), give("127.0.0.1:0")) + .await; + let given = server + .call(Method::PUT, SFTP, Some(&owner), give(" 127.0.0.1:41977 ")) + .await; + let switches = server.call(Method::GET, FEATURES, Some(&owner), None).await; + let answering = server.call(Method::GET, GROUPS, Some(&owner), None).await; + let taken_back = server + .call(Method::PUT, SFTP, Some(&owner), Some(json!({ "listen": null }))) + .await; + let waiting_again = server.call(Method::GET, GROUPS, Some(&owner), None).await; + + let nothing = json!({ + "listen": null, "pinned": false, "open": false, "fingerprint": null, "refused": null, + }); + assert_eq!(before.status, StatusCode::OK, "{}", before.text); + assert_eq!(before.json(), nothing); + assert_eq!(waiting.code(), "feature_disabled"); + assert_eq!(waiting.json()["params"]["held"], "notConfigured"); + assert_eq!(by_manager.code(), "owner_only"); + assert_eq!(a_name.status, StatusCode::BAD_REQUEST); + assert_eq!(a_name.code(), "invalid_sftp_listen"); + assert_eq!(a_name.json()["params"]["reason"], "address"); + assert_eq!(no_port.json()["params"]["reason"], "port"); + assert_eq!(given.status, StatusCode::OK, "{}", given.text); + // Nothing keeps the port in these tests, so it is given and not open. + assert_eq!( + given.json(), + json!({ + "listen": "127.0.0.1:41977", "pinned": false, "open": false, "fingerprint": null, + "refused": null, + }) + ); + let sftp = &switches.json()["features"][0]; + assert_eq!(sftp["feature"], "sftp"); + assert_eq!((&sftp["running"], &sftp["held"]), (&json!(true), &json!(null))); + assert_eq!(sftp["turnOn"], "now"); + assert_eq!(sftp["reach"]["listens"], json!(["127.0.0.1:41977"])); + assert_eq!(answering.status, StatusCode::OK, "{}", answering.text); + assert_eq!(taken_back.json(), nothing); + assert_eq!(waiting_again.code(), "feature_disabled"); + let audit = server + .call( + Method::GET, + "/api/audit?action=sftp_listen_changed", + Some(&owner), + None, + ) + .await; + let entries = audit.json()["entries"].clone(); + let details: Vec<_> = entries + .as_array() + .unwrap() + .iter() + .map(|entry| (entry["target"].clone(), entry["details"].clone())) + .collect(); + assert_eq!(details.len(), 2, "{entries}"); + assert!(details.contains(&( + json!("sftp"), + json!({ "listen": "127.0.0.1:41977", "previous": null }) + ))); + assert!(details.contains(&( + json!("sftp"), + json!({ "listen": null, "previous": "127.0.0.1:41977" }) + ))); +} + +#[tokio::test] +async fn an_address_observer_toml_names_is_not_the_panels_to_change() { + let server = Server::new().await; + let owner = server.owner().await; + + let shown = server.call(Method::GET, SFTP, Some(&owner), None).await; + let refused = server + .call( + Method::PUT, + SFTP, + Some(&owner), + Some(json!({ "listen": "127.0.0.1:41977" })), + ) + .await; + let after = server.call(Method::GET, SFTP, Some(&owner), None).await; + + assert_eq!( + shown.json(), + json!({ + "listen": "127.0.0.1:41976", "pinned": true, "open": false, "fingerprint": null, + "refused": null, + }) + ); + assert_eq!(refused.status, StatusCode::FORBIDDEN); + assert_eq!(refused.code(), "sftp_listen_pinned"); + assert_eq!(after.json(), shown.json()); +} diff --git a/crates/observer-web/src/tests/map.rs b/crates/observer-web/src/tests/map.rs index 49504869..a7605ee7 100644 --- a/crates/observer-web/src/tests/map.rs +++ b/crates/observer-web/src/tests/map.rs @@ -64,6 +64,7 @@ async fn the_map_opens_to_its_own_permission_and_its_image_to_the_owner() { assert_eq!(seen.status, StatusCode::OK, "{}", seen.text); assert_eq!(seen.json()["image"], json!(null)); assert_eq!(seen.json()["settings"]["everySeconds"], 2); + assert_eq!(seen.json()["settings"]["imagery"], "atlas"); assert_eq!(not_owner.status, StatusCode::FORBIDDEN); assert_eq!(reauth.status, StatusCode::NO_CONTENT, "{}", reauth.text); assert_eq!(added.status, StatusCode::OK, "{}", added.text); @@ -89,6 +90,7 @@ async fn the_settings_take_the_settings_permission_and_are_checked() { let watcher = server.member(&owner, "hopper", PLAYERS_MAP).await; let manager = server.member(&owner, "sarah", SETTINGS_MANAGE).await; let settings = json!({ + "imagery": "satellite", "bounds": { "west": -5640.0, "east": 6470.0, "south": -4140.0, "north": 7970.0 }, "newHours": 5, "veteranHours": 50, diff --git a/crates/observer-web/src/tests/servers.rs b/crates/observer-web/src/tests/servers.rs index 0bc2648d..70bbcb37 100644 --- a/crates/observer-web/src/tests/servers.rs +++ b/crates/observer-web/src/tests/servers.rs @@ -14,6 +14,7 @@ use super::live::{receive, send}; use super::{Server, Session}; const START: &str = "/api/servers/default/start"; +const EXTEND: &str = "/api/servers/default/start/extend"; const STOP: &str = "/api/servers/default/stop"; const RESTART: &str = "/api/servers/default/restart"; const CONSOLE: &str = "/api/servers/default/console"; @@ -90,15 +91,23 @@ async fn starting_stopping_and_restarting_take_server_control_and_say_why_they_c let owner = server.owner().await; let viewer = server.member(&owner, "grace", "3").await; - for path in [START, STOP, RESTART] { + for path in [START, STOP, EXTEND, RESTART] { let refused = server.call(Method::POST, path, Some(&viewer), None).await; assert_eq!(refused.status, StatusCode::FORBIDDEN, "{path}"); assert_eq!(refused.json()["params"]["permission"], "serverControl"); } let denied = server.audited("permission_denied").await; - assert_eq!(denied.len(), 3); + assert_eq!(denied.len(), 4); assert_eq!(denied[0]["target"], format!("POST {RESTART}")); + let extend = server.call(Method::POST, EXTEND, Some(&owner), None).await; + assert_eq!( + (extend.status, extend.code()), + (StatusCode::CONFLICT, json!("server_not_starting")), + "no start to give more time to" + ); + assert_eq!(server.audited("server_start_extended").await, Vec::::new()); + let start = server.call(Method::POST, START, Some(&owner), None).await; let stop = server.call(Method::POST, STOP, Some(&owner), None).await; let restart = server.call(Method::POST, RESTART, Some(&owner), None).await; diff --git a/crates/observer-web/src/tests/tools.rs b/crates/observer-web/src/tests/tools.rs index e7aa6af3..7a5d175c 100644 --- a/crates/observer-web/src/tests/tools.rs +++ b/crates/observer-web/src/tests/tools.rs @@ -16,6 +16,8 @@ const TOOLS: &str = "/api/tools"; const SCAN: &str = "/api/tools/duplicate-archetypes/scan"; const FIX: &str = "/api/tools/duplicate-archetypes/fix"; const UNDO: &str = "/api/tools/duplicate-archetypes/undo"; +const STAGE: &str = "/api/tools/duplicate-archetypes/stage"; +const WARNINGS: &str = "/api/servers/default/start/warnings"; const SETTINGS: &str = "/api/tools/duplicate-archetypes/settings"; impl Server { @@ -93,6 +95,9 @@ async fn the_tools_are_the_owner_s_and_what_writes_takes_the_password_again() { (Method::POST, SCAN, None), (Method::POST, FIX, Some(json!({ "plan": "0".repeat(64) }))), (Method::POST, UNDO, Some(json!({ "scope": "all" }))), + (Method::POST, STAGE, Some(json!({ "plan": "0".repeat(64) }))), + (Method::DELETE, STAGE, None), + (Method::GET, WARNINGS, None), ] { let refused = server.call(method, path, Some(&manager), body).await; assert_eq!(refused.status, StatusCode::FORBIDDEN, "{path}"); @@ -101,6 +106,20 @@ async fn the_tools_are_the_owner_s_and_what_writes_takes_the_password_again() { let page = server.call(Method::GET, TOOLS, Some(&owner), None).await; let scanned = server.call(Method::POST, SCAN, Some(&owner), None).await; let plan = scanned.json()["plan"].clone(); + let warnings = server.call(Method::GET, WARNINGS, Some(&owner), None).await; + assert_eq!( + warnings.json(), + json!({ "startedAt": null, "heavy": [], "missing": [], "truncated": false }), + "FXServer was not started yet" + ); + for (method, body) in [ + (Method::POST, Some(json!({ "plan": plan }))), + (Method::DELETE, None), + ] { + let staging = server.call(method, STAGE, Some(&owner), body).await; + assert_eq!(staging.status, StatusCode::FORBIDDEN); + assert_eq!(staging.code(), "reauth_required"); + } let without_password = [ server .call(Method::POST, FIX, Some(&owner), Some(json!({ "plan": plan }))) diff --git a/crates/observer/src/config.rs b/crates/observer/src/config.rs index 3c02a225..25fd88e5 100644 --- a/crates/observer/src/config.rs +++ b/crates/observer/src/config.rs @@ -21,7 +21,8 @@ pub struct Config { /// started from anywhere finds the same data. pub data_dir: PathBuf, pub http: HttpConfig, - /// SFTP opens a new network service, so it stays off until the owner adds this section. + /// Where SFTP listens, for a host that keeps the address out of the panel's reach. Without + /// this section the owner gives it in the panel. pub sftp: Option, /// The optional features the host turns off, such as `sftp = "off"`: the panel then cannot /// turn them on, even from the owner's account. @@ -203,7 +204,7 @@ mod tests { } #[test] - fn sftp_stays_off_until_its_section_names_where_to_listen() { + fn the_sftp_section_is_absent_by_default_and_names_where_to_listen() { let dir = tempfile::tempdir().unwrap(); let path = dir.path().join("observer.toml"); assert!(Config::load(&path).unwrap().sftp.is_none()); diff --git a/crates/observer/src/desktop/mod.rs b/crates/observer/src/desktop/mod.rs index b5c10036..6f0013ac 100644 --- a/crates/observer/src/desktop/mod.rs +++ b/crates/observer/src/desktop/mod.rs @@ -435,6 +435,7 @@ mod tests { launch, name: None, port: None, + start: None, } } diff --git a/crates/observer/src/main.rs b/crates/observer/src/main.rs index 762698ea..662544d1 100644 --- a/crates/observer/src/main.rs +++ b/crates/observer/src/main.rs @@ -8,6 +8,7 @@ mod lock; mod logging; mod run; mod setup; +mod sftp; mod shutdown; #[cfg(target_os = "linux")] mod systemd; diff --git a/crates/observer/src/run.rs b/crates/observer/src/run.rs index ac8ad034..01414aa6 100644 --- a/crates/observer/src/run.rs +++ b/crates/observer/src/run.rs @@ -8,11 +8,11 @@ use axum::Router; use jiff::tz::TimeZone; use observer_app::{ Accounts, Artifacts, ArtifactsSetup, ArtifactsTask, AuditLog, Backups, BackupsHook, BackupsSetup, Bans, - BansSetup, BeforeSpawn, Changes, ChangesSetup, Clock, DatabaseFiles, Diagnostics, DiagnosticsSetup, - Drops, DropsSetup, Features, FeaturesSetup, LiveMap, MapSetup, Metrics, MetricsSetup, OwnPaths, - PRE_LAUNCH_LIMIT, PasswordCost, Players, PlayersSetup, Running, Schedule, ScheduleSetup, ScheduleTask, - Servers, Setup, Sources, Statistics, Supervision, Switched, Takeover, Tools, ToolsHook, ToolsSetup, - Updates, UpdatesSetup, + BansSetup, BeforeSpawn, Changes, ChangesSetup, Clock, DatabaseFiles, DebugCounts, DebugSetup, + Diagnostics, DiagnosticsSetup, Drops, DropsSetup, Features, FeaturesSetup, LiveMap, MapSetup, Metrics, + MetricsSetup, OwnPaths, PRE_LAUNCH_LIMIT, PasswordCost, Players, PlayersSetup, Running, Schedule, + ScheduleSetup, ScheduleTask, Servers, Setup, Sources, Statistics, Supervision, Switched, Takeover, Tools, + ToolsHook, ToolsSetup, Updates, UpdatesSetup, }; use observer_artifacts::Http; use observer_artifacts::catalog::Catalog; @@ -20,6 +20,7 @@ use observer_artifacts::releases::{ReleaseKey, Releases}; use observer_domain::audit::StopSource; use observer_domain::features::Feature; use observer_fxserver::console::{ConsoleLog, LogLimits}; +use observer_fxserver::supervisor::Counting; use observer_store::{BackupKind, ChangesDatabase, Database}; use tokio::net::TcpListener; use tokio::sync::mpsc; @@ -30,7 +31,7 @@ use tracing::{error, info, warn}; use crate::config::{Config, DataPaths, HttpConfig}; use crate::shutdown::{Signals, StopRequest}; use crate::update::{self, Arrival, Probation, Relaunch, Returned}; -use crate::{BUILD, lock, logging, setup}; +use crate::{BUILD, lock, logging, setup, sftp}; /// In a build with the `test-catalog` feature, the version this Observer takes itself for, so /// that the test release is newer; the Observer it starts does not inherit it. @@ -161,6 +162,8 @@ async fn run_until_stopped( backups: backups_hook.clone(), })); settings.pre_launch_limit = PRE_LAUNCH_LIMIT; + let switches = features.clone(); + settings.counting = Some(Counting::new(move || switches.runs(Feature::Debug))); // A hung server's snapshot is the diagnostics' to read. if !diagnosing_at_start { settings.hang_snapshot = None; @@ -310,6 +313,14 @@ async fn run_until_stopped( _ => (None, None), }; + // Before FXServer boots, so that a start that counts is read from its first report. + let (debug, debugging) = DebugCounts::start(DebugSetup { + db: db.clone(), + clock: clock.clone(), + supervision: supervision.clone(), + features: features.clone(), + }) + .await?; // Before FXServer boots, so that no player gets past the bans. let (bans, banning) = Bans::start(BansSetup { db: db.clone(), @@ -345,21 +356,21 @@ async fn run_until_stopped( if let Err(error) = observer_sftp::sweep_uploads(&files).await { warn!(%error, "the temporary SFTP uploads could not all be removed"); } - // Right after the panel's port, so that a port taken stops Observer before FXServer starts. - let sftp_listener = match &config.sftp { + let sftp_port = sftp::Port { + features: features.clone(), + files: files.clone(), + refused: own_paths.paths().to_vec(), + }; + // Right after the panel's port, so that a port `observer.toml` names and another program + // holds stops Observer before FXServer starts. An address given in the panel is opened by + // the port's keeper instead: taken, it holds SFTP back, never Observer. + let sftp_pinned = match &config.sftp { Some(sftp) if features.runs(Feature::Sftp) => { - let host_key = observer_sftp::host_key(&files) - .await - .context("cannot read or create the SFTP host key")?; - let listener = TcpListener::bind(sftp.listen) + let serving = sftp_port + .open(sftp.listen) .await .with_context(|| format!("cannot listen for SFTP on {}", sftp.listen))?; - info!( - listen = %sftp.listen, - fingerprint = %host_key.fingerprint(observer_sftp::FINGERPRINT), - "SFTP listening" - ); - Some((listener, host_key)) + Some(serving) } _ => None, }; @@ -383,6 +394,7 @@ async fn run_until_stopped( files: files.clone(), tools: tools.clone(), features: features.clone(), + debug, diagnostics, metrics: metrics.clone(), players, @@ -431,24 +443,7 @@ async fn run_until_stopped( files.clone(), stop.clone(), )); - let sftp = sftp_listener.map(|(listener, host_key)| { - let closing = CancellationToken::new(); - let task = tokio::spawn(observer_sftp::serve( - listener, - files.clone(), - host_key, - own_paths.paths().to_vec(), - closing.clone(), - )); - features.switched(Feature::Sftp, move || async move { - closing.cancel(); - match task.await { - Ok(Ok(())) => {} - Ok(Err(error)) => error!(%error, "the SFTP server stopped with an error"), - Err(error) => error!(%error, "the SFTP server task failed"), - } - }) - }); + let sftp = tokio::spawn(sftp::keep(sftp_port, sftp_pinned, stop.clone())); let audit_feed = tokio::spawn(publish_audit(audit.clone(), stop.clone())); let host = Host::start(HostSetup { args, @@ -521,12 +516,15 @@ async fn run_until_stopped( close(dropping).await; // After the supervision: the bridge answers no one once FXServer is down. banning.close().await; + debugging.close().await; // After the supervision: a stop of FXServer ends a snapshot's job, which this joins. close(following).await; // After the supervision, so that FXServer's stop is in the statistics. close(sampling).await; stop.cancel(); - close(sftp).await; + if let Err(error) = sftp.await { + error!(%error, "the task that keeps SFTP's port failed"); + } live.shutdown().await; join(server, [backups, upkeep, audit_feed]).await?; diff --git a/crates/observer/src/sftp.rs b/crates/observer/src/sftp.rs new file mode 100644 index 00000000..7f207cf9 --- /dev/null +++ b/crates/observer/src/sftp.rs @@ -0,0 +1,105 @@ +//! Keeps SFTP's port as its switch and its address say: open while the feature runs, on the +//! address in force, closed the rest of the time. Neither needs Observer restarted. + +use std::net::SocketAddr; +use std::path::PathBuf; + +use anyhow::Context; +use observer_app::{Features, Files, Switched}; +use observer_domain::features::Feature; +use tokio::net::TcpListener; +use tokio_util::sync::CancellationToken; +use tracing::{error, info, warn}; + +/// The port while it is open. +pub(crate) struct Serving { + at: SocketAddr, + /// The fingerprint of the host key clients are shown. + fingerprint: String, + switched: Switched, +} + +/// What serving SFTP takes. +pub(crate) struct Port { + pub(crate) features: Features, + pub(crate) files: Files, + /// Observer's own folders and files, which no session reaches. + pub(crate) refused: Vec, +} + +impl Port { + /// Opens the port on `at` and serves SFTP there, until the feature is turned off or what is + /// returned is closed. + pub(crate) async fn open(&self, at: SocketAddr) -> anyhow::Result { + let host_key = observer_sftp::host_key(&self.files) + .await + .context("cannot read or create the SFTP host key")?; + let listener = TcpListener::bind(at).await?; + let fingerprint = host_key.fingerprint(observer_sftp::FINGERPRINT).to_string(); + info!(listen = %at, %fingerprint, "SFTP listening"); + let closing = CancellationToken::new(); + let task = tokio::spawn(observer_sftp::serve( + listener, + self.files.clone(), + host_key, + self.refused.clone(), + closing.clone(), + )); + let switched = self.features.switched(Feature::Sftp, move || async move { + closing.cancel(); + match task.await { + Ok(Ok(())) => {} + Ok(Err(error)) => error!(%error, "the SFTP server stopped with an error"), + Err(error) => error!(%error, "the SFTP server task failed"), + } + }); + Ok(Serving { + at, + fingerprint, + switched, + }) + } +} + +/// Opens, moves and closes the port until `stop`. `first` is the port `observer.toml` names, +/// which Observer's start opened already. +pub(crate) async fn keep(port: Port, first: Option, stop: CancellationToken) { + let mut running = port.features.running(); + let mut listen = port.features.sftp_listen(); + let mut serving = first; + if let Some(serving) = &serving { + port.features.sftp_opened(serving.at, serving.fingerprint.clone()); + } + loop { + let on = running.borrow_and_update().contains(&Feature::Sftp); + let at = *listen.borrow_and_update(); + let wanted = if on { at } else { None }; + if serving.as_ref().map(|serving| serving.at) != wanted { + if let Some(serving) = serving.take() { + serving.switched.close().await; + port.features.sftp_closed(); + } + if let Some(at) = wanted { + match port.open(at).await { + Ok(opened) => { + port.features.sftp_opened(at, opened.fingerprint.clone()); + serving = Some(opened); + } + Err(error) => { + let error = format!("{error:#}"); + warn!(listen = %at, %error, "SFTP's port did not open"); + port.features.sftp_refused(at, error); + } + } + } + } + tokio::select! { + () = stop.cancelled() => break, + changed = running.changed() => if changed.is_err() { break }, + changed = listen.changed() => if changed.is_err() { break }, + } + } + if let Some(serving) = serving { + serving.switched.close().await; + } +} diff --git a/crates/observer/src/systemd.rs b/crates/observer/src/systemd.rs index f259f2a4..9102b401 100644 --- a/crates/observer/src/systemd.rs +++ b/crates/observer/src/systemd.rs @@ -453,6 +453,7 @@ mod tests { launch: observer_domain::server::LaunchReadiness::Ready, name: None, port: None, + start: None, }; assert_eq!(status_line(&status), "FXServer online · 12 / 64 players"); diff --git a/crates/observer/tests/run.rs b/crates/observer/tests/run.rs index 876a6d6f..37376bcd 100644 --- a/crates/observer/tests/run.rs +++ b/crates/observer/tests/run.rs @@ -12,6 +12,8 @@ use std::process::{Child, Command, Stdio}; use std::sync::mpsc; use std::time::{Duration, Instant}; +use observer_fxserver::hook; +use observer_fxserver::layout::{Flavour, Layout}; use observer_store::Database; use observer_store::artifacts::{self, ArtifactState}; use serde_json::{Value, json}; @@ -513,6 +515,79 @@ fn sftp_turned_off_in_the_panel_closes_its_port_at_once_and_at_the_next_start() ); } +/// Without `[sftp]` in `observer.toml`, the owner gives SFTP its address in the panel: the port +/// opens at once and moves with the address. A port another program holds is held against SFTP, +/// never against Observer, and the address outlives Observer until it is taken back. +#[test] +fn sftp_takes_its_address_from_the_panel_and_its_port_follows_without_a_restart() { + let (mut observer, owner) = Observer::start(); + let panel = observer.port; + let give = |port: u16| { + let listen = json!({ "listen": format!("127.0.0.1:{port}") }); + call(panel, "PUT", "/settings/sftp", Some(&owner), Some(listen)) + }; + let (first, second) = (free_port(), free_port()); + + let waiting = call(panel, "GET", "/settings/sftp", Some(&owner), None); + let given = give(first); + let opened = TcpStream::connect(("127.0.0.1", first)).is_ok(); + let moved = give(second); + + assert_eq!( + waiting.body, + json!({ "listen": null, "pinned": false, "open": false, "fingerprint": null, "refused": null }) + ); + assert_eq!(given.status, 200, "{}", given.body); + assert_eq!(given.body["open"], true, "{}", given.body); + let fingerprint = given.body["fingerprint"].as_str().unwrap(); + assert!(fingerprint.starts_with("SHA256:"), "{fingerprint}"); + assert!(observer.log().contains(fingerprint), "the log names the same key"); + assert!(opened, "{}", observer.log()); + assert_eq!(moved.body["open"], true, "{}", moved.body); + assert_eq!( + moved.body["fingerprint"], fingerprint, + "the key stays, wherever the port is" + ); + assert!(TcpStream::connect(("127.0.0.1", second)).is_ok()); + assert!( + TcpListener::bind(("127.0.0.1", first)).is_ok(), + "the port it left is free by the time the panel hears back" + ); + + let held = TcpListener::bind("127.0.0.1:0").unwrap(); + let taken = held.local_addr().unwrap().port(); + let refused = give(taken); + let switches = call(panel, "GET", "/features", Some(&owner), None); + drop(held); + let again = give(taken); + + assert_eq!(refused.status, 200, "{}", refused.body); + assert_eq!(refused.body["open"], false, "{}", refused.body); + assert!(refused.body["refused"].is_string(), "{}", refused.body); + let sftp = &switches.body["features"][0]; + assert_eq!( + (&sftp["running"], &sftp["held"]), + (&json!(false), &json!("portRefused")) + ); + assert_eq!(again.body["open"], true, "{}", again.body); + assert_eq!(again.body["refused"], Value::Null); + + observer.respawn(); + wait_for( + "SFTP listening where the owner said, after a restart of Observer", + || TcpStream::connect(("127.0.0.1", taken)).is_ok(), + ); + let back = json!({ "listen": null }); + let taken_back = call(observer.port, "PUT", "/settings/sftp", Some(&owner), Some(back)); + + assert_eq!(taken_back.status, 200, "{}", taken_back.body); + assert_eq!(taken_back.body["listen"], Value::Null); + assert!( + TcpListener::bind(("127.0.0.1", taken)).is_ok(), + "the port is closed by the time the panel hears back" + ); +} + /// `[features] sftp = "off"` keeps SFTP's port closed though `[sftp]` names it, and even the /// owner's session cannot open it: the refusal is audited. #[test] @@ -547,3 +622,85 @@ fn a_feature_observer_toml_turns_off_never_runs_and_the_panel_cannot_turn_it_on( "{log}" ); } + +/// The owner turns Debug on: the next start puts Observer's two lines into the build's scheduler, +/// what a resource counts reads on the Events page, and a started resource that runs no Lua on the +/// server is named as one that does not report. The bridge, Observer's own, is neither. +#[test] +fn debug_turned_on_in_the_panel_counts_from_the_next_start_and_reads_on_its_pages() { + let (observer, owner) = Observer::start(); + let (build, data) = observer.lay_out_fake(); + let scheduler = hook::scheduler(&Layout::new(&build, Flavour::HOST)); + fs::create_dir_all(scheduler.parent().unwrap()).unwrap(); + fs::write( + &scheduler, + "Citizen.SetEventRoutine(function() end) +", + ) + .unwrap(); + for resource in ["chat", "clothes"] { + let folder = data.join("resources").join(resource); + fs::create_dir_all(&folder).unwrap(); + fs::write( + folder.join("fxmanifest.lua"), + "fx_version 'cerulean' +game 'gta5' +", + ) + .unwrap(); + } + fs::write( + data.join("d.cfg"), + "exec server.cfg +set fake_debug_silent clothes +ensure chat +ensure clothes + fake_print running d.cfg +", + ) + .unwrap(); + observer.select(35245); + observer.save(&owner, &data, "d.cfg"); + let get = |path: &str| call(observer.port, "GET", path, Some(&owner), None); + let post = |path: &str, body: Option| call(observer.port, "POST", path, Some(&owner), body); + + let off = get("/debug/status"); + let on = json!({ "enabled": true }); + let turned_on = call(observer.port, "PUT", "/features/debug", Some(&owner), Some(on)); + assert_eq!(turned_on.status, 200, "{}", turned_on.body); + let started = post("/servers/default/start", None); + assert_eq!(started.status, 204, "{}", started.body); + wait_for("the resources' first reports", || { + let status = get("/debug/status").body; + status["reporting"] == 1 && status["silentCount"] == 1 + }); + let row = json!({ "line": "fake_debug chat ev toA chat:addMessage 4 1240 310 0" }); + let typed = post("/servers/default/console", Some(row)); + assert_eq!(typed.status, 200, "{}", typed.body); + wait_for("the chat's event on the page", || { + get("/debug/events?window=1m").body["events"][0]["name"] == "chat:addMessage" + }); + + assert_eq!(off.status, 404, "{}", off.body); + assert_eq!(off.body["code"], "feature_disabled"); + let counted = fs::read_to_string(&scheduler).unwrap(); + assert!(counted.starts_with("--[[obs:hook:head "), "{counted}"); + let page = get("/debug/events?window=1m").body; + assert_eq!(page["status"]["state"], "counting", "{page}"); + assert_eq!( + page["status"]["silent"], + json!(["clothes"]), + "the bridge is no resource of the server's" + ); + let event = &page["events"][0]; + assert_eq!( + ( + &event["way"], + &event["resources"], + &event["count"], + &event["bytes"] + ), + (&json!("toAll"), &json!(["chat"]), &json!(4), &json!(1240)) + ); + assert_eq!(page["totals"]["toClients"], json!({ "count": 4, "bytes": 1240 })); +} diff --git a/resource/observer/hook.lua b/resource/observer/hook.lua new file mode 100644 index 00000000..ecfc633c --- /dev/null +++ b/resource/observer/hook.lua @@ -0,0 +1,726 @@ +-- Counts what a resource sends and receives, for Observer's Debug pages: events, state bag writes +-- and outgoing HTTP requests. Names, counts and sizes: never a payload, a value, a path or a query. +-- +-- This file is no script of the bridge, and the manifest does not list it. While the owner keeps +-- Debug on, Observer adds two lines to the build's citizen/scripting/lua/scheduler.lua at each +-- start: the first loads this file into every server Lua runtime, one per resource, and keeps the +-- table it returns; the last calls that table's `init` with the scheduler's event handlers. +-- +-- It runs inside other people's resources, so: it returns its table and never raises at the top +-- level, creates no global, allocates nothing where an event passes, never calls a resource's code +-- under pcall (an error must read as it always did), and never looks into a resource's functions. + +local bridge = ... + +local WIRE_TYPE = 'observer_debug' +local WIRE_VERSION = 1 +local HOOK_VERSION = '1' +local BRIDGE_CONVAR = 'observer_bridge' +-- What is counted under once a window holds more distinct names than it keeps. +local OVERFLOW = '__overflow__' +local FLUSH_MS = 1000 +local MAX_FLUSH_ERRORS = 10 +local MAX_EVENTS = 400 +local MAX_SENDERS = 200 +local MAX_REFUSED = 200 +local MAX_BAGS = 200 +local MAX_HOSTS = 50 +local LARGEST = 10 +-- A length no script meant: the string's own is taken instead. +local MAX_LENGTH = 4294967296 + +-- Taken before any resource runs, which may replace them. +local _G = _G +local Citizen = Citizen +local type = type +local pairs = pairs +local next = next +local tostring = tostring +local tonumber = tonumber +local rawget = rawget +local rawset = rawset +local pcall = pcall +local str_byte = string.byte +local str_sub = string.sub +local str_match = string.match +local str_upper = string.upper +local str_lower = string.lower +local math_floor = math.floor +local osLib = rawget(_G, 'os') +local citizenWait = type(Citizen) == 'table' and Citizen.Wait or nil +local createThread = type(Citizen) == 'table' and Citizen.CreateThread or nil + +local idle = { init = function() end } + +-- The server alone, and only the one Observer started with this very bridge: the same build may +-- run elsewhere, where nobody reads the reports. +local function startedByObserver() + return IsDuplicityVersion() and GetConvar(BRIDGE_CONVAR, '') == bridge +end +local asked, mine = pcall(startedByObserver) +if not asked or not mine or type(bridge) ~= 'string' or type(Citizen) ~= 'table' then + return idle +end + +-- The bridge is Observer's own: what it sends and receives is not the server's. +local function inTheBridge() + return GetCurrentResourceName() == bridge +end +local named, own = pcall(inTheBridge) +if named and own then + return idle +end + +-- The clock stays in its own unit where an event passes; microseconds are worked out at each report. +local clock, clockToUs, timerKind +do + local function probe(fn) + if type(fn) ~= 'function' then + return false + end + local ok, value = pcall(fn) + return ok and type(value) == 'number' + end + local function gameTimer() + return GetGameTimer() + end + local hasOs = type(osLib) == 'table' + if hasOs and probe(osLib.nanotime) then + clock, clockToUs, timerKind = osLib.nanotime, 0.001, 'nanotime' + elseif hasOs and probe(osLib.microtime) then + clock, clockToUs, timerKind = osLib.microtime, 1, 'microtime' + elseif hasOs and probe(osLib.clock) then + clock, clockToUs, timerKind = osLib.clock, 1000000, 'clock' + elseif probe(gameTimer) then + clock, clockToUs, timerKind = gameTimer, 1000, 'gametimer' + else + clock, clockToUs, timerKind = function() + return 0 + end, 1, 'none' + end +end + +-- Every wrapper passes straight through until `init`, and again once reporting broke. +local enabled = false +local initialized = false +local eventHandlers = nil +local jsonEncode = nil +local natives = {} +local seq = 0 +local lastFlush = 0 +local helloPending = true + +-- How deep the events dispatched inside one another are, to take a handler's own time alone. +local depth = 0 +local childTime = {} + +-- The window being counted, emptied at each report. +local events, eventKeys -- events[way][name] = { count, bytes, largest, time, longest } +local senders, senderKeys -- senders[name][player] = { count, bytes } +local refused, refusedKeys -- refused[name][player] = count +local bags, bagKeys -- bags[kind][key] = { count, bytes, largest, replicated } +local largest, largestCount, smallest -- { bag, key, bytes, replicated }, the smallest one's index +local hosts, hostKeys -- hosts[method][host] = { count, bytes } +local overflow + +local function emptyWindow() + events, eventKeys = {}, 0 + senders, senderKeys = {}, 0 + refused, refusedKeys = {}, 0 + bags, bagKeys = {}, 0 + largest, largestCount, smallest = {}, 0, 1 + hosts, hostKeys = {}, 0 + overflow = 0 +end +emptyWindow() + +local function global(name) + -- Through the natives loader's __index when the native was not asked for yet. + return _G[name] +end + +local function native(name) + local fn = natives[name] + if fn == nil then + local ok, value = pcall(global, name) + if ok and type(value) == 'function' then + fn = value + natives[name] = value + end + end + return fn +end + +local function lengthOf(length, data) + if type(length) == 'number' and length >= 0 and length < MAX_LENGTH then + return length + end + return type(data) == 'string' and #data or 0 +end + +-- A player's id as an integer, or nil when it is no id. +local function playerOf(value) + local id = tonumber(value) + if id == nil or not (id >= -1 and id < MAX_LENGTH) then + return nil + end + return math_floor(id) +end + +-- The counts of an event, and the name they are kept under. +local function eventOf(way, name) + local byName = events[way] + if byName == nil then + byName = {} + events[way] = byName + end + local counts = byName[name] + if counts == nil then + if eventKeys >= MAX_EVENTS then + overflow = overflow + 1 + name = OVERFLOW + counts = byName[name] + end + if counts == nil then + counts = { 0, 0, 0, 0, 0 } + byName[name] = counts + eventKeys = eventKeys + 1 + end + end + return counts, name +end + +local function countSender(name, player, length) + local byPlayer = senders[name] + local counts = byPlayer and byPlayer[player] + if counts == nil then + if senderKeys >= MAX_SENDERS then + overflow = overflow + 1 + name = OVERFLOW + byPlayer = senders[name] + counts = byPlayer and byPlayer[player] + end + if counts == nil then + if byPlayer == nil then + byPlayer = {} + senders[name] = byPlayer + end + counts = { 0, 0 } + byPlayer[player] = counts + senderKeys = senderKeys + 1 + end + end + counts[1] = counts[1] + 1 + counts[2] = counts[2] + length +end + +local function countRefused(name, player) + local byPlayer = refused[name] + local count = byPlayer and byPlayer[player] + if count == nil then + if refusedKeys >= MAX_REFUSED then + overflow = overflow + 1 + name = OVERFLOW + byPlayer = refused[name] + count = byPlayer and byPlayer[player] + end + if count == nil then + if byPlayer == nil then + byPlayer = {} + refused[name] = byPlayer + end + count = 0 + refusedKeys = refusedKeys + 1 + end + end + byPlayer[player] = count + 1 +end + +local function keepLargest(bag, key, length, replicated) + if largestCount < LARGEST then + largestCount = largestCount + 1 + largest[largestCount] = { bag, key, length, replicated } + else + local least = largest[smallest] + if length <= least[3] then + return + end + least[1], least[2], least[3], least[4] = bag, key, length, replicated + end + local index, least = 1, largest[1][3] + for i = 2, largestCount do + if largest[i][3] < least then + index, least = i, largest[i][3] + end + end + smallest = index +end + +local function countSend(one, all, name, target, payload, payloadLength) + local length = lengthOf(payloadLength, payload) + local counts = eventOf(playerOf(target) == -1 and all or one, type(name) == 'string' and name or tostring(name)) + counts[1] = counts[1] + 1 + counts[2] = counts[2] + length + if length > counts[3] then + counts[3] = length + end +end + +local function countBagWrite(bagName, keyName, value, valueLength, replicated) + local length = lengthOf(valueLength, value) + local bag = type(bagName) == 'string' and bagName or tostring(bagName) + local key = type(keyName) == 'string' and keyName or tostring(keyName) + local sent = (replicated == true or (type(replicated) == 'number' and replicated ~= 0)) and 1 or 0 + + local kind = 'other' + local first = str_byte(bag, 1) + if first == 103 and bag == 'global' then + kind = 'global' + elseif first == 112 and str_sub(bag, 1, 7) == 'player:' then + kind = 'player' + elseif first == 101 and str_sub(bag, 1, 7) == 'entity:' then + kind = 'entity' + elseif first == 108 and str_sub(bag, 1, 12) == 'localEntity:' then + kind = 'localEntity' + end + + local byKey = bags[kind] + if byKey == nil then + byKey = {} + bags[kind] = byKey + end + local name = key + local counts = byKey[name] + if counts == nil then + if bagKeys >= MAX_BAGS then + overflow = overflow + 1 + name = OVERFLOW + counts = byKey[name] + end + if counts == nil then + counts = { 0, 0, 0, 0 } + byKey[name] = counts + bagKeys = bagKeys + 1 + end + end + counts[1] = counts[1] + 1 + counts[2] = counts[2] + length + if length > counts[3] then + counts[3] = length + end + counts[4] = counts[4] + sent + + if largestCount < LARGEST or length > largest[smallest][3] then + keepLargest(bag, key, length, sent) + end +end + +local function countRequest(request) + if type(request) ~= 'table' then + return + end + local method = request.method + if type(method) == 'string' and #method > 0 and #method <= 16 then + method = str_upper(method) + else + method = 'GET' + end + + -- The host alone: a path or a query may hold a webhook's token or a key. + local host = '?' + local url = request.url + if type(url) == 'string' then + local authority = str_match(url, '^%a[%w+.-]*://([^/?#]*)') + if authority ~= nil then + authority = str_match(authority, '^.*@(.*)$') or authority + local name = str_match(authority, '^%[([^%]]*)%]') or str_match(authority, '^([^:]*)') + if name ~= nil and name ~= '' then + host = str_lower(name) + end + end + end + + local body = request.data + local length = type(body) == 'string' and #body or 0 + local byHost = hosts[method] + if byHost == nil then + byHost = {} + hosts[method] = byHost + end + local counts = byHost[host] + if counts == nil then + if hostKeys >= MAX_HOSTS then + overflow = overflow + 1 + host = OVERFLOW + counts = byHost[host] + end + if counts == nil then + counts = { 0, 0 } + byHost[host] = counts + hostKeys = hostKeys + 1 + end + end + counts[1] = counts[1] + 1 + counts[2] = counts[2] + length +end + +-- Passes on what the dispatch returned, untouched and without a table. +local function dispatched(started, level, counts, ...) + local total = clock() - started + depth = level - 1 + if level > 1 then + childTime[level - 1] = (childTime[level - 1] or 0) + total + end + local own = total - (childTime[level] or 0) + if own < 0 then + own = 0 + end + counts[4] = counts[4] + own + if own > counts[5] then + counts[5] = own + end + return ... +end + +local function triggered(started, level, counts, ...) + local total = clock() - started + depth = level - 1 + if level > 1 then + childTime[level - 1] = (childTime[level - 1] or 0) + total + end + counts[4] = counts[4] + total + if total > counts[5] then + counts[5] = total + end + return ... +end + +-- The scheduler's own routine, counted on its way in. The source is told apart as the scheduler +-- tells it: `net:` from a client, `internal-net:` from the engine, anything else local. +local function countedRoutine(routine) + return function(eventName, eventPayload, eventSource) + if not enabled then + return routine(eventName, eventPayload, eventSource) + end + + local way = 'in' + local player = nil + if type(eventSource) == 'string' then + local first = str_byte(eventSource, 1) + if first == 110 and str_sub(eventSource, 1, 3) == 'net' then + way = 'inN' + player = playerOf(str_sub(eventSource, 5)) + elseif first == 105 and str_sub(eventSource, 1, 12) == 'internal-net' then + way = 'inI' + end + end + + local name = eventName + if type(name) ~= 'string' then + name = tostring(name) + end + if eventHandlers ~= nil then + local entry = rawget(eventHandlers, eventName) + local handlers = type(entry) == 'table' and entry.handlers or nil + if handlers == nil then + return routine(eventName, eventPayload, eventSource) + end + if way == 'inN' and not entry.safeForNet then + -- The scheduler is about to refuse it: no resource opened this event to clients. + countRefused(name, player or -1) + elseif type(handlers) ~= 'table' or next(handlers) == nil then + return routine(eventName, eventPayload, eventSource) + end + end + + local length = type(eventPayload) == 'string' and #eventPayload or 0 + local counts + counts, name = eventOf(way, name) + counts[1] = counts[1] + 1 + counts[2] = counts[2] + length + if length > counts[3] then + counts[3] = length + end + if player ~= nil then + countSender(name, player, length) + end + + local level = depth + 1 + depth = level + childTime[level] = 0 + return dispatched(clock(), level, counts, routine(eventName, eventPayload, eventSource)) + end +end + +-- Each wrapper passes on any argument a later scheduler may add. +local function countedTriggerEvent(original) + return function(eventName, eventPayload, payloadLength, ...) + if not enabled then + return original(eventName, eventPayload, payloadLength, ...) + end + local length = lengthOf(payloadLength, eventPayload) + local counts = eventOf('loc', type(eventName) == 'string' and eventName or tostring(eventName)) + counts[1] = counts[1] + 1 + counts[2] = counts[2] + length + if length > counts[3] then + counts[3] = length + end + local level = depth + 1 + depth = level + childTime[level] = 0 + return triggered(clock(), level, counts, original(eventName, eventPayload, payloadLength, ...)) + end +end + +local function countedTriggerClientEvent(original) + return function(eventName, eventTarget, eventPayload, payloadLength, ...) + if enabled then + countSend('toC', 'toA', eventName, eventTarget, eventPayload, payloadLength) + end + return original(eventName, eventTarget, eventPayload, payloadLength, ...) + end +end + +local function countedTriggerLatentClientEvent(original) + return function(eventName, eventTarget, eventPayload, payloadLength, ...) + if enabled then + countSend('lat', 'latA', eventName, eventTarget, eventPayload, payloadLength) + end + return original(eventName, eventTarget, eventPayload, payloadLength, ...) + end +end + +local function countedSetStateBagValue(original) + return function(bagName, keyName, value, valueLength, replicated, ...) + if enabled then + countBagWrite(bagName, keyName, value, valueLength, replicated) + end + return original(bagName, keyName, value, valueLength, replicated, ...) + end +end + +local function countedHttpRequest(original) + return function(request, ...) + if enabled then + countRequest(request) + end + return original(request, ...) + end +end + +local wrapped = { + { name = 'TriggerEventInternal', count = countedTriggerEvent, done = false }, + { name = 'TriggerClientEventInternal', count = countedTriggerClientEvent, done = false }, + { name = 'TriggerLatentClientEventInternal', count = countedTriggerLatentClientEvent, done = false }, + { name = 'SetStateBagValue', count = countedSetStateBagValue, done = false }, + { name = 'PerformHttpRequestInternalEx', count = countedHttpRequest, done = false }, +} +local retried = false + +local function wrapNatives() + for i = 1, #wrapped do + local one = wrapped[i] + if not one.done then + local ok, original = pcall(global, one.name) + if ok and type(original) == 'function' then + rawset(_G, one.name, one.count(original)) + one.done = true + end + end + end +end + +local function encoder() + local jsonLib = rawget(_G, 'json') + if type(jsonLib) == 'table' and type(jsonLib.encode) == 'function' then + return jsonLib.encode + end + return nil +end + +local function microseconds(value) + return math_floor(value * clockToUs) +end + +local function flush() + if not retried then + -- A native the runtime had not loaded at `init` is there by the first report. + retried = true + wrapNatives() + end + depth = 0 + + local now = clock() + local elapsed = now - lastFlush + lastFlush = now + -- Taken first, so that an error below cannot make the same window fail again. + local wEvents, wSenders, wRefused, wBags, wLargest, wLargestCount, wHosts, wOverflow = + events, senders, refused, bags, largest, largestCount, hosts, overflow + emptyWindow() + + local hello = helloPending + if not hello and wOverflow == 0 and next(wEvents) == nil and next(wSenders) == nil + and next(wRefused) == nil and next(wBags) == nil and next(wHosts) == nil + then + return + end + + local report = { + type = WIRE_TYPE, + v = WIRE_VERSION, + seq = seq + 1, + dur = math_floor(elapsed * clockToUs / 1000), + } + + local list, count = {}, 0 + for way, byName in pairs(wEvents) do + for name, counts in pairs(byName) do + count = count + 1 + list[count] = { + way, name, counts[1], math_floor(counts[2]), math_floor(counts[3]), + microseconds(counts[4]), microseconds(counts[5]), + } + end + end + if count > 0 then + report.ev = list + end + + list, count = {}, 0 + for name, byPlayer in pairs(wSenders) do + for player, counts in pairs(byPlayer) do + count = count + 1 + list[count] = { name, player, counts[1], math_floor(counts[2]) } + end + end + if count > 0 then + report.np = list + end + + list, count = {}, 0 + for name, byPlayer in pairs(wRefused) do + for player, times in pairs(byPlayer) do + count = count + 1 + list[count] = { name, player, times } + end + end + if count > 0 then + report.un = list + end + + list, count = {}, 0 + for kind, byKey in pairs(wBags) do + for key, counts in pairs(byKey) do + count = count + 1 + list[count] = { kind, key, counts[1], math_floor(counts[2]), math_floor(counts[3]), counts[4] } + end + end + if count > 0 then + report.sb = list + end + + if wLargestCount > 0 then + list = {} + for i = 1, wLargestCount do + local write = wLargest[i] + list[i] = { write[1], write[2], math_floor(write[3]), write[4] } + end + report.sbl = list + end + + list, count = {}, 0 + for method, byHost in pairs(wHosts) do + for host, counts in pairs(byHost) do + count = count + 1 + list[count] = { method, host, counts[1], math_floor(counts[2]) } + end + end + if count > 0 then + report.http = list + end + + if wOverflow > 0 then + report.ovf = wOverflow + end + if hello then + report.hello = { hook = HOOK_VERSION, timer = timerKind } + end + + local encode = jsonEncode or encoder() + local trace = native('PrintStructuredTrace') + if encode == nil or trace == nil then + error('json.encode or PrintStructuredTrace is missing') + end + trace(encode(report)) + seq = seq + 1 + if hello then + helloPending = false + end +end + +local function flushForever() + local wait = citizenWait + if type(wait) ~= 'function' then + return + end + local errors = 0 + while true do + wait(FLUSH_MS) + if pcall(flush) then + errors = 0 + else + errors = errors + 1 + if errors >= MAX_FLUSH_ERRORS then + -- Nothing would ever report what the wrappers count: they pass through again. + enabled = false + emptyWindow() + return + end + end + end +end + +-- Called by the line Observer adds at the end of scheduler.lua, once the scheduler defined +-- everything. +local function init(context) + if initialized then + return + end + initialized = true + if type(context) == 'table' and type(context.eventHandlers) == 'table' then + eventHandlers = context.eventHandlers + end + + -- Now, before a resource's own script can replace them. + jsonEncode = encoder() + native('PrintStructuredTrace') + wrapNatives() + if type(citizenWait) ~= 'function' then + citizenWait = Citizen.Wait + end + if type(createThread) ~= 'function' then + createThread = Citizen.CreateThread + end + -- Without its thread nothing could be reported: every wrapper keeps passing through. + if type(citizenWait) ~= 'function' or type(createThread) ~= 'function' then + return + end + lastFlush = clock() + enabled = true + createThread(flushForever) +end + +-- Now, before the scheduler hands the runtime its routine: that one alone is wrapped, and the +-- function is put back at once. +local setEventRoutine = Citizen.SetEventRoutine +if type(setEventRoutine) == 'function' then + Citizen.SetEventRoutine = function(routine) + Citizen.SetEventRoutine = setEventRoutine + if type(routine) ~= 'function' then + return setEventRoutine(routine) + end + return setEventRoutine(countedRoutine(routine)) + end +end + +return { init = init } diff --git a/web/src/api/cfg.ts b/web/src/api/cfg.ts new file mode 100644 index 00000000..a0406de6 --- /dev/null +++ b/web/src/api/cfg.ts @@ -0,0 +1,67 @@ +import { queryOptions } from '@tanstack/react-query'; + +import { getJson, request, send } from './client'; +import type { CfgDocument } from './generated/CfgDocument'; +import type { CfgDraft } from './generated/CfgDraft'; +import type { CfgRestore } from './generated/CfgRestore'; +import type { CfgReview } from './generated/CfgReview'; +import type { CfgSave } from './generated/CfgSave'; +import type { CfgVersion } from './generated/CfgVersion'; +import type { TextDiff } from './generated/TextDiff'; + +const PATH = '/servers/default/cfg'; + +/** + * The launch's cfg, its secrets as marks. Read once for an edit: a save names the file it started + * from, and is refused when the file is another by then. + */ +export const cfgQuery = queryOptions({ + queryKey: ['cfg'], + queryFn: ({ signal }) => getJson(PATH, signal), + staleTime: Infinity, + refetchOnWindowFocus: false, + retry: false, +}); + +const draft = async (path: string, text: string, signal: AbortSignal) => + (await (await request('POST', path, { body: { text } satisfies CfgDraft, signal })).json()) as T; + +/** What a text of the cfg sets and starts and what its scan finds. Nothing is written. */ +export const cfgReviewQuery = (text: string) => + queryOptions({ + queryKey: ['cfg', 'review', text], + queryFn: ({ signal }) => draft(`${PATH}/check`, text, signal), + staleTime: Infinity, + gcTime: 0, + retry: false, + }); + +/** The lines a save of the text would change in the file. Nothing is written. */ +export const cfgDiffQuery = (text: string) => + queryOptions({ + queryKey: ['cfg', 'diff', text], + queryFn: ({ signal }) => draft(`${PATH}/diff`, text, signal), + staleTime: Infinity, + gcTime: 0, + retry: false, + }); + +/** Refused with `cfg_changed` when the file is no longer the one `base` names. */ +export const saveCfg = (save: CfgSave) => send('PUT', PATH, save); + +export const cfgVersionsQuery = queryOptions({ + queryKey: ['cfg', 'versions'], + queryFn: ({ signal }) => getJson(`${PATH}/versions`, signal), +}); + +/** The lines putting a version back would change in the file. */ +export const cfgVersionDiffQuery = (id: number, base: string) => + queryOptions({ + queryKey: ['cfg', 'versions', id, base], + queryFn: ({ signal }) => getJson(`${PATH}/versions/${id}`, signal), + staleTime: Infinity, + retry: false, + }); + +export const restoreCfg = (id: number, restore: CfgRestore) => + send('POST', `${PATH}/versions/${id}/restore`, restore); diff --git a/web/src/api/debug.ts b/web/src/api/debug.ts new file mode 100644 index 00000000..4d6d4de1 --- /dev/null +++ b/web/src/api/debug.ts @@ -0,0 +1,41 @@ +import { queryOptions, type Query } from '@tanstack/react-query'; + +import { getJson, send } from './client'; +import type { DebugBags } from './generated/DebugBags'; +import type { DebugEvents } from './generated/DebugEvents'; +import type { DebugHttp } from './generated/DebugHttp'; +import type { DebugWindow } from './generated/DebugWindow'; +import type { KnownHost } from './generated/KnownHost'; + +/** The resources report every second, and Observer totals them in slices of five. */ +const EVERY_MS = 2_000; + +/** A page asks again while it is live, and stops once its feature answers that it is off. */ +const every = + (live: boolean) => + ({ state }: Pick) => + live && state.status !== 'error' ? EVERY_MS : false; + +export const debugEventsQuery = (window: DebugWindow, live: boolean) => + queryOptions({ + queryKey: ['debug', 'events', window], + queryFn: ({ signal }) => getJson(`/debug/events?window=${window}`, signal), + refetchInterval: every(live), + }); + +export const debugBagsQuery = (window: DebugWindow, live: boolean) => + queryOptions({ + queryKey: ['debug', 'bags', window], + queryFn: ({ signal }) => getJson(`/debug/bags?window=${window}`, signal), + refetchInterval: every(live), + }); + +export const debugHttpQuery = (window: DebugWindow, live: boolean) => + queryOptions({ + queryKey: ['debug', 'http', window], + queryFn: ({ signal }) => getJson(`/debug/http?window=${window}`, signal), + refetchInterval: every(live), + }); + +/** The owner knows why this resource calls this host: it stops showing as new. */ +export const knowHost = (known: KnownHost) => send('POST', '/debug/http/known', known); diff --git a/web/src/api/features.ts b/web/src/api/features.ts index 119a71ae..e165ca70 100644 --- a/web/src/api/features.ts +++ b/web/src/api/features.ts @@ -4,6 +4,8 @@ import { getJson, send } from './client'; import type { Feature } from './generated/Feature'; import type { FeatureList } from './generated/FeatureList'; import type { FeatureSwitch } from './generated/FeatureSwitch'; +import type { SftpListen } from './generated/SftpListen'; +import type { SftpListenChange } from './generated/SftpListenChange'; /** Every optional feature, in a fixed order, with what it runs and whether it runs. */ export const featuresQuery = queryOptions({ @@ -14,3 +16,13 @@ export const featuresQuery = queryOptions({ /** Answered once the feature's tasks have stopped, which can take a while. */ export const switchFeature = (feature: Feature, enabled: boolean) => send('PUT', `/features/${feature}`, { enabled } satisfies FeatureSwitch); + +/** Where SFTP listens, who says so, and what became of its port. */ +export const sftpListenQuery = queryOptions({ + queryKey: ['features', 'sftp'], + queryFn: ({ signal }) => getJson('/settings/sftp', signal), +}); + +/** Answered once the port opened, was refused or closed. No address closes it. */ +export const setSftpListen = (listen: string | null) => + send('PUT', '/settings/sftp', { listen } satisfies SftpListenChange); diff --git a/web/src/api/generated/AssetMemory.ts b/web/src/api/generated/AssetMemory.ts new file mode 100644 index 00000000..5b78b9a3 --- /dev/null +++ b/web/src/api/generated/AssetMemory.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * The memory FXServer measures a streamed file by. + */ +export type AssetMemory = "physical" | "virtual"; diff --git a/web/src/api/generated/AuditAction.ts b/web/src/api/generated/AuditAction.ts index 50efba41..6aaf945a 100644 --- a/web/src/api/generated/AuditAction.ts +++ b/web/src/api/generated/AuditAction.ts @@ -3,4 +3,4 @@ /** * Every action the audit log records. Stored by name, so a variant is never renamed. */ -export type AuditAction = "setup_completed" | "setup_link_created" | "login_succeeded" | "login_failed" | "logout" | "session_revoked" | "password_changed" | "user_created" | "user_deleted" | "user_permissions_changed" | "user_password_reset" | "user_unlocked" | "permission_denied" | "audit_exported" | "reauthenticated" | "launch_settings_changed" | "server_started" | "server_stopped" | "server_restarted" | "console_command_sent" | "console_log_downloaded" | "server_crash_loop" | "schedule_changed" | "scheduled_restart_skipped" | "scheduled_restart_restored" | "scheduled_restart_delayed" | "scheduled_restart_missed" | "one_off_restart_planned" | "one_off_restart_cancelled" | "build_install_started" | "build_install_cancelled" | "build_install_finished" | "build_selected" | "pending_build_set" | "pending_build_cancelled" | "pending_build_applied" | "pending_build_dropped" | "probation_passed" | "probation_failed" | "build_rolled_back" | "build_deleted" | "artifact_settings_changed" | "observer_stopped" | "tree_process_started" | "hang_snapshot_taken" | "crash_file_downloaded" | "crash_files_deleted" | "tree_records_full" | "file_session_opened" | "file_session_closed" | "file_login_failed" | "file_written" | "file_deleted" | "folder_created" | "folder_removed" | "file_renamed" | "file_refused" | "file_records_full" | "file_group_created" | "file_group_changed" | "file_group_deleted" | "file_access_changed" | "service_access_created" | "service_access_changed" | "service_access_deleted" | "file_key_added" | "file_key_removed" | "file_session_disconnected" | "file_settings_changed" | "file_host_key_created" | "player_forgotten" | "players_imported" | "players_retention_changed" | "player_drops_exported" | "player_banned" | "ban_lifted" | "bans_settings_changed" | "map_image_changed" | "map_settings_changed" | "drop_rates_exported" | "drops_retention_changed" | "change_contents_viewed" | "change_settings_changed" | "change_contents_purged" | "change_verify_requested" | "change_check_requested" | "tool_settings_changed" | "tool_files_changed" | "tool_files_restored" | "feature_changed" | "backup_settings_changed" | "backup_destination_created" | "backup_destination_changed" | "backup_destination_deleted" | "backup_finished" | "backup_cancelled" | "backup_downloaded" | "backup_deleted" | "backups_expired" | "statistics_link_dismissed" | "statistics_link_restored" | "observer_update_started" | "observer_update_cancelled" | "observer_update_finished"; +export type AuditAction = "setup_completed" | "setup_link_created" | "login_succeeded" | "login_failed" | "logout" | "session_revoked" | "password_changed" | "user_created" | "user_deleted" | "user_permissions_changed" | "user_password_reset" | "user_unlocked" | "permission_denied" | "audit_exported" | "reauthenticated" | "launch_settings_changed" | "server_started" | "server_stopped" | "server_restarted" | "server_start_extended" | "server_cfg_changed" | "server_cfg_restored" | "console_command_sent" | "console_log_downloaded" | "server_crash_loop" | "schedule_changed" | "scheduled_restart_skipped" | "scheduled_restart_restored" | "scheduled_restart_delayed" | "scheduled_restart_missed" | "one_off_restart_planned" | "one_off_restart_cancelled" | "build_install_started" | "build_install_cancelled" | "build_install_finished" | "build_selected" | "pending_build_set" | "pending_build_cancelled" | "pending_build_applied" | "pending_build_dropped" | "probation_passed" | "probation_failed" | "build_rolled_back" | "build_deleted" | "artifact_settings_changed" | "observer_stopped" | "tree_process_started" | "hang_snapshot_taken" | "crash_file_downloaded" | "crash_files_deleted" | "tree_records_full" | "file_session_opened" | "file_session_closed" | "file_login_failed" | "file_written" | "file_deleted" | "folder_created" | "folder_removed" | "file_renamed" | "file_refused" | "file_records_full" | "file_group_created" | "file_group_changed" | "file_group_deleted" | "file_access_changed" | "service_access_created" | "service_access_changed" | "service_access_deleted" | "file_key_added" | "file_key_removed" | "file_session_disconnected" | "file_settings_changed" | "file_host_key_created" | "player_forgotten" | "players_imported" | "players_retention_changed" | "player_drops_exported" | "player_banned" | "ban_lifted" | "bans_settings_changed" | "map_image_changed" | "map_settings_changed" | "drop_rates_exported" | "drops_retention_changed" | "change_contents_viewed" | "change_settings_changed" | "change_contents_purged" | "change_verify_requested" | "change_check_requested" | "tool_settings_changed" | "tool_files_changed" | "tool_files_restored" | "tool_fix_staged" | "tool_fix_unstaged" | "feature_changed" | "sftp_listen_changed" | "debug_host_known" | "backup_settings_changed" | "backup_destination_created" | "backup_destination_changed" | "backup_destination_deleted" | "backup_finished" | "backup_cancelled" | "backup_downloaded" | "backup_deleted" | "backups_expired" | "statistics_link_dismissed" | "statistics_link_restored" | "observer_update_started" | "observer_update_cancelled" | "observer_update_finished"; diff --git a/web/src/api/generated/BagKind.ts b/web/src/api/generated/BagKind.ts new file mode 100644 index 00000000..db7724d6 --- /dev/null +++ b/web/src/api/generated/BagKind.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * The bag a write went to. + */ +export type BagKind = "global" | "player" | "entity" | "localEntity" | "other"; diff --git a/web/src/api/generated/BagResourceRow.ts b/web/src/api/generated/BagResourceRow.ts new file mode 100644 index 00000000..9c1fdc29 --- /dev/null +++ b/web/src/api/generated/BagResourceRow.ts @@ -0,0 +1,3 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type BagResourceRow = { resource: string, count: number, bytes: number, replicatedBytes: number, keys: number, }; diff --git a/web/src/api/generated/BagRow.ts b/web/src/api/generated/BagRow.ts new file mode 100644 index 00000000..e4575853 --- /dev/null +++ b/web/src/api/generated/BagRow.ts @@ -0,0 +1,8 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { BagKind } from "./BagKind"; + +export type BagRow = { key: string, kind: BagKind, resources: Array, resourceCount: number, count: number, bytes: number, largest: number, +/** + * How many of the writes went out to clients. + */ +replicated: number, }; diff --git a/web/src/api/generated/BagTotals.ts b/web/src/api/generated/BagTotals.ts new file mode 100644 index 00000000..c49786c4 --- /dev/null +++ b/web/src/api/generated/BagTotals.ts @@ -0,0 +1,7 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type BagTotals = { writes: number, bytes: number, +/** + * Of the writes that went out to clients. + */ +replicatedBytes: number, keys: number, resources: number, }; diff --git a/web/src/api/generated/CfgDocument.ts b/web/src/api/generated/CfgDocument.ts new file mode 100644 index 00000000..3e4be3ad --- /dev/null +++ b/web/src/api/generated/CfgDocument.ts @@ -0,0 +1,27 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { CfgReview } from "./CfgReview"; + +/** + * `GET /api/servers/default/cfg`. + */ +export type CfgDocument = { +/** + * The cfg, named from the server folder. + */ +file: string, +/** + * The file's text, each secret replaced by its mark, every line ended by `\n`. + */ +text: string, +/** + * The SHA-256 of the file as read. A save names it, and is refused once the file is another. + */ +base: string, bytes: number, modifiedAt: string | null, +/** + * How many secrets the text hides. + */ +secrets: number, check: CfgReview, +/** + * How many earlier versions are kept. + */ +versions: number, }; diff --git a/web/src/api/generated/CfgDraft.ts b/web/src/api/generated/CfgDraft.ts new file mode 100644 index 00000000..7e08b30a --- /dev/null +++ b/web/src/api/generated/CfgDraft.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * `POST /api/servers/default/cfg/check` and `/cfg/diff`: the text as edited, marks included. + */ +export type CfgDraft = { text: string, }; diff --git a/web/src/api/generated/CfgEndpointLine.ts b/web/src/api/generated/CfgEndpointLine.ts new file mode 100644 index 00000000..0e1e1c40 --- /dev/null +++ b/web/src/api/generated/CfgEndpointLine.ts @@ -0,0 +1,4 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { EndpointProtocol } from "./EndpointProtocol"; + +export type CfgEndpointLine = { line: number, protocol: EndpointProtocol, address: string, start: number, end: number, alone: boolean, }; diff --git a/web/src/api/generated/CfgOutline.ts b/web/src/api/generated/CfgOutline.ts new file mode 100644 index 00000000..979f5ac2 --- /dev/null +++ b/web/src/api/generated/CfgOutline.ts @@ -0,0 +1,24 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { CfgEndpointLine } from "./CfgEndpointLine"; +import type { CfgResourceLine } from "./CfgResourceLine"; +import type { CfgSetting } from "./CfgSetting"; + +/** + * The lines of a cfg the panel edits by field. Lines count from 1; `start` and `end` say where a + * value lies in its line, in UTF-16 code units, its quotes included. + */ +export type CfgOutline = { +/** + * Every `set` and its kin, and the convars the panel has a field for set by their name alone, + * in the order of the file. The last one of a name is the one FXServer keeps. + */ +settings: Array, endpoints: Array, +/** + * Every line that starts a resource or a category, and those commented out that would start + * one the folder holds. + */ +resources: Array, +/** + * The file has more of them than are listed. + */ +truncated: boolean, }; diff --git a/web/src/api/generated/CfgRefusal.ts b/web/src/api/generated/CfgRefusal.ts new file mode 100644 index 00000000..5a716fcd --- /dev/null +++ b/web/src/api/generated/CfgRefusal.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * Why the cfg cannot be edited from the panel, as the `reason` of `cfg_not_editable`. + */ +export type CfgRefusal = "notConfigured" | "missing" | "notAFile" | "unreadable" | "readOnly" | "tooLarge" | "notUtf8" | "holdsMarks"; diff --git a/web/src/api/generated/CfgResourceLine.ts b/web/src/api/generated/CfgResourceLine.ts new file mode 100644 index 00000000..1295c9c1 --- /dev/null +++ b/web/src/api/generated/CfgResourceLine.ts @@ -0,0 +1,19 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type CfgResourceLine = { line: number, +/** + * The resource, or the category in brackets, as written. + */ +name: string, +/** + * False for a line commented out. + */ +enabled: boolean, +/** + * The line holds nothing else: it can be commented out, or put back, as a whole. + */ +alone: boolean, +/** + * Neither the server folder nor the build holds it. + */ +missing: boolean, }; diff --git a/web/src/api/generated/CfgRestore.ts b/web/src/api/generated/CfgRestore.ts new file mode 100644 index 00000000..400e986c --- /dev/null +++ b/web/src/api/generated/CfgRestore.ts @@ -0,0 +1,10 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * `POST /api/servers/default/cfg/versions/{id}/restore`. + */ +export type CfgRestore = { +/** + * [`CfgDocument::base`] of the file the version replaces. + */ +base: string, }; diff --git a/web/src/api/generated/CfgReview.ts b/web/src/api/generated/CfgReview.ts new file mode 100644 index 00000000..b23f6b2b --- /dev/null +++ b/web/src/api/generated/CfgReview.ts @@ -0,0 +1,13 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { CfgFinding } from "./CfgFinding"; +import type { CfgOutline } from "./CfgOutline"; + +/** + * What a text of the cfg sets and starts, and what the scan of it and of the files it runs finds. + */ +export type CfgReview = { outline: CfgOutline, findings: Array, +/** + * The lines a save of the text would add to the file and take from it: a line changed + * counts once in each. + */ +added: number, removed: number, }; diff --git a/web/src/api/generated/CfgSave.ts b/web/src/api/generated/CfgSave.ts new file mode 100644 index 00000000..43ead085 --- /dev/null +++ b/web/src/api/generated/CfgSave.ts @@ -0,0 +1,10 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * `PUT /api/servers/default/cfg`. + */ +export type CfgSave = { text: string, +/** + * [`CfgDocument::base`] of the text the edit started from. + */ +base: string, }; diff --git a/web/src/api/generated/CfgSetting.ts b/web/src/api/generated/CfgSetting.ts new file mode 100644 index 00000000..9571a8fa --- /dev/null +++ b/web/src/api/generated/CfgSetting.ts @@ -0,0 +1,19 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type CfgSetting = { line: number, +/** + * `set`, `sets`, `setr` or `seta`; null when the convar's name is the command. + */ +command: string | null, +/** + * As written. + */ +name: string, +/** + * Null for a secret. + */ +value: string | null, start: number, end: number, +/** + * The line holds nothing else that FXServer runs. + */ +alone: boolean, }; diff --git a/web/src/api/generated/CfgVersion.ts b/web/src/api/generated/CfgVersion.ts new file mode 100644 index 00000000..233bed7e --- /dev/null +++ b/web/src/api/generated/CfgVersion.ts @@ -0,0 +1,10 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * The cfg as it was before a save replaced it. + */ +export type CfgVersion = { id: number, +/** + * When it was replaced, and by whom. + */ +replacedAt: string, replacedBy: string, bytes: number, lines: number, }; diff --git a/web/src/api/generated/DebugBags.ts b/web/src/api/generated/DebugBags.ts new file mode 100644 index 00000000..9b4d104c --- /dev/null +++ b/web/src/api/generated/DebugBags.ts @@ -0,0 +1,11 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { BagResourceRow } from "./BagResourceRow"; +import type { BagRow } from "./BagRow"; +import type { BagTotals } from "./BagTotals"; +import type { DebugStatus } from "./DebugStatus"; +import type { LargestWrite } from "./LargestWrite"; + +/** + * `GET /api/debug/bags`. + */ +export type DebugBags = { status: DebugStatus, seconds: number, totals: BagTotals, keys: Array, resources: Array, largest: Array, truncated: boolean, }; diff --git a/web/src/api/generated/DebugEvents.ts b/web/src/api/generated/DebugEvents.ts new file mode 100644 index 00000000..4052105a --- /dev/null +++ b/web/src/api/generated/DebugEvents.ts @@ -0,0 +1,25 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { DebugStatus } from "./DebugStatus"; +import type { EventPlayerRow } from "./EventPlayerRow"; +import type { EventResourceRow } from "./EventResourceRow"; +import type { EventRow } from "./EventRow"; +import type { EventTotals } from "./EventTotals"; +import type { RefusedRow } from "./RefusedRow"; +import type { TrafficPoint } from "./TrafficPoint"; + +/** + * `GET /api/debug/events`. + */ +export type DebugEvents = { status: DebugStatus, +/** + * The window's length. + */ +seconds: number, totals: EventTotals, series: Array, +/** + * By traffic, the first two hundred. + */ +events: Array, resources: Array, players: Array, refused: Array, +/** + * A resource or Observer kept less than there was. + */ +truncated: boolean, }; diff --git a/web/src/api/generated/DebugHttp.ts b/web/src/api/generated/DebugHttp.ts new file mode 100644 index 00000000..1ac3ec54 --- /dev/null +++ b/web/src/api/generated/DebugHttp.ts @@ -0,0 +1,13 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { DebugStatus } from "./DebugStatus"; +import type { HttpRow } from "./HttpRow"; +import type { HttpTotals } from "./HttpTotals"; + +/** + * `GET /api/debug/http`. + */ +export type DebugHttp = { status: DebugStatus, seconds: number, totals: HttpTotals, requests: Array, +/** + * Until when every host is taken as known: the first day Debug counted. + */ +learningUntil: string | null, truncated: boolean, }; diff --git a/web/src/api/generated/DebugQuery.ts b/web/src/api/generated/DebugQuery.ts new file mode 100644 index 00000000..ea716b15 --- /dev/null +++ b/web/src/api/generated/DebugQuery.ts @@ -0,0 +1,7 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { DebugWindow } from "./DebugWindow"; + +/** + * `?window=` of the Debug routes. + */ +export type DebugQuery = { window: DebugWindow, }; diff --git a/web/src/api/generated/DebugState.ts b/web/src/api/generated/DebugState.ts new file mode 100644 index 00000000..c9d2080e --- /dev/null +++ b/web/src/api/generated/DebugState.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * Where the counting stands. + */ +export type DebugState = "waiting" | "counting" | "refused"; diff --git a/web/src/api/generated/DebugStatus.ts b/web/src/api/generated/DebugStatus.ts new file mode 100644 index 00000000..8cea6bd9 --- /dev/null +++ b/web/src/api/generated/DebugStatus.ts @@ -0,0 +1,28 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { DebugState } from "./DebugState"; + +/** + * What every Debug page says under its title. + */ +export type DebugStatus = { state: DebugState, +/** + * Whether FXServer runs now. + */ +running: boolean, +/** + * When the first report of this start came in. + */ +since: string | null, +/** + * How many resources report. + */ +reporting: number, +/** + * The started resources that do not: in JavaScript or C#, or with no script. The first + * hundred by name. + */ +silent: Array, silentCount: number, +/** + * Why the scheduler was left as it was, in the host's words. + */ +refused: string | null, }; diff --git a/web/src/api/generated/DebugWindow.ts b/web/src/api/generated/DebugWindow.ts new file mode 100644 index 00000000..c999962f --- /dev/null +++ b/web/src/api/generated/DebugWindow.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * How far back a page looks. + */ +export type DebugWindow = "1m" | "5m" | "15m"; diff --git a/web/src/api/generated/ErrorCode.ts b/web/src/api/generated/ErrorCode.ts index eb0063dc..8c321803 100644 --- a/web/src/api/generated/ErrorCode.ts +++ b/web/src/api/generated/ErrorCode.ts @@ -4,4 +4,4 @@ * Why the API refused a request. The panel turns the code into a message; the API never sends * sentences. */ -export type ErrorCode = "unauthenticated" | "invalid_credentials" | "temporary_password_expired" | "too_many_attempts" | "busy" | "too_many_sockets" | "password_change_required" | "reauth_required" | "wrong_password" | "forbidden" | "owner_only" | "self_edit" | "owner_protected" | "escalation" | "hierarchy" | "conflict" | "not_found" | "name_taken" | "invalid_username" | "password_too_short" | "password_too_long" | "password_unchanged" | "unknown_permissions" | "invalid_launch_settings" | "launch_not_configured" | "launch_failed" | "server_running" | "server_not_running" | "server_stopping" | "shutting_down" | "command_refused" | "too_many_commands" | "already_set_up" | "invalid_setup_token" | "origin_mismatch" | "unknown_host" | "json_required" | "invalid_request" | "internal" | "invalid_schedule" | "nothing_to_skip" | "already_skipped" | "nothing_to_restore" | "nothing_to_delay" | "delay_too_long" | "no_one_off" | "job_running" | "no_job" | "job_committing" | "catalog_unavailable" | "build_not_listed" | "not_downloadable" | "already_installed" | "end_of_life" | "build_not_installed" | "build_broken" | "kvp_rollback" | "no_scheduled_restart" | "already_selected" | "no_pending" | "build_protected" | "build_in_use" | "invalid_artifact_settings" | "no_build_selected" | "invalid_import_folder" | "crash_not_found" | "crash_file_not_found" | "crash_file_too_large" | "setup_blocked" | "invalid_rule" | "invalid_public_key" | "key_in_use" | "too_many_keys" | "invalid_access" | "player_not_found" | "address_search" | "invalid_search" | "invalid_players_settings" | "invalid_players_file" | "invalid_drops_settings" | "ban_not_found" | "ban_not_active" | "invalid_ban" | "invalid_bans_settings" | "invalid_map_image" | "no_map_image" | "invalid_map_settings" | "changes_unavailable" | "start_not_found" | "content_hidden" | "content_not_kept" | "diff_too_large" | "check_running" | "check_too_soon" | "invalid_glob" | "plan_changed" | "server_not_stopped" | "tool_not_available" | "feature_disabled" | "feature_locked" | "backup_running" | "invalid_backup_settings" | "invalid_backup_destination" | "backup_not_configured" | "backup_copy_unavailable" | "update_not_available" | "update_running" | "no_update" | "update_restarting" | "update_blocked"; +export type ErrorCode = "unauthenticated" | "invalid_credentials" | "temporary_password_expired" | "too_many_attempts" | "busy" | "too_many_sockets" | "password_change_required" | "reauth_required" | "wrong_password" | "forbidden" | "owner_only" | "self_edit" | "owner_protected" | "escalation" | "hierarchy" | "conflict" | "not_found" | "name_taken" | "invalid_username" | "password_too_short" | "password_too_long" | "password_unchanged" | "unknown_permissions" | "invalid_launch_settings" | "launch_not_configured" | "launch_failed" | "server_running" | "server_not_running" | "server_stopping" | "server_not_starting" | "cfg_not_editable" | "cfg_changed" | "invalid_cfg_text" | "cfg_version_not_found" | "shutting_down" | "command_refused" | "too_many_commands" | "already_set_up" | "invalid_setup_token" | "origin_mismatch" | "unknown_host" | "json_required" | "invalid_request" | "internal" | "invalid_schedule" | "nothing_to_skip" | "already_skipped" | "nothing_to_restore" | "nothing_to_delay" | "delay_too_long" | "no_one_off" | "job_running" | "no_job" | "job_committing" | "catalog_unavailable" | "build_not_listed" | "not_downloadable" | "already_installed" | "end_of_life" | "build_not_installed" | "build_broken" | "kvp_rollback" | "no_scheduled_restart" | "already_selected" | "no_pending" | "build_protected" | "build_in_use" | "invalid_artifact_settings" | "no_build_selected" | "invalid_import_folder" | "crash_not_found" | "crash_file_not_found" | "crash_file_too_large" | "setup_blocked" | "invalid_rule" | "invalid_public_key" | "key_in_use" | "too_many_keys" | "invalid_access" | "player_not_found" | "address_search" | "invalid_search" | "invalid_players_settings" | "invalid_players_file" | "invalid_drops_settings" | "ban_not_found" | "ban_not_active" | "invalid_ban" | "invalid_bans_settings" | "invalid_map_image" | "no_map_image" | "invalid_map_settings" | "changes_unavailable" | "start_not_found" | "content_hidden" | "content_not_kept" | "diff_too_large" | "check_running" | "check_too_soon" | "invalid_glob" | "plan_changed" | "server_not_stopped" | "tool_not_available" | "feature_disabled" | "feature_locked" | "sftp_listen_pinned" | "invalid_sftp_listen" | "backup_running" | "invalid_backup_settings" | "invalid_backup_destination" | "backup_not_configured" | "backup_copy_unavailable" | "update_not_available" | "update_running" | "no_update" | "update_restarting" | "update_blocked"; diff --git a/web/src/api/generated/EventPlayerRow.ts b/web/src/api/generated/EventPlayerRow.ts new file mode 100644 index 00000000..4c7470b4 --- /dev/null +++ b/web/src/api/generated/EventPlayerRow.ts @@ -0,0 +1,7 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type EventPlayerRow = { player: number, count: number, bytes: number, +/** + * The event they sent most. + */ +top: string, }; diff --git a/web/src/api/generated/EventResourceRow.ts b/web/src/api/generated/EventResourceRow.ts new file mode 100644 index 00000000..b2609679 --- /dev/null +++ b/web/src/api/generated/EventResourceRow.ts @@ -0,0 +1,8 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { Flow } from "./Flow"; + +export type EventResourceRow = { resource: string, toClients: Flow, fromClients: Flow, +/** + * Events received from the server's resources and from the engine. + */ +local: number, handlerUs: number, }; diff --git a/web/src/api/generated/EventRow.ts b/web/src/api/generated/EventRow.ts new file mode 100644 index 00000000..85abb833 --- /dev/null +++ b/web/src/api/generated/EventRow.ts @@ -0,0 +1,20 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { EventSender } from "./EventSender"; +import type { EventWay } from "./EventWay"; + +/** + * An event in one way, over the resources that sent or received it. + */ +export type EventRow = { name: string, way: EventWay, +/** + * The first three by traffic. + */ +resources: Array, resourceCount: number, count: number, bytes: number, +/** + * The largest single payload. + */ +largest: number, handlerUs: number, +/** + * Who sent it most, for an event from clients: five at most. + */ +senders: Array, }; diff --git a/web/src/api/generated/EventSender.ts b/web/src/api/generated/EventSender.ts new file mode 100644 index 00000000..79b57218 --- /dev/null +++ b/web/src/api/generated/EventSender.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * A client that sent an event. + */ +export type EventSender = { player: number, count: number, bytes: number, }; diff --git a/web/src/api/generated/EventTotals.ts b/web/src/api/generated/EventTotals.ts new file mode 100644 index 00000000..a758f677 --- /dev/null +++ b/web/src/api/generated/EventTotals.ts @@ -0,0 +1,16 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { Flow } from "./Flow"; + +export type EventTotals = { toClients: Flow, fromClients: Flow, +/** + * Events received from the server's own resources. + */ +local: number, +/** + * In the handlers of the events received, in microseconds. + */ +handlerUs: number, +/** + * Events clients sent that no resource opened to them. + */ +refused: number, refusedPlayers: number, }; diff --git a/web/src/api/generated/EventWay.ts b/web/src/api/generated/EventWay.ts new file mode 100644 index 00000000..09f9e6e1 --- /dev/null +++ b/web/src/api/generated/EventWay.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * Which way an event went. + */ +export type EventWay = "fromClients" | "toClient" | "toAll" | "local" | "fromEngine"; diff --git a/web/src/api/generated/Feature.ts b/web/src/api/generated/Feature.ts index 2cfc306e..b5d909af 100644 --- a/web/src/api/generated/Feature.ts +++ b/web/src/api/generated/Feature.ts @@ -1,3 +1,3 @@ // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. -export type Feature = "sftp" | "changes" | "tools" | "diagnostics" | "metrics" | "players" | "drops" | "updates"; +export type Feature = "sftp" | "changes" | "tools" | "diagnostics" | "metrics" | "players" | "drops" | "updates" | "debug"; diff --git a/web/src/api/generated/FeatureHold.ts b/web/src/api/generated/FeatureHold.ts index a7dba2c4..ce8157bb 100644 --- a/web/src/api/generated/FeatureHold.ts +++ b/web/src/api/generated/FeatureHold.ts @@ -3,4 +3,4 @@ /** * Why a feature does not run although the owner left it on. */ -export type FeatureHold = "host" | "notConfigured" | "requires"; +export type FeatureHold = "host" | "notConfigured" | "portRefused" | "requires"; diff --git a/web/src/api/generated/FeatureReach.ts b/web/src/api/generated/FeatureReach.ts index 3c6c8329..377b0111 100644 --- a/web/src/api/generated/FeatureReach.ts +++ b/web/src/api/generated/FeatureReach.ts @@ -9,7 +9,7 @@ import type { Topic } from "./Topic"; */ export type FeatureReach = { /** - * The addresses it listens on, as `observer.toml` gives them. + * The addresses it listens on. */ listens: Array, tasks: Array, whileOff: Array, files: Array, serverFolder: ServerFolder, /** diff --git a/web/src/api/generated/FeatureTask.ts b/web/src/api/generated/FeatureTask.ts index ac69dcfc..f7af4c5f 100644 --- a/web/src/api/generated/FeatureTask.ts +++ b/web/src/api/generated/FeatureTask.ts @@ -3,4 +3,4 @@ /** * What a feature does in the background, for the panel to name. */ -export type FeatureTask = "sftpServer" | "sftpUpkeep" | "uploadsSweep" | "snapshots" | "changesUpkeep" | "toolsAtStart" | "toolsHeal" | "processRecords" | "connections" | "crashAnalysis" | "hangSnapshot" | "diagnosticsUpkeep" | "sampler" | "metricsUpkeep" | "playersFollower" | "dropsWorker" | "updateCheck" | "updateProbation"; +export type FeatureTask = "sftpServer" | "sftpUpkeep" | "uploadsSweep" | "snapshots" | "changesUpkeep" | "toolsAtStart" | "toolsHeal" | "processRecords" | "connections" | "crashAnalysis" | "hangSnapshot" | "diagnosticsUpkeep" | "sampler" | "metricsUpkeep" | "playersFollower" | "dropsWorker" | "updateCheck" | "updateProbation" | "hookLines" | "hookCounts"; diff --git a/web/src/api/generated/Flow.ts b/web/src/api/generated/Flow.ts new file mode 100644 index 00000000..a3be76f7 --- /dev/null +++ b/web/src/api/generated/Flow.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * A count of events with the bytes they carried. + */ +export type Flow = { count: number, bytes: number, }; diff --git a/web/src/api/generated/HeavyAsset.ts b/web/src/api/generated/HeavyAsset.ts new file mode 100644 index 00000000..8aa2ce62 --- /dev/null +++ b/web/src/api/generated/HeavyAsset.ts @@ -0,0 +1,19 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { AssetMemory } from "./AssetMemory"; + +/** + * A streamed file FXServer said is heavy as it started the file's resource. + */ +export type HeavyAsset = { resource: string, +/** + * Within the resource, as FXServer names it. + */ +file: string, +/** + * In tenths of a MiB, as FXServer prints it. + */ +tenths: number, memory: AssetMemory, +/** + * FXServer says it can keep models from loading. + */ +oversized: boolean, }; diff --git a/web/src/api/generated/HttpRow.ts b/web/src/api/generated/HttpRow.ts new file mode 100644 index 00000000..0db4f584 --- /dev/null +++ b/web/src/api/generated/HttpRow.ts @@ -0,0 +1,11 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type HttpRow = { host: string, method: string, resource: string, count: number, bytes: number, +/** + * When this resource first called this host, as far as Observer saw. + */ +firstCalled: string | null, +/** + * First called after Observer's day of learning, and not marked as known since. + */ +new: boolean, }; diff --git a/web/src/api/generated/HttpTotals.ts b/web/src/api/generated/HttpTotals.ts new file mode 100644 index 00000000..e648f31c --- /dev/null +++ b/web/src/api/generated/HttpTotals.ts @@ -0,0 +1,11 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type HttpTotals = { requests: number, +/** + * Of the request bodies. + */ +bytes: number, hosts: number, resources: number, +/** + * The hosts no resource had called before, and nobody said they know. + */ +newHosts: number, }; diff --git a/web/src/api/generated/KnownHost.ts b/web/src/api/generated/KnownHost.ts new file mode 100644 index 00000000..d94197ea --- /dev/null +++ b/web/src/api/generated/KnownHost.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * `POST /api/debug/http/known`: the owner knows why this resource calls this host. + */ +export type KnownHost = { resource: string, host: string, }; diff --git a/web/src/api/generated/LargestWrite.ts b/web/src/api/generated/LargestWrite.ts new file mode 100644 index 00000000..e2d28ab6 --- /dev/null +++ b/web/src/api/generated/LargestWrite.ts @@ -0,0 +1,7 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type LargestWrite = { +/** + * The bag's own name, such as `player:41`. + */ +bag: string, key: string, resource: string, bytes: number, replicated: boolean, }; diff --git a/web/src/api/generated/MapImagery.ts b/web/src/api/generated/MapImagery.ts new file mode 100644 index 00000000..4dfac9d2 --- /dev/null +++ b/web/src/api/generated/MapImagery.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * What the map draws under the players. + */ +export type MapImagery = "atlas" | "satellite" | "roads" | "image" | "grid"; diff --git a/web/src/api/generated/MapSettings.ts b/web/src/api/generated/MapSettings.ts index cd01c20d..22a6f477 100644 --- a/web/src/api/generated/MapSettings.ts +++ b/web/src/api/generated/MapSettings.ts @@ -1,9 +1,11 @@ // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. import type { MapBounds } from "./MapBounds"; +import type { MapImagery } from "./MapImagery"; -export type MapSettings = { +export type MapSettings = { imagery: MapImagery, /** - * `None` until the image is calibrated: the map then shows a grid of game coordinates. + * Where the owner's image sits; `None` until it is calibrated: the map then shows a grid of + * game coordinates in its place. */ bounds: MapBounds | null, /** diff --git a/web/src/api/generated/MissingFile.ts b/web/src/api/generated/MissingFile.ts new file mode 100644 index 00000000..a3fb3625 --- /dev/null +++ b/web/src/api/generated/MissingFile.ts @@ -0,0 +1,15 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * A file a manifest names that its resource does not hold, as FXServer said when it started the + * resource. + */ +export type MissingFile = { resource: string, +/** + * What the manifest asks for it as: `client_script`, `file`. + */ +kind: string, file: string, +/** + * The manifest and the line that name it. + */ +manifest: string, line: number, }; diff --git a/web/src/api/generated/RefusedRow.ts b/web/src/api/generated/RefusedRow.ts new file mode 100644 index 00000000..9a0ed76a --- /dev/null +++ b/web/src/api/generated/RefusedRow.ts @@ -0,0 +1,7 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type RefusedRow = { name: string, +/** + * Null when the scheduler did not say who. + */ +player: number | null, resource: string, count: number, }; diff --git a/web/src/api/generated/ServerStatus.ts b/web/src/api/generated/ServerStatus.ts index ecdc1a8f..31d4eabd 100644 --- a/web/src/api/generated/ServerStatus.ts +++ b/web/src/api/generated/ServerStatus.ts @@ -3,6 +3,7 @@ import type { HealthProblem } from "./HealthProblem"; import type { LaunchReadiness } from "./LaunchReadiness"; import type { PlayerCount } from "./PlayerCount"; import type { ServerState } from "./ServerState"; +import type { StartProgress } from "./StartProgress"; import type { StateCause } from "./StateCause"; /** @@ -36,4 +37,8 @@ name: string | null, /** * The game port, as of the last run. */ -port: number | null, }; +port: number | null, +/** + * Only while FXServer is starting, from its first second. + */ +start: StartProgress | null, }; diff --git a/web/src/api/generated/SftpListen.ts b/web/src/api/generated/SftpListen.ts new file mode 100644 index 00000000..10456620 --- /dev/null +++ b/web/src/api/generated/SftpListen.ts @@ -0,0 +1,26 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * `GET /api/settings/sftp`, and the answer to a change: where SFTP listens, and who says so. + */ +export type SftpListen = { +/** + * The address in force, such as `0.0.0.0:40126`; `None` until someone gives one. + */ +listen: string | null, +/** + * Whether `observer.toml` names it, which the panel cannot change. + */ +pinned: boolean, +/** + * Whether the port is open on that address now. + */ +open: boolean, +/** + * The fingerprint of the host key a client is shown on the open port. + */ +fingerprint: string | null, +/** + * Why the port did not open, in the host's words. + */ +refused: string | null, }; diff --git a/web/src/api/generated/SftpListenChange.ts b/web/src/api/generated/SftpListenChange.ts new file mode 100644 index 00000000..28d3d9de --- /dev/null +++ b/web/src/api/generated/SftpListenChange.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * `PUT /api/settings/sftp`. No address closes the port and forgets the one there was. + */ +export type SftpListenChange = { listen: string | null, }; diff --git a/web/src/api/generated/SlowResource.ts b/web/src/api/generated/SlowResource.ts new file mode 100644 index 00000000..c4830b5d --- /dev/null +++ b/web/src/api/generated/SlowResource.ts @@ -0,0 +1,7 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type SlowResource = { name: string, +/** + * From the resource started before it to this one. + */ +ms: number, }; diff --git a/web/src/api/generated/StagedFix.ts b/web/src/api/generated/StagedFix.ts new file mode 100644 index 00000000..0628b72d --- /dev/null +++ b/web/src/api/generated/StagedFix.ts @@ -0,0 +1,16 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * A fix the owner prepared while FXServer ran. Before the next start that is not a recovery from + * a crash, a fresh scan runs: the fix is written if it plans exactly what the owner saw, and + * dropped either way. + */ +export type StagedFix = { +/** + * The `plan` of the report the owner saw. + */ +plan: string, +/** + * The groups it fixes. + */ +fixes: number, stagedAt: string, stagedBy: string, }; diff --git a/web/src/api/generated/StartCommand.ts b/web/src/api/generated/StartCommand.ts new file mode 100644 index 00000000..3c8321f8 --- /dev/null +++ b/web/src/api/generated/StartCommand.ts @@ -0,0 +1,26 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * The command of the cfg a start is at. + */ +export type StartCommand = { +/** + * The cfg file the command is written in, named from the server folder. + */ +file: string, line: number, +/** + * Its argument as written: a resource, or a category in brackets. + */ +target: string, +/** + * When FXServer came to it. + */ +since: string, +/** + * How many of the resources it starts are up or refused to start, out of `of`. + */ +done: number, of: number, +/** + * The resource whose server scripts FXServer is loading, when it said so. + */ +resource: string | null, }; diff --git a/web/src/api/generated/StartProgress.ts b/web/src/api/generated/StartProgress.ts new file mode 100644 index 00000000..500092a6 --- /dev/null +++ b/web/src/api/generated/StartProgress.ts @@ -0,0 +1,44 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { SlowResource } from "./SlowResource"; +import type { StartCommand } from "./StartCommand"; +import type { StartedResource } from "./StartedResource"; + +/** + * How far the start under way has come, read from what FXServer prints while it runs its cfg: + * each resource once it is up, and nothing of the one it is starting. + */ +export type StartProgress = { +/** + * Resources FXServer said it started. + */ +started: number, +/** + * Resources it could not start. + */ +failed: number, +/** + * How many the server folder holds, once FXServer has said. + */ +found: number | null, +/** + * How many the cfg's commands start. Null when Observer could not read them all. + */ +planned: number | null, last: StartedResource | null, +/** + * Worked out from the resources each command starts. Null before the first command and + * after the last, and when Observer could not read every command. + */ +at: StartCommand | null, +/** + * The five slowest, the slowest first. + */ +slowest: Array, +/** + * When Observer takes the start for stuck and restarts it, should FXServer start no resource + * until then. Null with hang detection off, and once the cfg has run. + */ +stuckAt: string | null, +/** + * What `POST /servers/default/start/extend` adds to `stuckAt`, in minutes. + */ +extensionMinutes: number, }; diff --git a/web/src/api/generated/StartWarnings.ts b/web/src/api/generated/StartWarnings.ts new file mode 100644 index 00000000..a9612870 --- /dev/null +++ b/web/src/api/generated/StartWarnings.ts @@ -0,0 +1,17 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { HeavyAsset } from "./HeavyAsset"; +import type { MissingFile } from "./MissingFile"; + +/** + * `GET /api/servers/default/start/warnings`: what FXServer warned about as it started its + * resources, the latest time Observer started it. Read from its console, kept in memory. + */ +export type StartWarnings = { +/** + * Null when Observer has not started FXServer since it started itself. + */ +startedAt: string | null, heavy: Array, missing: Array, +/** + * FXServer warned of more than Observer keeps. + */ +truncated: boolean, }; diff --git a/web/src/api/generated/StartedResource.ts b/web/src/api/generated/StartedResource.ts new file mode 100644 index 00000000..581d3fdb --- /dev/null +++ b/web/src/api/generated/StartedResource.ts @@ -0,0 +1,10 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * The last resource FXServer said it started. + */ +export type StartedResource = { name: string, at: string, +/** + * What FXServer counted after its name as it started. + */ +warnings: number, }; diff --git a/web/src/api/generated/ToolView.ts b/web/src/api/generated/ToolView.ts index 92d7e838..50197bf4 100644 --- a/web/src/api/generated/ToolView.ts +++ b/web/src/api/generated/ToolView.ts @@ -1,4 +1,5 @@ // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { StagedFix } from "./StagedFix"; import type { ToolChange } from "./ToolChange"; import type { ToolKind } from "./ToolKind"; import type { ToolRun } from "./ToolRun"; @@ -42,4 +43,8 @@ closed: Array, /** * What the backups of these changes take on the disk. */ -backupBytes: number, }; +backupBytes: number, +/** + * The fix that waits for the next start of FXServer. + */ +staged: StagedFix | null, }; diff --git a/web/src/api/generated/TrafficPoint.ts b/web/src/api/generated/TrafficPoint.ts new file mode 100644 index 00000000..d3a4c790 --- /dev/null +++ b/web/src/api/generated/TrafficPoint.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * The bytes that went to and came from clients in one slice of the window. + */ +export type TrafficPoint = { at: string, seconds: number, toClients: number, fromClients: number, }; diff --git a/web/src/api/server.ts b/web/src/api/server.ts index a9aa864d..07ce1eec 100644 --- a/web/src/api/server.ts +++ b/web/src/api/server.ts @@ -3,6 +3,7 @@ import { queryOptions } from '@tanstack/react-query'; import { getJson, request, send } from './client'; import type { CommandAck } from './generated/CommandAck'; import type { LogFile } from './generated/LogFile'; +import type { StartWarnings } from './generated/StartWarnings'; const SERVER = '/servers/default'; @@ -10,6 +11,15 @@ const SERVER = '/servers/default'; export const startServer = () => send('POST', `${SERVER}/start`); export const stopServer = () => send('POST', `${SERVER}/stop`); export const restartServer = () => send('POST', `${SERVER}/restart`); +/** Gives the start under way more time before Observer takes it for stuck. */ +export const extendStart = () => send('POST', `${SERVER}/start/extend`); + +/** What FXServer warned about as it last started its resources: the tools list it. */ +export const startWarningsQuery = queryOptions({ + queryKey: ['start-warnings'], + queryFn: ({ signal }) => getJson(`${SERVER}/start/warnings`, signal), + staleTime: 10_000, +}); /** Audited before FXServer gets it; the answer says whether FXServer read it yet. */ export const sendCommand = (line: string) => send('POST', `${SERVER}/console`, { line }); diff --git a/web/src/api/tools.ts b/web/src/api/tools.ts index 44fe9dc6..c0baa949 100644 --- a/web/src/api/tools.ts +++ b/web/src/api/tools.ts @@ -28,6 +28,16 @@ export const scanArchetypes = () => send('POST', `${PATH}/scan`); export const fixArchetypes = (plan: string) => send('POST', `${PATH}/fix`, { plan } satisfies ToolFixRequest); +/** + * Keeps the plan for the next start of FXServer, which writes it before FXServer starts if a + * fresh scan plans exactly that. Refused with `plan_changed` when the plan is not the latest report's. + */ +export const stageArchetypes = (plan: string) => + send('POST', `${PATH}/stage`, { plan } satisfies ToolFixRequest); + +/** Drops the fix staged for the next start. */ +export const unstageArchetypes = () => send('DELETE', `${PATH}/stage`); + /** Puts back what the scope names, with FXServer stopped. */ export const undoArchetypes = (scope: ToolUndoScope) => send('POST', `${PATH}/undo`, { scope } satisfies ToolUndoRequest); diff --git a/web/src/cfg/CfgEditor.tsx b/web/src/cfg/CfgEditor.tsx new file mode 100644 index 00000000..c7f1c9ae --- /dev/null +++ b/web/src/cfg/CfgEditor.tsx @@ -0,0 +1,235 @@ +import { Search } from 'lucide-react'; +import { useEffect, useMemo, useRef, useState } from 'react'; + +import type { CfgSpan } from '../api/generated/CfgSpan'; +import type { Severity } from '../api/generated/Severity'; +import { inputClass } from '../components/Field'; +import { useI18n } from '../i18n/I18nProvider'; +import { cn } from '../lib/cn'; +import { TOKEN_CLASS } from '../setup/CfgBrowser'; +import { lineTokens } from '../setup/cfgSyntax'; + +const LINE_HEIGHT = 20; +const SECRET = /<>/gu; + +/** The text and the colours over it must lay out alike, character for character. */ +const TYPE = + 'm-0 border-0 p-0 pr-16 pl-3 font-mono text-[12.5px] leading-5 tracking-normal whitespace-pre [font-variant-ligatures:none] [tab-size:4]'; + +const GUTTER = { + blocking: 'text-danger-ink font-semibold', + warning: 'text-warning font-semibold', + note: 'text-accent-ink', +} satisfies Record; + +/** Scrolls the lines so that `line` shows a third of the way down. */ +const reveal = (view: HTMLElement | null, line: number) => { + if (view !== null) { + view.scrollTop = Math.max(0, (line - 1) * LINE_HEIGHT - view.clientHeight / 3); + } +}; + +/** A line as the preview of the setup colours it, a secret's mark kept as the text field holds it. */ +const coloured = (line: string) => { + const marks = line.match(SECRET) ?? []; + const spans: CfgSpan[] = []; + line.split(SECRET).forEach((piece, index) => { + if (piece !== '') { + spans.push({ text: piece, secret: false }); + } + if (index < marks.length) { + spans.push({ text: '', secret: true }); + } + }); + let mark = 0; + let at = 0; + return lineTokens(spans).map((token) => { + const text = token.kind === 'secret' ? (marks[mark++] ?? '') : token.text; + const key = at; + at += text.length; + return { key, kind: token.kind, text }; + }); +}; + +/** + * The cfg as text: a plain text field, for the keyboard, the clipboard and assistive technology + * to work as they do anywhere, under the same text coloured. `marks` flags the lines the checks + * name; `jump` puts the caret on a line and brings it into view. + */ +export function CfgEditor({ + label, + text, + marks, + jump, + onChange, +}: { + label: string; + text: string; + marks: ReadonlyMap; + /** A line to go to; `at` tells two requests for one line apart. */ + jump: { line: number; at: number } | null; + onChange: (text: string) => void; +}) { + const { t, format } = useI18n(); + const words = t.cfg.file; + const area = useRef(null); + const scroller = useRef(null); + const lines = useMemo(() => text.split('\n'), [text]); + const [caret, setCaret] = useState({ line: 1, column: 1 }); + const [query, setQuery] = useState(''); + const [missed, setMissed] = useState(false); + + const place = () => { + const field = area.current; + if (field !== null) { + const before = field.value.slice(0, field.selectionStart); + setCaret({ line: before.split('\n').length, column: before.length - before.lastIndexOf('\n') }); + } + }; + const select = (start: number, end: number) => { + const field = area.current; + if (field !== null) { + field.focus({ preventScroll: true }); + field.setSelectionRange(start, end); + reveal(scroller.current, field.value.slice(0, start).split('\n').length); + place(); + } + }; + + useEffect(() => { + const field = area.current; + if (jump === null || field === null) { + return; + } + const held = field.value.split('\n'); + const line = Math.min(Math.max(jump.line, 1), held.length); + const start = held.slice(0, line - 1).reduce((sum, each) => sum + each.length + 1, 0); + field.focus({ preventScroll: true }); + field.setSelectionRange(start, start + (held[line - 1]?.length ?? 0)); + reveal(scroller.current, line); + setCaret({ line, column: 1 }); + }, [jump]); + + const find = () => { + const field = area.current; + if (field === null || query === '') { + return; + } + const held = field.value.toLowerCase(); + const needle = query.toLowerCase(); + const next = held.indexOf(needle, field.selectionEnd); + const found = next === -1 ? held.indexOf(needle) : next; + setMissed(found === -1); + if (found !== -1) { + select(found, found + needle.length); + } + }; + + return ( +
+
+ {label} + + {words.caret( + format.number(caret.line, { useGrouping: false }), + format.number(caret.column, { useGrouping: false }), + )} + + +
{ + event.preventDefault(); + find(); + }} + > +
+
+ {missed && {words.notFound}} +
+ +
+
+
+ +
+ +