diff --git a/apps/signage/DEBUGGING.md b/apps/signage/DEBUGGING.md index 7df5a76d8a9..c5673361883 100644 --- a/apps/signage/DEBUGGING.md +++ b/apps/signage/DEBUGGING.md @@ -85,6 +85,9 @@ makes the display request use `?preview=true`. | Old version running | `updates.new_version`, `updates.reload_pending` (a reload waits for the network and for play-through content to finish), `updates.last_check` | | Blank screen after a reboot | Likely offline boot — check `online`, then whether cached credentials exist | | Player reloading itself | `watchdog.recent_reloads` and `watchdog.last_error` — something fatal stalled a core loop | +| Paused and does not resume | Pause and resume messages are obeyed only from the parent frame. Check what embeds the player and `players[].state` | +| Plugin cut short, or held long | A play-through plugin advances on `finished`, or after a limit. Look for `did not report finished in time` in the console | +| Blank screen, no `window.signage` | The application did not start. Look for `Application failed to start` in the console; it reloads with a backoff | ## Recovery watchdog @@ -105,19 +108,24 @@ boot that never completes is most often a bad cached build. That deadline only applies once the device has been bootstrapped to a display — one sitting on the picker is waiting for a person, not broken. -| Guard | Value | -| ------------------------ | ---------------------------------------------------------- | -| Stall thresholds | poll 10 min, schedule 5 min, playback 3 min, visible 5 min | -| Boot deadline | 5 min from start with nothing on screen | -| Grace before recovering | 5 min | -| Recoveries allowed | 3 per hour, then 1 per hour | -| Back to 3 per hour after | 2 hours with no recovery | +| Guard | Value | +| Stall thresholds | poll 10 min, schedule 5 min, playback 3 min, visible 5 min | +| Boot deadline | 5 min from start with nothing on screen | +| Grace before recovering | 5 min | +| Recoveries allowed | 3 per hour, then 1 per hour | +| Back to 3 per hour after | 2 hours with no recovery | Once recoveries are throttled the next one clears the application cache first — unregistering the service worker and deleting its caches — in case the cached build is what is wrong. That only happens if `location.href` returns a 200, so a player is never left with no cached application and no way to fetch a new one; -if the server cannot be reached it falls back to a plain reload. +if the server cannot be reached it falls back to a plain reload. A server that +does not answer within 15 seconds counts as unreachable. + +A recovery that has not replaced the page after 2 minutes counts as failed: a +cache clear that hung, or a reload the server never answered. The watchdog then +reloads again and starts its checks again, inside the same limits, so a failed +recovery cannot stop the watchdog until someone restarts the device. A recovery reload does **not** wait for the network, unlike an update reload. A stalled player should restart whether or not the backend is up, and it can boot @@ -132,6 +140,32 @@ Failed initialisation — the app giving up because it cannot load the current user — is routed through the same limits, so it cannot restart the player every thirty seconds on its own. +The watchdog starts inside the application, so it cannot see a start that fails +before the application exists. That case has its own retry: the player reloads +after 10 seconds, and the wait doubles after each consecutive failure to a +maximum of 5 minutes. The count is in `sessionStorage["SIGNAGE.boot_failures"]` +and is removed after a successful start. + +### Content that holds the screen + +A play-through plugin advances when it reports `finished`. If it never sends a +plugin message (for example, the page did not load), it advances after its +configured duration, like a static plugin. If it sends messages but never +reports `finished`, it advances after twice its configured duration, held +between 5 and 60 minutes. After that limit, it also stops holding back an +update reload. + +When the plugin is the only item, there is nothing to advance to. A lone +play-through plugin that never sent a plugin message is then treated as a +failed load: it is removed from the screen and loaded again after 30 seconds, +until it responds. A lone plugin that responds but never reports `finished` +stays on screen, as any single item does. + +Pause and resume messages (US-SIG-024) are obeyed only from the parent frame. +Webpages and plugins on screen cannot pause the player. This is important +because a paused player still checks in with the watchdog, so the watchdog does +not recover it. + `watchdog.booted`, `watchdog.recoveries_throttled` and `watchdog.last_recovery` show where in that sequence a player is. @@ -145,15 +179,15 @@ recovered and you want to know what from. ## Storage -| Location | Holds | -| ------------------------------------------------------ | ----------------------------------------- | -| `localStorage["PlaceOS.SIGNAGE.display_details."]` | Last known display payload, used offline | -| `localStorage["PlaceOS.SIGNAGE.cached_files"]` | Media cache index (urls, sizes, owners) | -| `localStorage["PlaceOS.SIGNAGE.display"]` | Bootstrapped display id | -| `localStorage["PLACEOS.org.*"]` | Cached zone data and last known authority | -| `localStorage["PlaceOS.SIGNAGE.watchdog_reloads"]` | Timestamps of automatic recoveries | -| `sessionStorage["SIGNAGE.debug"]`, `["SIGNAGE.muted"]` | Debug and mute state | -| IndexedDB `SignageMedia` → `files` | The cached media files themselves | +| Location | Holds | +| `localStorage["PlaceOS.SIGNAGE.display_details."]` | Last known display payload, used offline | +| `localStorage["PlaceOS.SIGNAGE.cached_files"]` | Media cache index (urls, sizes, owners) | +| `localStorage["PlaceOS.SIGNAGE.display"]` | Bootstrapped display id | +| `localStorage["PLACEOS.org.*"]` | Cached zone data and last known authority | +| `localStorage["PlaceOS.SIGNAGE.watchdog_reloads"]` | Timestamps of automatic recoveries | +| `sessionStorage["SIGNAGE.debug"]`, `["SIGNAGE.muted"]` | Debug and mute state | +| `sessionStorage["SIGNAGE.boot_failures"]` | Consecutive failed starts, for the backoff | +| IndexedDB `SignageMedia` → `files` | The cached media files themselves | ## Resetting diff --git a/apps/signage/USER_STORIES.md b/apps/signage/USER_STORIES.md index cd3744f690a..0129228e020 100644 --- a/apps/signage/USER_STORIES.md +++ b/apps/signage/USER_STORIES.md @@ -38,6 +38,7 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device - The app also reads `OSK.enabled` from localStorage and enables the virtual keyboard when the value is `true`. - The bootstrap screen can clear stored signage bootstrap data when opened with `?clear=true`. - Clearing removes both the current display key and the legacy `PlaceOS.SIGNAGE.building` key. +- If the application fails to start, it reloads after 10 seconds. The wait doubles after each consecutive failure, to a maximum of 5 minutes, and resets after a successful start. --- @@ -161,6 +162,7 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device - Webpage items play for their configured effective duration. - A single valid webpage item remains loaded instead of reloading on every loop. - Upcoming webpage items can be preloaded on the inactive output shortly before transition. +- Preloading does not start while the current item is still waiting to be revealed or is in transition. - Webpage media is not cached as a local file. --- @@ -179,6 +181,9 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device - If a plugin does not report load or ready, the player sends config after a 15 second wait. - Static plugins follow the configured effective duration. - Play-through plugins advance when they report `finished`. +- A play-through plugin that never sends a plugin message advances after its configured duration, like a static plugin. +- If that plugin is the only item, it is removed from the screen and loaded again after 30 seconds. +- A play-through plugin that does not report `finished` advances after twice its configured duration, but not before 5 minutes and not after 60 minutes. - Interactive plugins can request a new playback duration through plugin interaction events. - Upcoming plugin items can be preloaded on the inactive output shortly before transition. - Fatal plugin errors advance to the next media item. @@ -425,6 +430,8 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device **Acceptance Criteria:** - The signage panel listens for object postMessage payloads. +- Only messages from the parent frame are accepted. Messages from other windows, such as webpage or plugin content on screen, are ignored. +- A player that is not in a frame ignores all pause and resume messages. - A payload with `type: 'signage:pause'` pauses all player instances. - A payload with `type: 'signage:resume'` resumes all player instances. - Unknown payloads are ignored. @@ -483,6 +490,7 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device - Object URLs outside the nearby window are revoked. - If an active item's URL is not ready, the player waits and retries item selection. - Webpage and plugin outputs are prepared on the inactive layer near the end of the current item so they can be revealed after loading. +- An item that is still waiting to be revealed keeps its output. The next item is not prepared until the current item is on screen. --- diff --git a/apps/signage/src/app/media-player.component.ts b/apps/signage/src/app/media-player.component.ts index 05cff89dedf..a13703b8cd8 100644 --- a/apps/signage/src/app/media-player.component.ts +++ b/apps/signage/src/app/media-player.component.ts @@ -50,6 +50,12 @@ const INTERACTIVE_PRELOAD_LEAD_TIME = 10 * 1000; const WEBPAGE_REVEAL_DELAY = 3 * 1000; /** Max wait for plugin load/ready before continuing playback anyway */ const PLUGIN_LOAD_TIMEOUT = 15 * 1000; +/** + * Bounds on how long a play-through plugin may run without reporting that it + * finished. It gets twice its scheduled duration, held between these. + */ +const PLAY_THROUGH_MIN_LIMIT = 5 * 60 * 1000; +const PLAY_THROUGH_MAX_LIMIT = 60 * 60 * 1000; /** Minimum spacing between attempts to resolve a URL that failed to resolve */ const URL_RETRY_DELAY = 1000; @@ -356,6 +362,8 @@ export class MediaPlayerComponent private _item_output = new Map(); private _output_items: [MediaPlayerItem, MediaPlayerItem] = [null, null]; private _ready_output_items = new Set(); + /** Plugin outputs that have sent at least one plugin protocol message */ + private _responded_output_items = new Set(); public get playlist_items() { return this._item_playlist; @@ -628,7 +636,8 @@ export class MediaPlayerComponent * Whether the item on screen plays to completion, so interrupting it now * would be noticed. Images and webpages hold a static frame and can be * replaced without anyone seeing a difference; videos and plugins that - * report when they finish cannot. + * report when they finish cannot. A plugin held on screen past its limit, + * as a lone item is, is not about to finish and does not count. */ public isMidPlayThroughItem() { const item = this.active_item; @@ -636,7 +645,13 @@ export class MediaPlayerComponent if (item.type === 'video') return true; if (item.type === 'plugin') { const playback = item.plugin?.playback_type; - return playback === 'playsthrough' || playback === 'interactive'; + const limit = + playback === 'playsthrough' + ? this._playThroughLimit(item) + : playback === 'interactive' + ? this._effectivePlaybackDuration(item) + : 0; + return time() - this._item_start < limit; } return false; } @@ -745,13 +760,17 @@ export class MediaPlayerComponent this.progress_start.set(0); this.setPlaylistItem(0); } - // For playsthrough plugins, advance when plugin signals finished + // For playsthrough plugins, advance when plugin signals finished, or + // once it has overrun its limit so a hung plugin cannot hold the + // screen forever if ( item?.type === 'plugin' && item.plugin?.playback_type === 'playsthrough' ) { if (this._plugin_finished) { this.nextItem(); + } else if (now > this._item_start + this._playThroughLimit(item)) { + this._handleOverrunPlugin(item); } return; } @@ -967,6 +986,7 @@ export class MediaPlayerComponent if (item) { this._item_output.delete(item.id); this._ready_output_items.delete(this._outputKey(output, item)); + this._responded_output_items.delete(this._outputKey(output, item)); } this._output_items[output] = null; } @@ -1150,9 +1170,13 @@ export class MediaPlayerComponent if (!item || item.type !== 'plugin') return; if (this._item_output.get(item.id) !== output) return; log('MediaPlayer', `Plugin status: ${status}`, [item.name]); + if (status !== 'unknown') { + this._responded_output_items.add(this._outputKey(output, item)); + } if (status === 'ready') { this._handlePluginReady(item, output); - } else if (status === 'finished') { + } else if (status === 'finished' && this.active_item?.id === item.id) { + // A preloaded plugin finishing must not end the one on screen this._plugin_finished = true; } } @@ -1191,16 +1215,55 @@ export class MediaPlayerComponent log('MediaPlayer', `Plugin error: ${error.message}`, [error], 'error'); if (!error.fatal) return; if (item?.type === 'plugin') { - this._markNotShown(item); - this._handled_error_cycle = this._currentMediaCycle(); - this._clearDeferredReveal(); - this._clearOutput(output); - this._skipFailedMedia(this._item_start || time()); + this._failPluginItem(item, output); } else { this.nextItem(); } } + /** + * Treat a plugin as failed to load: remove it from screen and skip it, or + * retry it after a delay when there is nothing else to show. + */ + private _failPluginItem(item: MediaPlayerItem, output: 0 | 1) { + this._markNotShown(item); + this._handled_error_cycle = this._currentMediaCycle(); + this._clearDeferredReveal(); + this._clearOutput(output); + this._skipFailedMedia(this._item_start || time()); + } + + /** + * A play-through plugin that has overrun its limit. With other items to + * show, move on. A lone one is held like any single item, unless it never + * sent a plugin message: then it can never finish and is most likely an + * error page, so it is retried the same way as a fatal plugin error. + */ + private _handleOverrunPlugin(item: MediaPlayerItem) { + if (!this._shouldHoldSingleInteractiveItem(item)) { + log( + 'MediaPlayer', + `Plugin "${item.name}" did not report finished in time; continuing.`, + [item.plugin?.uri], + 'warn', + ); + this.nextItem(); + return; + } + // Already removed from screen and waiting for its retry + const output = this._item_output.get(item.id); + if (output === undefined || this._pluginResponded(item, output)) { + return; + } + log( + 'MediaPlayer', + `Plugin "${item.name}" never responded; retrying.`, + [item.plugin?.uri], + 'warn', + ); + this._failPluginItem(item, output); + } + private _showPlugin(item: MediaPlayerItem, output: 0 | 1) { log('MediaPlayer', `Showing plugin: ${item.name}`, [item.plugin?.name]); this._item_output.set(item.id, output); @@ -1269,6 +1332,29 @@ export class MediaPlayerComponent return this._playback_duration || item?.duration || 15 * 1000; } + /** Whether the plugin on `output` has sent any plugin protocol message */ + private _pluginResponded(item: MediaPlayerItem, output: 0 | 1) { + return this._responded_output_items.has(this._outputKey(output, item)); + } + + /** + * How long a play-through plugin may hold the screen without reporting + * that it finished. One that never sent a plugin message - a page that + * failed to load, or not a plugin at all - cannot report it, so it gets + * its scheduled duration like a static item. One that did gets twice + * that, within bounds, so a long run is not cut short but a hung plugin + * cannot hold the screen forever. + */ + private _playThroughLimit(item: MediaPlayerItem) { + const duration = this._effectivePlaybackDuration(item); + const output = this._item_output.get(item.id) ?? this.active_output(); + if (!this._pluginResponded(item, output)) return duration; + return Math.min( + Math.max(duration * 2, PLAY_THROUGH_MIN_LIMIT), + PLAY_THROUGH_MAX_LIMIT, + ); + } + private _resetPlayback(playback_duration = 0) { if (playback_duration > 0) { this._playback_duration = playback_duration; @@ -1457,6 +1543,15 @@ export class MediaPlayerComponent private _shouldPreloadUpcomingInteractiveContent() { if (!this._item_real_start) return false; + // Until the current item is revealed it occupies the inactive output, + // and preloading there would replace it with the next item + if ( + this.defer_reveal() || + this.in_animation() || + this.pending_output() !== this.active_output() + ) { + return false; + } const item = this.active_item; const remaining = this._effectivePlaybackDuration(item) - @@ -1704,6 +1799,7 @@ export class MediaPlayerComponent this._item_output.clear(); this._output_items = [null, null]; this._ready_output_items.clear(); + this._responded_output_items.clear(); this._setOutputPlugin(0, null); this._setOutputPlugin(1, null); } diff --git a/apps/signage/src/app/signage.component.ts b/apps/signage/src/app/signage.component.ts index 994ca836c26..24bbe784df9 100644 --- a/apps/signage/src/app/signage.component.ts +++ b/apps/signage/src/app/signage.component.ts @@ -250,7 +250,14 @@ export class SignagePanelComponent extends AsyncHandler implements OnInit { sessionStorage.setItem(MUTE_STORAGE_KEY, `${muted}`); } + /** + * Pause and resume commands from the shell that embeds the player. Only + * the parent frame is obeyed: webpages and plugins on screen post messages + * to this window too, and a paused player looks healthy to the watchdog, + * so one that paused it would freeze the display for good. + */ private readonly _remote_message_handler = (event: MessageEvent) => { + if (window.parent === window || event?.source !== window.parent) return; const data = event?.data; if (!data || typeof data !== 'object') return; if (data.type === REMOTE_PAUSE) this._setPlaybackState('PAUSED'); diff --git a/apps/signage/src/app/template.component.ts b/apps/signage/src/app/template.component.ts index 0b755d554a0..be61027eca8 100644 --- a/apps/signage/src/app/template.component.ts +++ b/apps/signage/src/app/template.component.ts @@ -249,7 +249,13 @@ export class SignageTemplateComponent extends AsyncHandler implements OnInit { ); } + /** + * Unsaved layouts from the manager preview, which embeds the player in an + * iframe. Only the parent frame is listened to, so content on screen + * cannot replace the layout. + */ private readonly _preview_message_handler = (event: MessageEvent) => { + if (window.parent === window || event?.source !== window.parent) return; const data = event?.data; if (!this.debug() || data?.type !== PREVIEW_LAYOUTS_MESSAGE) return; this._preview_layouts.set( @@ -260,9 +266,11 @@ export class SignageTemplateComponent extends AsyncHandler implements OnInit { /** * Ask the embedding manager preview for its unsaved layouts. The manager * cannot know when this listener is ready, so the player asks first. + * Only previews run in debug mode, so a player embedded anywhere else + * says nothing to its parent. */ private _requestPreviewLayouts() { - if (window.parent === window) return; + if (!this.debug() || window.parent === window) return; window.parent.postMessage({ type: PREVIEW_READY_MESSAGE }, '*'); } diff --git a/apps/signage/src/app/watchdog.ts b/apps/signage/src/app/watchdog.ts index f4805a2f241..41ea0e02f05 100644 --- a/apps/signage/src/app/watchdog.ts +++ b/apps/signage/src/app/watchdog.ts @@ -71,7 +71,19 @@ const MAX_RECOVERIES_PER_WINDOW = 3; const RECOVERY_THROTTLE_MS = 60 * MINUTES; /** Quiet period after which the recovery history is forgotten */ const RECOVERY_RESET_MS = 2 * 60 * MINUTES; +/** Longest wait for the server check before clearing the application cache */ +const REACHABLE_TIMEOUT_MS = 15 * SECONDS; +/** + * How long a recovery may take before it counts as failed. A reload that + * works replaces the page well before this. + */ +const RECOVERY_TIMEOUT_MS = 2 * MINUTES; const RECOVERY_KEY = 'PlaceOS.SIGNAGE.watchdog_reloads'; +/** Consecutive failed application starts, for the boot retry backoff */ +const BOOT_FAILURES_KEY = 'SIGNAGE.boot_failures'; +/** First wait before reloading after a failed start; doubles each time */ +const BOOT_RETRY_BASE_MS = 10 * SECONDS; +const BOOT_RETRY_MAX_MS = 5 * MINUTES; const log = scoped_log('Watchdog'); @@ -112,10 +124,13 @@ let _stalled_since = 0; let _last_check = 0; let _started_at = 0; let _timer: ReturnType | undefined; +let _recovery_timer: ReturnType | undefined; +/** Increments for each recovery; only the newest one may reload the page */ +let _recovery_generation = 0; let _listening = false; let _recovering = false; let _reload: () => void = () => location.reload(); -let _hard_reload: () => Promise = () => clearCachesAndReload(); +let _clear_cache: () => Promise = () => clearApplicationCache(); /** Record that a piece of core machinery is still running */ export function recordHeartbeat(signal: WatchdogSignal) { @@ -215,23 +230,27 @@ function resetHeartbeats(now: number) { } /** - * Reload, clearing the application cache first. Used once plain reloads have - * failed to shift the problem, in case the cached build is what is wrong. - * Only clears the cache when the server can be reached, so a player is never - * left with no cached application and no way to fetch a new one. - * - * Reloads the current URL rather than navigating to the base path: the route - * that says which display to show, and whether to show it in debug mode, is in - * the hash. Dropping it leaves the player on the display picker instead of - * back on its content. + * Clear the application cache before a recovery reload. Used once plain + * reloads have failed to shift the problem, in case the cached build is what + * is wrong. Only clears the cache when the server can be reached, so a player + * is never left with no cached application and no way to fetch a new one. A + * server that accepts the request but never answers counts as unreachable. + * Returns whether the cache was cleared; the caller reloads either way. */ -export async function clearCachesAndReload(): Promise { +export async function clearApplicationCache(): Promise { let reachable = false; + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), REACHABLE_TIMEOUT_MS); try { - const response = await fetch(location.href, { cache: 'reload' }); + const response = await fetch(location.href, { + cache: 'reload', + signal: controller.signal, + }); reachable = response.ok; } catch { reachable = false; + } finally { + clearTimeout(timeout); } if (!reachable) { log.warn('Server unreachable; not clearing the application cache.'); @@ -249,7 +268,6 @@ export async function clearCachesAndReload(): Promise { } catch (error) { log.warn('Failed to clear the application cache.', error); } - _reload(); return true; } @@ -318,15 +336,34 @@ function recover(now: number, reasons: string[], prefer_hard: boolean) { last_error: _last_error, }); _recovering = true; - // Only clear the application cache when the server can serve a - // replacement; `hardReload` checks that and reports back. + const generation = ++_recovery_generation; + // If the page is still here after the timeout, the reload never happened: + // a cache clear that hung, or a navigation the server never answered. + // Reload again and let the checks run, so a failed recovery cannot leave + // the watchdog latched off until someone power-cycles the device. + clearTimeout(_recovery_timer); + _recovery_timer = setTimeout(() => { + log.error('Recovery did not reload the page; trying again.'); + // A cache clear still running belongs to the abandoned attempt and + // must not start a second reload when it finishes + _recovery_generation++; + _recovering = false; + _reload(); + }, RECOVERY_TIMEOUT_MS); + // Reloads the current URL rather than navigating to the base path: the + // route that says which display to show, and whether in debug mode, is + // in the hash. Dropping it leaves the player on the display picker. if (!prefer_hard && !throttled) { _reload(); return true; } - _hard_reload().then((cleared) => { - if (!cleared) _reload(); - }); + // Reload whether or not the cache could be cleared; the clear itself + // only goes ahead when the server can serve a replacement + _clear_cache() + .catch(() => false) + .then(() => { + if (generation === _recovery_generation) _reload(); + }); return true; } @@ -342,7 +379,7 @@ export function requestRecovery(reason: string, prefer_hard = false) { export interface WatchdogActions { reload?: () => void; - hardReload?: () => Promise; + clearCache?: () => Promise; /** * Whether this device is supposed to be showing content. A player that has * never been bootstrapped is legitimately waiting for someone to pick a @@ -354,7 +391,7 @@ export interface WatchdogActions { /** Start watching. Returns a callback that stops it again. */ export function startWatchdog(actions: WatchdogActions = {}) { _reload = actions.reload || (() => location.reload()); - _hard_reload = actions.hardReload || clearCachesAndReload; + _clear_cache = actions.clearCache || clearApplicationCache; const expectedToRun = actions.isExpectedToRun || (() => false); stopWatchdog(); if (!_listening) { @@ -371,6 +408,12 @@ export function startWatchdog(actions: WatchdogActions = {}) { export function stopWatchdog() { if (_timer) clearInterval(_timer); _timer = undefined; + // The latch must not outlive the timer that releases it, and a stopped + // watchdog must not reload when an earlier cache clear finishes + clearTimeout(_recovery_timer); + _recovery_timer = undefined; + _recovery_generation++; + _recovering = false; if (_listening) { _listening = false; window.removeEventListener('error', onWindowError); @@ -438,6 +481,45 @@ export function watchdogState() { }; } +/** + * Reload after the application failed to start. The watchdog starts inside + * the application, so a start that fails never reaches it and nothing else + * would recover the blank screen. The wait doubles with each consecutive + * failure, up to a cap, and the count lives in session storage so it survives + * the reloads it causes. Returns the wait in milliseconds. + */ +export function scheduleBootRetry( + reload: () => void = () => location.reload(), +) { + let failures = 0; + try { + failures = Number(sessionStorage.getItem(BOOT_FAILURES_KEY)) || 0; + sessionStorage.setItem(BOOT_FAILURES_KEY, `${failures + 1}`); + } catch { + // Ignore privacy-mode failures; retry at the base delay. + } + const delay = Math.min( + BOOT_RETRY_BASE_MS * 2 ** Math.min(failures, 10), + BOOT_RETRY_MAX_MS, + ); + // Not the scoped log: it prints only once settings enable debug, and + // settings never load when the application fails to start. + console.error( + `[Watchdog] Application failed to start; reloading in ${delay / 1000}s.`, + ); + setTimeout(reload, delay); + return delay; +} + +/** Forget earlier failed starts. Called once the application has started. */ +export function resetBootRetries() { + try { + sessionStorage.removeItem(BOOT_FAILURES_KEY); + } catch { + // Ignore privacy-mode failures. + } +} + function onWindowError(event: ErrorEvent) { recordFatalError(event.message || 'Unhandled error'); } diff --git a/apps/signage/src/main.ts b/apps/signage/src/main.ts index aaac0e26ee1..45f94a3fa6f 100644 --- a/apps/signage/src/main.ts +++ b/apps/signage/src/main.ts @@ -3,12 +3,18 @@ import { bootstrapApplication } from '@angular/platform-browser'; import { AppComponent } from './app/app.component'; import { appConfig } from './app/app.config'; +import { resetBootRetries, scheduleBootRetry } from './app/watchdog'; import { environment } from './environments/environment'; if (environment.production) { enableProdMode(); } -bootstrapApplication(AppComponent, appConfig).catch((err) => - console.error(err), -); +// The recovery watchdog starts inside the application, so a failed start is +// retried here or the display stays blank until someone power-cycles it. +bootstrapApplication(AppComponent, appConfig) + .then(() => resetBootRetries()) + .catch((err) => { + console.error(err); + scheduleBootRetry(); + }); diff --git a/apps/signage/src/tests/media-player.component.spec.ts b/apps/signage/src/tests/media-player.component.spec.ts index 6f6f27ecc28..1a9886226c9 100644 --- a/apps/signage/src/tests/media-player.component.spec.ts +++ b/apps/signage/src/tests/media-player.component.spec.ts @@ -765,6 +765,8 @@ describe('MediaPlayerComponent', () => { spectator.component.index.set(0); spectator.component.active_output.set(0); spectator.component.pending_output.set(0); + // The current item is on screen, not waiting to be revealed + spectator.component['_clearDeferredReveal'](); spectator.component['_output_items'] = [items[0], null]; spectator.component['_item_start'] = Date.now() - 6_000; spectator.component['_item_real_start'] = Date.now() - 6_000; @@ -782,6 +784,41 @@ describe('MediaPlayerComponent', () => { expect(spectator.component.output_plugins()[1]).toBeNull(); }); + it('should not preload over an item that is still waiting to be revealed', () => { + const items = [ + create_item('media-1'), + create_item('webpage-1', { type: 'webpage', duration: 15_000 }), + create_item('webpage-2', { type: 'webpage' }), + ]; + load_playlist(items); + spectator.component['_item_urls'] = { + 'media-1': 'blob:media-1' as any, + 'webpage-1': 'blob:webpage-1' as any, + 'webpage-2': 'blob:webpage-2' as any, + }; + spectator.component.index.set(0); + spectator.component.hold_over_item.set(false); + spectator.component.state.set('PLAYING'); + spectator.component['clearTimeout']('wait-for-url'); + + // The webpage loads into the inactive output while the image stays up + spectator.component.setPlaylistItem(1); + const pending = spectator.component.pending_output(); + expect(pending).not.toBe(spectator.component.active_output()); + expect(spectator.component.defer_reveal()).toBe(true); + + // Its load is slow enough to reach the preload window for the next item + spectator.component['_item_real_start'] = Date.now() - 6_000; + spectator.component['_processURLs'](); + + expect(spectator.component['_output_items'][pending].id).toBe( + 'webpage-1', + ); + expect( + spectator.component['_web_element'](pending).nativeElement.src, + ).toBe('blob:webpage-1'); + }); + it('should keep the preloaded webpage output invisible until it is active', () => { const items = [ create_item('webpage-1', { @@ -796,6 +833,8 @@ describe('MediaPlayerComponent', () => { spectator.component.index.set(0); spectator.component.active_output.set(0); spectator.component.pending_output.set(0); + // The current item is on screen, not waiting to be revealed + spectator.component['_clearDeferredReveal'](); spectator.component['_output_items'] = [items[0], null]; spectator.component['_item_start'] = Date.now() - 6_000; spectator.component['_item_real_start'] = Date.now() - 6_000; @@ -842,6 +881,8 @@ describe('MediaPlayerComponent', () => { spectator.component.index.set(0); spectator.component.active_output.set(0); spectator.component.pending_output.set(0); + // The current item is on screen, not waiting to be revealed + spectator.component['_clearDeferredReveal'](); spectator.component['_output_items'] = [items[0], null]; spectator.component['_item_start'] = Date.now() - 6_000; spectator.component['_item_real_start'] = Date.now() - 6_000; @@ -1166,6 +1207,117 @@ describe('MediaPlayerComponent', () => { expect(spectator.component['_item_start']).toBe(5_000); }); + describe('play-through plugins', () => { + const play_through = () => + create_item('plugin-1', { + type: 'plugin', + duration: 20_000, + plugin: { + id: 'plugin-1', + name: 'Story', + uri: 'https://plugins.example/story', + playback_type: 'playsthrough', + } as any, + }); + + /** Show the plugin, ahead of an image, as though playback started `ago` ms back */ + const show = (ago: number) => { + load_playlist([play_through(), create_item('image-1')]); + spectator.component.index.set(0); + spectator.component.state.set('PLAYING'); + spectator.component['_item_start'] = Date.now() - ago; + return spectator.component['_item_output'].get('plugin-1'); + }; + + it('should advance when the plugin reports it finished', () => { + const output = show(1_000); + const next_item_spy = vi + .spyOn(spectator.component, 'nextItem') + .mockImplementation(() => undefined); + spectator.component.onPluginStatus('ready', output); + + spectator.component['_updateItem'](); + expect(next_item_spy).not.toHaveBeenCalled(); + + spectator.component.onPluginStatus('finished', output); + spectator.component['_updateItem'](); + expect(next_item_spy).toHaveBeenCalled(); + }); + + it('should advance after a bound when a running plugin never finishes', () => { + const output = show(1_000); + const next_item_spy = vi + .spyOn(spectator.component, 'nextItem') + .mockImplementation(() => undefined); + spectator.component.onPluginStatus('ready', output); + + // Well past its scheduled duration, but it may legitimately run long + spectator.component['_item_start'] = Date.now() - 4 * 60_000; + spectator.component['_updateItem'](); + expect(next_item_spy).not.toHaveBeenCalled(); + expect(spectator.component.isMidPlayThroughItem()).toBe(true); + + spectator.component['_item_start'] = Date.now() - 5 * 60_000 - 1; + spectator.component['_updateItem'](); + expect(next_item_spy).toHaveBeenCalled(); + // Nor does it hold back an update any longer + expect(spectator.component.isMidPlayThroughItem()).toBe(false); + }); + + it('should retry a lone plugin that never responds instead of holding it', () => { + vi.useFakeTimers(); + const item = play_through(); + load_playlist([item]); + spectator.component.state.set('PLAYING'); + const output = spectator.component['_item_output'].get('plugin-1'); + // The frame loads an error page that never speaks the protocol + spectator.component.onPluginLoad(output); + expect(spectator.component['_shown_item_id']).toBe('plugin-1'); + + spectator.component['_item_start'] = Date.now() - 20_001; + spectator.component['_updateItem'](); + + // Taken off screen, so the player no longer reports it as shown + expect(spectator.component.output_plugins()[output]).toBeNull(); + expect(spectator.component['_shown_item_id']).toBe(''); + + vi.advanceTimersByTime(30_000); + expect(spectator.component.output_plugins()[output]).toBe( + item.plugin, + ); + }); + + it('should keep holding a lone plugin that is running but never finishes', () => { + const item = play_through(); + load_playlist([item]); + spectator.component.state.set('PLAYING'); + const output = spectator.component['_item_output'].get('plugin-1'); + spectator.component.onPluginStatus('ready', output); + + spectator.component['_item_start'] = Date.now() - 60 * 60_000; + spectator.component['_updateItem'](); + + expect(spectator.component.output_plugins()[output]).toBe( + item.plugin, + ); + }); + + it('should advance after its duration when the plugin never responds', () => { + show(1_000); + const next_item_spy = vi + .spyOn(spectator.component, 'nextItem') + .mockImplementation(() => undefined); + + spectator.component['_item_start'] = Date.now() - 19_000; + spectator.component['_updateItem'](); + expect(next_item_spy).not.toHaveBeenCalled(); + + spectator.component['_item_start'] = Date.now() - 20_001; + spectator.component['_updateItem'](); + expect(next_item_spy).toHaveBeenCalled(); + }); + }); + it('should clear the plugin output and skip on a fatal plugin error', () => { vi.useFakeTimers(); const plugin_item = create_item('plugin-1', { diff --git a/apps/signage/src/tests/signage.component.spec.ts b/apps/signage/src/tests/signage.component.spec.ts index 8c50ab007a7..c3bb4361f17 100644 --- a/apps/signage/src/tests/signage.component.spec.ts +++ b/apps/signage/src/tests/signage.component.spec.ts @@ -7,6 +7,7 @@ import { import { SettingsService } from '@placeos/common'; import { MockProvider } from 'ng-mocks'; +import { MediaPlayerComponent } from '../app/media-player.component'; import { SignagePanelComponent } from '../app/signage.component'; import { SignageService } from '../app/signage.service'; @@ -252,6 +253,52 @@ describe('SignagePanelComponent', () => { }); }); + describe('remote playback commands', () => { + let shell_frame: HTMLIFrameElement; + let shell: Window; + + const post = (type: string, source: Window) => + window.dispatchEvent( + new MessageEvent('message', { data: { type }, source }), + ); + + const player_state = () => + spectator.query(MediaPlayerComponent)?.state(); + + beforeEach(() => { + // Stand in for a shell that embeds the player in an iframe + shell_frame = document.createElement('iframe'); + document.body.appendChild(shell_frame); + shell = shell_frame.contentWindow as Window; + vi.spyOn(window, 'parent', 'get').mockReturnValue(shell); + build_component(); + }); + + afterEach(() => { + vi.restoreAllMocks(); + shell_frame.remove(); + }); + + it('should pause and resume when the parent shell asks', () => { + post('signage:pause', shell); + expect(player_state()).toBe('PAUSED'); + + post('signage:resume', shell); + expect(player_state()).toBe('PLAYING'); + }); + + it('should ignore commands from any other window', () => { + const content_frame = document.createElement('iframe'); + document.body.appendChild(content_frame); + + post('signage:pause', content_frame.contentWindow as Window); + post('signage:pause', window); + + expect(player_state()).toBe('PLAYING'); + content_frame.remove(); + }); + }); + it('should always forward player events to the signage service', () => { build_component(); const event = { type: 'media_count', ref_id: 'media-1' } as const; diff --git a/apps/signage/src/tests/template.component.spec.ts b/apps/signage/src/tests/template.component.spec.ts index f6518b59580..a8a7f104b0a 100644 --- a/apps/signage/src/tests/template.component.spec.ts +++ b/apps/signage/src/tests/template.component.spec.ts @@ -95,7 +95,30 @@ describe('SignageTemplateComponent', () => { }); }); - afterEach(() => vi.restoreAllMocks()); + let shell_frame: HTMLIFrameElement | null = null; + + /** Embed the player in a stand-in manager preview, its parent frame */ + const embedInShell = () => { + shell_frame = document.createElement('iframe'); + document.body.appendChild(shell_frame); + const shell = shell_frame.contentWindow as Window; + vi.spyOn(window, 'parent', 'get').mockReturnValue(shell); + return shell; + }; + + const postLayouts = (layouts: unknown, source: Window) => + window.dispatchEvent( + new MessageEvent('message', { + data: { type: 'signage:template-layouts', layouts }, + source, + }), + ); + + afterEach(() => { + vi.restoreAllMocks(); + shell_frame?.remove(); + shell_frame = null; + }); it('loads the template, background, and layout plugins', async () => { spectator = create_component({ @@ -307,18 +330,14 @@ describe('SignageTemplateComponent', () => { }); it('previews posted layouts only in debug mode', async () => { + const shell = embedInShell(); spectator = create_component({ params: { template_id: 'template-1', system_id: 'display-1' }, }); await vi.waitFor(() => { expect(spectator.component.layout_items()).toHaveLength(1); }); - const post = (layouts: unknown) => - window.dispatchEvent( - new MessageEvent('message', { - data: { type: 'signage:template-layouts', layouts }, - }), - ); + const post = (layouts: unknown) => postLayouts(layouts, shell); post([]); expect(spectator.component.layout_items()).toHaveLength(1); @@ -351,6 +370,7 @@ describe('SignageTemplateComponent', () => { ? Promise.reject(new Error('Not found')) : Promise.resolve(pending), ); + const shell = embedInShell(); spectator = create_component({ params: { template_id: 'template-1', system_id: 'display-1' }, queryParams: { debug: 'true' }, @@ -363,15 +383,9 @@ describe('SignageTemplateComponent', () => { {}, ); - window.dispatchEvent( - new MessageEvent('message', { - data: { - type: 'signage:template-layouts', - layouts: [ - { position: 'top', y_pos: 0.2, plugin_id: 'plugin-1' }, - ], - }, - }), + postLayouts( + [{ position: 'top', y_pos: 0.2, plugin_id: 'plugin-1' }], + shell, ); await spectator.fixture.whenStable(); @@ -384,6 +398,53 @@ describe('SignageTemplateComponent', () => { }); }); + it('ignores preview layouts from any window but the parent', async () => { + embedInShell(); + debug.set(true); + spectator = create_component({ + params: { template_id: 'template-1', system_id: 'display-1' }, + }); + await vi.waitFor(() => { + expect(spectator.component.layout_items()).toHaveLength(1); + }); + const content_frame = document.createElement('iframe'); + document.body.appendChild(content_frame); + + postLayouts([], content_frame.contentWindow as Window); + postLayouts([], window); + + expect(spectator.component.layout_items()).toHaveLength(1); + content_frame.remove(); + }); + + it('does not ask the parent for preview layouts outside debug mode', async () => { + const ask = vi.spyOn(embedInShell(), 'postMessage'); + spectator = create_component({ + params: { template_id: 'template-1', system_id: 'display-1' }, + }); + + await vi.waitFor(() => { + expect(spectator.component.template()?.id).toBe('template-1'); + }); + + expect(ask).not.toHaveBeenCalled(); + }); + + it('asks the parent for preview layouts in debug mode', async () => { + const ask = vi.spyOn(embedInShell(), 'postMessage'); + spectator = create_component({ + params: { template_id: 'template-1', system_id: 'display-1' }, + queryParams: { debug: 'true' }, + }); + + await vi.waitFor(() => { + expect(ask).toHaveBeenCalledWith( + { type: 'signage:template-preview-ready' }, + '*', + ); + }); + }); + it('uses the stored display when opening a template without one', () => { localStorage.setItem('PlaceOS.SIGNAGE.display', 'display-2'); spectator = create_component({ params: { template_id: 'template-1' } }); diff --git a/apps/signage/src/tests/watchdog.spec.ts b/apps/signage/src/tests/watchdog.spec.ts index 12e789357e8..e212f64bf3e 100644 --- a/apps/signage/src/tests/watchdog.spec.ts +++ b/apps/signage/src/tests/watchdog.spec.ts @@ -1,9 +1,11 @@ import { - clearCachesAndReload, + clearApplicationCache, recordFatalError, recordHeartbeat, requestRecovery, + resetBootRetries, resetWatchdog, + scheduleBootRetry, stalledSignals, startWatchdog, watchdogState, @@ -13,7 +15,7 @@ const MINUTE = 60 * 1000; describe('recovery watchdog', () => { let reload: any; - let hard_reload: any; + let clear_cache: any; let stop: () => void; let expected_to_run: boolean; @@ -22,7 +24,7 @@ describe('recovery watchdog', () => { stop?.(); stop = startWatchdog({ reload, - hardReload: hard_reload, + clearCache: clear_cache, isExpectedToRun: () => expected_to_run, }); }; @@ -64,8 +66,9 @@ describe('recovery watchdog', () => { vi.useFakeTimers(); localStorage.clear(); resetWatchdog(); - reload = vi.fn(); - hard_reload = vi.fn(async () => true); + // A reload that works ends the page, and the watchdog with it + reload = vi.fn(() => stop()); + clear_cache = vi.fn(async () => true); expected_to_run = true; stop = () => undefined; start(); @@ -83,14 +86,15 @@ describe('recovery watchdog', () => { // A failed boot is most often a bad cached build, so it goes straight // to clearing the cache rather than spending plain reloads first - expect(hard_reload).toHaveBeenCalledTimes(1); - expect(reload).not.toHaveBeenCalled(); + expect(clear_cache).toHaveBeenCalledTimes(1); + expect(clear_cache).toHaveBeenCalledBefore(reload); + expect(reload).toHaveBeenCalledTimes(1); }); it('should give the player time to boot before recovering', async () => { await vi.advanceTimersByTimeAsync(4 * MINUTE); - expect(hard_reload).not.toHaveBeenCalled(); + expect(clear_cache).not.toHaveBeenCalled(); expect(reload).not.toHaveBeenCalled(); }); @@ -99,7 +103,7 @@ describe('recovery watchdog', () => { await vi.advanceTimersByTimeAsync(60 * MINUTE); - expect(hard_reload).not.toHaveBeenCalled(); + expect(clear_cache).not.toHaveBeenCalled(); expect(reload).not.toHaveBeenCalled(); }); @@ -110,7 +114,7 @@ describe('recovery watchdog', () => { await vi.advanceTimersByTimeAsync(4 * MINUTE); - expect(hard_reload).not.toHaveBeenCalled(); + expect(clear_cache).not.toHaveBeenCalled(); }); it('should recover when content stops being visible', async () => { @@ -156,12 +160,12 @@ describe('recovery watchdog', () => { }); it('should fall back to a plain reload when a failed boot cannot clear the cache', async () => { - hard_reload = vi.fn(async () => false); + clear_cache = vi.fn(async () => false); start(); await vi.advanceTimersByTimeAsync(6 * MINUTE); - expect(hard_reload).toHaveBeenCalledTimes(1); + expect(clear_cache).toHaveBeenCalledTimes(1); expect(reload).toHaveBeenCalledTimes(1); }); @@ -235,7 +239,7 @@ describe('recovery watchdog', () => { for (let attempt = 0; attempt < 4; attempt++) await stallAgain(); expect(watchdogState().recoveries_throttled).toBe(true); reload.mockClear(); - hard_reload.mockClear(); + clear_cache.mockClear(); // Half an hour later, still stalled resetWatchdog(); @@ -246,12 +250,12 @@ describe('recovery watchdog', () => { await runStalled(); expect(reload).not.toHaveBeenCalled(); - expect(hard_reload).not.toHaveBeenCalled(); + expect(clear_cache).not.toHaveBeenCalled(); }); it('should clear the application cache once recoveries are throttled', async () => { for (let attempt = 0; attempt < 4; attempt++) await stallAgain(); - expect(hard_reload).not.toHaveBeenCalled(); + expect(clear_cache).not.toHaveBeenCalled(); expect(watchdogState().recoveries_throttled).toBe(true); resetWatchdog(); @@ -261,11 +265,11 @@ describe('recovery watchdog', () => { beat(); await runStalled(); - expect(hard_reload).toHaveBeenCalledTimes(1); + expect(clear_cache).toHaveBeenCalledTimes(1); }); it('should fall back to a plain reload when the cache cannot be cleared', async () => { - hard_reload = vi.fn(async () => false); + clear_cache = vi.fn(async () => false); for (let attempt = 0; attempt < 4; attempt++) await stallAgain(); reload.mockClear(); @@ -276,7 +280,7 @@ describe('recovery watchdog', () => { beat(); await runStalled(); - expect(hard_reload).toHaveBeenCalledTimes(1); + expect(clear_cache).toHaveBeenCalledTimes(1); expect(reload).toHaveBeenCalledTimes(1); }); @@ -296,7 +300,60 @@ describe('recovery watchdog', () => { expect(watchdogState().recoveries_in_last_hour).toBe(1); // Back to plain reloads rather than cache clearing expect(reload).toHaveBeenCalledTimes(4); - expect(hard_reload).not.toHaveBeenCalled(); + expect(clear_cache).not.toHaveBeenCalled(); + }); + + it('should reload anyway when clearing the cache never finishes', async () => { + clear_cache = vi.fn(() => new Promise(() => undefined)); + // Nor does the reload that follows: the server never answers it + reload = vi.fn(); + start(); + + // The failed boot is recovered at five minutes and never completes + await vi.advanceTimersByTimeAsync(6 * MINUTE); + expect(clear_cache).toHaveBeenCalledTimes(1); + expect(reload).not.toHaveBeenCalled(); + expect(watchdogState().recovering).toBe(true); + + // Two minutes after it started, it reloads anyway and lets go + await vi.advanceTimersByTimeAsync(MINUTE); + expect(reload).toHaveBeenCalledTimes(1); + expect(watchdogState().recovering).toBe(false); + + // So the next check can try again instead of waiting forever + await vi.advanceTimersByTimeAsync(30 * 1000); + expect(clear_cache).toHaveBeenCalledTimes(2); + }); + + it('should not reload again when an abandoned cache clear finishes late', async () => { + let finish: (cleared: boolean) => void = () => undefined; + clear_cache = vi.fn( + () => new Promise((resolve) => (finish = resolve)), + ); + // The reload never completes either, so this page stays + reload = vi.fn(); + start(); + + await vi.advanceTimersByTimeAsync(7 * MINUTE); + expect(reload).toHaveBeenCalledTimes(1); + + // The first attempt finally gives up while the fallback reload is + // still loading; it must not start a competing reload + finish(false); + await vi.advanceTimersByTimeAsync(0); + + expect(reload).toHaveBeenCalledTimes(1); + }); + + it('should fall back to a plain reload when clearing the cache throws', async () => { + clear_cache = vi.fn(async () => { + throw new Error('denied'); + }); + start(); + + await vi.advanceTimersByTimeAsync(6 * MINUTE); + + expect(reload).toHaveBeenCalledTimes(1); }); it('should allow a recovery to be requested directly', () => { @@ -449,26 +506,22 @@ describe('cache clearing recovery', () => { vi.unstubAllGlobals(); }); - it('should reload the current url, keeping the route it displays', async () => { - // The display to show, and whether to show it in debug mode, live in - // the hash, so recovering must not navigate to the base path - expect(await clearCachesAndReload()).toBe(true); + it('should remove the service worker and its caches, leaving the reload to the caller', async () => { + expect(await clearApplicationCache()).toBe(true); expect(unregister).toHaveBeenCalledTimes(1); expect(delete_cache).toHaveBeenCalledTimes(2); - expect(reload).toHaveBeenCalledTimes(1); + expect(reload).not.toHaveBeenCalled(); }); - it('should still reload when the cache cannot be cleared', async () => { + it('should carry on when the caches cannot be deleted', async () => { vi.stubGlobal('caches', { keys: async () => { throw new Error('denied'); }, }); - expect(await clearCachesAndReload()).toBe(true); - - expect(reload).toHaveBeenCalledTimes(1); + expect(await clearApplicationCache()).toBe(true); }); it('should not clear the cache when the server cannot be reached', async () => { @@ -479,22 +532,90 @@ describe('cache clearing recovery', () => { }), ); - expect(await clearCachesAndReload()).toBe(false); + expect(await clearApplicationCache()).toBe(false); expect(unregister).not.toHaveBeenCalled(); expect(delete_cache).not.toHaveBeenCalled(); expect(reload).not.toHaveBeenCalled(); }); + it('should not clear the cache when the server never answers', async () => { + vi.useFakeTimers(); + vi.stubGlobal( + 'fetch', + vi.fn( + (_: string, init: RequestInit) => + new Promise((_resolve, reject) => + init.signal?.addEventListener('abort', () => + reject(new Error('aborted')), + ), + ), + ), + ); + + const result = clearApplicationCache(); + await vi.advanceTimersByTimeAsync(15 * 1000); + + expect(await result).toBe(false); + expect(unregister).not.toHaveBeenCalled(); + expect(reload).not.toHaveBeenCalled(); + vi.useRealTimers(); + }); + it('should not clear the cache when the server errors', async () => { vi.stubGlobal( 'fetch', vi.fn(async () => ({ ok: false })), ); - expect(await clearCachesAndReload()).toBe(false); + expect(await clearApplicationCache()).toBe(false); expect(unregister).not.toHaveBeenCalled(); expect(reload).not.toHaveBeenCalled(); }); }); + +describe('boot retry', () => { + let console_error: ReturnType; + + beforeEach(() => { + vi.useFakeTimers(); + sessionStorage.clear(); + console_error = vi.spyOn(console, 'error').mockImplementation(() => {}); + }); + + afterEach(() => { + sessionStorage.clear(); + console_error.mockRestore(); + vi.useRealTimers(); + }); + + it('should reload after a failed start, waiting longer each time up to a cap', () => { + const reload = vi.fn(); + + const delays = Array.from({ length: 8 }, () => + scheduleBootRetry(reload), + ); + + expect(delays).toEqual([ + 10_000, 20_000, 40_000, 80_000, 160_000, 300_000, 300_000, 300_000, + ]); + vi.advanceTimersByTime(300_000); + expect(reload).toHaveBeenCalledTimes(8); + // Printed without debug mode, which needs settings that never loaded + expect(console_error).toHaveBeenCalledWith( + expect.stringContaining( + 'Application failed to start; reloading in 10s', + ), + ); + }); + + it('should start from the shortest wait again after a successful start', () => { + scheduleBootRetry(vi.fn()); + scheduleBootRetry(vi.fn()); + + resetBootRetries(); + + expect(scheduleBootRetry(vi.fn())).toBe(10_000); + }); +});