fix(signage): recover from stalled plugins, hung recovery and failed boot

apps/signage/DEBUGGING.md

@@ -85,6 +85,9 @@ makes the display request use `?preview=true`.
 | Old version running                | `updates.new_version`, `updates.reload_pending` (a reload waits for the network and for play-through content to finish), `updates.last_check`                               |
 | Blank screen after a reboot        | Likely offline boot — check `online`, then whether cached credentials exist                                                                                                 |
 | Player reloading itself            | `watchdog.recent_reloads` and `watchdog.last_error` — something fatal stalled a core loop                                                                                   |
+| Paused and does not resume         | Pause and resume messages are obeyed only from the parent frame. Check what embeds the player and `players[].state`                                                         |
+| Plugin cut short, or held long     | A play-through plugin advances on `finished`, or after a limit. Look for `did not report finished in time` in the console                                                   |
+| Blank screen, no `window.signage`  | The application did not start. Look for `Application failed to start` in the console; it reloads with a backoff                                                             |
 
 ## Recovery watchdog
 
@@ -105,19 +108,24 @@ boot that never completes is most often a bad cached build. That deadline only
 applies once the device has been bootstrapped to a display — one sitting on the
 picker is waiting for a person, not broken.
 
-| Guard                    | Value                                                      |
-| ------------------------ | ---------------------------------------------------------- |
-| Stall thresholds         | poll 10 min, schedule 5 min, playback 3 min, visible 5 min |
-| Boot deadline            | 5 min from start with nothing on screen                    |
-| Grace before recovering  | 5 min                                                      |
-| Recoveries allowed       | 3 per hour, then 1 per hour                                |
-| Back to 3 per hour after | 2 hours with no recovery                                   |
+| Guard | Value |
+| Stall thresholds | poll 10 min, schedule 5 min, playback 3 min, visible 5 min |
+| Boot deadline | 5 min from start with nothing on screen |
+| Grace before recovering | 5 min |
+| Recoveries allowed | 3 per hour, then 1 per hour |
+| Back to 3 per hour after | 2 hours with no recovery |
 
 Once recoveries are throttled the next one clears the application cache first —
 unregistering the service worker and deleting its caches — in case the cached
 build is what is wrong. That only happens if `location.href` returns a 200, so a
 player is never left with no cached application and no way to fetch a new one;
-if the server cannot be reached it falls back to a plain reload.
+if the server cannot be reached it falls back to a plain reload. A server that
+does not answer within 15 seconds counts as unreachable.
+
+A recovery that has not replaced the page after 2 minutes counts as failed: a
+cache clear that hung, or a reload the server never answered. The watchdog then
+reloads again and starts its checks again, inside the same limits, so a failed
+recovery cannot stop the watchdog until someone restarts the device.
 
 A recovery reload does **not** wait for the network, unlike an update reload. A
 stalled player should restart whether or not the backend is up, and it can boot
@@ -132,6 +140,32 @@ Failed initialisation — the app giving up because it cannot load the current
 user — is routed through the same limits, so it cannot restart the player every
 thirty seconds on its own.
 
+The watchdog starts inside the application, so it cannot see a start that fails
+before the application exists. That case has its own retry: the player reloads
+after 10 seconds, and the wait doubles after each consecutive failure to a
+maximum of 5 minutes. The count is in `sessionStorage["SIGNAGE.boot_failures"]`
+and is removed after a successful start.
+
+### Content that holds the screen
+
+A play-through plugin advances when it reports `finished`. If it never sends a
+plugin message (for example, the page did not load), it advances after its
+configured duration, like a static plugin. If it sends messages but never
+reports `finished`, it advances after twice its configured duration, held
+between 5 and 60 minutes. After that limit, it also stops holding back an
+update reload.
+
+When the plugin is the only item, there is nothing to advance to. A lone
+play-through plugin that never sent a plugin message is then treated as a
+failed load: it is removed from the screen and loaded again after 30 seconds,
+until it responds. A lone plugin that responds but never reports `finished`
+stays on screen, as any single item does.
+
+Pause and resume messages (US-SIG-024) are obeyed only from the parent frame.
+Webpages and plugins on screen cannot pause the player. This is important
+because a paused player still checks in with the watchdog, so the watchdog does
+not recover it.
+
 `watchdog.booted`, `watchdog.recoveries_throttled` and `watchdog.last_recovery`
 show where in that sequence a player is.
 
@@ -145,15 +179,15 @@ recovered and you want to know what from.
 
 ## Storage
 
-| Location                                               | Holds                                     |
-| ------------------------------------------------------ | ----------------------------------------- |
-| `localStorage["PlaceOS.SIGNAGE.display_details.<id>"]` | Last known display payload, used offline  |
-| `localStorage["PlaceOS.SIGNAGE.cached_files"]`         | Media cache index (urls, sizes, owners)   |
-| `localStorage["PlaceOS.SIGNAGE.display"]`              | Bootstrapped display id                   |
-| `localStorage["PLACEOS.org.*"]`                        | Cached zone data and last known authority |
-| `localStorage["PlaceOS.SIGNAGE.watchdog_reloads"]`     | Timestamps of automatic recoveries        |
-| `sessionStorage["SIGNAGE.debug"]`, `["SIGNAGE.muted"]` | Debug and mute state                      |
-| IndexedDB `SignageMedia` → `files`                     | The cached media files themselves         |
+| Location | Holds |
+| `localStorage["PlaceOS.SIGNAGE.display_details.<id>"]` | Last known display payload, used offline |
+| `localStorage["PlaceOS.SIGNAGE.cached_files"]` | Media cache index (urls, sizes, owners) |
+| `localStorage["PlaceOS.SIGNAGE.display"]` | Bootstrapped display id |
+| `localStorage["PLACEOS.org.*"]` | Cached zone data and last known authority |
+| `localStorage["PlaceOS.SIGNAGE.watchdog_reloads"]` | Timestamps of automatic recoveries |
+| `sessionStorage["SIGNAGE.debug"]`, `["SIGNAGE.muted"]` | Debug and mute state |
+| `sessionStorage["SIGNAGE.boot_failures"]` | Consecutive failed starts, for the backoff |
+| IndexedDB `SignageMedia` → `files` | The cached media files themselves |
 
 ## Resetting
 

apps/signage/USER_STORIES.md

@@ -38,6 +38,7 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device
 - The app also reads `OSK.enabled` from localStorage and enables the virtual keyboard when the value is `true`.
 - The bootstrap screen can clear stored signage bootstrap data when opened with `?clear=true`.
 - Clearing removes both the current display key and the legacy `PlaceOS.SIGNAGE.building` key.
+- If the application fails to start, it reloads after 10 seconds. The wait doubles after each consecutive failure, to a maximum of 5 minutes, and resets after a successful start.
 
 ---
 
@@ -161,6 +162,7 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device
 - Webpage items play for their configured effective duration.
 - A single valid webpage item remains loaded instead of reloading on every loop.
 - Upcoming webpage items can be preloaded on the inactive output shortly before transition.
+- Preloading does not start while the current item is still waiting to be revealed or is in transition.
 - Webpage media is not cached as a local file.
 
 ---
@@ -179,6 +181,9 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device
 - If a plugin does not report load or ready, the player sends config after a 15 second wait.
 - Static plugins follow the configured effective duration.
 - Play-through plugins advance when they report `finished`.
+- A play-through plugin that never sends a plugin message advances after its configured duration, like a static plugin.
+- If that plugin is the only item, it is removed from the screen and loaded again after 30 seconds.
+- A play-through plugin that does not report `finished` advances after twice its configured duration, but not before 5 minutes and not after 60 minutes.
 - Interactive plugins can request a new playback duration through plugin interaction events.
 - Upcoming plugin items can be preloaded on the inactive output shortly before transition.
 - Fatal plugin errors advance to the next media item.
@@ -425,6 +430,8 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device
 **Acceptance Criteria:**
 
 - The signage panel listens for object postMessage payloads.
+- Only messages from the parent frame are accepted. Messages from other windows, such as webpage or plugin content on screen, are ignored.
+- A player that is not in a frame ignores all pause and resume messages.
 - A payload with `type: 'signage:pause'` pauses all player instances.
 - A payload with `type: 'signage:resume'` resumes all player instances.
 - Unknown payloads are ignored.
@@ -483,6 +490,7 @@ The Signage app is a kiosk-style digital signage player. It bootstraps a device
 - Object URLs outside the nearby window are revoked.
 - If an active item's URL is not ready, the player waits and retries item selection.
 - Webpage and plugin outputs are prepared on the inactive layer near the end of the current item so they can be revealed after loading.
+- An item that is still waiting to be revealed keeps its output. The next item is not prepared until the current item is on screen.
 
 ---
 

apps/signage/src/app/media-player.component.ts

@@ -50,6 +50,12 @@ const INTERACTIVE_PRELOAD_LEAD_TIME = 10 * 1000;
 const WEBPAGE_REVEAL_DELAY = 3 * 1000;
 /** Max wait for plugin load/ready before continuing playback anyway */
 const PLUGIN_LOAD_TIMEOUT = 15 * 1000;
+/**
+ * Bounds on how long a play-through plugin may run without reporting that it
+ * finished. It gets twice its scheduled duration, held between these.
+ */
+const PLAY_THROUGH_MIN_LIMIT = 5 * 60 * 1000;
+const PLAY_THROUGH_MAX_LIMIT = 60 * 60 * 1000;
 /** Minimum spacing between attempts to resolve a URL that failed to resolve */
 const URL_RETRY_DELAY = 1000;
 
@@ -356,6 +362,8 @@ export class MediaPlayerComponent
     private _item_output = new Map<string, 0 | 1>();
     private _output_items: [MediaPlayerItem, MediaPlayerItem] = [null, null];
     private _ready_output_items = new Set<string>();
+    /** Plugin outputs that have sent at least one plugin protocol message */
+    private _responded_output_items = new Set<string>();
 
     public get playlist_items() {
         return this._item_playlist;
@@ -628,15 +636,22 @@ export class MediaPlayerComponent
      * Whether the item on screen plays to completion, so interrupting it now
      * would be noticed. Images and webpages hold a static frame and can be
      * replaced without anyone seeing a difference; videos and plugins that
-     * report when they finish cannot.
+     * report when they finish cannot. A plugin held on screen past its limit,
+     * as a lone item is, is not about to finish and does not count.
      */
     public isMidPlayThroughItem() {
         const item = this.active_item;
         if (!item || this.state() !== 'PLAYING') return false;
         if (item.type === 'video') return true;
         if (item.type === 'plugin') {
             const playback = item.plugin?.playback_type;
-            return playback === 'playsthrough' || playback === 'interactive';
+            const limit =
+                playback === 'playsthrough'
+                    ? this._playThroughLimit(item)
+                    : playback === 'interactive'
+                      ? this._effectivePlaybackDuration(item)
+                      : 0;
+            return time() - this._item_start < limit;
         }
         return false;
     }
@@ -745,13 +760,17 @@ export class MediaPlayerComponent
             this.progress_start.set(0);
             this.setPlaylistItem(0);
         }
-        // For playsthrough plugins, advance when plugin signals finished
+        // For playsthrough plugins, advance when plugin signals finished, or
+        // once it has overrun its limit so a hung plugin cannot hold the
+        // screen forever
         if (
             item?.type === 'plugin' &&
             item.plugin?.playback_type === 'playsthrough'
         ) {
             if (this._plugin_finished) {
                 this.nextItem();
+            } else if (now > this._item_start + this._playThroughLimit(item)) {
+                this._handleOverrunPlugin(item);
             }
             return;
         }
@@ -967,6 +986,7 @@ export class MediaPlayerComponent
         if (item) {
             this._item_output.delete(item.id);
             this._ready_output_items.delete(this._outputKey(output, item));
+            this._responded_output_items.delete(this._outputKey(output, item));
         }
         this._output_items[output] = null;
     }
@@ -1150,9 +1170,13 @@ export class MediaPlayerComponent
         if (!item || item.type !== 'plugin') return;
         if (this._item_output.get(item.id) !== output) return;
         log('MediaPlayer', `Plugin status: ${status}`, [item.name]);
+        if (status !== 'unknown') {
+            this._responded_output_items.add(this._outputKey(output, item));
+        }
         if (status === 'ready') {
             this._handlePluginReady(item, output);
-        } else if (status === 'finished') {
+        } else if (status === 'finished' && this.active_item?.id === item.id) {
+            // A preloaded plugin finishing must not end the one on screen
             this._plugin_finished = true;
         }
     }
@@ -1191,16 +1215,55 @@ export class MediaPlayerComponent
         log('MediaPlayer', `Plugin error: ${error.message}`, [error], 'error');
         if (!error.fatal) return;
         if (item?.type === 'plugin') {
-            this._markNotShown(item);
-            this._handled_error_cycle = this._currentMediaCycle();
-            this._clearDeferredReveal();
-            this._clearOutput(output);
-            this._skipFailedMedia(this._item_start || time());
+            this._failPluginItem(item, output);
         } else {
             this.nextItem();
         }
     }
 
+    /**
+     * Treat a plugin as failed to load: remove it from screen and skip it, or
+     * retry it after a delay when there is nothing else to show.
+     */
+    private _failPluginItem(item: MediaPlayerItem, output: 0 | 1) {
+        this._markNotShown(item);
+        this._handled_error_cycle = this._currentMediaCycle();
+        this._clearDeferredReveal();
+        this._clearOutput(output);
+        this._skipFailedMedia(this._item_start || time());
+    }
+
+    /**
+     * A play-through plugin that has overrun its limit. With other items to
+     * show, move on. A lone one is held like any single item, unless it never
+     * sent a plugin message: then it can never finish and is most likely an
+     * error page, so it is retried the same way as a fatal plugin error.
+     */
+    private _handleOverrunPlugin(item: MediaPlayerItem) {
+        if (!this._shouldHoldSingleInteractiveItem(item)) {
+            log(
+                'MediaPlayer',
+                `Plugin "${item.name}" did not report finished in time; continuing.`,
+                [item.plugin?.uri],
+                'warn',
+            );
+            this.nextItem();
+            return;
+        }
+        // Already removed from screen and waiting for its retry
+        const output = this._item_output.get(item.id);
+        if (output === undefined || this._pluginResponded(item, output)) {
+            return;
+        }
+        log(
+            'MediaPlayer',
+            `Plugin "${item.name}" never responded; retrying.`,
+            [item.plugin?.uri],
+            'warn',
+        );
+        this._failPluginItem(item, output);
+    }
+
     private _showPlugin(item: MediaPlayerItem, output: 0 | 1) {
         log('MediaPlayer', `Showing plugin: ${item.name}`, [item.plugin?.name]);
         this._item_output.set(item.id, output);
@@ -1269,6 +1332,29 @@ export class MediaPlayerComponent
         return this._playback_duration || item?.duration || 15 * 1000;
     }
 
+    /** Whether the plugin on `output` has sent any plugin protocol message */
+    private _pluginResponded(item: MediaPlayerItem, output: 0 | 1) {
+        return this._responded_output_items.has(this._outputKey(output, item));
+    }
+
+    /**
+     * How long a play-through plugin may hold the screen without reporting
+     * that it finished. One that never sent a plugin message - a page that
+     * failed to load, or not a plugin at all - cannot report it, so it gets
+     * its scheduled duration like a static item. One that did gets twice
+     * that, within bounds, so a long run is not cut short but a hung plugin
+     * cannot hold the screen forever.
+     */
+    private _playThroughLimit(item: MediaPlayerItem) {
+        const duration = this._effectivePlaybackDuration(item);
+        const output = this._item_output.get(item.id) ?? this.active_output();
+        if (!this._pluginResponded(item, output)) return duration;
+        return Math.min(
+            Math.max(duration * 2, PLAY_THROUGH_MIN_LIMIT),
+            PLAY_THROUGH_MAX_LIMIT,
+        );
+    }
+
     private _resetPlayback(playback_duration = 0) {
         if (playback_duration > 0) {
             this._playback_duration = playback_duration;
@@ -1457,6 +1543,15 @@ export class MediaPlayerComponent
 
     private _shouldPreloadUpcomingInteractiveContent() {
         if (!this._item_real_start) return false;
+        // Until the current item is revealed it occupies the inactive output,
+        // and preloading there would replace it with the next item
+        if (
+            this.defer_reveal() ||
+            this.in_animation() ||
+            this.pending_output() !== this.active_output()
+        ) {
+            return false;
+        }
         const item = this.active_item;
         const remaining =
             this._effectivePlaybackDuration(item) -
@@ -1704,6 +1799,7 @@ export class MediaPlayerComponent
         this._item_output.clear();
         this._output_items = [null, null];
         this._ready_output_items.clear();
+        this._responded_output_items.clear();
         this._setOutputPlugin(0, null);
         this._setOutputPlugin(1, null);
     }

apps/signage/src/app/signage.component.ts

@@ -250,7 +250,14 @@ export class SignagePanelComponent extends AsyncHandler implements OnInit {
         sessionStorage.setItem(MUTE_STORAGE_KEY, `${muted}`);
     }
 
+    /**
+     * Pause and resume commands from the shell that embeds the player. Only
+     * the parent frame is obeyed: webpages and plugins on screen post messages
+     * to this window too, and a paused player looks healthy to the watchdog,
+     * so one that paused it would freeze the display for good.
+     */
     private readonly _remote_message_handler = (event: MessageEvent) => {
+        if (window.parent === window || event?.source !== window.parent) return;
         const data = event?.data;
         if (!data || typeof data !== 'object') return;
         if (data.type === REMOTE_PAUSE) this._setPlaybackState('PAUSED');