Skip to content

fix(signage): recover from stalled plugins, hung recovery and failed boot - #518

Merged
MrYuion merged 2 commits into
developfrom
fix/signage-stalls
Oct 2, 2026
Merged

MrYuion merged 2 commits into
developfrom
fix/signage-stalls

Conversation

@MrYuion

@MrYuion MrYuion commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

The player could freeze with no way to recover except a power cycle:

  • Watchdog: a recovery set the _recovering latch, then waited on a server check with no timeout. A server that accepted the connection but never answered stopped all later checks.
  • Play-through plugins: a plugin that 404ed or never sent finished stayed on screen forever. It also blocked update reloads, and the watchdog did not see a stall.
  • Preload: the next item was preloaded while the current webpage or plugin waited to be revealed. The preload replaced the pending item, so the wrong item showed.
  • Pause messages: any window, including webpages and plugins on screen, could pause the player with signage:pause. A paused player looks healthy to the watchdog.
  • Startup: when the application failed to start, the error was only logged. The watchdog starts inside the application, so nothing reloaded the blank screen.

Changes

  • watchdog.ts:
    • The server check times out after 15 s.
    • Every recovery arms a 2 minute fallback reload that also releases the latch.
    • A failed cache clear falls back to a plain reload.
  • media-player.component.ts:
    • Play-through plugins advance after a time limit: their duration if they never sent a message, or 2× their duration (5 to 60 min) if they never sent finished. isMidPlayThroughItem() is false after the limit.
    • No preload while a reveal is pending or an animation runs.
  • signage.component.ts, template.component.ts:
    • Pause, resume and preview layout messages are accepted only from window.parent.
    • The preview-ready request is sent only in debug mode.
  • Review follow-up:
    • A lone play-through plugin that never sent a plugin message is retried like a fatal plugin error.
    • The cache clear no longer reloads; recover does the single reload, guarded by a recovery generation.
  • main.ts: a failed start reloads after 10 s, doubling to 5 min. The count is in sessionStorage["SIGNAGE.boot_failures"] and is cleared after a successful start.

USER_STORIES.md (US-SIG-002, 009, 010, 024, 027) and DEBUGGING.md are updated.

Testing

  • Unit tests for each case, with fake timers. Each new test fails on develop.
    • The watchdog spec's reload mock now stops the watchdog, as a real reload does. Four tests depended on the watchdog not acting after a recovery, which was the bug.
    • Three preload tests preloaded during a pending reveal. They now finish the reveal first.
  • nx test signage passes (385 tests). nx build signage passes. nx lint signage has 12 errors that are also on develop.
  • Tested against a local PlaceOS stack with Playwright:
    • Pause messages: from the same window and from a webpage item they are ignored; from a parent frame they work.
    • Template preview: messages with and without debug.
    • Plugins: a 404 plugin advances after its duration; a plugin that never sends finished advances after 5 min.
    • Preload: a slow webpage is revealed before the next item.
    • Hung recovery: requests to / that never answer. The fallback fired at 120 s, and the server check timed out at exactly 15 s.
    • Boot retry: a failed locale chunk. Reloads came at 10 s, 20 s, 40 s and 80 s.
    • A plain playlist plays as before.

Merge order

Independent of #517 and #519, but all three add rows to the symptom table in apps/signage/DEBUGGING.md. The second and third to merge need a rebase. Based on develop.


Changes made by Claude Opus 5.5 (1M context) in Claude Code, running in T3 Code.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
frontend-templates Ignored Ignored Preview Oct 2, 2026 12:31am UTC

@greptile-apps

greptile-apps Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High risk] Adds recovery logic for failed application startup and stalled playback.

The PR appears safe to merge; the remaining issue makes two debugging tables harder to read.

Findings

  1. P2 Debugging tables do not render ▶

Summary

The signage player now has recovery paths for stalled watchdog checks, stuck plugins, and failed application startup. It also protects pending content reveals and accepts playback and preview messages only from the parent frame.

  • Watchdog recovery and failed-start retries now have time limits.
  • Play-through plugins can time out, and preloading waits for reveals and transitions.
  • Pause, resume, and preview-layout messages are limited to the parent frame.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Watchdog detects a stall] --> B[Start recovery timer]
  B --> C{Clear cache needed?}
  C -- No --> D[Reload page]
  C -- Yes --> E[Check server and clear cache]
  E --> F[Reload if attempt is current]
  B --> G[Two minutes pass]
  G --> H[Abandon attempt and reload]
Loading

Reviews (3) · Last reviewed commit: "fix(signage): retry a lone silent plugin..."

Comment thread apps/signage/src/app/media-player.component.ts Outdated
Comment thread apps/signage/src/app/watchdog.ts
…boot

- Watchdog: the server check before a cache clear times out after
  15 s, and every recovery arms a 2 minute fallback reload. A server
  that never answered latched the watchdog off until a power cycle.
- Play-through plugins advance after a bound when they never report
  finished. A plugin that 404ed or hung held the screen forever and
  blocked update reloads.
- Interactive content is not preloaded while the current item waits to
  be revealed. The preload replaced the pending item.
- Pause, resume and template preview messages are accepted only from
  the parent frame. Content on screen could freeze the player. The
  preview request is sent only in debug mode.
- A failed application start reloads with a capped backoff. The
  watchdog starts inside the application, so nothing recovered it.
- A lone play-through plugin that never sent a plugin message is
  retried once it passes its limit, the same way as a fatal plugin
  error. It was held on screen forever, and an error page still counted
  as content shown, so the watchdog never recovered it.
- The cache clear no longer reloads. `recover` does the single reload,
  guarded by a recovery generation. The fallback timer and
  stopWatchdog advance the generation, so a cache clear from an
  abandoned attempt cannot start a second reload.
@MrYuion
MrYuion force-pushed the fix/signage-stalls branch from f3621c0 to 83655e4 Compare October 2, 2026 00:24
Comment thread apps/signage/DEBUGGING.md
Comment on lines +111 to +112
| Guard | Value |
| Stall thresholds | poll 10 min, schedule 5 min, playback 3 min, visible 5 min |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Debugging tables do not render

The recovery guard table is missing the separator row beneath its heading, so readers see pipe-separated text instead of a table. The storage table below has the same problem. Restore both separator rows so operators can scan the checks.

@greptile-apps

greptile-apps Bot commented Oct 2, 2026

Copy link
Copy Markdown

Want your agent to iterate on Greptile's feedback? Try greploops.

@MrYuion
MrYuion merged commit 553dbf3 into develop Oct 2, 2026
6 checks passed
@MrYuion
MrYuion deleted the fix/signage-stalls branch October 2, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant