Skip to content

Fix event subscriptions stopping after a reconnect - #3548

Open
Ekhorn wants to merge 2 commits into
mainfrom
bugfix/event-provider-reconnect
Open

Ekhorn wants to merge 2 commits into
mainfrom
bugfix/event-provider-reconnect

Conversation

@Ekhorn

@Ekhorn Ekhorn commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

Events could stop until the page was reloaded after a connection drop. The event provider reconnected and reported CONNECTED, but every subscribe failed with There's already a pending subscription and nothing was sent to the server.

Cause: a socket error or close while a subscribe was waiting for its reply left _subscribeDeferred set, because _beforeDisconnect didn't clear it. Only a deliberate _doDisconnect did. The interrupted subscribe was queued again, but _doSubscribe rejected it and every later subscribe before sending a frame. The rejected _attributeEventPromise or _assetEventPromise then stayed cached, so later calls got the same rejection.

Two related problems:

Evidence: confirmed through Web Inspector on an iPhone, in the app using core 1.28.1, whose event.ts behaves the same as main. The stuck provider was CONNECTED with _subscribeDeferred set and _pendingSubscription null, and it had two open sockets, only one of them held by the provider. A fresh subscribe was rejected with no frames sent. Clearing _subscribeDeferred and _attributeEventPromise by hand brought events back without a reload. In the manager, the reproduction below makes creating an asset hang, with Uncaught (in promise) There's already a pending subscription in the console.

Evidence for #1200: reproduced 3 times on a staging server without this PR, each time after the connection dropped. The provider reconnected and reported CONNECTED, but the new connection showed no frames at all. After a forced reconnect, none of the 3 subscriptions the page held was sent again. Queuing them again by hand brought events back without a reload.

Behavior change for consumers: subscriptions now survive a reconnect and keep their ID, so a consumer that holds on to its subscription keeps getting events. A consumer that resubscribes on reconnect still ends with one server subscription, but may send an extra SUBSCRIBE and UNSUBSCRIBE. unsubscribe now also covers a subscription still being restored, otherwise that consumer would end up with two subscriptions and receive every event twice. Components using the subscribe mixin unsubscribe on DISCONNECTED and subscribe again on CONNECTED. I traced that flow through the code, but no test covers it.

Testing

The 11 new tests use a fake socket, so each test controls the event order. Removing any single change makes at least one test fail.

The issue is confirmed resolved on a test server running this branch, and in the app on 1.28.1 with the fix backported, in Chromium and Firefox.

How the fix was manually tested on the app
  1. Get 1.28.1 with the fix

    git checkout 1.28.1
    git fetch origin main && git fetch origin pull/3548/head
    git diff origin/main...FETCH_HEAD -- ui/component/core/src/event.ts | git apply --ignore-whitespace -C1
  2. Build the package

    ./gradlew :ui:component:rest:generateTypeScript && yarn install && cd ui/component/core && npx tsc -b && yarn pack --out openremote-core-1.28.1-event-provider-fix.tgz

    Writes to ui/component/core/openremote-core-1.28.1-event-provider-fix.tgz.

  3. Build the app with it

    1. Copy the tarball into the project.
    2. In package.json resolutions, set:
      "@openremote/core": "file:./openremote-core-1.28.1-event-provider-fix.tgz"
    3. Run yarn install, then ./gradlew clean installDist. The bundle ends up at deployment/build/image/manager/app/<app>/bundle.*.js.
  4. Serve it over the deployed app

    Chromium:

    1. In DevTools, go to Sources, then Overrides, click Select folder for overrides, then Allow. Make sure Enable Local Overrides is ticked.
    2. In Network, tick Disable cache.
    3. In Sources, then Page, right-click <app>/bundle.<deployed hash>.js and choose Override content. DevTools creates
      <folder>/<host>/<app>/bundle.<deployed hash>.js.
    4. Copy your build over it, keeping the deployed name:
      cp deployment/build/image/manager/app/<app>/bundle.*.js <folder>/<host>/<app>/bundle.<deployed hash>.js
    5. Reload, run block 1, and check that typeof __p._discardWebSocket gives 'function'.

    Firefox: in Debugger, right-click the bundle, choose Add script override and save it. Copy your build over the saved file, then reload with the cache disabled.

Reproduction

These steps reproduce the failing subscribes. They work in any browser and in any app built on core, including the manager. The steps for #1200 are in this issue comment.

  1. Find the event provider: Paste it, then switch to another tab and back, or do something that subscribes (waiting for Found the event provider. to be logged).
(() => {
  const apply = Function.prototype.apply;
  const found = (provider) => {
    Function.prototype.apply = apply;
    window.__p = provider;
    console.log('Found the event provider.');
  };
  Function.prototype.apply = function (self, args) {
    try {
      if (window.OpenRemoteConsole && self?._console === window.OpenRemoteConsole) found(self._events);
      else if (self?._subscribeDeferred !== undefined) found(self);
    } catch {}
    return Reflect.apply(this, self, args ?? []);
  };
  setTimeout(() => {
    if (Function.prototype.apply !== apply) {
      Function.prototype.apply = apply;
      console.log('Event provider not found.');
    }
  }, 60000);
  console.log('Switch to another tab and back, or do something that subscribes.');
})();
Shorter Safari only alternative

In Safari's Web Inspector, including iOS web views, this finds the event provider straight away, with no tab switch needed:

window.__p = queryHolders(OpenRemoteConsole).find((o) => { try { return o._console === OpenRemoteConsole; } catch {} })._events;
  1. Reproduce: Without the fixes in this PR the following will log: There's already a pending subscription
__p.subscribe({ eventType: 'alarm' }, () => {}).catch(() => {});
__p._webSocket.close();
  1. Check
function checkEvents() {
  if (__p.status !== 'CONNECTED') return console.log(`Not connected (${__p.status}). Run checkEvents() again once it has reconnected.`);
  if (!__p._subscribeDeferred) return console.log('OK: no subscribe is waiting for a reply.');
  if (__p._pendingSubscription) return console.log('BUSY: a subscribe is waiting for its reply. Run checkEvents() again in a few seconds.');
  console.log('STUCK: the provider waits for the reply to a subscribe that was lost with an earlier connection, so it rejects every new subscribe until the page is reloaded.');
}
checkEvents(); // It's defined as checkEvents(), so you can run it again later.
  1. Recover
__p._subscribeDeferred = null;
__p._attributeEventPromise = __p._assetEventPromise = undefined;
__p._webSocket.close();

Changelog

Checklist

  • 1. Acceptance criteria of the linked issue(s) are met
  • 2. Tests are written and all tests pass
  • 3. Changes are manually tested by you and the reviewer
  • 4. Documentation is written or updated

A socket error or close during a subscribe left the pending reply state
set, so every later subscribe was rejected before anything was sent.
Events only came back after a page reload.

- Clear the pending subscribe reply on an unexpected disconnect
- Stop reusing a rejected shared asset or attribute subscription, and
  remove the callback of the failed call
- Detach and close the previous socket before opening a new one
- Restore subscriptions after a reconnect
- Let unsubscribe cancel a subscription that is still being restored
- Add component tests and npmTest for the core package
@Ekhorn Ekhorn self-assigned this Nov 5, 2026
@wborn wborn added the Bug Error or issue in the system label Nov 5, 2026
@Ekhorn
Ekhorn marked this pull request as ready for review November 6, 2026 11:12
@Ekhorn
Ekhorn requested a review from a team November 6, 2026 11:14

@wborn wborn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an initial AI-assisted review.

The changes address the reported WebSocket subscription issues in a straightforward way:

  • Clearing the pending subscription state on disconnect prevents subscriptions from becoming permanently stuck.
  • Detaching handlers and closing replaced WebSockets prevents stale socket events from interfering with the active connection.
  • Correcting the Map iteration restores existing subscriptions after reconnecting.
  • Handling cancellation during subscription restoration prevents obsolete subscriptions from remaining active.

The new regression tests cover the main failure scenarios and several important reconnect and cancellation edge cases. The relevant CI checks are passing.

One minor edge case remains around cancelling an initial queued subscription, but this does not appear to affect existing consumers and should not block merging.

No blocking issues were identified. Manual maintainer review is still needed before merge.

Comment thread ui/component/core/src/event.ts Resolved
@Ekhorn

Ekhorn commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

I also made a refactored implementation of ui/component/core/src/event.ts, but I've not extensively tested it: 55f0fe8

The reduction is 332 insertions(+), 628 deletions(-), though I'm not certain it's fully compatible with existing apps.

Unsubscribing a subscription the server hadn't confirmed yet left its
subscribe promise pending. It now resolves with the ID, as it already
did when the reply arrived after the unsubscribe.
@sonarqubecloud

sonarqubecloud Bot commented Nov 6, 2026

Copy link
Copy Markdown

@Ekhorn
Ekhorn requested review from a team and wborn November 6, 2026 14:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Error or issue in the system

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bugreport: Map-view never receives AttributeEvents automatically anymore after being once offline

2 participants