Skip to content

Fix flaky JetStream tests, and give up on unanswered pulls after 10s - #198

Merged
christopherread merged 6 commits into
v7from
fix/nats-jetstream-delete-race
Nov 15, 2026
Merged

christopherread merged 6 commits into
v7from
fix/nats-jetstream-delete-race

Conversation

@christopherread

@christopherread christopherread commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Fixes the intermittent timeout in TestNatsJetStreamReconnect.ShouldReportAnErrorToSubscribersWhenTheSharedCommandConsumerIsDeleted from the v7.0.0 tag build, where it failed after 30s 088ms.

Why the test was wrong

The test deleted the shared command consumer as soon as the host had started, while the bus's first pull could still be on its way to the server.

  • Deleting a consumer answers the pulls already waiting on it with 409 Consumer Deleted. The bus handles that: it fetches the consumer again, gets 10014 and reports it.
  • But in nats-server 2.15.1 (server/consumer.go), processNextMsgReq only queues a pull. The consumer registers it later in processNextMsgRequest, which returns without answering once stopWithFlags has cleared o.mset. A pull that arrives while the consumer is being deleted is dropped with no reply and no heartbeats.
  • NATS.Net then gives up on that pull only after it misses two idle heartbeats. With the default 30s expiry that's 2 × 15s = 30s, the same as the test's own timeout. After that the next pull gets 503 and the error is reported, but by then the test has already failed.

So the bus does report the deletion; the test just measured a delete that raced the first pull, which can take up to the client's heartbeat timeout. ShouldReportAnErrorToSubscribersWhenTheCommandStreamIsDeleted has the same race (deleting a stream stops its consumers the same way), so it gets the same fix.

The fix

Before deleting, both tests now wait until the consumer has a pull waiting (ConsumerInfo.NumWaiting > 0), using a new WaitUntilPullingAsync helper. ShouldRecreateTheBusesOwnEventConsumerWhenItIsDeleted, which deletes the bus's own event consumer, gave the bus a fixed 500ms to start pulling first. That's a guess a slow machine could miss, so it now waits with the same helper. So does ShouldKeepReceivingEventsAfterTheServerRestarts, which gave the bus 500ms to start pulling before killing the server. It now names the bus's instance so it knows which event consumer to wait on. If the restart came first, the test wouldn't check that a bus that was receiving keeps receiving. And TestNatsJetStreamHosting.ShouldNotReportAnErrorToSubscribersWhenTheBusStops gave the bus 500ms before stopping it, so it now waits for its event and command pulls. The helper now lives in JetStreamServices, so both test classes use it. There are no 500ms waits left in the JetStream tests.

And: the bus gives up on an unanswered pull after 10s, not 30s

The bus did report the deletion, but up to 30s late. Any pull the server drops would hold up the receive loop that long. MessageSource now pulls with IdleHeartbeat = 5s, so the client gives up on a pull that gets no heartbeats after 10s. The loop then pulls again and finds the consumer gone. The pulls still expire after 30s; the only cost is a small heartbeat from the server every 5s for each waiting pull. No public API change.

The new unit test TestJetStreamMessageSourceRetry.ShouldGiveUpOnAPullThatGoesUnansweredWithinTenSeconds checks the heartbeat each pull asks for. It failed first (the bus used the client's default), then passed.

And: the test server's port clash

RestartableJetStreamServer took a free port from the OS, freed it, then started nats-server on it. The OS gives outgoing connections ports from that same range, so one could take the port in between. nats-server then exited with address already in use, or the startup probe connected to itself and the server looked started when it wasn't.

  • Servers now pick a port between 20000 and 32000, below the range the OS gives outgoing connections (from 32768 on Linux, 49152 on macOS and Windows).
  • On a first start, a server that can't listen on its port picks another, because servers starting at the same time can still clash. Restarts keep their port.
  • A server counts as started once it logs Server is ready, not when something answers on its port.
  • A server that can't start now includes the last lines it wrote in the exception, so it says why.
  • It listens on 127.0.0.1 only.

The new test TestRestartableJetStreamServer.ShouldStartManyServersAtOnce starts 30 servers at a time, 10 times, while connections churn through ports. It failed 2 runs in 10 against the old harness and passed 30 in 30 after. It runs in a collection of its own with parallelization disabled: when it ran alongside the others, its load once delayed ShouldKeepARunningCommandInProgressPastItsAckWait's acknowledgements past a 1s AckWait.

Reproduced and verified locally

  • A scratch sweep (not committed) started a bus on its own nats-server, deleted the consumer 0–15ms after the host started, and timed how long the error took, 100 times per run. Without the wait, some deletions about 13–15ms after start took 30,260ms. With the wait, 900 deletions over 9 runs all reported quickly.
  • With the test fix: both deletion tests passed 20 runs in a row, and the NATS Explicit suite passed 7 runs (44/44 each).
  • With the event test on the helper too: it passed 20 runs in a row, the NATS Explicit suite passed 6 more runs (44/44 each), and the unit tests pass.
  • With the restart test on the helper too: it passed 20 runs in a row, the NATS Explicit suite passed 6 more runs (44/44 each), and the unit tests pass.
  • With the hosting test on the helper too: it passed 20 runs in a row, the NATS Explicit suite passed 6 more runs (44/44 each), and the unit tests pass.
  • With the harness fix: the NATS Explicit suite passed 13 runs (45/45 each, once the new test ran alone), the unit tests pass, and the build has 0 warnings.
  • With the shorter heartbeat, I ran the sweep again without the test's wait, 2,400 deletions in all. The deletions that still raced the first pull (6 of them) were reported in ~10,300ms instead of 30,260ms. The NATS Explicit suite passed 8 runs (44/44 each), the NATS unit tests pass (39), and dotnet build Obvs.sln -c Release has 0 warnings.

The test fix has no new failing test: the scratch sweep was its reproduction.

🤖 Generated with Claude Code

Deleting a consumer answers the pulls waiting on it with 409 Consumer
Deleted, but nats-server drops a pull that's still queued for the
consumer as it closes, without an answer. The client then gives up on
that pull only after missing two idle heartbeats, 30 seconds by default,
which is the tests' own timeout. The deletion tests deleted right after
the host started, while the bus's first pull could be on its way, so
they sometimes timed out (the v7.0.0 tag build).

They now wait until the consumer has a pull waiting before deleting it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@christopherread christopherread added this to the v7 milestone Nov 15, 2026
nats-server drops a pull that reaches a consumer as it's deleted,
without an answer, and the client gives up on a pull only after it
misses two of the server's idle heartbeats: 30 seconds with the
client's default for a 30 second pull. Until then the bus doesn't
notice the consumer has gone. Pulls now ask for a heartbeat every
5 seconds, so the bus pulls again, and finds the consumer gone,
after 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@christopherread christopherread changed the title Fix the flaky JetStream consumer deletion test: wait for the bus's pull first Fix the flaky JetStream consumer deletion test, and give up on unanswered pulls after 10s Nov 15, 2026
christopherread and others added 4 commits November 15, 2026 12:28
ShouldRecreateTheBusesOwnEventConsumerWhenItIsDeleted gave the bus
500ms to start pulling before deleting its event consumer, a guess that
a slow machine could miss, and then the deletion could race the bus's
first pull. It now waits until the consumer has a pull waiting, as the
other deletion tests do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ShouldKeepReceivingEventsAfterTheServerRestarts gave the bus 500ms to
start pulling before killing the server. On a slow machine the restart
could come first, and the test wouldn't check that a bus that was
receiving keeps receiving. It now names the bus's instance, so it knows
its event consumer, and waits until that consumer has a pull waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g test

ShouldNotReportAnErrorToSubscribersWhenTheBusStops gave the bus 500ms
to start pulling before stopping it. On a slow machine the bus could
stop first, and the test wouldn't check that stopping doesn't fail
receive loops that are pulling. It now waits until its event and
command consumers each have a pull waiting.

WaitUntilPullingAsync moves to JetStreamServices so both test classes
use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RestartableJetStreamServer took a free port from the OS, freed it, then
started nats-server on it. The OS gives outgoing connections ports from
the same range, so one could take the port first: nats-server then
exited with 'address already in use', or the startup probe connected to
itself and the server looked started when it wasn't.

Servers now pick a port between 20000 and 32000, below the range the OS
gives outgoing connections, and a first start that can't listen on its
port picks another (servers starting at the same time can still clash).
Restarts keep their port. A server is started once it logs that it's
ready, not when something answers on its port, and a server that
can't start says why, with the last lines it wrote.

TestRestartableJetStreamServer starts 30 servers at a time, 10 times,
while connections churn through ports. It failed 2 runs in 10 before
and passed 30 in 30 after. It runs by itself, as its load held up
another test's acknowledgements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@christopherread christopherread changed the title Fix the flaky JetStream consumer deletion test, and give up on unanswered pulls after 10s Fix flaky JetStream tests, and give up on unanswered pulls after 10s Nov 15, 2026
@christopherread
christopherread merged commit 8ae0e23 into v7 Nov 15, 2026
1 check passed
@christopherread
christopherread deleted the fix/nats-jetstream-delete-race branch November 15, 2026 21:04
@christopherread christopherread self-assigned this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant