Repository navigation
Fix flaky JetStream tests, and give up on unanswered pulls after 10s - #198
Merged
Merged
Conversation
Deleting a consumer answers the pulls waiting on it with 409 Consumer Deleted, but nats-server drops a pull that's still queued for the consumer as it closes, without an answer. The client then gives up on that pull only after missing two idle heartbeats, 30 seconds by default, which is the tests' own timeout. The deletion tests deleted right after the host started, while the bus's first pull could be on its way, so they sometimes timed out (the v7.0.0 tag build). They now wait until the consumer has a pull waiting before deleting it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nats-server drops a pull that reaches a consumer as it's deleted, without an answer, and the client gives up on a pull only after it misses two of the server's idle heartbeats: 30 seconds with the client's default for a 30 second pull. Until then the bus doesn't notice the consumer has gone. Pulls now ask for a heartbeat every 5 seconds, so the bus pulls again, and finds the consumer gone, after 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ShouldRecreateTheBusesOwnEventConsumerWhenItIsDeleted gave the bus 500ms to start pulling before deleting its event consumer, a guess that a slow machine could miss, and then the deletion could race the bus's first pull. It now waits until the consumer has a pull waiting, as the other deletion tests do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ShouldKeepReceivingEventsAfterTheServerRestarts gave the bus 500ms to start pulling before killing the server. On a slow machine the restart could come first, and the test wouldn't check that a bus that was receiving keeps receiving. It now names the bus's instance, so it knows its event consumer, and waits until that consumer has a pull waiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g test ShouldNotReportAnErrorToSubscribersWhenTheBusStops gave the bus 500ms to start pulling before stopping it. On a slow machine the bus could stop first, and the test wouldn't check that stopping doesn't fail receive loops that are pulling. It now waits until its event and command consumers each have a pull waiting. WaitUntilPullingAsync moves to JetStreamServices so both test classes use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RestartableJetStreamServer took a free port from the OS, freed it, then started nats-server on it. The OS gives outgoing connections ports from the same range, so one could take the port first: nats-server then exited with 'address already in use', or the startup probe connected to itself and the server looked started when it wasn't. Servers now pick a port between 20000 and 32000, below the range the OS gives outgoing connections, and a first start that can't listen on its port picks another (servers starting at the same time can still clash). Restarts keep their port. A server is started once it logs that it's ready, not when something answers on its port, and a server that can't start says why, with the last lines it wrote. TestRestartableJetStreamServer starts 30 servers at a time, 10 times, while connections churn through ports. It failed 2 runs in 10 before and passed 30 in 30 after. It runs by itself, as its load held up another test's acknowledgements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the intermittent timeout in
TestNatsJetStreamReconnect.ShouldReportAnErrorToSubscribersWhenTheSharedCommandConsumerIsDeletedfrom thev7.0.0tag build, where it failed after 30s 088ms.Why the test was wrong
The test deleted the shared command consumer as soon as the host had started, while the bus's first pull could still be on its way to the server.
409 Consumer Deleted. The bus handles that: it fetches the consumer again, gets 10014 and reports it.server/consumer.go),processNextMsgReqonly queues a pull. The consumer registers it later inprocessNextMsgRequest, which returns without answering oncestopWithFlagshas clearedo.mset. A pull that arrives while the consumer is being deleted is dropped with no reply and no heartbeats.So the bus does report the deletion; the test just measured a delete that raced the first pull, which can take up to the client's heartbeat timeout.
ShouldReportAnErrorToSubscribersWhenTheCommandStreamIsDeletedhas the same race (deleting a stream stops its consumers the same way), so it gets the same fix.The fix
Before deleting, both tests now wait until the consumer has a pull waiting (
ConsumerInfo.NumWaiting > 0), using a newWaitUntilPullingAsynchelper.ShouldRecreateTheBusesOwnEventConsumerWhenItIsDeleted, which deletes the bus's own event consumer, gave the bus a fixed 500ms to start pulling first. That's a guess a slow machine could miss, so it now waits with the same helper. So doesShouldKeepReceivingEventsAfterTheServerRestarts, which gave the bus 500ms to start pulling before killing the server. It now names the bus's instance so it knows which event consumer to wait on. If the restart came first, the test wouldn't check that a bus that was receiving keeps receiving. AndTestNatsJetStreamHosting.ShouldNotReportAnErrorToSubscribersWhenTheBusStopsgave the bus 500ms before stopping it, so it now waits for its event and command pulls. The helper now lives inJetStreamServices, so both test classes use it. There are no 500ms waits left in the JetStream tests.And: the bus gives up on an unanswered pull after 10s, not 30s
The bus did report the deletion, but up to 30s late. Any pull the server drops would hold up the receive loop that long.
MessageSourcenow pulls withIdleHeartbeat = 5s, so the client gives up on a pull that gets no heartbeats after 10s. The loop then pulls again and finds the consumer gone. The pulls still expire after 30s; the only cost is a small heartbeat from the server every 5s for each waiting pull. No public API change.The new unit test
TestJetStreamMessageSourceRetry.ShouldGiveUpOnAPullThatGoesUnansweredWithinTenSecondschecks the heartbeat each pull asks for. It failed first (the bus used the client's default), then passed.And: the test server's port clash
RestartableJetStreamServertook a free port from the OS, freed it, then started nats-server on it. The OS gives outgoing connections ports from that same range, so one could take the port in between. nats-server then exited withaddress already in use, or the startup probe connected to itself and the server looked started when it wasn't.Server is ready, not when something answers on its port.The new test
TestRestartableJetStreamServer.ShouldStartManyServersAtOncestarts 30 servers at a time, 10 times, while connections churn through ports. It failed 2 runs in 10 against the old harness and passed 30 in 30 after. It runs in a collection of its own with parallelization disabled: when it ran alongside the others, its load once delayedShouldKeepARunningCommandInProgressPastItsAckWait's acknowledgements past a 1s AckWait.Reproduced and verified locally
dotnet build Obvs.sln -c Releasehas 0 warnings.The test fix has no new failing test: the scratch sweep was its reproduction.
🤖 Generated with Claude Code