Repository navigation
out_forward: start retry after a failed ACK when heartbeat_type is none #5627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
SahinurDEV
wants to merge
8
commits into
fluent:master
Choose a base branch
from
SahinurDEV:fix-5619-out-forward-ack-retry-none-heartbeat
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
295f6c4
out_forward: start retry after a failed ACK when heartbeat_type is none
SahinurDEV 9bb8423
out_forward: detach the node after a failed ACK instead of counting a…
SahinurDEV 8a61903
out_forward: do not regard a received ack as missing in the ack reader
SahinurDEV 54cfc28
out_forward: drop stale rebalances and skip a retry already counted b…
SahinurDEV 6e6e1eb
out_forward: do not let an older weight array rebuild replace a newer…
SahinurDEV 0420c88
out_forward: do not count a retry twice for a late ack failure
SahinurDEV abed3b4
Merge branch 'fix-5619-out-forward-ack-retry-none-heartbeat' of the fork
SahinurDEV a2a3990
test_out_forward: use Array.new instead of times.map (RuboCop)
SahinurDEV File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MUST: This rebuilds the weight array from the ack thread. With
heartbeat_type none,rebalance_if_availability_changedalso rebuilds it from the flush threads. On master, after start, only the event loop rebuilds it (the heartbeat timer and service discovery), so two rebuilds never overlap. Now a rebuild that read the node list earlier can publish its array after a newer one.I reproduced this with dynamic service discovery and three servers (
failed,removed,retained):failedfails, and thisservice_discovery_rebalancereads the current services. I paused the ack thread there.failedandremoved, and publishes a weight array with onlyretained.After that, master sends only to
retained. With this PR,removedis still selected.availability_changed?only checks the current services, so the check before a write does not notice it, and the stale array stays until another rebuild. A lock around the assignment of@weight_arrayalone would not help, because the node list is read before it.Please make sure that an older rebuild cannot replace a newer one, and add a test that forces this order. The standby case (
delivers every chunk to the healthy standby server) should keep passing.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for reproducing this with a forced order. Fixed in 6e6e1eb.
LoadBalancer#rebuild_weight_arraynow holds a rebuild lock while it reads the node list, builds the array, and publishes it. As you pointed out, locking only the assignment wouldn't help, so out_forward gives the load balancer anodes_provider(service_discovery_services). Every rebuild reads the list under the lock and ignores the list the caller read earlier. That covers the ack thread, the flush threads, the heartbeat timer, and the service discovery rebuild throughManager#rebalance. A rebuild that read the old list has to publish before the next rebuild can take the lock. Any later rebuild, such as the one service discovery runs after removingfailedandremoved, then reads the new list and publishes last. The service discovery helper and itsrebalance(services)interface are unchanged.Tests:
concurrent rebuilds of the weight array (#5627)/a rebuild that read the old node list does not replace a newer oneintest_out_forward.rbfollows your steps. It uses three servers (failed,removed,retained) and pauses the ack-thread rebuild right after it has read the node list. Meanwhile, another thread sendsSERVICE_OUTforfailedandremovedand callsManager#run_once. The test then resumes the paused rebuild and checks that the weight array and the selected nodes contain onlyretained. It fails on 8a61903 (removedis still in the array) and passes now.test_load_balancer.rbcheck that the provider is used and that the list is read under the lock.delivers every chunk to the healthy standby serverand the othernever sends the ackanda failed acktests still pass.