Repository navigation
Conversation
acts_as_secondary defines rollback_write to delegate to the primary with rollback_write(chunk_id, update_retry). Output#rollback_write has taken update_retry as a keyword argument since v0.14.25, so the call always raised ArgumentError. A delayed commit output used as <secondary>, such as out_forward with require_ack_response, could not roll back a chunk after an ack timeout, and the chunk was not sent again. Fixes fluent#5633 Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the fix and the clear reproduction. Passing update_retry as a keyword is correct, and the new test fails on master as described.
In one configuration, making this rollback work changes behavior that works on master. If the secondary is out_forward with heartbeat_type none and its destination never acks, the retry no longer reaches retry_timeout or retry_max_times. Fluentd then stays on the secondary, even after the primary recovers. Details are in the inline comment. I also left a note on the new test.
| singleton_class.module_eval do | ||
| define_method(:commit_write){ |chunk_id| @primary_instance.commit_write(chunk_id, delayed: delayed_commit, secondary: true) } | ||
| define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry) } | ||
| define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry: update_retry) } |
There was a problem hiding this comment.
MUST: This fix changes how the retry ends in this configuration:
- The secondary is
out_forwardwithrequire_ack_response trueandheartbeat_type none. - Its destination accepts data but never acks.
- The secondary's
ack_response_timeoutis shorter than the primary'sdelayed_commit_timeout. This holds with the default values when the primary is alsoout_forwardwithrequire_ack_response: 190 seconds against 192 seconds.
On master, the delegation raises ArgumentError before the primary's rollback_write runs. So the chunk's entry stays in the primary's @dequeued_chunks. After the primary's delayed_commit_timeout, try_rollback_write takes the chunk back and calls update_retry_state. When the retry reaches its limit, master drops the queued chunks and clears the retry state. Later chunks then go to the primary.
With this PR, the ack timeout calls rollback_write(chunk_id, update_retry: false) on the primary. It returns the chunk to the queue and removes its entry, but it does not update the retry state. Nothing else updates it in this configuration. So retry_timeout and retry_max_times are never reached, and Fluentd keeps using the secondary, even after the primary recovers. With the default heartbeat_type transport, this does not happen. The disabled node raises NoNodesAvailable, and that error moves the retry forward.
update_retry: false comes from #1665, so that an ack failure on the primary does not count as a flush error. In the secondary phase, though, the retry state already exists. If the secondary never succeeds, only the retry limit ends that phase.
I ran Fluentd for 25 seconds with the configuration below. Nothing listens on port 24224 for the first 6 seconds. Then a server starts there and acks every chunk. The server on port 24225 reads the data but never acks.
<source>
@type sample
tag test
size 1
rate 1
</source>
<match test>
@type forward
heartbeat_type none
require_ack_response true
ack_response_timeout 1
<server>
host 127.0.0.1
port 24224
</server>
<buffer>
@type memory
flush_interval 1s
retry_type periodic
retry_wait 1s
retry_timeout 4s
retry_secondary_threshold 0.25
</buffer>
<secondary>
@type forward
heartbeat_type none
require_ack_response true
ack_response_timeout 1
<server>
host 127.0.0.1
port 24225
</server>
</secondary>
</match>
- master (d2c7287): The retry ends, and chunks go to the primary from about 9 seconds after start. The primary received 5 chunks.
- This PR: The primary received nothing. The secondary kept receiving the same 6 chunks, 24 sends in total.
I also checked the retry state with a script that runs the same plugins directly. With this PR, retry.steps stays at 0. With retry_type exponential_backoff and retry_max_times 2, master clears the queue after 2 steps. This PR is still at 0 steps after 16 seconds.
Could you keep retry_timeout and retry_max_times effective in this configuration, without changing how an ack failure on the primary is handled? Please add a test that goes through out_forward's ack timeout on the secondary, and check that each limit still ends the retry. The new test calls rollback_write directly, so it does not cover this.
| priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false}) | ||
| secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'}) | ||
| @i.configure(config_element('ROOT','',{},[priconf,secconf])) | ||
| @i.register(:prefer_buffered_processing){ true } | ||
| @i.register(:prefer_delayed_commit){ false } | ||
| @i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" } | ||
| @i.register(:write){|chunk| raise "yay, your #write must fail" } | ||
| @i.secondary.register(:prefer_delayed_commit){ true } | ||
| @i.start | ||
| @i.after_start | ||
|
|
||
| @i.interrupt_flushes | ||
|
|
There was a problem hiding this comment.
SHOULD: This test leaves the flush thread running. interrupt_flushes stops automatic enqueueing and the timeout rollback, but the flush thread still calls try_flush. It wakes up about 1 second after start. If the test thread is that slow, the flush thread can take the rolled-back chunk and call the failing write. It then calls update_retry_state with three arguments. The mock rejects that call, and the flush thread dies. Teardown then loops in submit_flush_all forever, because @retry is still nil and the chunk is still queued.
I reproduced this by adding sleep 4 right after rollback_write. The update_retry false case failed, and the process hung until timeout killed it.
The test calls rollback_write directly, so it does not need the flush thread or the failing write. With the change below, it passed with the same sleep 4. It still fails on master, and also when the delegation ignores update_retry. queued_chunks_limit_size defaults to flush_thread_count, so it has to be set here. The update_retry false case also gets about 1 second faster, because teardown no longer flushes the chunk through the failing write.
| priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false}) | |
| secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'}) | |
| @i.configure(config_element('ROOT','',{},[priconf,secconf])) | |
| @i.register(:prefer_buffered_processing){ true } | |
| @i.register(:prefer_delayed_commit){ false } | |
| @i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" } | |
| @i.register(:write){|chunk| raise "yay, your #write must fail" } | |
| @i.secondary.register(:prefer_delayed_commit){ true } | |
| @i.start | |
| @i.after_start | |
| @i.interrupt_flushes | |
| priconf = config_element('buffer','tag',{'flush_thread_count' => 0, 'queued_chunks_limit_size' => 1}) | |
| secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'}) | |
| @i.configure(config_element('ROOT','',{},[priconf,secconf])) | |
| @i.register(:prefer_buffered_processing){ true } | |
| @i.start | |
| @i.after_start | |
A secondary runs while the primary is retrying. With out_forward as the secondary, an ack timeout rolls the chunk back with update_retry: false, so retry_timeout and retry_max_times were never reached when the secondary never acked, and Fluentd stayed on the secondary even after the primary recovered. A rollback through the secondary now always updates the retry state of the primary. An ack failure on the primary still does not. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
|
Thanks for the detailed case. Pushed 2bd73f2. Retry limit with out_forward as the secondary. The rollback that the secondary delegates to the primary now always updates the retry state, even when the plugin passes I ran your configuration for 25 seconds, with a fluentd
New test in Unit test. Applied your suggestion:
|
Which issue(s) this PR fixes:
Fixes #5633
What this PR does / why we need it:
acts_as_secondarydefinesrollback_writeon the secondary so that it delegates to the primary, but it passedupdate_retryas a positional argument.Output#rollback_write(chunk_id, update_retry: true)has taken it as a keyword since 59a47a1, so the delegation always raisedArgumentError: wrong number of arguments (given 2, expected 1).A delayed commit output used as
<secondary>, such asout_forwardwithrequire_ack_response true, callsrollback_writewhen the ack times out. The error was logged asunexpected error while receiving ack. The chunk was not taken back, so it was not sent again.This passes
update_retry: update_retry, as the issue suggests.Tests:
test_output_as_buffered_secondary.rb. The secondary dequeues a chunk and callsrollback_writewithupdate_retry: falseand with the default. The chunk must go back to the queue, andupdate_retry_statemust be called only with the default. On master both cases raise theArgumentError.test/plugin/test_output_as_buffered_secondary.rb15 tests,test_output.rb141 tests,test_output_as_buffered_retries.rb14 tests: 0 failures.Docs Changes:
None.
Release Note:
Fix
rollback_writeon a<secondary>output, which always raised ArgumentError.