Skip to content

output: Pass update_retry as a keyword in the secondary rollback_write - #5636

Open
lbarrosby wants to merge 2 commits into
fluent:masterfrom
lbarrosby:fix-secondary-rollback-write
Open

lbarrosby wants to merge 2 commits into
fluent:masterfrom
lbarrosby:fix-secondary-rollback-write

Conversation

@lbarrosby

Copy link
Copy Markdown
Contributor

Which issue(s) this PR fixes:
Fixes #5633

What this PR does / why we need it:

acts_as_secondary defines rollback_write on the secondary so that it delegates to the primary, but it passed update_retry as a positional argument. Output#rollback_write(chunk_id, update_retry: true) has taken it as a keyword since 59a47a1, so the delegation always raised ArgumentError: wrong number of arguments (given 2, expected 1).

A delayed commit output used as <secondary>, such as out_forward with require_ack_response true, calls rollback_write when the ack times out. The error was logged as unexpected error while receiving ack. The chunk was not taken back, so it was not sent again.

This passes update_retry: update_retry, as the issue suggests.

Tests:

  • New data-driven test in test_output_as_buffered_secondary.rb. The secondary dequeues a chunk and calls rollback_write with update_retry: false and with the default. The chunk must go back to the queue, and update_retry_state must be called only with the default. On master both cases raise the ArgumentError.
  • test/plugin/test_output_as_buffered_secondary.rb 15 tests, test_output.rb 141 tests, test_output_as_buffered_retries.rb 14 tests: 0 failures.
  • The reproduction from the issue (primary port closed, secondary to a server that never acks), 22 s:
master: silent server received 1134 bytes, 9 x "unexpected error while receiving ack ... ArgumentError"
branch: silent server received 3285 bytes, no error

Docs Changes:
None.

Release Note:
Fix rollback_write on a <secondary> output, which always raised ArgumentError.

acts_as_secondary defines rollback_write to delegate to the primary
with rollback_write(chunk_id, update_retry). Output#rollback_write has
taken update_retry as a keyword argument since v0.14.25, so the call
always raised ArgumentError. A delayed commit output used as
<secondary>, such as out_forward with require_ack_response, could not
roll back a chunk after an ack timeout, and the chunk was not sent
again.

Fixes fluent#5633

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@Watson1978 Watson1978 added the bug Something isn't working label Nov 16, 2026

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix and the clear reproduction. Passing update_retry as a keyword is correct, and the new test fails on master as described.

In one configuration, making this rollback work changes behavior that works on master. If the secondary is out_forward with heartbeat_type none and its destination never acks, the retry no longer reaches retry_timeout or retry_max_times. Fluentd then stays on the secondary, even after the primary recovers. Details are in the inline comment. I also left a note on the new test.

Comment thread lib/fluent/plugin/output.rb Outdated
singleton_class.module_eval do
define_method(:commit_write){ |chunk_id| @primary_instance.commit_write(chunk_id, delayed: delayed_commit, secondary: true) }
define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry) }
define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry: update_retry) }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: This fix changes how the retry ends in this configuration:

  • The secondary is out_forward with require_ack_response true and heartbeat_type none.
  • Its destination accepts data but never acks.
  • The secondary's ack_response_timeout is shorter than the primary's delayed_commit_timeout. This holds with the default values when the primary is also out_forward with require_ack_response: 190 seconds against 192 seconds.

On master, the delegation raises ArgumentError before the primary's rollback_write runs. So the chunk's entry stays in the primary's @dequeued_chunks. After the primary's delayed_commit_timeout, try_rollback_write takes the chunk back and calls update_retry_state. When the retry reaches its limit, master drops the queued chunks and clears the retry state. Later chunks then go to the primary.

With this PR, the ack timeout calls rollback_write(chunk_id, update_retry: false) on the primary. It returns the chunk to the queue and removes its entry, but it does not update the retry state. Nothing else updates it in this configuration. So retry_timeout and retry_max_times are never reached, and Fluentd keeps using the secondary, even after the primary recovers. With the default heartbeat_type transport, this does not happen. The disabled node raises NoNodesAvailable, and that error moves the retry forward.

update_retry: false comes from #1665, so that an ack failure on the primary does not count as a flush error. In the secondary phase, though, the retry state already exists. If the secondary never succeeds, only the retry limit ends that phase.

I ran Fluentd for 25 seconds with the configuration below. Nothing listens on port 24224 for the first 6 seconds. Then a server starts there and acks every chunk. The server on port 24225 reads the data but never acks.

<source>
  @type sample
  tag test
  size 1
  rate 1
</source>
<match test>
  @type forward
  heartbeat_type none
  require_ack_response true
  ack_response_timeout 1
  <server>
    host 127.0.0.1
    port 24224
  </server>
  <buffer>
    @type memory
    flush_interval 1s
    retry_type periodic
    retry_wait 1s
    retry_timeout 4s
    retry_secondary_threshold 0.25
  </buffer>
  <secondary>
    @type forward
    heartbeat_type none
    require_ack_response true
    ack_response_timeout 1
    <server>
      host 127.0.0.1
      port 24225
    </server>
  </secondary>
</match>
  • master (d2c7287): The retry ends, and chunks go to the primary from about 9 seconds after start. The primary received 5 chunks.
  • This PR: The primary received nothing. The secondary kept receiving the same 6 chunks, 24 sends in total.

I also checked the retry state with a script that runs the same plugins directly. With this PR, retry.steps stays at 0. With retry_type exponential_backoff and retry_max_times 2, master clears the queue after 2 steps. This PR is still at 0 steps after 16 seconds.

Could you keep retry_timeout and retry_max_times effective in this configuration, without changing how an ack failure on the primary is handled? Please add a test that goes through out_forward's ack timeout on the secondary, and check that each limit still ends the retry. The new test calls rollback_write directly, so it does not cover this.

Comment on lines +461 to +473
priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false})
secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'})
@i.configure(config_element('ROOT','',{},[priconf,secconf]))
@i.register(:prefer_buffered_processing){ true }
@i.register(:prefer_delayed_commit){ false }
@i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" }
@i.register(:write){|chunk| raise "yay, your #write must fail" }
@i.secondary.register(:prefer_delayed_commit){ true }
@i.start
@i.after_start

@i.interrupt_flushes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHOULD: This test leaves the flush thread running. interrupt_flushes stops automatic enqueueing and the timeout rollback, but the flush thread still calls try_flush. It wakes up about 1 second after start. If the test thread is that slow, the flush thread can take the rolled-back chunk and call the failing write. It then calls update_retry_state with three arguments. The mock rejects that call, and the flush thread dies. Teardown then loops in submit_flush_all forever, because @retry is still nil and the chunk is still queued.

I reproduced this by adding sleep 4 right after rollback_write. The update_retry false case failed, and the process hung until timeout killed it.

The test calls rollback_write directly, so it does not need the flush thread or the failing write. With the change below, it passed with the same sleep 4. It still fails on master, and also when the delegation ignores update_retry. queued_chunks_limit_size defaults to flush_thread_count, so it has to be set here. The update_retry false case also gets about 1 second faster, because teardown no longer flushes the chunk through the failing write.

Suggested change
priconf = config_element('buffer','tag',{'flush_interval' => 1, 'retry_type' => :periodic, 'retry_wait' => 3, 'retry_timeout' => 60, 'retry_randomize' => false})
secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'})
@i.configure(config_element('ROOT','',{},[priconf,secconf]))
@i.register(:prefer_buffered_processing){ true }
@i.register(:prefer_delayed_commit){ false }
@i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" }
@i.register(:write){|chunk| raise "yay, your #write must fail" }
@i.secondary.register(:prefer_delayed_commit){ true }
@i.start
@i.after_start
@i.interrupt_flushes
priconf = config_element('buffer','tag',{'flush_thread_count' => 0, 'queued_chunks_limit_size' => 1})
secconf = config_element('secondary','',{'@type' => 'output_secondary_test2'})
@i.configure(config_element('ROOT','',{},[priconf,secconf]))
@i.register(:prefer_buffered_processing){ true }
@i.start
@i.after_start

A secondary runs while the primary is retrying. With out_forward as the
secondary, an ack timeout rolls the chunk back with update_retry: false,
so retry_timeout and retry_max_times were never reached when the
secondary never acked, and Fluentd stayed on the secondary even after
the primary recovered.

A rollback through the secondary now always updates the retry state of
the primary. An ack failure on the primary still does not.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed case. Pushed 2bd73f2.

Retry limit with out_forward as the secondary. The rollback that the secondary delegates to the primary now always updates the retry state, even when the plugin passes update_retry: false. In the secondary phase the retry state already exists, and only its limit ends that phase, as you wrote. The primary's own rollback_write is unchanged, so an ack failure on the primary still does not count as a flush error. On master the same update happens later, through try_rollback_write after the primary's delayed_commit_timeout.

I ran your configuration for 25 seconds, with a fluentd in_forward started on port 24224 after 6 seconds and a server on 24225 that reads and never acks:

  • master (d2c7287): the limit was reached, 7 records reached the primary, 13 connections to the secondary.
  • previous head (2200f8a): 0 records reached the primary, 56 connections to the secondary.
  • this commit: the limit was reached at about 4 seconds, 23 records reached the primary, 4 connections to the secondary.

New test in test_out_forward.rb, as secondary with a destination that never acks. The primary is out_forward to a closed port, and the secondary is out_forward with heartbeat_type none, require_ack_response true and ack_response_timeout 1s to a TCP server that never acks. One case uses exponential_backoff with retry_max_times 2, and the other uses periodic with retry_timeout 4s. Each waits until the retry state is cleared and the buffer is empty. Both pass here and on master, and time out on 2200f8a.

Unit test. Applied your suggestion: flush_thread_count 0 and queued_chunks_limit_size 1, without the failing write or interrupt_flushes. With the change above, both the update_retry: false case and the default case now expect one update_retry_state call, and the name says so. It still fails on master with ArgumentError.

test_output_as_buffered_secondary.rb, test_output_as_buffered_retries.rb, test_out_forward.rb, test_output.rb and test_output_as_buffered.rb pass. test_out_secondary_file.rb has 3 encoding failures here, and they are the same on master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A <secondary> output cannot roll back a chunk: rollback_write always raises ArgumentError

2 participants