Repository navigation
Description
Describe the bug
When an output is used as <secondary>, Output#acts_as_secondary defines rollback_write on it to delegate to the primary:
define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry) }https://github.com/fluent/fluentd/blob/d2c7287a/lib/fluent/plugin/output.rb#L256
Output#rollback_write takes update_retry as a keyword argument (def rollback_write(chunk_id, update_retry: true), line 1178). So this call always raises ArgumentError: wrong number of arguments (given 2, expected 1). rollback_write became a keyword argument in 59a47a1 (v0.14.25 and v1.0.0), and this delegation was not updated.
A delayed commit output, such as out_forward with require_ack_response true, calls rollback_write when the ack does not arrive. When this output is a <secondary>, the call raises, and AckHandler#collect_response logs unexpected error while receiving ack. The chunk is not rolled back, so it is not sent again. In the test below, it stayed in the delayed commit state until the shutdown.
To Reproduce
- Make nothing listen on port 24224, so the primary fails.
- Run a server on port 24225 that reads the data but never sends an ack:
ruby -rsocket -e 's = TCPServer.new("127.0.0.1", 24225); loop { c = s.accept; Thread.new(c) { |x| loop { x.readpartial(65536) } rescue nil } }'
- Run Fluentd with the configuration below for about 20 seconds.
Results on master (d2c7287):
- After the primary fails, the secondary sends the chunk. Each time the ack times out, the error below is logged, about every 2 seconds.
- The chunk is not sent again. The silent server received 1014 bytes in 20 seconds.
With @primary_instance.rollback_write(chunk_id, update_retry: update_retry) instead, the error is gone. The chunk is sent again after each ack timeout, and the silent server received 3096 bytes in the same time.
Expected behavior
rollback_write on a <secondary> output rolls back the chunk, as on the primary. Passing update_retry as a keyword argument fixes it, as above.
Your Environment
- Fluentd version: master (d2c7287a). v1.20.1 and v1.19.4 have the same line (not run).
- Package version: -
- Operating system: Linux
- Kernel version: -Your Configuration
<source>
@type sample
tag test
size 1
rate 1
</source>
<match test>
@type forward
heartbeat_type none
require_ack_response true
<server>
host 127.0.0.1
port 24224
</server>
<buffer>
@type memory
flush_interval 1s
retry_type periodic
retry_wait 1s
retry_timeout 60s
retry_secondary_threshold 0.05
</buffer>
<secondary>
@type forward
heartbeat_type none
require_ack_response true
ack_response_timeout 2
<server>
host 127.0.0.1
port 24225
</server>
</secondary>
</match>
Your Error Log
[error]: unexpected error while receiving ack error_class=ArgumentError error="wrong number of arguments (given 2, expected 1)"
[error]: lib/fluent/plugin/output.rb:256:in 'block (2 levels) in acts_as_secondary'Additional context
I found this while reviewing #5627.
Activity