Skip to content

A <secondary> output cannot roll back a chunk: rollback_write always raises ArgumentError #5633

Open

Description

@Watson1978

Describe the bug

When an output is used as <secondary>, Output#acts_as_secondary defines rollback_write on it to delegate to the primary:

define_method(:rollback_write){ |chunk_id, update_retry: true| @primary_instance.rollback_write(chunk_id, update_retry) }

https://github.com/fluent/fluentd/blob/d2c7287a/lib/fluent/plugin/output.rb#L256

Output#rollback_write takes update_retry as a keyword argument (def rollback_write(chunk_id, update_retry: true), line 1178). So this call always raises ArgumentError: wrong number of arguments (given 2, expected 1). rollback_write became a keyword argument in 59a47a1 (v0.14.25 and v1.0.0), and this delegation was not updated.

A delayed commit output, such as out_forward with require_ack_response true, calls rollback_write when the ack does not arrive. When this output is a <secondary>, the call raises, and AckHandler#collect_response logs unexpected error while receiving ack. The chunk is not rolled back, so it is not sent again. In the test below, it stayed in the delayed commit state until the shutdown.

To Reproduce

  1. Make nothing listen on port 24224, so the primary fails.
  2. Run a server on port 24225 that reads the data but never sends an ack:
ruby -rsocket -e 's = TCPServer.new("127.0.0.1", 24225); loop { c = s.accept; Thread.new(c) { |x| loop { x.readpartial(65536) } rescue nil } }'
  1. Run Fluentd with the configuration below for about 20 seconds.

Results on master (d2c7287):

  • After the primary fails, the secondary sends the chunk. Each time the ack times out, the error below is logged, about every 2 seconds.
  • The chunk is not sent again. The silent server received 1014 bytes in 20 seconds.

With @primary_instance.rollback_write(chunk_id, update_retry: update_retry) instead, the error is gone. The chunk is sent again after each ack timeout, and the silent server received 3096 bytes in the same time.

Expected behavior

rollback_write on a <secondary> output rolls back the chunk, as on the primary. Passing update_retry as a keyword argument fixes it, as above.

Your Environment

- Fluentd version: master (d2c7287a). v1.20.1 and v1.19.4 have the same line (not run).
- Package version: -
- Operating system: Linux
- Kernel version: -

Your Configuration

<source>
  @type sample
  tag test
  size 1
  rate 1
</source>
<match test>
  @type forward
  heartbeat_type none
  require_ack_response true
  <server>
    host 127.0.0.1
    port 24224
  </server>
  <buffer>
    @type memory
    flush_interval 1s
    retry_type periodic
    retry_wait 1s
    retry_timeout 60s
    retry_secondary_threshold 0.05
  </buffer>
  <secondary>
    @type forward
    heartbeat_type none
    require_ack_response true
    ack_response_timeout 2
    <server>
      host 127.0.0.1
      port 24225
    </server>
  </secondary>
</match>

Your Error Log

[error]: unexpected error while receiving ack error_class=ArgumentError error="wrong number of arguments (given 2, expected 1)"
  [error]: lib/fluent/plugin/output.rb:256:in 'block (2 levels) in acts_as_secondary'

Additional context

I found this while reviewing #5627.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions