Skip to content

in_tcp: drop only the line the parser fails on - #5640

Open
rafayqayyum wants to merge 1 commit into
fluent:masterfrom
rafayqayyum:in-tcp-rescue-parser-errors
Open

rafayqayyum wants to merge 1 commit into
fluent:masterfrom
rafayqayyum:in-tcp-rescue-parser-errors

Conversation

@rafayqayyum

Copy link
Copy Markdown

Which issue(s) this PR fixes:
Fixes #5629

What this PR does / why we need it:

in_tcp called @parser.parse(msg) without a rescue, in both the batch-emit and the single-emit (<extract> with tag_key) paths. When the parser raises on a line — for example the json parser with time_format and a time field that doesn't match — the error goes up to the server helper, which logs unexpected error on reading data and closes the connection. With batch emit none of the lines in that read are emitted; with single emit the lines after the bad one are lost.

Both paths now call a parse_message helper that rescues the error and logs failed to parse message at warn level, with the message truncated like pattern not matched. Only the bad line is dropped, the other lines are emitted, and the connection stays open. Emit errors are unaffected, since EventRouter#emit and #emit_stream already rescue them.

The new tests send the three lines from the issue in one write, then a fourth line on the same socket, for both batch and single emit, and expect k1, k3 and k4 plus the warning.

Docs Changes:
None.

Release Note:
in_tcp: a line the parser fails on is dropped with a warning instead of closing the connection and losing the other lines.

Copilot AI balanced review requested due to automatic review settings November 15, 2026 13:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. Rescuing the parser error per line fixes #5629 in both paths. The new tests fail on master and pass with this change.

One problem: in the single-emit path, the rescue also catches emit errors. See the inline comment.

Also, the DCO check fails because the commit has no Signed-off-by line. Please add your sign-off to each commit (for example, git commit --amend -s) and force-push.

Comment on lines +224 to +228
def parse_message(msg, &block)
@parser.parse(msg, &block)
rescue => e
log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize
end

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: This rescue also covers the block. So in the single-emit path, it catches errors from router.emit on line 155.

The description says emit errors are unaffected because EventRouter rescues them. But without an @ERROR label, RootAgent#handle_emits_error logs emit transaction failed and re-raises the error. On master, the error reaches the server helper. It logs unexpected error on reading data and closes the connection. With this change, the error is logged as a parse failure, and the connection stays open.

To reproduce, use this config and send {"k":"1"} to {"k":"8"} as eight lines in one write:

<source>
  @type tcp
  port 5170
  bind 127.0.0.1
  tag test
  <parse>
    @type json
  </parse>
  <extract>
    tag_key tag
  </extract>
</source>
<match test>
  @type stdout
  <buffer>
    @type memory
    chunk_limit_size 64
    total_limit_size 64
    flush_mode interval
    flush_interval 1h
    overflow_action throw_exception
  </buffer>
</match>

Each line after the buffer is full logs this, and the connection stays open:

[warn]: failed to parse message error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" message="{\"k\":\"3\"}" size=9

In the batch-emit path, router.emit_stream is outside parse_message, so its errors are not affected.

Please rescue only the errors from the parser, and re-raise the errors from the block. One way is to track whether the block is running. This keeps the block running inside @parser.parse, as on master:

Suggested change
def parse_message(msg, &block)
@parser.parse(msg, &block)
rescue => e
log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize
end
def parse_message(msg)
yielding = false
@parser.parse(msg) do |time, record|
yielding = true
result = yield time, record
yielding = false
result
end
rescue => e
raise if yielding
log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize
end

Do not reset yielding in an ensure clause. It must stay true when the block raises.

Please also add a test for this case. For example, use the single-emit config and stub d.instance.router.emit to raise. Then check that d.run raises that error, and that failed to parse message is not logged. The test driver re-raises errors from the server callback, so d.run raises on master too.

def parse_message(msg, &block)
@parser.parse(msg, &block)
rescue => e
log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Could you add log.debug_backtrace(e.backtrace) after this warn? in_tail does the same for an invalid line (lib/fluent/plugin/in_tail/line_feeder.rb). On master, the server helper logged the backtrace. With this change, it is not logged at any level, so a bug in a third-party parser is hard to locate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

in_tcp: a parser error drops the other lines in the same read and closes the connection

3 participants