Repository navigation
in_tcp: drop only the line the parser fails on - #5640
rafayqayyum wants to merge 1 commit into
Conversation
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the fix. Rescuing the parser error per line fixes #5629 in both paths. The new tests fail on master and pass with this change.
One problem: in the single-emit path, the rescue also catches emit errors. See the inline comment.
Also, the DCO check fails because the commit has no Signed-off-by line. Please add your sign-off to each commit (for example, git commit --amend -s) and force-push.
| def parse_message(msg, &block) | ||
| @parser.parse(msg, &block) | ||
| rescue => e | ||
| log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize | ||
| end |
There was a problem hiding this comment.
MUST: This rescue also covers the block. So in the single-emit path, it catches errors from router.emit on line 155.
The description says emit errors are unaffected because EventRouter rescues them. But without an @ERROR label, RootAgent#handle_emits_error logs emit transaction failed and re-raises the error. On master, the error reaches the server helper. It logs unexpected error on reading data and closes the connection. With this change, the error is logged as a parse failure, and the connection stays open.
To reproduce, use this config and send {"k":"1"} to {"k":"8"} as eight lines in one write:
<source>
@type tcp
port 5170
bind 127.0.0.1
tag test
<parse>
@type json
</parse>
<extract>
tag_key tag
</extract>
</source>
<match test>
@type stdout
<buffer>
@type memory
chunk_limit_size 64
total_limit_size 64
flush_mode interval
flush_interval 1h
overflow_action throw_exception
</buffer>
</match>
Each line after the buffer is full logs this, and the connection stays open:
[warn]: failed to parse message error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" message="{\"k\":\"3\"}" size=9
In the batch-emit path, router.emit_stream is outside parse_message, so its errors are not affected.
Please rescue only the errors from the parser, and re-raise the errors from the block. One way is to track whether the block is running. This keeps the block running inside @parser.parse, as on master:
| def parse_message(msg, &block) | |
| @parser.parse(msg, &block) | |
| rescue => e | |
| log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize | |
| end | |
| def parse_message(msg) | |
| yielding = false | |
| @parser.parse(msg) do |time, record| | |
| yielding = true | |
| result = yield time, record | |
| yielding = false | |
| result | |
| end | |
| rescue => e | |
| raise if yielding | |
| log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize | |
| end |
Do not reset yielding in an ensure clause. It must stay true when the block raises.
Please also add a test for this case. For example, use the single-emit config and stub d.instance.router.emit to raise. Then check that d.run raises that error, and that failed to parse message is not logged. The test driver re-raises errors from the server callback, so d.run raises on master too.
| def parse_message(msg, &block) | ||
| @parser.parse(msg, &block) | ||
| rescue => e | ||
| log.warn "failed to parse message", error: e, message: msg[...MAX_LOGGED_MESSAGE_LENGTH], size: msg.bytesize |
There was a problem hiding this comment.
NIT: Could you add log.debug_backtrace(e.backtrace) after this warn? in_tail does the same for an invalid line (lib/fluent/plugin/in_tail/line_feeder.rb). On master, the server helper logged the backtrace. With this change, it is not logged at any level, so a bug in a third-party parser is hard to locate.
Which issue(s) this PR fixes:
Fixes #5629
What this PR does / why we need it:
in_tcpcalled@parser.parse(msg)without a rescue, in both the batch-emit and the single-emit (<extract>withtag_key) paths. When the parser raises on a line — for example the json parser withtime_formatand a time field that doesn't match — the error goes up to the server helper, which logsunexpected error on reading dataand closes the connection. With batch emit none of the lines in that read are emitted; with single emit the lines after the bad one are lost.Both paths now call a
parse_messagehelper that rescues the error and logsfailed to parse messageat warn level, with the message truncated likepattern not matched. Only the bad line is dropped, the other lines are emitted, and the connection stays open. Emit errors are unaffected, sinceEventRouter#emitand#emit_streamalready rescue them.The new tests send the three lines from the issue in one write, then a fourth line on the same socket, for both batch and single emit, and expect k1, k3 and k4 plus the warning.
Docs Changes:
None.
Release Note:
in_tcp: a line the parser fails on is dropped with a warning instead of closing the connection and losing the other lines.