Repository navigation
parser_syslog: require the space after VERSION to detect RFC 5424 - #5642
rafayqayyum wants to merge 1 commit into
Conversation
Signed-off-by: Rafay Qayyum <rafayqayyum786@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for fixing #5631. I confirmed that the message from the issue now parses with both parser engines when time_format matches it, and that the new test fails without the change.
One compatibility problem needs a fix. The detection now requires a literal space right after VERSION. On master, auto sent messages with a tab or a newline there to the RFC 5424 parsers, and some of them were parsed. With this change they go to the RFC 3164 path and are rejected. Details and a suggested fix are inline.
| EOS | ||
|
|
||
| REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2}/ | ||
| REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2} / |
There was a problem hiding this comment.
MUST: This makes auto reject some messages that master parses. On master, every message that starts with <PRI> and a non-zero digit goes to the RFC 5424 parsers. parse_rfc5424_regex finds VERSION with RFC5424_PRI_REGEXP (/\A<(?<pri>\d{1,3})>\d\d{0,2}\s/), and its final \s also accepts a tab or a newline after VERSION. The string parser parse_rfc5424 skips to the first space after PRI. The new detection accepts only a space right after VERSION, so these messages now go to the RFC 3164 path.
For example, take in_syslog with only message_format auto in <parse> (so the default parser_engine regexp), and send this message over UDP. It has a tab after VERSION.
"<14>1\t2026-04-25T16:43:29Z host app 123 ID - msg"- master (d2c7287): emits
syslog.user.infowith{"host" => "host", "ident" => "app", "pid" => "123", "msgid" => "ID", "extradata" => "-", "message" => "msg"}. - This PR: drops it and logs
[error]: invalid input ... error_class=Fluent::TimeParser::TimeParseError.
With parser_engine string, "<14>1\t 2026-04-25T16:43:29Z host app 123 ID - msg" (a tab and then a space) is parsed on master and returns no record with this PR. A newline in place of the tab behaves the same way with both engines.
Using \s here keeps master's behavior for these messages. It still fixes the message from #5631, because the year has four digits and VERSION has at most three.
| REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2} / | |
| REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2}\s/ |
Please also add the following regression test to SyslogParserTest in test/plugin/test_parser_syslog.rb. It passes on master, fails on the current HEAD, and passes with the suggestion above. With both changes, all tests in the file pass.
data(
'regexp/tab' => ['regexp', "\t"],
'regexp/newline' => ['regexp', "\n"],
'string/tab and space' => ['string', "\t "],
'string/newline and space' => ['string', "\n "],
)
def test_auto_detects_rfc5424_with_non_space_separator(data)
parser_engine, separator = data
@parser.configure(
'parser_engine' => parser_engine,
'message_format' => 'auto',
'with_priority' => true,
)
time = record = nil
@parser.instance.parse("<14>1#{separator}2026-04-25T16:43:29Z host app 123 ID - msg") do |t, r|
time, record = t, r
end
assert_equal(event_time('2026-04-25T16:43:29Z', format: '%Y-%m-%dT%H:%M:%S%z'), time)
assert_equal([14, 'host', 'app', '123', 'ID', '-', 'msg'], record.values_at('pri', 'host', 'ident', 'pid', 'msgid', 'extradata', 'message'))
end
Which issue(s) this PR fixes:
Fixes #5631
What this PR does / why we need it:
With
message_format auto,REGEXP_DETECT_RFC5424(/\A\<[0-9]{1,3}\>[1-9]\d{0,2}/) only checks that a digit follows PRI. An RFC 3164 message whose timestamp starts with the year, such as<13>2026-04-25T16:43:29+00:00 web01 nginx: hellofrom rsyslog'sRSYSLOG_ForwardFormat, was detected as RFC 5424 and then failed to parse.The detection now also requires the space after VERSION, which RFC 5424's header always has (
<PRI>VERSION SP TIMESTAMP ...), so such messages fall back to RFC 3164 and parse withtime_format.<14>1alone is still rejected. The new test covers both parser engines with the message from the issue.Docs Changes:
None.
Release Note:
parser_syslog:
message_format autodetects RFC 5424 only when VERSION is followed by a space, so RFC 3164 messages with a year-first timestamp are parsed.