Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion lib/fluent/plugin/parser_syslog.rb
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ class SyslogParser < Parser
\A<(?<pri>[0-9]{1,3})\>[1-9]\d{0,2} %s\z
EOS

REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2}/
REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2} /

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: This makes auto reject some messages that master parses. On master, every message that starts with <PRI> and a non-zero digit goes to the RFC 5424 parsers. parse_rfc5424_regex finds VERSION with RFC5424_PRI_REGEXP (/\A<(?<pri>\d{1,3})>\d\d{0,2}\s/), and its final \s also accepts a tab or a newline after VERSION. The string parser parse_rfc5424 skips to the first space after PRI. The new detection accepts only a space right after VERSION, so these messages now go to the RFC 3164 path.

For example, take in_syslog with only message_format auto in <parse> (so the default parser_engine regexp), and send this message over UDP. It has a tab after VERSION.

"<14>1\t2026-04-25T16:43:29Z host app 123 ID - msg"
  • master (d2c7287): emits syslog.user.info with {"host" => "host", "ident" => "app", "pid" => "123", "msgid" => "ID", "extradata" => "-", "message" => "msg"}.
  • This PR: drops it and logs [error]: invalid input ... error_class=Fluent::TimeParser::TimeParseError.

With parser_engine string, "<14>1\t 2026-04-25T16:43:29Z host app 123 ID - msg" (a tab and then a space) is parsed on master and returns no record with this PR. A newline in place of the tab behaves the same way with both engines.

Using \s here keeps master's behavior for these messages. It still fixes the message from #5631, because the year has four digits and VERSION has at most three.

Suggested change
REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2} /
REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2}\s/

Please also add the following regression test to SyslogParserTest in test/plugin/test_parser_syslog.rb. It passes on master, fails on the current HEAD, and passes with the suggestion above. With both changes, all tests in the file pass.

  data(
    'regexp/tab' => ['regexp', "\t"],
    'regexp/newline' => ['regexp', "\n"],
    'string/tab and space' => ['string', "\t "],
    'string/newline and space' => ['string', "\n "],
  )
  def test_auto_detects_rfc5424_with_non_space_separator(data)
    parser_engine, separator = data
    @parser.configure(
      'parser_engine' => parser_engine,
      'message_format' => 'auto',
      'with_priority' => true,
    )

    time = record = nil
    @parser.instance.parse("<14>1#{separator}2026-04-25T16:43:29Z host app 123 ID - msg") do |t, r|
      time, record = t, r
    end
    assert_equal(event_time('2026-04-25T16:43:29Z', format: '%Y-%m-%dT%H:%M:%S%z'), time)
    assert_equal([14, 'host', 'app', '123', 'ID', '-', 'msg'], record.values_at('pri', 'host', 'ident', 'pid', 'msgid', 'extradata', 'message'))
  end


RFC3164_WITHOUT_TIME_AND_PRI_REGEXP = /(?<host>[^ ]*) (?<ident>[^ :\[]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
RFC3164_CAPTURES = RFC3164_WITHOUT_TIME_AND_PRI_REGEXP.names.freeze
Expand Down
17 changes: 17 additions & 0 deletions test/plugin/test_parser_syslog.rb
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,23 @@ def test_auto_detects_rfc3164_when_a_later_line_looks_like_rfc5424(parser_engine
assert_equal([13, 'web01', 'nginx'], record.values_at('pri', 'host', 'ident'))
end

data('regexp' => 'regexp', 'string' => 'string')
def test_auto_detects_rfc3164_with_a_timestamp_starting_with_the_year(parser_engine)
@parser.configure(
'parser_engine' => parser_engine,
'message_format' => 'auto',
'with_priority' => true,
'time_format' => '%Y-%m-%dT%H:%M:%S%z',
)

time = record = nil
@parser.instance.parse('<13>2026-04-25T16:43:29+00:00 web01 nginx: hello') do |t, r|
time, record = t, r
end
assert_equal(event_time('2026-04-25T16:43:29+00:00', format: '%Y-%m-%dT%H:%M:%S%z'), time)
assert_equal([13, 'web01', 'nginx', 'hello'], record.values_at('pri', 'host', 'ident', 'message'))
end

data(
'rfc5424' => ['rfc5424', 'time_format'],
'auto' => ['auto', 'rfc5424_time_format'],
Expand Down