Skip to content

parser_syslog: require the space after VERSION to detect RFC 5424 - #5642

Open
rafayqayyum wants to merge 1 commit into
fluent:masterfrom
rafayqayyum:syslog-detect-rfc5424-version-space
Open

rafayqayyum wants to merge 1 commit into
fluent:masterfrom
rafayqayyum:syslog-detect-rfc5424-version-space

Conversation

@rafayqayyum

Copy link
Copy Markdown

Which issue(s) this PR fixes:
Fixes #5631

What this PR does / why we need it:

With message_format auto, REGEXP_DETECT_RFC5424 (/\A\<[0-9]{1,3}\>[1-9]\d{0,2}/) only checks that a digit follows PRI. An RFC 3164 message whose timestamp starts with the year, such as <13>2026-04-25T16:43:29+00:00 web01 nginx: hello from rsyslog's RSYSLOG_ForwardFormat, was detected as RFC 5424 and then failed to parse.

The detection now also requires the space after VERSION, which RFC 5424's header always has (<PRI>VERSION SP TIMESTAMP ...), so such messages fall back to RFC 3164 and parse with time_format. <14>1 alone is still rejected. The new test covers both parser engines with the message from the issue.

Docs Changes:
None.

Release Note:
parser_syslog: message_format auto detects RFC 5424 only when VERSION is followed by a space, so RFC 3164 messages with a year-first timestamp are parsed.

Signed-off-by: Rafay Qayyum <rafayqayyum786@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 15, 2026 13:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing #5631. I confirmed that the message from the issue now parses with both parser engines when time_format matches it, and that the new test fails without the change.

One compatibility problem needs a fix. The detection now requires a literal space right after VERSION. On master, auto sent messages with a tab or a newline there to the RFC 5424 parsers, and some of them were parsed. With this change they go to the RFC 3164 path and are rejected. Details and a suggested fix are inline.

EOS

REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2}/
REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2} /

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: This makes auto reject some messages that master parses. On master, every message that starts with <PRI> and a non-zero digit goes to the RFC 5424 parsers. parse_rfc5424_regex finds VERSION with RFC5424_PRI_REGEXP (/\A<(?<pri>\d{1,3})>\d\d{0,2}\s/), and its final \s also accepts a tab or a newline after VERSION. The string parser parse_rfc5424 skips to the first space after PRI. The new detection accepts only a space right after VERSION, so these messages now go to the RFC 3164 path.

For example, take in_syslog with only message_format auto in <parse> (so the default parser_engine regexp), and send this message over UDP. It has a tab after VERSION.

"<14>1\t2026-04-25T16:43:29Z host app 123 ID - msg"
  • master (d2c7287): emits syslog.user.info with {"host" => "host", "ident" => "app", "pid" => "123", "msgid" => "ID", "extradata" => "-", "message" => "msg"}.
  • This PR: drops it and logs [error]: invalid input ... error_class=Fluent::TimeParser::TimeParseError.

With parser_engine string, "<14>1\t 2026-04-25T16:43:29Z host app 123 ID - msg" (a tab and then a space) is parsed on master and returns no record with this PR. A newline in place of the tab behaves the same way with both engines.

Using \s here keeps master's behavior for these messages. It still fixes the message from #5631, because the year has four digits and VERSION has at most three.

Suggested change
REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2} /
REGEXP_DETECT_RFC5424 = /\A\<[0-9]{1,3}\>[1-9]\d{0,2}\s/

Please also add the following regression test to SyslogParserTest in test/plugin/test_parser_syslog.rb. It passes on master, fails on the current HEAD, and passes with the suggestion above. With both changes, all tests in the file pass.

  data(
    'regexp/tab' => ['regexp', "\t"],
    'regexp/newline' => ['regexp', "\n"],
    'string/tab and space' => ['string', "\t "],
    'string/newline and space' => ['string', "\n "],
  )
  def test_auto_detects_rfc5424_with_non_space_separator(data)
    parser_engine, separator = data
    @parser.configure(
      'parser_engine' => parser_engine,
      'message_format' => 'auto',
      'with_priority' => true,
    )

    time = record = nil
    @parser.instance.parse("<14>1#{separator}2026-04-25T16:43:29Z host app 123 ID - msg") do |t, r|
      time, record = t, r
    end
    assert_equal(event_time('2026-04-25T16:43:29Z', format: '%Y-%m-%dT%H:%M:%S%z'), time)
    assert_equal([14, 'host', 'app', '123', 'ID', '-', 'msg'], record.values_at('pri', 'host', 'ident', 'pid', 'msgid', 'extradata', 'message'))
  end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parser_syslog: message_format auto parses RFC 3164 messages with an RFC 3339 timestamp as RFC 5424

3 participants