Skip to content

parser_syslog: anchor the RFC 5424 header regexp at the match position - #5641

Open
rafayqayyum wants to merge 1 commit into
fluent:masterfrom
rafayqayyum:syslog-anchor-rfc5424-regexp
Open

rafayqayyum wants to merge 1 commit into
fluent:masterfrom
rafayqayyum:syslog-anchor-rfc5424-regexp

Conversation

@rafayqayyum

Copy link
Copy Markdown

Which issue(s) this PR fixes:
Fixes #5630

What this PR does / why we need it:

With message_format rfc5424 and parser_engine regexp, the header fields after the timestamp are matched with RFC5424_WITHOUT_TIME_AND_PRI_REGEXP.match(text, idx). Regexp#match with a position doesn't anchor the match there, so when the text at idx doesn't match, the regexp matches later in the message: <14>1 2026-04-25T16:43:29Z hôst app 123 ID - msg gave "host" => "st", and first line\nhost app 123 ID - second line dropped first line and parsed the second line.

The regexp now starts with \G, so the fields must start at idx; otherwise the message is reported as not matched, as the issue suggests. The constant is only matched with a position (and exposed through patterns), so nothing else changes. The new tests use the issue's two inputs.

Docs Changes:
None.

Release Note:
parser_syslog: the regexp engine no longer matches the RFC 5424 header later in the message.

Copilot AI balanced review requested due to automatic review settings November 15, 2026 13:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Signed-off-by: Rafay Qayyum <rafayqayyum786@gmail.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. Both inputs from #5630 are now rejected, and the new tests fail on master and pass with this change.

I found one kind of input that master parses but this change rejects. Please see the inline comment.

RFC3164_PRI_REGEXP = /\A<(?<pri>[0-9]{1,3})>/

RFC5424_WITHOUT_TIME_AND_PRI_REGEXP = /(?<host>[!-~]{1,255}) (?<ident>[!-~]{1,48}) (?<pid>[!-~]{1,128}) (?<msgid>[!-~]{1,32}) (?<extradata>(?:\-|(?:\[.*?(?<!\\)\])))(?: (?<message>.+))?\z/m
RFC5424_WITHOUT_TIME_AND_PRI_REGEXP = /\G(?<host>[!-~]{1,255}) (?<ident>[!-~]{1,48}) (?<pid>[!-~]{1,128}) (?<msgid>[!-~]{1,32}) (?<extradata>(?:\-|(?:\[.*?(?<!\\)\])))(?: (?<message>.+))?\z/m

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: \G also rejects messages that have more than one space or tab between TIMESTAMP and HOSTNAME. Master parses them, because the unanchored match skipped the extra whitespace. Two examples, as Ruby strings:

"<14>1 2026-04-25T16:43:29Z  host app 123 ID - msg"
"<14>1 2026-04-25T16:43:29Z \thost app 123 ID - msg"

With message_format rfc5424 (or auto), with_priority true and the default regexp engine, master parses both into "host" => "host" and "message" => "msg". With this change, the parser yields nil, nil for both. With the default emit_unmatched_lines false, in_syslog then logs failed to parse message and drops the record.

Allowing only spaces and tabs at the match position keeps these messages. Both inputs from #5630 are still rejected, and so is a header after a line break:

Suggested change
RFC5424_WITHOUT_TIME_AND_PRI_REGEXP = /\G(?<host>[!-~]{1,255}) (?<ident>[!-~]{1,48}) (?<pid>[!-~]{1,128}) (?<msgid>[!-~]{1,32}) (?<extradata>(?:\-|(?:\[.*?(?<!\\)\])))(?: (?<message>.+))?\z/m
RFC5424_WITHOUT_TIME_AND_PRI_REGEXP = /\G[ \t]*(?<host>[!-~]{1,255}) (?<ident>[!-~]{1,48}) (?<pid>[!-~]{1,128}) (?<msgid>[!-~]{1,32}) (?<extradata>(?:\-|(?:\[.*?(?<!\\)\])))(?: (?<message>.+))?\z/m

Could you also add a test that these two messages are still parsed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parser_syslog: the regexp engine for RFC 5424 can match the header later in the message and corrupt the record

3 participants