Repository navigation
fix: keep runtime label lookups on the plugin's own configuration during reload - #5628
sarankumar1325 wants to merge 2 commits into
Conversation
…ing reload event_emitter_router looked up labels in Engine.root_agent, which reload_config replaces before the old plugins stop. A runtime lookup (as rewrite_tag_filter and concat do) could then emit into a new label before its <match> was added, cache that tag as unmatched, and drop events of that tag until the next reload. Look up labels through the plugin's context_router, and clear EventRouter's MatchCache when a rule is added. Fixes fluent#5594 Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the fix. I confirmed it with the script in #5594. On master, the event during the reload and all 5 events after it are lost. With this PR, all 6 arrive.
One case changes from master, though. An event that the old configuration sends late in its shutdown now goes to an old label whose plugins have already dropped their router, and it is lost with NoMethodError. On master, it goes to the new configuration. #5594 mentions this case too. Please see the inline comments.
| root_agent = event_emitter_root_agent | ||
| if label_name == "@ROOT" | ||
| Engine.root_agent.event_router | ||
| root_agent.event_router | ||
| else | ||
| Engine.root_agent.find_label(label_name).event_router | ||
| root_agent.find_label(label_name).event_router |
There was a problem hiding this comment.
MUST: An event that the old configuration sends late in its shutdown now goes to an old label whose plugins have already dropped their router. The event is lost with NoMethodError. On master, it goes to the new configuration.
During Engine#reload_config, RootAgent#shutdown of the old configuration runs after_shutdown on its plugins one by one, and EventEmitter#after_shutdown sets @router to nil. The Thread helper does not join the threads of a plugin before close, which runs after after_shutdown. Output#after_shutdown joins the flush threads, and the <label> sections run after_shutdown in the order they are defined (@ERROR comes last). So the thread of an old input, or a flush thread of a buffered output in a later <label>, can still emit after the plugins of a label have run after_shutdown. #5594 mentions this case at the end of "Additional context".
I reproduced it with this configuration. label_at_runtime is the output in your test in test_engine.rb, and count records the events.
<source>
@type sample
tag t
</source>
<match t>
@type label_at_runtime
to @L
</match>
<label @L>
<match **>
@type relabel
@label @C
</match>
</label>
<label @C>
<match **>
@type count
</match>
</label>
In each case below, the event reaches the new configuration on master. With this PR, the old relabel raises NoMethodError: undefined method 'emit_stream' for nil, and the event is lost.
- The thread of
in_sampleis inside the oldlabel_at_runtimewhen the reload starts. It looks up@Lafter the oldrelabelhas runafter_shutdown. I held the thread until thecloseof the old input. - The same thread looks up
@Lafter the new configuration is configured, but before the old one startsafter_shutdown. It callsemit_streamafter the oldrelabelhas runafter_shutdown. - This one needs no hook, and uses another configuration. A buffered output in
<label @L2>looks up@L1inwrite, and@L1hasrelabel.writetook 80 ms,flush_at_shutdownwas false, and the reload started right after a flush started.
Please keep these events from being lost, as on master, and add tests for these cases. A check only at the time of the lookup does not cover case 2, because the router was taken before the old configuration started after_shutdown. In a test, you can make the timing certain as your test does with add_match.
| engine = Fluent::EngineClass.new | ||
| engine.init(system_config) | ||
| engine.configure(conf) | ||
| engine.root_agent.start |
There was a problem hiding this comment.
SHOULD: These tests do not go through Fluent::Engine, so they do not test the cause of #5594. On master, event_emitter_router reads the global Fluent::Engine.root_agent, but this test creates another EngineClass. So on master, both tests fail with NoMethodError: undefined method 'find_label' for nil. That includes the second test, which should pass on master.
With the stub below, the first test fails on master as in #5594 ([], all 6 events lost), and the second one passes on master. Both pass with this PR.
| engine.root_agent.start | |
| engine.root_agent.start | |
| stub(Fluent::Engine).root_agent { engine.root_agent } |
| def event_emitter_root_agent | ||
| router = self.context_router | ||
| handler = router.respond_to?(:emit_error_handler) ? router.emit_error_handler : nil | ||
| if handler.respond_to?(:find_label) | ||
| handler | ||
| elsif handler.respond_to?(:root_agent) && handler.root_agent | ||
| handler.root_agent | ||
| else | ||
| Engine.root_agent | ||
| end | ||
| end |
There was a problem hiding this comment.
SHOULD: This guesses the agent of the plugin from emit_error_handler with respond_to? checks, and it does so on every call.
- When no check matches, it falls back to
Engine.root_agentwithout any sign, and the reload bug comes back for that plugin. In Ruby, I would let the agents answer this themselves. For example,Labelalready hasroot_agent, andRootAgent#root_agentcould returnself. - For a plugin in a
RootAgentor aLabel, the owning root agent does not change afterconfigure. But it is computed again on every runtime lookup, and plugins likerewrite_tag_filterlook up the label for each event stream. In my measurement, one call ofevent_emitter_router("@L")took about 195 ns on master and 300 to 350 ns with this PR. When I cached the owning root agent, it went back to about 200 ns. The fallback toEngine.root_agentshould stay dynamic, so please do not cache it. - It becomes a public method of every plugin that uses this helper, but it is only used in
event_emitter_router. - The new unit tests cover only the branch for
RootAgent. A plugin in a<label>goes through thehandler.root_agentbranch, and no test checks it. Also, the unit tests mock onlyfind_label, so they need an update if the agents answerroot_agent.
| super | ||
| end | ||
| end | ||
| Fluent::Agent.prepend(EmitDuringAddMatchHook) |
There was a problem hiding this comment.
NIT: This prepends the hook to Fluent::Agent for the whole test process, because rake test loads all test files into one process. The hook can stay inside the test with rr. For example, replace the thread-local setup in the test with this block, after old_output is assigned. Then EmitDuringAddMatchHook, this prepend, and the thread-local cleanup in teardown can be removed.
stub.proxy(Fluent::Label).new do |label|
if label.context == "@L"
label.singleton_class.prepend(Module.new do
define_method(:add_match) do |*args|
old_output.process("t", Fluent::OneEventStream.new(Fluent::EventTime.now, { "message" => "during" }))
super(*args)
end
end)
end
label
endAddress review on fluent#5628. - event_emitter_router returns the label's EventRouter itself outside a reload. For a plugin of the old configuration during a reload, it returns a LabelRouter that decides on each emit: the old configuration's label while it runs, and Engine.root_agent's label once the old configuration has started to shut down (as on master). So a router taken before the old shutdown and used after it no longer hits a relabel without a router. - The agents answer their owning RootAgent (RootAgent#root_agent returns self, Label#root_agent already exists). The owner is cached in the plugin; the Engine.root_agent fallback stays dynamic. The helper is private. - RootAgent#shutdown_started? is set when the whole configuration starts shutting down. - Tests: reload tests go through Fluent::Engine.root_agent (stubbed), the add_match hook is scoped with rr instead of a global prepend, and new tests cover a late lookup and a router taken during reload, for plugins in <match> and in <label>. Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the update. I checked 0df0c69. The three cases in my last review now deliver the event, as on master. The script in #5594 still delivers all 6 events. The reload tests go through Fluent::Engine.root_agent, the owning root agent is cached, and the hook is scoped to the test.
One case is still lost, and I would like your opinion on how to handle it.
An old input emits t from its thread after the new configuration is configured, but before the old one starts to shut down. The runtime lookup returns the old @L, and the event waits in the <filter> there. Meanwhile, the old relabel in @L runs after_shutdown and drops its router. When the filter returns, relabel raises NoMethodError: undefined method 'emit_stream' for nil, and the event is lost. On master, the lookup returns the new @L, and the event arrives. Events that @label or relabel send into an old label have the same window on master.
<match t>
@type label_at_runtime
to @L
</match>
<label @L>
<filter **>
@type slow_filter
</filter>
<match **>
@type relabel
@label @C
</match>
</label>
<label @C>
<match **>
@type count
</match>
</label>
In my test, slow_filter waits until the close of the old input.
I tried one way: keep the router of a plugin until close. That is, remove @router = nil from EventEmitter#after_shutdown, and keep clearing it in close and terminate. Inputs close before labels, and the Thread helper waits up to one second per thread in close by default. In my test, the event leaves the filter within that time, so it reaches the old @C. If the filter waits longer, the event can still reach relabel after its router is cleared. With this change on top of 0df0c69:
- The case above and all the cases in my last review deliver the event. The same case with
relabelin place oflabel_at_runtimedelivers it too. - The tests in this PR pass. So do the tests of the root agent, the event router, outputs, buffered outputs, secondary, filters, inputs, relabel, copy, multi output, exec_filter and in_sample.
This changes the router lifetime for all plugins that use the event emitter helper, also in a normal shutdown. In my test of a normal shutdown, a late event now passes through relabel and reaches a buffered output after both have run after_shutdown. The emit returns without an error, but the event is never flushed, even with flush_at_shutdown true. A file buffer keeps it in a chunk. A memory buffer loses it during the shutdown. On master, the emit raises NoMethodError at relabel.
So I see two ways:
- Add this change to this PR. Then I suggest tests for both cases: a late emit while the old configuration shuts down, and an event that enters the old label before the shutdown and leaves its filter after
after_shutdown. The current engine tests for late emits send them after the reload finishes, so they do not check when the switch happens. - Keep this PR as it is, and change when the router is dropped in a separate PR. That change also covers
@labelandrelabel. This leaves the runtime lookup regression above in this PR until the follow-up fixes it.
Would you prefer option 1 or option 2? Please let me know before changing the code.
One note about my previous request. It moved the switch to the start of the shutdown. In my test, an old input emits from its shutdown through a runtime label lookup. The event now reaches the new configuration before it starts, and its memory buffer drops the event on startup. Your previous commit delivered it to the old configuration. The current behavior matches master. I am not asking for a change to this memory buffer case in this PR.
Which issue(s) this PR fixes:
Fixes #5594
What this PR does / why we need it:
During a graceful reload,
Engine#reload_configmakes the new configurationEngine.root_agentbefore it is configured and before the old plugins stop. Plugins that callevent_emitter_routerat runtime (for example a rulelabelortimeout_label) then send events to the new label. If that happens before the new<match>is added, the event is dropped as "no patterns matched", andEventRouter'sMatchCachekeeps thatNoMatchMatchfor the tag. The new configuration then drops its own events of that tag until the cache expires (1,024 other tags) or the next reload.@labelandrelabelare not affected, because they resolve the label once inconfigure.This PR:
context_router(set on the plugin beforeconfigure), so a runtime lookup stays on the configuration the plugin belongs to. New plugins still see the new labels inconfigure, becausecontext_routeris already the new agent. The event emitted during reload goes to the old label, as with@labelandrelabel.MatchCacheinadd_rule, so a tag that arrived before a<match>was added is not stuck as unmatched.Docs Changes:
None. The issue also mentioned documenting this as a limitation if it was not fixed; this change removes the limitation instead.
Release Note:
Runtime label lookups during a graceful reload stay on the plugin's own configuration, so a tag is no longer left unmatched in the new configuration after a reload.