Skip to content

RDK-62645: Make Ping/Trace non-blocking and run them without a shell - #386

Open
tukken-comcast wants to merge 10 commits into
developfrom
topic/RDK-62645
Open

tukken-comcast wants to merge 10 commits into
developfrom
topic/RDK-62645

Conversation

@tukken-comcast

Copy link
Copy Markdown
Contributor

Ping and Trace ran ping/traceroute synchronously on the NetworkManager implementation's only COM-RPC dispatch thread, so a 15-20 s Trace blocked every other NetworkManager and legacy Network method until it finished. The JSON-RPC API (parameters and result fields) is unchanged.

  • Implementation: a diagnostics engine runs at most one Ping and one Trace child at a time on its own worker threads; further requests of the same kind wait in a FIFO queue of 4 and get ERROR_INPROGRESS only when it is full. Children get a deadline derived from the request (capped at 120 s) and are killed with their process group on timeout or shutdown.
  • Interface: add IPingCallback, ITraceCallback, PingAsync and TraceAsync (appended, @JSON:omit). The synchronous Ping/Trace remain for COM-RPC clients, use the same engine, and are marked deprecated.
  • Plugins: org.rdk.NetworkManager Ping/Trace and org.rdk.Network ping/trace reply later on the caller's connection, so the request no longer holds a thread; in-process direct callers (channel ~0) still get the result synchronously. Pending callers get an error reply when either plugin is deactivated or the implementation goes away. DeferredReply.h is the only code that depends on Thunder's reply-later convention.

Security:

  • Fix command injection: the endpoint was placed into a shell command line. ping/traceroute are now run with fork/execvp and an argument vector, the endpoint is validated (IP address or hostname, no leading '-'), and "--" ends option parsing.
  • Fix a stack over-read in Trace output handling that could leak memory contents into the result for lines of 1023 bytes or more.

Other changes:

  • If ping cannot be started, the result now has success:false and an error instead of only the endpoint.
  • Deferred error replies are sent with HTTP 200 instead of 202 over HTTP POST; the JSON-RPC body is unchanged.
  • definition/docs: drop the never-populated guid result field and the malformed onTraceResponse entry; document the per-kind queue.

Tests: L1 tests for endpoint validation, argument vectors, output parsing, the engine and the process runner; L2 Ping/Trace tests use an injected process runner and check result fields, the deferred reply path and the reply helper; legacy ping/trace tests use PingAsync/TraceAsync.

@tukken-comcast
tukken-comcast requested a review from a team as a code owner November 12, 2026 13:45
Copilot AI balanced review requested due to automatic review settings November 12, 2026 13:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

External completion callbacks can deadlock synchronous Ping or Trace requests, and Stop() cannot unblock the stalled workers.

4 open issues
What changed in this PR

This PR moves Ping and Trace off NetworkManager’s COM-RPC dispatch thread and replaces shell commands with validated argument vectors. It adds deferred replies for both plugins while retaining synchronous COM-RPC methods.

Changes:

  • Adds bounded diagnostic queues, worker threads, process deadlines, and shutdown cleanup.
  • Introduces asynchronous callbacks and shared response helpers.
  • Expands diagnostic tests and updates API documentation.
File Description
tests/​mocks/​INetworkManagerMock.h Mocks asynchronous diagnostic methods.
tests/​l2Test/​rdk/​l2_test_rdkproxy.cpp Tests injected runners, results, queues, and deferred replies.
tests/​l2Test/​rdk/​CMakeLists.txt Links diagnostics into RDK tests.
tests/​l2Test/​libnm/​CMakeLists.txt Links diagnostics into libnm tests.
tests/​l2Test/​legacy/​l2_test_LegacyPlugin_NetworkAPIs.cpp Tests legacy APIs through asynchronous callbacks.
tests/​l1Test/​l1_test_diagnostics.cpp Covers validation, parsing, queues, and process execution.
tests/​l1Test/​CMakeLists.txt Adds diagnostic unit tests and implementation.
plugin/​NetworkManagerJsonRpc.cpp Routes Ping and Trace through deferred replies.
plugin/​NetworkManagerImplementation.h Declares diagnostic engine integration and async methods.
plugin/​NetworkManagerImplementation.cpp Replaces shell execution with engine-backed diagnostics.
plugin/​NetworkManagerDiagnosticsReply.h Adapts callbacks to JSON-RPC replies.
plugin/​NetworkManagerDiagnosticsJson.h Centralizes diagnostic result formatting.
plugin/​NetworkManagerDiagnostics.h Defines process runner and queue interfaces.
plugin/​NetworkManagerDiagnostics.cpp Implements validation, execution, parsing, and workers.
plugin/​NetworkManager.h Declares context-aware handlers and reply registry.
plugin/​NetworkManager.cpp Initializes and drains pending replies during lifecycle changes.
plugin/​DeferredReply.h Tracks and completes deferred requests.
plugin/​CMakeLists.txt Builds the diagnostics engine.
legacy/​LegacyNetworkAPIs.h Declares deferred legacy diagnostic handlers.
legacy/​LegacyNetworkAPIs.cpp Migrates legacy diagnostics to asynchronous callbacks.
interface/​INetworkManager.h Adds callback interfaces and asynchronous methods.
docs/​NetworkManagerPlugin.md Documents queues and removes unused result fields.
definition/​NetworkManager.json Aligns definitions with queue behavior and result fields.

Review effort: Balanced

Comment thread plugin/NetworkManagerDiagnostics.cpp Resolved
Comment thread plugin/NetworkManagerDiagnostics.cpp Resolved
Comment thread plugin/NetworkManagerDiagnostics.cpp Resolved
Comment thread tests/l2Test/rdk/l2_test_rdkproxy.cpp Resolved
Copilot AI balanced review requested due to automatic review settings November 13, 2026 14:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A blocked notifier lets workers bypass the admission limit by accumulating results in the unbounded _finished queue, risking unbounded memory growth and reply delays.

1 open issue
4 resolved since last review

Review effort: Balanced

Comment thread plugin/NetworkManagerDiagnostics.cpp Resolved
Copilot AI balanced review requested due to automatic review settings November 16, 2026 08:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A starter error can overwrite a direct caller’s completed Close() or Drain() reply because the failure branch does not check whether the entry is already done.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread plugin/DeferredReply.h Resolved
tukken-comcast and others added 9 commits November 19, 2026 19:07
Ping and Trace ran ping/traceroute synchronously on the NetworkManager
implementation's only COM-RPC dispatch thread, so a 15-20 s Trace blocked
every other NetworkManager and legacy Network method until it finished.
The JSON-RPC API (parameters and result fields) is unchanged.

- Implementation: a diagnostics engine runs at most one Ping and one
  Trace child at a time on its own worker threads; further requests of
  the same kind wait in a FIFO queue of 4 and get ERROR_INPROGRESS only
  when it is full. Children get a deadline derived from the request
  (capped at 120 s) and are killed with their process group on timeout
  or shutdown.
- Interface: add IDiagnosticsCallback, PingAsync and TraceAsync
  (appended, @JSON:omit). The synchronous Ping/Trace remain for COM-RPC
  clients, use the same engine, and are marked deprecated.
- Plugins: org.rdk.NetworkManager Ping/Trace and org.rdk.Network
  ping/trace reply later on the caller's connection, so the request no
  longer holds a thread; in-process direct callers (channel ~0) still get
  the result synchronously. Pending callers get an error reply when
  either plugin is deactivated or the implementation goes away.
  NetworkManagerDeferredReply.h is the only code that depends on
  Thunder's reply-later convention.

Security:
- Fix command injection: the endpoint was placed into a shell command
  line. ping/traceroute are now run with fork/execvp and an argument
  vector, the endpoint is validated (IP address or hostname, no leading
  '-'), and "--" ends option parsing.
- Fix a stack over-read in Trace output handling that could leak memory
  contents into the result for lines of 1023 bytes or more.

Other changes:
- If ping cannot be started, the result now has success:false and an
  error instead of only the endpoint.
- Deferred error replies are sent with HTTP 200 instead of 202 over
  HTTP POST; the JSON-RPC body is unchanged.
- definition/docs: drop the never-populated guid result field and the
  malformed onTraceResponse entry; document the per-kind queue.

Tests: L1 tests for endpoint validation, argument vectors, output parsing,
the engine and the process runner; L2 Ping/Trace tests use an injected
process runner and check result fields, the deferred reply path and the
reply helper; legacy ping/trace tests use PingAsync/TraceAsync.
- Avoid a duplicate reply when PingAsync/TraceAsync fails after a drain
  (plugin deactivation or implementation loss) has already answered the
  request: the handler now returns "reply later" instead of a second,
  synchronous error for the same request id.
- Keep the process runner's deadline and stop flag in force after the
  child exits: if a background descendant still holds the output pipe,
  the runner keeps polling and kills the process group when either
  fires, instead of blocking in read().
- Accept absolute hostnames with a single trailing dot (example.com.).
- org.rdk.NetworkManager.1.Trace returns ERROR_INPROGRESS as a JSON-RPC
  error when the Trace queue is full, like Ping and the legacy methods,
  instead of {"success":false}; other Trace failures are unchanged.

Regression tests cover each case.
- DeferredReply::Table::Invoke() now holds its own reference to the
  table for the whole call. During plugin deactivation, Close() can
  wake a waiting direct caller and the plugin and the implementation
  can then drop the last other references, which destroyed the table's
  mutex and pending map under the waiter.
- Run ping with -q. Only the summary is parsed; with a large count the
  per-packet lines could fill the 1 MiB output cap within the deadline,
  so the summary was lost and the result had no statistics.

Tests: a reply-table test that reproduces the shutdown order (AddressSanitizer
reports a use-after-free without the fix) and an argument test for -q.
…e callbacks

The deferred-reply table implemented IDiagnosticsCallback itself, so no
other JSON-RPC method could use it, and Ping/Trace results crossed
COM-RPC as JSON strings built in the implementation. Neither interface
has been released, so both are changed now. JSON-RPC results are
byte-for-byte unchanged.

- DeferredReply.h replaces NetworkManagerDeferredReply.h and depends
  only on Thunder and the plugin's Module.h:
  - Registry (was Table), one per plugin: tracks pending requests,
    defers or waits for direct callers, answers all on teardown, and
    returns how many it answered (the plugins log it).
  - PendingReply: move-only right to answer one request; destroying it
    unanswered answers with ERROR_UNAVAILABLE, except inside start(),
    where the start error code wins.
  - CallbackReply<INTERFACE>: base for COM-RPC callback adapters.
  Each request registers its own failure shaper; adapters format
  successful results.
- Interface: IDiagnosticsCallback is replaced by IPingCallback
  (Completed(rc, PingResult)) and ITraceCallback (Completed(rc, raw
  traceroute output)). PingResult uses plain members with a presence
  flag because the R4.4.3 stub generator has no OptionalType/vector.
- Implementation: parses ping output into PingResult; no JSON on the
  async path. The deprecated sync Ping/Trace still return the same JSON.
- NetworkManagerDiagnosticsJson.h builds the Ping/Trace result JSON;
  NetworkManagerDiagnosticsReply.h holds the Ping/Trace adapters shared
  by org.rdk.NetworkManager and org.rdk.Network, which differ only in a
  decorator (Trace "success", legacy "target").

Tests: golden tests pin the exact Ping/Trace result bytes (added and
passing before the refactor); registry tests cover close, abandon,
start failure inside start(), drain during start, completion inside
start(), failure shaping and direct callers; legacy ping/trace tests
complete through the typed callbacks and assert exact results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Diagnostics::Decorator is a plain function pointer. The decorators
  are named functions; the legacy one copies "endpoint" into "target"
  instead of capturing the endpoint in a lambda.
- NetworkManagerImplementation: the PingAsync/TraceAsync and sync
  Ping/Trace completion lambdas are named, one statement per line.
- Wrap the long lines added on this branch in the changed source files.

No behaviour change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Deliver PingAsync/TraceAsync results on a separate notifier thread
  instead of the Ping/Trace lane worker. Completed() is a COM-RPC call
  to another process; if it called the synchronous Ping (or Trace), the
  sync request queued behind it on the same lane, which was blocked in
  that callback. The deprecated sync Ping/Trace only wake a local
  waiter, so they still complete on the lane worker (Notify::INLINE).
  Stop() now drops queued jobs before joining the workers and drops
  results that were finished but not yet delivered.
- Reject endpoints containing a NUL character: inet_pton() and execvp()
  only saw the part before it, so "8.8.8.8\0;x" passed validation and
  pinged 8.8.8.8.
- Include ping's 2 ms probe interval in the deadline. With timeout 0 the
  deadline was a fixed 5 s, killing a healthy ping with a large count.
- Test: SentMessage() no longer blocks forever when no reply arrives.

Tests: an engine test where a dispatched callback submits a sync job of
the same kind and waits for it (fails without the fix), an embedded-NUL
endpoint test and a zero-timeout deadline case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
While the notifier thread was blocked in a slow Completed() callback,
the Ping/Trace workers kept running queued jobs and buffering their
results, so Submit() kept admitting new requests. Each buffered result
holds a callback and up to 1 MiB of output, with no upper bound.

Results waiting for the notifier now count toward the per-kind limit of
4, so Submit() returns ERROR_INPROGRESS once a lane holds 4 jobs that
are queued or awaiting delivery. Workers still never wait for the
notifier, so INLINE (sync Ping/Trace) jobs keep running.

Test: an engine test blocks the notifier and submits jobs one at a
time; without the fix all 20 are admitted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…uite

The Ping/Trace plugin tests and the DeferredReply registry tests do not
depend on the backend: they exercise the plugin, NetworkManagerImplementation
and DeferredReply.h through an injected process runner. They lived in the
RDK (IARM) proxy suite only because the old popen-based Ping/Trace tests
were there. Move them to libnm_proxy_l2_test, the suite for the default
libnm backend; its fixture has the same members, so the tests are
unchanged.

l2_test_rdkproxy.cpp is now the upstream file minus the obsolete
popen-based Ping/Trace tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For a direct (channel ~0) caller the pending entry stays in the registry
until Invoke() collects it. If Drain()/Close() or a completion answered
it while start() was still running and start() then returned an error,
Invoke() replaced that answer with the start error. Connected callers
were not affected: their answered entry is already removed, so the
start error is ignored. Now the first answer wins for direct callers
too.

Test: a direct-caller counterpart of StartFailureAfterDrainDoesNotReplyTwice.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 13:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An execvp failure can produce an empty trace reported as successful, leaving callers without a launch-failure diagnostic.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread plugin/NetworkManagerDiagnostics.cpp Resolved
When ping/traceroute ran through popen(), a missing program produced
the shell's "sh: traceroute6: not found" in the captured output. Since
they run through execvp(), the child exited with 127 and printed
nothing, so Trace returned an empty result list with no explanation.

The child now writes "<program>: not found" (ENOENT) or
"<program>: cannot execute" to the captured output before exiting. Both
messages are built before fork(), so the child does not allocate.

Tests: the missing-program test checks the message, and a new test
covers a file that is not executable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 14:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the mocked deferred-reply tests bypass HTTP dispatch, leaving actual reply delivery and HTTP status behavior unverified.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants