Repository navigation
RDK-62645: Make Ping/Trace non-blocking and run them without a shell - #386
tukken-comcast wants to merge 10 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
External completion callbacks can deadlock synchronous Ping or Trace requests, and Stop() cannot unblock the stalled workers.
4 open issues
What changed in this PR
This PR moves Ping and Trace off NetworkManager’s COM-RPC dispatch thread and replaces shell commands with validated argument vectors. It adds deferred replies for both plugins while retaining synchronous COM-RPC methods.
Changes:
- Adds bounded diagnostic queues, worker threads, process deadlines, and shutdown cleanup.
- Introduces asynchronous callbacks and shared response helpers.
- Expands diagnostic tests and updates API documentation.
| File | Description |
|---|---|
| tests/mocks/INetworkManagerMock.h | Mocks asynchronous diagnostic methods. |
| tests/l2Test/rdk/l2_test_rdkproxy.cpp | Tests injected runners, results, queues, and deferred replies. |
| tests/l2Test/rdk/CMakeLists.txt | Links diagnostics into RDK tests. |
| tests/l2Test/libnm/CMakeLists.txt | Links diagnostics into libnm tests. |
| tests/l2Test/legacy/l2_test_LegacyPlugin_NetworkAPIs.cpp | Tests legacy APIs through asynchronous callbacks. |
| tests/l1Test/l1_test_diagnostics.cpp | Covers validation, parsing, queues, and process execution. |
| tests/l1Test/CMakeLists.txt | Adds diagnostic unit tests and implementation. |
| plugin/NetworkManagerJsonRpc.cpp | Routes Ping and Trace through deferred replies. |
| plugin/NetworkManagerImplementation.h | Declares diagnostic engine integration and async methods. |
| plugin/NetworkManagerImplementation.cpp | Replaces shell execution with engine-backed diagnostics. |
| plugin/NetworkManagerDiagnosticsReply.h | Adapts callbacks to JSON-RPC replies. |
| plugin/NetworkManagerDiagnosticsJson.h | Centralizes diagnostic result formatting. |
| plugin/NetworkManagerDiagnostics.h | Defines process runner and queue interfaces. |
| plugin/NetworkManagerDiagnostics.cpp | Implements validation, execution, parsing, and workers. |
| plugin/NetworkManager.h | Declares context-aware handlers and reply registry. |
| plugin/NetworkManager.cpp | Initializes and drains pending replies during lifecycle changes. |
| plugin/DeferredReply.h | Tracks and completes deferred requests. |
| plugin/CMakeLists.txt | Builds the diagnostics engine. |
| legacy/LegacyNetworkAPIs.h | Declares deferred legacy diagnostic handlers. |
| legacy/LegacyNetworkAPIs.cpp | Migrates legacy diagnostics to asynchronous callbacks. |
| interface/INetworkManager.h | Adds callback interfaces and asynchronous methods. |
| docs/NetworkManagerPlugin.md | Documents queues and removes unused result fields. |
| definition/NetworkManager.json | Aligns definitions with queue behavior and result fields. |
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
A blocked notifier lets workers bypass the admission limit by accumulating results in the unbounded _finished queue, risking unbounded memory growth and reply delays.
1 open issue
4 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
A starter error can overwrite a direct caller’s completed Close() or Drain() reply because the failure branch does not check whether the entry is already done.
1 open issue
1 resolved since last review
Review effort: Balanced
Ping and Trace ran ping/traceroute synchronously on the NetworkManager implementation's only COM-RPC dispatch thread, so a 15-20 s Trace blocked every other NetworkManager and legacy Network method until it finished. The JSON-RPC API (parameters and result fields) is unchanged. - Implementation: a diagnostics engine runs at most one Ping and one Trace child at a time on its own worker threads; further requests of the same kind wait in a FIFO queue of 4 and get ERROR_INPROGRESS only when it is full. Children get a deadline derived from the request (capped at 120 s) and are killed with their process group on timeout or shutdown. - Interface: add IDiagnosticsCallback, PingAsync and TraceAsync (appended, @JSON:omit). The synchronous Ping/Trace remain for COM-RPC clients, use the same engine, and are marked deprecated. - Plugins: org.rdk.NetworkManager Ping/Trace and org.rdk.Network ping/trace reply later on the caller's connection, so the request no longer holds a thread; in-process direct callers (channel ~0) still get the result synchronously. Pending callers get an error reply when either plugin is deactivated or the implementation goes away. NetworkManagerDeferredReply.h is the only code that depends on Thunder's reply-later convention. Security: - Fix command injection: the endpoint was placed into a shell command line. ping/traceroute are now run with fork/execvp and an argument vector, the endpoint is validated (IP address or hostname, no leading '-'), and "--" ends option parsing. - Fix a stack over-read in Trace output handling that could leak memory contents into the result for lines of 1023 bytes or more. Other changes: - If ping cannot be started, the result now has success:false and an error instead of only the endpoint. - Deferred error replies are sent with HTTP 200 instead of 202 over HTTP POST; the JSON-RPC body is unchanged. - definition/docs: drop the never-populated guid result field and the malformed onTraceResponse entry; document the per-kind queue. Tests: L1 tests for endpoint validation, argument vectors, output parsing, the engine and the process runner; L2 Ping/Trace tests use an injected process runner and check result fields, the deferred reply path and the reply helper; legacy ping/trace tests use PingAsync/TraceAsync.
- Avoid a duplicate reply when PingAsync/TraceAsync fails after a drain
(plugin deactivation or implementation loss) has already answered the
request: the handler now returns "reply later" instead of a second,
synchronous error for the same request id.
- Keep the process runner's deadline and stop flag in force after the
child exits: if a background descendant still holds the output pipe,
the runner keeps polling and kills the process group when either
fires, instead of blocking in read().
- Accept absolute hostnames with a single trailing dot (example.com.).
- org.rdk.NetworkManager.1.Trace returns ERROR_INPROGRESS as a JSON-RPC
error when the Trace queue is full, like Ping and the legacy methods,
instead of {"success":false}; other Trace failures are unchanged.
Regression tests cover each case.
- DeferredReply::Table::Invoke() now holds its own reference to the table for the whole call. During plugin deactivation, Close() can wake a waiting direct caller and the plugin and the implementation can then drop the last other references, which destroyed the table's mutex and pending map under the waiter. - Run ping with -q. Only the summary is parsed; with a large count the per-packet lines could fill the 1 MiB output cap within the deadline, so the summary was lost and the result had no statistics. Tests: a reply-table test that reproduces the shutdown order (AddressSanitizer reports a use-after-free without the fix) and an argument test for -q.
…e callbacks
The deferred-reply table implemented IDiagnosticsCallback itself, so no
other JSON-RPC method could use it, and Ping/Trace results crossed
COM-RPC as JSON strings built in the implementation. Neither interface
has been released, so both are changed now. JSON-RPC results are
byte-for-byte unchanged.
- DeferredReply.h replaces NetworkManagerDeferredReply.h and depends
only on Thunder and the plugin's Module.h:
- Registry (was Table), one per plugin: tracks pending requests,
defers or waits for direct callers, answers all on teardown, and
returns how many it answered (the plugins log it).
- PendingReply: move-only right to answer one request; destroying it
unanswered answers with ERROR_UNAVAILABLE, except inside start(),
where the start error code wins.
- CallbackReply<INTERFACE>: base for COM-RPC callback adapters.
Each request registers its own failure shaper; adapters format
successful results.
- Interface: IDiagnosticsCallback is replaced by IPingCallback
(Completed(rc, PingResult)) and ITraceCallback (Completed(rc, raw
traceroute output)). PingResult uses plain members with a presence
flag because the R4.4.3 stub generator has no OptionalType/vector.
- Implementation: parses ping output into PingResult; no JSON on the
async path. The deprecated sync Ping/Trace still return the same JSON.
- NetworkManagerDiagnosticsJson.h builds the Ping/Trace result JSON;
NetworkManagerDiagnosticsReply.h holds the Ping/Trace adapters shared
by org.rdk.NetworkManager and org.rdk.Network, which differ only in a
decorator (Trace "success", legacy "target").
Tests: golden tests pin the exact Ping/Trace result bytes (added and
passing before the refactor); registry tests cover close, abandon,
start failure inside start(), drain during start, completion inside
start(), failure shaping and direct callers; legacy ping/trace tests
complete through the typed callbacks and assert exact results.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Diagnostics::Decorator is a plain function pointer. The decorators are named functions; the legacy one copies "endpoint" into "target" instead of capturing the endpoint in a lambda. - NetworkManagerImplementation: the PingAsync/TraceAsync and sync Ping/Trace completion lambdas are named, one statement per line. - Wrap the long lines added on this branch in the changed source files. No behaviour change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Deliver PingAsync/TraceAsync results on a separate notifier thread instead of the Ping/Trace lane worker. Completed() is a COM-RPC call to another process; if it called the synchronous Ping (or Trace), the sync request queued behind it on the same lane, which was blocked in that callback. The deprecated sync Ping/Trace only wake a local waiter, so they still complete on the lane worker (Notify::INLINE). Stop() now drops queued jobs before joining the workers and drops results that were finished but not yet delivered. - Reject endpoints containing a NUL character: inet_pton() and execvp() only saw the part before it, so "8.8.8.8\0;x" passed validation and pinged 8.8.8.8. - Include ping's 2 ms probe interval in the deadline. With timeout 0 the deadline was a fixed 5 s, killing a healthy ping with a large count. - Test: SentMessage() no longer blocks forever when no reply arrives. Tests: an engine test where a dispatched callback submits a sync job of the same kind and waits for it (fails without the fix), an embedded-NUL endpoint test and a zero-timeout deadline case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
While the notifier thread was blocked in a slow Completed() callback, the Ping/Trace workers kept running queued jobs and buffering their results, so Submit() kept admitting new requests. Each buffered result holds a callback and up to 1 MiB of output, with no upper bound. Results waiting for the notifier now count toward the per-kind limit of 4, so Submit() returns ERROR_INPROGRESS once a lane holds 4 jobs that are queued or awaiting delivery. Workers still never wait for the notifier, so INLINE (sync Ping/Trace) jobs keep running. Test: an engine test blocks the notifier and submits jobs one at a time; without the fix all 20 are admitted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…uite The Ping/Trace plugin tests and the DeferredReply registry tests do not depend on the backend: they exercise the plugin, NetworkManagerImplementation and DeferredReply.h through an injected process runner. They lived in the RDK (IARM) proxy suite only because the old popen-based Ping/Trace tests were there. Move them to libnm_proxy_l2_test, the suite for the default libnm backend; its fixture has the same members, so the tests are unchanged. l2_test_rdkproxy.cpp is now the upstream file minus the obsolete popen-based Ping/Trace tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For a direct (channel ~0) caller the pending entry stays in the registry until Invoke() collects it. If Drain()/Close() or a completion answered it while start() was still running and start() then returned an error, Invoke() replaced that answer with the start error. Connected callers were not affected: their answered entry is already removed, so the start error is ignored. Now the first answer wins for direct callers too. Test: a direct-caller counterpart of StartFailureAfterDrainDoesNotReplyTwice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
f7b81f1 to
bcbbe52
Compare
When ping/traceroute ran through popen(), a missing program produced the shell's "sh: traceroute6: not found" in the captured output. Since they run through execvp(), the child exited with 127 and printed nothing, so Trace returned an empty result list with no explanation. The child now writes "<program>: not found" (ENOENT) or "<program>: cannot execute" to the captured output before exiting. Both messages are built before fork(), so the child does not allocate. Tests: the missing-program test checks the message, and a new test covers a file that is not executable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because the mocked deferred-reply tests bypass HTTP dispatch, leaving actual reply delivery and HTTP status behavior unverified.
No open issues 🎉
1 resolved since last review
Review effort: Balanced


Ping and Trace ran ping/traceroute synchronously on the NetworkManager implementation's only COM-RPC dispatch thread, so a 15-20 s Trace blocked every other NetworkManager and legacy Network method until it finished. The JSON-RPC API (parameters and result fields) is unchanged.
Security:
Other changes:
Tests: L1 tests for endpoint validation, argument vectors, output parsing, the engine and the process runner; L2 Ping/Trace tests use an injected process runner and check result fields, the deferred reply path and the reply helper; legacy ping/trace tests use PingAsync/TraceAsync.