Skip to content

feat(ffe): add shared Agentless EVP fallback transport [FFLSDK-185] - #2517

Draft
leoromanovsky wants to merge 7 commits into
mainfrom
leo.romanovsky/fflsdk-185-libdatadog-evp-fallback-v2
Draft

feat(ffe): add shared Agentless EVP fallback transport [FFLSDK-185]#2517
leoromanovsky wants to merge 7 commits into
mainfrom
leo.romanovsky/fflsdk-185-libdatadog-evp-fallback-v2

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the shared libdatadog sidecar transport and identity-bearing FFI/IPC contract needed for PHP Feature Flags exposures and flag-evaluation telemetry to reach Event Platform intake in Agentless mode.

Base: 1413485bc8728dde0859fdcd799a38a5d953c919

Head: 536240cb5b8a59e5bfc609b039363d7417158412

Motivation

We are shipping an Agentless, CDN-delivered Feature Flags configuration mode to simplify customer deployments. The SDK's core telemetry still needs a safe network path off the process: discover a compatible running Agent or serverless proxy when present, then fall back to authenticated direct EVP intake when no compatible local route is available.

libdatadog owns the shared sidecar transport used by PHP, so this layer must preserve Agent-backed behavior while providing the discovery, fallback, credential isolation, producer identity, and retry semantics required for Agentless delivery.

Changes

  • Add a per-session FfeEvpTransport shared by the exposure and flag-evaluation writers.
  • Keep Agent/Remote Configuration traffic on the historical local EVP v2 route.
  • In Agentless mode, probe Agent /info, prefer v4 over v2, and require both DD-EVP-ORIGIN and DD-EVP-ORIGIN-VERSION in evp_proxy_allowed_headers before choosing a local route.
  • Preserve configured Agent URL prefixes for discovery and local event routes while keeping direct intake paths exact.
  • Fall back to canonical https://event-platform-intake.<site>/api/v2/{exposures|flagevaluation} only when a valid direct endpoint and API key are present.
  • Add an identity-bearing FFI/IPC configuration path; reject Agentless delivery through the identity-less compatibility path.
  • Isolate direct credentials from local Agent requests and isolate transport/deduplication state by session and logical SDK producer.
  • Export and verify the new FFI symbols for the configuration and event APIs.
  • Keep the optional Tokio console subscriber gated on both its Cargo feature and Tokio's required tokio_unstable compiler cfg, so ordinary --all-features builds can start the sidecar instead of panicking.

Decisions

  • Direct fallback is gated by the Feature Flags configuration source, not the tracing transport mode.
  • Agent/Remote Configuration stays fixed to EVP v2 and never retains direct-intake credentials.
  • Agentless discovery is v4, then v2, then direct. Direct selection is sticky; an unavailable route is re-probed after a 30-second cooldown.
  • The current batch is replayed directly only after a definitive pre-send failure or local 404/405. Ambiguous transport failures and local 403, 429, and 5xx responses are never replayed; they switch only future batches to direct intake when credentials exist, or enter the bounded unavailable cooldown otherwise.
  • Both logical producer headers are required because intake identity belongs to the SDK, not the sidecar process. Header capability matching is whitespace-trimmed and case-insensitive.
  • Direct endpoints require HTTPS, a canonical event-platform-intake.<site> authority, no port/userinfo/path/query, and a non-empty valid API key. Redirects are not followed.
  • This PR intentionally stops at the shared libdatadog transport/ABI boundary. PHP adoption will separately supply the source, direct endpoint/key, and PHP SDK identity through the identity-bearing setter. Until that follow-up lands, this PR does not activate Agentless EVP delivery in dd-trace-php and does not claim PHP end-to-end signal proof.

Validation

  • cargo +nightly-2026-07-26 fmt --all -- --check — pass.
  • cargo check -p datadog-sidecar -p datadog-sidecar-ffi -p libdd-ffe — pass.
  • cargo +stable clippy -p datadog-sidecar --all-targets --all-features -- -D warnings — pass.
  • cargo +stable clippy -p datadog-sidecar-ffi --all-targets --all-features -- -D warnings — pass.
  • cargo +stable clippy -p libdd-ffe --all-targets --all-features -- -D warnings — pass.
  • cargo test -p datadog-sidecar --no-fail-fast — 68 unit and 5 doc tests pass.
  • cargo test -p datadog-sidecar-ffi --no-fail-fast — 19 tests pass; 1 platform-specific test ignored.
  • cargo test -p libdd-ffe --no-fail-fast — 77 unit and 10 doc tests pass.
  • Focused route-selector suite — 20/20 pass, including header capability rejection/acceptance, prefixed v4/v2 routes, credential isolation, concurrency, cooldown, and replay classification.
  • Public FFI construction test — pass, including rejection of identity-less Agentless configuration and successful submission through both valid configuration symbols. The test remains fire-and-forget because ordinary spawned-sidecar test binaries on Windows and CentOS cannot reliably perform blocking round trips; the repository's dedicated connection test already carries its own platform/runtime requirements.
  • In-process sender to bincode IPC to handler to session test — pass for both the Agent compatibility request and identity-bearing Agentless request. Blocking pings order each assertion, proving the installed session transport and logical producer identity came from the preceding IPC message.
  • Portable IPC-test teardown (536240cb): cargo test -p datadog-sidecar service::sidecar_server::tests::ffe_evp_config_reaches_the_session_through_ipc -- --exact — 1/1 pass on Linux. The ordered IPC and session-state assertions are unchanged; after they pass, the test explicitly cancels and verifies cancellation of its idle server task rather than waiting for peer-close notification, which macOS SOCK_DGRAM socketpairs do not provide. cargo +nightly-2026-07-26 fmt --all -- --check and cargo +stable clippy -p datadog-sidecar --all-targets --all-features -- -D warnings also pass. GitHub macOS confirmation is pending on this head.
  • Cross-platform IPC codec test — pass for both configuration request variants.
  • Exact CI nextest selection (libdd-profiling-ffi, libdd-ffe, datadog-sidecar, datadog-sidecar-ffi, libdd-ffe-ffi, and libdd-ffe-test-suite with --all-features) — 231/231 pass, 1 platform-specific test skipped. This includes the public FFI construction test, in-process handler/session IPC proof, and cross-platform request codec coverage.
  • RUSTFLAGS='--cfg tokio_unstable' cargo check -p datadog-sidecar --features tokio-console — pass, confirming the console subscriber remains enabled when its complete opt-in is present.
  • cargo build --release -p datadog-sidecar-ffi — pass; nm -D --defined-only confirms ddog_sidecar_session_set_ffe_evp_config and ddog_sidecar_session_set_ffe_evp_config_with_identity are exported.
  • cargo ffi-test: 11 suites passed and 2 were skipped in the aggregate run. The FFE suite initially failed only because its pinned test-data submodule was not initialized; after initializing exact SHA 2c0c8d55, the focused FFE ABI suite passed 1/1 and produced 58 artifacts. The unrelated profile_intern 10,000,000-sample stress fixture exceeded the runner's 300-second default timeout; an extended 900-second attempt remained silent and was stopped, so this is classified as a test-duration/infrastructure limitation rather than a product failure.

Additional Notes

Tracks FFLSDK-185. The eventual dd-trace-php adoption and cross-language system-test enablement remain separate rollout steps so this transport change has a narrow ownership and review boundary.

How to test the change?

Run the commands listed under Validation. The route-selector unit tests provide deterministic request-by-request assertions; the FFI test exercises the public ABI, and the in-process IPC test proves the sender installs both transport configurations in real session state.

Route Feature Flags exposures and flag evaluations through a shared per-session selector with Agent v4/v2 discovery and credential-isolated direct fallback. Add logical producer identity to the IPC/FFI configuration contract and preserve fixed EVP v2 behavior for Agent-backed sessions.

Environment: Datadog workspace
Require Agent discovery to advertise both EVP identity headers, preserve configured Agent URL prefixes, and reject Agentless delivery through the identity-less FFI/IPC path.

Environment: Datadog workspace
@linear-code

linear-code Bot commented Sep 12, 2026

Copy link
Copy Markdown

FFLSDK-185

@leoromanovsky leoromanovsky changed the title [FFLSDK-185] Add shared Agentless EVP fallback transport feat(ffe): add shared Agentless EVP fallback transport [FFLSDK-185] Sep 12, 2026
@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

📚 Documentation Check Results

⚠️ 6608 documentation warning(s) found

📦 datadog-sidecar-ffi - 3257 warning(s)

📦 datadog-sidecar - 2964 warning(s)

📦 libdd-ffe - 387 warning(s)


Updated: 2026-09-12 06:13:42 UTC | Commit: 8640628 | missing-docs job results

Keep 403, 429, and 5xx batches non-replayed while moving later Agentless batches to direct intake, or through the bounded unavailable cooldown when direct credentials are absent.

Environment: Datadog workspace
@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

🔒 Cargo Deny Results

⚠️ 7 issue(s) found, showing only errors (advisories, bans, sources)

📦 datadog-sidecar-ffi - 3 error(s)

Show output
error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:36:1
   │
36 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     ├── datadog-sidecar v0.0.1
     │   └── datadog-sidecar-ffi v0.0.1
     ├── (dev) libdd-ffe v2.0.0
     │   └── datadog-sidecar v0.0.1 (*)
     └── libdd-ipc v2.0.0
         ├── datadog-sidecar v0.0.1 (*)
         └── datadog-sidecar-ffi v0.0.1 (*)

error[unsound]: Potential use-after-free due to lack of panic safety in `LruCache::pop()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:233:1
    │
233 │ lru 0.16.4 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0253
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0253
    ├ `LruCache::pop()` in `lru` was not panic-safe. If the `Drop` implementation of a stored key panics during `pop()`, `self.detach()` is never called, leaving dangling pointers in the internal doubly-linked list.
      
      A subsequent cache operation that triggers eviction can then dereference these dangling pointers:
      - The node is freed from the map, but remains linked in the LRU list due to the skipped `detach()` call
      - When a new insertion causes eviction, the LRU traversal encounters the dangling pointer
      - This results in a write to already-freed memory during the eviction process
      
      ## Impact
      
      - **CWE-416 (Use-After-Free):** memory corruption when subsequent cache operations access freed node pointers in the linked list
      - **CWE-415 (Double Free):** potential heap corruption when the same memory is freed multiple times
      
      Both types of undefined behavior can be invoked in safe Rust, but only if unwinding panics are enabled and `std::panic::catch_unwind` is used with key types that have potentially-panicking `Drop` implementations.
      
      ## Fix
      
      Fixed in `lru` 0.18.2 by detaching the node from the linked list before freeing it and dropping the key ([lru-rs#238](https://github.com/jeromefroe/lru-rs/pull/238)).
    ├ Announcement: https://github.com/jeromefroe/lru-rs/pull/238
    ├ Solution: Upgrade to >=0.18.2 (try `cargo update -p lru`)
    ├ lru v0.16.4
      └── libdd-ffe v2.0.0
          └── datadog-sidecar v0.0.1
              └── datadog-sidecar-ffi v0.0.1

error[unmaintained]: paste - no longer maintained
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:275:1
    │
275 │ paste 1.0.15 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
    │
    ├ ID: RUSTSEC-2024-0436
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2024-0436
    ├ The creator of the crate `paste` has stated in the [`README.md`](https://github.com/dtolnay/paste/blob/master/README.md) 
      that this project is not longer maintained as well as archived the repository
      
      ## Possible Alternative(s)
      
      - [`pastey`]: a fork of paste and is aimed to be a drop-in replacement with additional features for paste crate
      - [`with_builtin_macros`]: crate providing a [superset of `paste`'s functionality including general `macro_rules!` eager expansions](https://docs.rs/with_builtin_macros/0.1.0/with_builtin_macros/macro.with_eager_expansions.html)  and `concat!`/`concat_idents!` macros
      
      [`pastey`]: https://crates.io/crates/pastey
      [`with_builtin_macros`]: https://crates.io/crates/with_builtin_macros
    ├ Announcement: https://github.com/dtolnay/paste
    ├ Solution: No safe upgrade is available!
    ├ paste v1.0.15
      ├── datadog-sidecar-ffi v0.0.1
      ├── libdd-libunwind-sys v1.0.3
      │   └── libdd-crashtracker v3.0.0
      │       ├── datadog-sidecar v0.0.1
      │       │   └── datadog-sidecar-ffi v0.0.1 (*)
      │       ├── datadog-sidecar-ffi v0.0.1 (*)
      │       └── libdd-crashtracker-ffi v44.0.0
      │           ├── datadog-sidecar v0.0.1 (*)
      │           └── datadog-sidecar-ffi v0.0.1 (*)
      ├── libdd-telemetry-ffi v44.0.0
      │   └── datadog-sidecar-ffi v0.0.1 (*)
      └── rmp v0.8.14
          ├── libdd-trace-utils v12.0.0
          │   ├── (dev) datadog-sidecar v0.0.1 (*)
          │   ├── (dev) datadog-sidecar-ffi v0.0.1 (*)
          │   ├── libdd-data-pipeline v10.0.0
          │   │   ├── datadog-sidecar v0.0.1 (*)
          │   │   └── libdd-live-debugger v1.0.0
          │   │       ├── datadog-sidecar v0.0.1 (*)
          │   │       └── datadog-sidecar-ffi v0.0.1 (*)
          │   ├── libdd-data-pipeline-core v1.0.0
          │   │   └── libdd-data-pipeline v10.0.0 (*)
          │   ├── libdd-trace-obfuscation v8.0.0
          │   │   ├── libdd-data-pipeline v10.0.0 (*)
          │   │   ├── libdd-data-pipeline-core v1.0.0 (*)
          │   │   └── libdd-trace-stats v9.0.0
          │   │       ├── datadog-sidecar v0.0.1 (*)
          │   │       ├── libdd-data-pipeline v10.0.0 (*)
          │   │       └── libdd-ipc v2.0.0
          │   │           ├── datadog-sidecar v0.0.1 (*)
          │   │           └── datadog-sidecar-ffi v0.0.1 (*)
          │   ├── libdd-trace-stats v9.0.0 (*)
          │   └── (dev) libdd-trace-utils v12.0.0 (*)
          ├── rmp-serde v1.3.0
          │   ├── datadog-sidecar v0.0.1 (*)
          │   ├── datadog-sidecar-ffi v0.0.1 (*)
          │   ├── libdd-data-pipeline v10.0.0 (*)
          │   ├── (dev) libdd-tinybytes v1.1.3
          │   │   ├── datadog-sidecar v0.0.1 (*)
          │   │   ├── datadog-sidecar-ffi v0.0.1 (*)
          │   │   ├── libdd-data-pipeline v10.0.0 (*)
          │   │   ├── (dev) libdd-data-pipeline-core v1.0.0 (*)
          │   │   ├── libdd-ipc v2.0.0 (*)
          │   │   ├── (dev) libdd-tinybytes v1.1.3 (*)
          │   │   ├── (dev) libdd-trace-obfuscation v8.0.0 (*)
          │   │   └── libdd-trace-utils v12.0.0 (*)
          │   ├── libdd-trace-stats v9.0.0 (*)
          │   └── libdd-trace-utils v12.0.0 (*)
          └── rmpv v1.3.0
              └── libdd-trace-utils v12.0.0 (*)

advisories FAILED, bans ok, sources ok

📦 datadog-sidecar - 2 error(s)

Show output
error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:36:1
   │
36 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     ├── datadog-sidecar v0.0.1
     ├── (dev) libdd-ffe v2.0.0
     │   └── datadog-sidecar v0.0.1 (*)
     └── libdd-ipc v2.0.0
         └── datadog-sidecar v0.0.1 (*)

error[unsound]: Potential use-after-free due to lack of panic safety in `LruCache::pop()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:231:1
    │
231 │ lru 0.16.4 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0253
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0253
    ├ `LruCache::pop()` in `lru` was not panic-safe. If the `Drop` implementation of a stored key panics during `pop()`, `self.detach()` is never called, leaving dangling pointers in the internal doubly-linked list.
      
      A subsequent cache operation that triggers eviction can then dereference these dangling pointers:
      - The node is freed from the map, but remains linked in the LRU list due to the skipped `detach()` call
      - When a new insertion causes eviction, the LRU traversal encounters the dangling pointer
      - This results in a write to already-freed memory during the eviction process
      
      ## Impact
      
      - **CWE-416 (Use-After-Free):** memory corruption when subsequent cache operations access freed node pointers in the linked list
      - **CWE-415 (Double Free):** potential heap corruption when the same memory is freed multiple times
      
      Both types of undefined behavior can be invoked in safe Rust, but only if unwinding panics are enabled and `std::panic::catch_unwind` is used with key types that have potentially-panicking `Drop` implementations.
      
      ## Fix
      
      Fixed in `lru` 0.18.2 by detaching the node from the linked list before freeing it and dropping the key ([lru-rs#238](https://github.com/jeromefroe/lru-rs/pull/238)).
    ├ Announcement: https://github.com/jeromefroe/lru-rs/pull/238
    ├ Solution: Upgrade to >=0.18.2 (try `cargo update -p lru`)
    ├ lru v0.16.4
      └── libdd-ffe v2.0.0
          └── datadog-sidecar v0.0.1

advisories FAILED, bans ok, sources ok

📦 libdd-ffe - 2 error(s)

Show output
error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:21:1
   │
21 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     └── (dev) libdd-ffe v2.0.0

error[unsound]: Potential use-after-free due to lack of panic safety in `LruCache::pop()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:131:1
    │
131 │ lru 0.16.4 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0253
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0253
    ├ `LruCache::pop()` in `lru` was not panic-safe. If the `Drop` implementation of a stored key panics during `pop()`, `self.detach()` is never called, leaving dangling pointers in the internal doubly-linked list.
      
      A subsequent cache operation that triggers eviction can then dereference these dangling pointers:
      - The node is freed from the map, but remains linked in the LRU list due to the skipped `detach()` call
      - When a new insertion causes eviction, the LRU traversal encounters the dangling pointer
      - This results in a write to already-freed memory during the eviction process
      
      ## Impact
      
      - **CWE-416 (Use-After-Free):** memory corruption when subsequent cache operations access freed node pointers in the linked list
      - **CWE-415 (Double Free):** potential heap corruption when the same memory is freed multiple times
      
      Both types of undefined behavior can be invoked in safe Rust, but only if unwinding panics are enabled and `std::panic::catch_unwind` is used with key types that have potentially-panicking `Drop` implementations.
      
      ## Fix
      
      Fixed in `lru` 0.18.2 by detaching the node from the linked list before freeing it and dropping the key ([lru-rs#238](https://github.com/jeromefroe/lru-rs/pull/238)).
    ├ Announcement: https://github.com/jeromefroe/lru-rs/pull/238
    ├ Solution: Upgrade to >=0.18.2 (try `cargo update -p lru`)
    ├ lru v0.16.4
      └── libdd-ffe v2.0.0

advisories FAILED, bans ok, sources ok

Updated: 2026-09-12 06:12:57 UTC | Commit: 8640628 | dependency-check job results

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Sep 12, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

Required checks pass | allchecks

View more details · View in GitHub Actions

❄️ 1 New flaky test detected

service::sidecar_server::tests::ffe_evp_config_reaches_the_session_through_ipc from datadog-sidecar
thread &#39;service::sidecar_server::tests::ffe_evp_config_reaches_the_session_through_ipc&#39; panicked at datadog-sidecar/src/service/sidecar_server.rs:1703:14

thread &#39;service::sidecar_server::tests::ffe_evp_config_reaches_the_session_through_ipc&#39; panicked at datadog-sidecar/src/service/sidecar_server.rs:1703:14:
server should stop when the client disconnects: Elapsed(())
note: run with \`RUST_BACKTRACE=1\` environment variable to display a backtrace

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 95.69%
Overall Coverage: 78.52% (+0.75%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 536240c | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Sep 12, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-09-12 06:21:05

Comparing candidate commit 536240c in PR branch leo.romanovsky/fflsdk-185-libdatadog-evp-fallback-v2 with baseline commit 1413485 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:flagevaluation_evp/coalescer/typical/100flags_50users_10fields

  • unstable execution_time [-3.549µs; +15.069µs] or [-1.941%; +8.243%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 536240c 1789193443 leo.romanovsky/fflsdk-185-libdatadog-evp-fallback-v2
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time 21.472ms 21.603ms ± 0.070ms 21.594ms ± 0.048ms 21.646ms 21.732ms 21.781ms 21.816ms 1.03% 0.509 -0.220 0.33% 0.005ms 1 200
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput 114593.635op/s 115728.174op/s ± 376.903op/s 115771.373op/s ± 256.856op/s 116015.120op/s 116261.742op/s 116410.040op/s 116432.720op/s 0.57% -0.494 -0.244 0.32% 26.651op/s 1 200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time 96.131ms 97.482ms ± 0.582ms 97.476ms ± 0.201ms 97.644ms 98.151ms 98.269ms 103.317ms 5.99% 4.873 49.018 0.60% 0.041ms 1 200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput 9678.978op/s 10258.687op/s ± 59.583op/s 10258.899op/s ± 21.223op/s 10282.943op/s 10338.004op/s 10388.042op/s 10402.480op/s 1.40% -4.405 43.309 0.58% 4.213op/s 1 200
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time 593.931µs 596.095µs ± 0.649µs 596.092µs ± 0.410µs 596.497µs 597.168µs 597.595µs 597.752µs 0.28% -0.085 0.253 0.11% 0.046µs 1 200
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput 167293.473op/s 167758.666op/s ± 182.533op/s 167759.212op/s ± 115.383op/s 167875.570op/s 168047.337op/s 168139.761op/s 168369.830op/s 0.36% 0.092 0.258 0.11% 12.907op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time [21.593ms; 21.612ms] or [-0.045%; +0.045%] None None None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput [115675.939op/s; 115780.409op/s] or [-0.045%; +0.045%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time [97.401ms; 97.562ms] or [-0.083%; +0.083%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput [10250.430op/s; 10266.945op/s] or [-0.080%; +0.080%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time [596.005µs; 596.185µs] or [-0.015%; +0.015%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput [167733.369op/s; 167783.963op/s] or [-0.015%; +0.015%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 536240c 1789193443 leo.romanovsky/fflsdk-185-libdatadog-evp-fallback-v2
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sdk_test_data/rules-based execution_time 255.691µs 261.821µs ± 12.385µs 258.622µs ± 1.184µs 260.813µs 274.749µs 308.411µs 387.086µs 49.67% 6.868 58.075 4.72% 0.876µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sdk_test_data/rules-based execution_time [260.105µs; 263.537µs] or [-0.656%; +0.656%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 536240c 1789193443 leo.romanovsky/fflsdk-185-libdatadog-evp-fallback-v2
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
single_flag_killswitch/rules-based execution_time 156.275ns 159.611ns ± 2.542ns 159.259ns ± 1.607ns 160.991ns 164.043ns 167.686ns 169.487ns 6.42% 1.077 1.370 1.59% 0.180ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
single_flag_killswitch/rules-based execution_time [159.259ns; 159.963ns] or [-0.221%; +0.221%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 536240c 1789193443 leo.romanovsky/fflsdk-185-libdatadog-evp-fallback-v2
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time 5.387ms 5.413ms ± 0.013ms 5.412ms ± 0.005ms 5.418ms 5.423ms 5.428ms 5.559ms 2.71% 6.944 74.056 0.24% 0.001ms 1 200
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput 449709.099op/s 461840.994op/s ± 1106.778op/s 461918.439op/s ± 423.960op/s 462315.700op/s 462925.990op/s 463503.662op/s 464105.500op/s 0.47% -6.743 71.163 0.24% 78.261op/s 1 200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time 7.945ms 7.959ms ± 0.006ms 7.959ms ± 0.003ms 7.962ms 7.967ms 7.986ms 8.002ms 0.54% 2.123 11.301 0.08% 0.000ms 1 200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput 124976.171op/s 125643.166op/s ± 101.724op/s 125648.164op/s ± 49.278op/s 125696.674op/s 125791.257op/s 125825.778op/s 125869.074op/s 0.18% -2.102 11.156 0.08% 7.193op/s 1 200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time 171.677µs 188.581µs ± 67.166µs 183.949µs ± 0.241µs 184.257µs 184.897µs 185.242µs 1133.440µs 516.17% 14.024 194.783 35.53% 4.749µs 1 200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput 88226.951op/s 541734.981op/s ± 32611.982op/s 543629.599op/s ± 711.114op/s 544207.591op/s 545056.656op/s 578797.121op/s 582490.482op/s 7.15% -13.519 185.899 6.00% 2306.015op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time [5.411ms; 5.415ms] or [-0.034%; +0.034%] None None None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput [461687.605op/s; 461994.383op/s] or [-0.033%; +0.033%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time [7.958ms; 7.960ms] or [-0.011%; +0.011%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput [125629.068op/s; 125657.264op/s] or [-0.011%; +0.011%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time [179.273µs; 197.890µs] or [-4.936%; +4.936%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput [537215.274op/s; 546254.688op/s] or [-0.834%; +0.834%] None None None

Baseline

Baseline benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 1413485 1789164958 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time 21.371ms 21.487ms ± 0.056ms 21.485ms ± 0.044ms 21.527ms 21.581ms 21.602ms 21.616ms 0.61% 0.045 -0.728 0.26% 0.004ms 1 200
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput 115654.613op/s 116351.277op/s ± 303.861op/s 116359.535op/s ± 240.377op/s 116599.493op/s 116844.618op/s 116928.759op/s 116982.978op/s 0.54% -0.035 -0.731 0.26% 21.486op/s 1 200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time 97.090ms 98.065ms ± 0.504ms 98.074ms ± 0.266ms 98.290ms 98.834ms 99.419ms 101.424ms 3.42% 1.630 9.029 0.51% 0.036ms 1 200
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput 9859.609op/s 10197.599op/s ± 52.027op/s 10196.348op/s ± 27.780op/s 10224.922op/s 10281.834op/s 10287.015op/s 10299.759op/s 1.01% -1.506 8.125 0.51% 3.679op/s 1 200
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time 599.186µs 600.486µs ± 0.616µs 600.516µs ± 0.508µs 601.013µs 601.469µs 601.601µs 602.227µs 0.28% 0.093 -0.927 0.10% 0.044µs 1 200
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput 166050.330op/s 166531.989op/s ± 170.746op/s 166523.406op/s ± 140.920op/s 166675.851op/s 166775.831op/s 166849.518op/s 166892.983op/s 0.22% -0.090 -0.929 0.10% 12.074op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time [21.479ms; 21.495ms] or [-0.036%; +0.036%] None None None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput [116309.165op/s; 116393.389op/s] or [-0.036%; +0.036%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time [97.995ms; 98.135ms] or [-0.071%; +0.071%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput [10190.389op/s; 10204.810op/s] or [-0.071%; +0.071%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time [600.401µs; 600.571µs] or [-0.014%; +0.014%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput [166508.325op/s; 166555.653op/s] or [-0.014%; +0.014%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 1413485 1789164958 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sdk_test_data/rules-based execution_time 254.515µs 261.022µs ± 12.488µs 257.865µs ± 1.170µs 260.077µs 275.925µs 308.119µs 386.347µs 49.83% 6.791 56.752 4.77% 0.883µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sdk_test_data/rules-based execution_time [259.292µs; 262.753µs] or [-0.663%; +0.663%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 1413485 1789164958 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
single_flag_killswitch/rules-based execution_time 155.975ns 159.675ns ± 3.084ns 159.022ns ± 1.503ns 160.470ns 166.891ns 169.727ns 169.879ns 6.83% 1.565 2.249 1.93% 0.218ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
single_flag_killswitch/rules-based execution_time [159.247ns; 160.102ns] or [-0.268%; +0.268%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 1413485 1789164958 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time 5.322ms 5.351ms ± 0.025ms 5.345ms ± 0.011ms 5.363ms 5.374ms 5.406ms 5.633ms 5.38% 6.996 74.180 0.47% 0.002ms 1 200
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput 443846.219op/s 467232.777op/s ± 2147.465op/s 467717.474op/s ± 991.968op/s 468427.517op/s 469085.455op/s 469494.648op/s 469775.667op/s 0.44% -6.619 68.556 0.46% 151.849op/s 1 200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time 7.936ms 7.949ms ± 0.007ms 7.948ms ± 0.003ms 7.952ms 7.960ms 7.980ms 7.987ms 0.48% 2.223 8.279 0.09% 0.001ms 1 200
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput 125207.923op/s 125800.903op/s ± 114.170op/s 125811.405op/s ± 52.858op/s 125865.560op/s 125944.590op/s 125986.696op/s 126000.793op/s 0.15% -2.209 8.199 0.09% 8.073op/s 1 200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time 180.491µs 182.821µs ± 0.703µs 182.691µs ± 0.303µs 183.106µs 183.874µs 184.277µs 188.629µs 3.25% 2.732 22.801 0.38% 0.050µs 1 200
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput 530141.177op/s 546989.884op/s ± 2081.850op/s 547371.944op/s ± 908.258op/s 548119.710op/s 548746.684op/s 551520.819op/s 554044.553op/s 1.22% -2.538 21.069 0.38% 147.209op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time [5.347ms; 5.354ms] or [-0.066%; +0.066%] None None None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput [466935.159op/s; 467530.395op/s] or [-0.064%; +0.064%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time [7.948ms; 7.950ms] or [-0.013%; +0.013%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput [125785.080op/s; 125816.726op/s] or [-0.013%; +0.013%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time [182.724µs; 182.919µs] or [-0.053%; +0.053%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput [546701.360op/s; 547278.408op/s] or [-0.053%; +0.053%] None None None

Keep the spawned-sidecar ordering assertion on supported test binaries and verify both new request variants through platform-neutral wire round trips.\n\nEnvironment: Datadog workspace
Replace the spawned-sidecar ping that cannot reliably perform blocking round trips in the ordinary Windows and CentOS test harnesses with an in-process socket test that proves both EVP configuration APIs reach session state.

Environment: Datadog workspace
@dd-octo-sts

dd-octo-sts Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 9.02 MB 9.02 MB 0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 96.01 MB 96.01 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 107.42 MB 107.42 MB 0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 12.19 MB 12.19 MB 0% (0 B) 👌
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 29.04 MB 29.04 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 96.08 KB 96.08 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 191.77 MB 191.77 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 818.09 MB 818.09 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 9.69 MB 9.69 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 96.08 KB 96.08 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 27.51 MB 27.51 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 55.58 MB 55.58 MB 0% (0 B) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 25.41 MB 25.41 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 97.58 KB 97.58 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 197.08 MB 197.11 MB +.01% (+24.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 800.38 MB 800.38 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 7.51 MB 7.51 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 97.58 KB 97.58 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 29.63 MB 29.63 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 52.55 MB 52.55 MB 0% (0 B) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 85.97 MB 85.97 MB 0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 10.03 MB 10.03 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 101.85 MB 101.85 MB 0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 12.25 MB 12.25 MB 0% (0 B) 👌

Cancel the idle test server explicitly after both ordered IPC assertions so macOS socketpair disconnect semantics cannot time out teardown.

Environment: Datadog workspace
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant