Skip to content

feat(wall): support signal suppression for unfiltered threads - #663

Open
kaahos wants to merge 34 commits into
mainfrom
paul.fournillon/wallclock-all-threads
Open

kaahos wants to merge 34 commits into
mainfrom
paul.fournillon/wallclock-all-threads

Conversation

@kaahos

@kaahos kaahos commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?:

Extends wall-clock signal suppression to threads sampled through the existing explicit filter= mode.

When wall profiling runs with both filter= and wallprecheck=true, the profiler now:

  • Tracks thread identity and lifecycle independently from context-filter membership.
  • Registers Java threads that already exist when profiling starts.
  • Registers new threads through the existing thread lifecycle callbacks.
  • Allows block and park hooks to lazily associate pre-existing threads with their registry slots.
  • Makes the thread metadata available to both the ASGCT and JVMTI wall-clock paths.
  • Suppresses repeated samples only for profiler-owned blocking runs that remain outside the context window.
  • Validates thread identity, slot lifecycle, and context transitions before applying suppression.

The existing context-filtered behavior remains unchanged. This PR does not change the meaning of filter=.

Motivation:

With filter=, wall-clock profiling already samples every thread. However, threads outside an active context did not have the lifecycle metadata required to recognize and suppress repeated samples from the same profiler-owned blocking run.

This change closes that metadata gap. It also provides the thread lifecycle machinery needed by the follow-up TaskBlock work for non-context threads.

Implementation notes:

  • Thread identity is indexed by native TID while context-window membership remains separate.
  • Lifecycle generations prevent reused slots or TIDs from inheriting stale state.
  • Context-window epochs prevent suppression if a thread enters or leaves a context during a blocking run.
  • Registry tracking is activated only for wall recordings using explicit filter= together with wallprecheck=true.
  • Tracking is disabled again when wall profiling is absent, precheck is disabled, wall startup fails, or a later recording uses another configuration.
  • Both ASGCT and JVMTI wall-clock collection paths use the same registry validation.

How to test the change?:

  • ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug_threadFilter_ut :ddprof-lib:gtestDebug_wallprecheck_args_ut
  • ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests="*UnfilteredWallPrecheck*"
  • ./.claude/commands/build-and-summarize testDebug

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles credentials
    of any kind, I've requested a security review.
  • This PR doesn't touch any of that.
  • JIRA: [JIRA-XXXX]

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #36609406833 | Commit: 4d55a0d | Duration: 22m 2s (longest job)

❌ 9 of 76 test jobs failed

Status Overview

JDK glibc-aarch64/asan glibc-aarch64/debug glibc-aarch64/tsan glibc-amd64/asan glibc-amd64/debug glibc-amd64/tsan musl-aarch64/debug musl-amd64/debug
8 - - - ❌ ✅ ✅ - -
8-ibm - - - ✅ ✅ ✅ - -
8-j9 ✅ ✅ ✅ ✅ ✅ ✅ - -
8-librca - - - - - - ✅ ✅
8-orcl - - - ❌ ✅ ✅ - -
11 - - - ❌ ✅ ✅ - -
11-j9 ✅ ✅ ✅ ✅ ✅ ✅ - -
11-librca - - - - - - ✅ ✅
17 ❌ ✅ ✅ ❌ ✅ ✅ - -
17-graal ✅ ✅ ✅ ✅ ✅ ✅ - -
17-j9 ✅ ✅ ✅ ✅ ✅ ✅ - -
17-librca - - - - - - ✅ ✅
21 ❌ ✅ ✅ ❌ ✅ ✅ - -
21-graal ✅ ✅ ✅ ✅ ✅ ✅ - -
21-librca - - - - - - ✅ ✅
25 ❌ ✅ ✅ ❌ ✅ ✅ - -
25-graal ✅ ✅ ✅ ✅ ✅ ✅ - -
25-librca - - - - - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

❌ Failing tests

Test Cells Message
MegamorphicCallTest._5_ glibc-21-asan-amd64 org.opentest4j.AssertionFailedError: Test execution #5 (of up to 5 with at least 1 successes) failed ~> test fails - see

Retried 9 of 70 cells.

Failed Jobs

Summary: Total: 76 | Passed: 67 | Failed: 9


Updated: 2026-09-29 18:28:39 UTC

@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 3 times, most recently from 3902a14 to f302dd5 Compare July 19, 2026 12:00
@kaahos kaahos changed the title feat(wall): add configurable wall thread scope feat(wall): support signal suppression for unfiltered threads Jul 19, 2026
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 2 times, most recently from 0ad425c to fb1c44d Compare July 19, 2026 18:43
@dd-octo-sts

dd-octo-sts Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit fb1c44d)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125547869 Commit: fb1c44de77197780eae60267ff98bd38a907a558

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +5.9% (2679→2836 ms)
  • 🔴 future-genetic (JDK 21): runtime +3.8% (2083→2162 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10210 ms (21 iters) ✅ 10256 ms (21 iters) ≈ +0.5% (±10.9%) — / —
akka-uct 25 ✅ 8953 ms (24 iters) ✅ 8890 ms (24 iters) ≈ -0.7% (±10.2%) — / —
finagle-chirper 21 ✅ 6000 ms (33 iters) ✅ 5960 ms (33 iters) ≈ -0.7% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5431 ms (36 iters) ✅ 5443 ms (36 iters) ≈ +0.2% (±24.6%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2679 ms (70 iters) ✅ 2836 ms (66 iters) 🔴 +5.9% — / —
fj-kmeans 25 ✅ 2847 ms (66 iters) ✅ 2828 ms (66 iters) ≈ -0.7% (±2.6%) — / —
future-genetic 21 ✅ 2083 ms (89 iters) ✅ 2162 ms (86 iters) 🔴 +3.8% — / —
future-genetic 25 ✅ 2076 ms (90 iters) ✅ 2122 ms (88 iters) ≈ +2.2% (±2.5%) — / —
naive-bayes 21 ✅ 1252 ms (136 iters) ✅ 1248 ms (137 iters) ≈ -0.3% (±32.7%) — / —
naive-bayes 25 ✅ 1008 ms (170 iters) ✅ 1023 ms (167 iters) ≈ +1.5% (±31.7%) — / —
reactors 21 ✅ 16312 ms (15 iters) ✅ 16357 ms (15 iters) ≈ +0.3% (±8.8%) — / —
reactors 25 ✅ 18416 ms (15 iters) ✅ 18780 ms (15 iters) ≈ +2% (±3.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1891 / 2004 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2305 / 2283 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 2 8183 / 8674 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 8087 / 8641 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1270 / 1278 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 3 1282 / 1275 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 3 2966 / 3007 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2992 / 2972 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 6 3491 / 3501 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 5 3488 / 3493 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1711 / 1786 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1819 / 1920 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from fb1c44d to 1118f62 Compare July 19, 2026 21:16
@dd-octo-sts

dd-octo-sts Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 1118f62)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125554643 Commit: 1118f62831a4384bdf0283ef01a626c5bdc48af5

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -3.9% (2808→2698 ms)
  • 🟢 future-genetic (JDK 25): runtime -4.2% (2088→2001 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10294 ms (21 iters) ✅ 10251 ms (21 iters) ≈ -0.4% (±11.4%) — / —
akka-uct 25 ✅ 8933 ms (24 iters) ✅ 8835 ms (24 iters) ≈ -1.1% (±10%) — / —
finagle-chirper 21 ✅ 6016 ms (33 iters) ✅ 6034 ms (33 iters) ≈ +0.3% (±25.3%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5462 ms (36 iters) ✅ 5452 ms (36 iters) ≈ -0.2% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2808 ms (66 iters) ✅ 2698 ms (70 iters) 🟢 -3.9% — / —
fj-kmeans 25 ✅ 2812 ms (66 iters) ✅ 2813 ms (66 iters) ≈ +0% (±2.6%) — / —
future-genetic 21 ✅ 2067 ms (90 iters) ✅ 2083 ms (90 iters) ≈ +0.8% (±2.7%) — / —
future-genetic 25 ✅ 2088 ms (89 iters) ✅ 2001 ms (93 iters) 🟢 -4.2% — / —
naive-bayes 21 ✅ 1239 ms (138 iters) ✅ 1231 ms (138 iters) ≈ -0.6% (±32.7%) — / —
naive-bayes 25 ✅ 1019 ms (168 iters) ✅ 1011 ms (169 iters) ≈ -0.8% (±32.1%) — / —
reactors 21 ✅ 16149 ms (15 iters) ✅ 16215 ms (15 iters) ≈ +0.4% (±8.4%) — / —
reactors 25 ✅ 18598 ms (15 iters) ✅ 18345 ms (15 iters) ≈ -1.4% (±5.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 2 1978 / 1973 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2287 / 2449 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8978 / 8724 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 8633 / 8297 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 1 1257 / 1281 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 4 1280 / 1268 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 3 2909 / 2923 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 5 2847 / 2887 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 7 / 4 3492 / 3493 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 2 3463 / 3473 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1761 / 1721 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1916 / 1880 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 40dbe8e)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125596529 Commit: 40dbe8e833888192b6026558590bce1cf0ad1c9f

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -2.7% (2823→2748 ms)
  • 🔴 future-genetic (JDK 21): runtime +4.1% (2041→2125 ms)
  • 🔴 future-genetic (JDK 25): runtime +5.8% (1994→2110 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10340 ms (21 iters) ✅ 10312 ms (21 iters) ≈ -0.3% (±11.6%) — / —
akka-uct 25 ✅ 8871 ms (24 iters) ✅ 8849 ms (24 iters) ≈ -0.2% (±10.1%) — / —
finagle-chirper 21 ✅ 6017 ms (33 iters) ✅ 6015 ms (33 iters) ≈ -0% (±25%) ⚠️ W:3 / ⚠️ W:4
finagle-chirper 25 ✅ 5470 ms (36 iters) ✅ 5509 ms (36 iters) ≈ +0.7% (±24.2%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2823 ms (66 iters) ✅ 2748 ms (68 iters) 🟢 -2.7% — / —
fj-kmeans 25 ✅ 2856 ms (66 iters) ✅ 2847 ms (66 iters) ≈ -0.3% (±2.6%) — / —
future-genetic 21 ✅ 2041 ms (90 iters) ✅ 2125 ms (88 iters) 🔴 +4.1% — / —
future-genetic 25 ✅ 1994 ms (93 iters) ✅ 2110 ms (88 iters) 🔴 +5.8% — / —
naive-bayes 21 ✅ 1247 ms (137 iters) ✅ 1259 ms (135 iters) ≈ +1% (±33.5%) — / —
naive-bayes 25 ✅ 1027 ms (167 iters) ✅ 1018 ms (168 iters) ≈ -0.9% (±31.5%) — / —
reactors 21 ✅ 16583 ms (15 iters) ✅ 16798 ms (15 iters) ≈ +1.3% (±7.5%) — / —
reactors 25 ✅ 18582 ms (15 iters) ✅ 18610 ms (15 iters) ≈ +0.2% (±5.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / 6 1978 / 1998 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2344 / 2271 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8863 / 8312 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8368 / 8738 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 7 / 1 1284 / 1242 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 4 1302 / 1289 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 1 2979 / 2980 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 2847 / 2862 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 5 3520 / 3534 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 3 / 3 3523 / 3509 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 2 1682 / 1809 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1806 / 1871 ✅ / ✅ ✅ / ✅

Copilot AI review requested due to automatic review settings July 21, 2026 07:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Extends wall-clock “owned-block” signal suppression to unfiltered (explicit filter=) wall recordings when wallprecheck=true, by introducing a thread registry keyed by native TID and integrating it into both ASGCT and JVMTI wall-clock sampling paths.

Changes:

  • Added unfiltered wall thread registry support (native TID indexing, lifecycle generations, recording epochs) and wired it into profiler lifecycle + wall-clock precheck logic.
  • Added a bounded candidate-selection/backfill mechanism for precheck to avoid reverting to O(N) registry work per wall tick under heavy suppression.
  • Added Java integration tests, C++ unit tests, and a JMH benchmark to validate behavior and measure overhead.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java New integration coverage for suppression behavior in explicit-empty-filter wall recordings.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java Verifies registry activation does not leak across recording restarts and handles stopped gaps safely.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java Ensures the unfiltered path works when wall sampling delegates stacks via JVMTI.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java Confirms J9 wall engine capability gating prevents activation of unfiltered tracking.
ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java Adds beforeProfilerStart() hook to allow test setup before profiler thread callbacks are enabled.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java Exposes owned-block hooks to benchmarks without widening core API surface.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java Adds throughput benchmark comparing wallprecheck=false/true as owned-block population grows.
ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp Adds capability tests and verifies explicit-empty vs omitted filter parsing behavior.
ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp Unit-tests the new bounded candidate selection logic.
ddprof-lib/src/test/cpp/threadFilter_ut.cpp Expands tests for registry/epoch/lifecycle behavior and adjusts TID usage in recovery test.
ddprof-lib/src/main/cpp/wallClockCandidateSelector.h Adds reusable “randomized prefix without replacement” candidate selection with stats.
ddprof-lib/src/main/cpp/wallClock.h Integrates candidate selector and adds bounded-visit backfill path to the wall timer loop.
ddprof-lib/src/main/cpp/wallClock.cpp Updates precheck suppression to validate registry identity + context transitions; supports lazy registry lookups.
ddprof-lib/src/main/cpp/threadFilter.h Introduces registry state (TID index, epochs, lifecycle generation) and APIs for unfiltered tracking.
ddprof-lib/src/main/cpp/threadFilter.cpp Implements lock-free lookups + mutex-serialized writers for TID indexing, epoch refresh, and retirement.
ddprof-lib/src/main/cpp/profiler.h Declares bootstrap of existing Java threads for registry population.
ddprof-lib/src/main/cpp/profiler.cpp Boots existing Java threads into registry when unfiltered tracking is active; deactivates registry on stop/failure.
ddprof-lib/src/main/cpp/jvmThread.h Adds capability check for cross-thread native TID lookup support.
ddprof-lib/src/main/cpp/jvmThread.cpp Implements native TID lookup support predicate.
ddprof-lib/src/main/cpp/javaApi.cpp Ensures TLS slot binding is validated/refreshed before using owned-block and filter APIs.
ddprof-lib/src/main/cpp/engine.h Adds wall-engine capability hook for unfiltered precheck support.
ddprof-lib/src/main/cpp/counters.h Adds counters for registry bootstrap and bounded-lookup/backfill diagnostics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ddprof-lib/src/main/cpp/wallClockCandidateSelector.h
Comment thread ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp
@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit d82291f)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125875202 Commit: d82291ff62852b63091350a05ebf26283fa39029

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10141 ms (21 iters) ✅ 10278 ms (21 iters) ≈ +1.4% (±11.9%) — / —
akka-uct 25 ✅ 8800 ms (24 iters) ✅ 8939 ms (24 iters) ≈ +1.6% (±9.7%) — / —
finagle-chirper 21 ✅ 6003 ms (33 iters) ✅ 6060 ms (33 iters) ≈ +0.9% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5463 ms (36 iters) ✅ 5482 ms (36 iters) ≈ +0.3% (±24.1%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2780 ms (68 iters) ✅ 2771 ms (68 iters) ≈ -0.3% (±2.7%) — / —
fj-kmeans 25 ✅ 2752 ms (68 iters) ✅ 2824 ms (66 iters) ≈ +2.6% (±2.7%) — / —
future-genetic 21 ✅ 2082 ms (90 iters) ✅ 2035 ms (91 iters) ≈ -2.3% (±2.6%) — / —
future-genetic 25 ✅ 2101 ms (88 iters) ✅ 2055 ms (90 iters) ≈ -2.2% (±2.6%) — / —
naive-bayes 21 ✅ 1299 ms (132 iters) ✅ 1232 ms (138 iters) ≈ -5.2% (±31.7%) — / —
naive-bayes 25 ✅ 1011 ms (169 iters) ✅ 1013 ms (169 iters) ≈ +0.2% (±31.4%) — / —
reactors 21 ✅ 16511 ms (15 iters) ✅ 16445 ms (15 iters) ≈ -0.4% (±9.8%) — / —
reactors 25 ✅ 18516 ms (15 iters) ✅ 18146 ms (15 iters) ≈ -2% (±6.2%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 2 1925 / 1914 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 3 2117 / 2331 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8401 / 8842 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8557 / 8073 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 3 1243 / 1272 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1265 / 1283 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 1 3054 / 3036 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2890 / 2867 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 3 3522 / 3464 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 5 3452 / 3463 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1679 / 1552 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1933 / 1805 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit ec76f58)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125883187 Commit: ec76f5829d243f516dee70833f3dfbe7a2b9eaa3

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -4.1% (2839→2724 ms)
  • 🔴 future-genetic (JDK 21): runtime +3% (2066→2127 ms)
  • 🔴 future-genetic (JDK 25): runtime +5.8% (2006→2122 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10350 ms (21 iters) ✅ 10305 ms (21 iters) ≈ -0.4% (±11.3%) — / —
akka-uct 25 ✅ 8864 ms (24 iters) ✅ 8976 ms (24 iters) ≈ +1.3% (±10.4%) — / —
finagle-chirper 25 ✅ 5472 ms (36 iters) ✅ 5418 ms (36 iters) ≈ -1% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2839 ms (66 iters) ✅ 2724 ms (68 iters) 🟢 -4.1% — / —
fj-kmeans 25 ✅ 2851 ms (66 iters) ✅ 2836 ms (66 iters) ≈ -0.5% (±2.6%) — / —
future-genetic 21 ✅ 2066 ms (89 iters) ✅ 2127 ms (87 iters) 🔴 +3% — / —
future-genetic 25 ✅ 2006 ms (92 iters) ✅ 2122 ms (87 iters) 🔴 +5.8% — / —
naive-bayes 21 ✅ 1277 ms (134 iters) ✅ 1289 ms (133 iters) ≈ +0.9% (±32.4%) — / —
naive-bayes 25 ✅ 1021 ms (167 iters) ✅ 1023 ms (168 iters) ≈ +0.2% (±31.6%) — / —
reactors 21 ✅ 15903 ms (15 iters) ✅ 16452 ms (15 iters) ≈ +3.5% (±8.8%) — / —
reactors 25 ✅ 18032 ms (15 iters) ✅ 18628 ms (15 iters) ≈ +3.3% (±4.7%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 3 2089 / 2428 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 7 / 2 8520 / 8487 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 4 / 2 8382 / 8319 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 4 / 4 1278 / 1277 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / ✅ 1290 / 1289 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 2 2953 / 3032 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 2 2846 / 2856 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 3 3534 / 3561 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 3 3480 / 3510 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1614 / 1718 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 3 1962 / 1876 ✅ / ✅ ✅ / ✅

@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@kaahos
kaahos marked this pull request as ready for review August 17, 2026 14:03
@kaahos
kaahos requested a review from a team as a code owner August 17, 2026 14:03

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a68d3f22c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp Outdated

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗿 🤖 🔴

Sphinx Review found 2 critical/high severity finding(s) that must be addressed.

Comment thread ddprof-lib/src/main/cpp/javaApi.cpp

if (_thread_filter.enabled()) {
_thread_filter.unregisterThread(slot_id);
if (slot_id >= 0) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗿 🤖 🔴

[Sphinx Review — MEDIUM] onThreadEnd() now unconditionally calls _thread_filter.unregisterThread()/unregisterThreadByTid() (both mutex-guarded) for every JVMTI ThreadEnd event whenever any profiling engine is active, instead of being gated behind the thread-filter registry's active/enabled state as before.

Suggestion: Gate the else-branch (and ideally both branches) behind _thread_filter.registryActive() (or a cheaper lock-free check) so thread teardown for profiling sessions that never use the registry does not pay for a mutex acquisition.

Comment thread ddprof-lib/src/main/cpp/profiler.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/profiler.cpp
Comment thread ddprof-lib/src/main/cpp/profiler.cpp
Comment thread ddprof-lib/src/main/cpp/threadFilter.h
Comment thread ddprof-lib/src/main/cpp/wallClock.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/wallClock.cpp Outdated
Comment thread ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp
Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckTest.java Outdated
@jbachorik jbachorik added the sphinx:critical Sphinx: critical — human review required label Aug 18, 2026
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from af36f26 to 2a142a6 Compare August 18, 2026 18:41
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from 2a142a6 to 391fdd2 Compare August 18, 2026 18:58
@kaahos
kaahos requested a review from jbachorik August 18, 2026 19:03
kaahos and others added 5 commits August 19, 2026 10:05
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts:
#	ddprof-lib/src/main/cpp/javaApi.cpp
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from 5c765fa to 1b94815 Compare August 20, 2026 13:01

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you, please, also take a look if it makes sense to add new adversaries in the chaos tests, or some fuzzing tests?

Comment thread ddprof-lib/src/main/cpp/engine.h Outdated
Comment thread ddprof-lib/src/main/cpp/frames.h
Comment thread ddprof-lib/src/main/cpp/frames.h
Comment thread ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java Outdated
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/wallClockCandidateSelector.h
@kaahos
kaahos requested a review from jbachorik August 21, 2026 14:03
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from e0ec41e to 26ae49b Compare August 21, 2026 14:16
…-DEBUG builds

- flightRecorder.cpp: restore "<unloaded>" stale-jmethodID label (was reverted to "jvmtiError"), undoing #744
- JMethodIDInvalidationStressTest: restore assertUnloadedFrameLabel() regression guard + imports + gating assumeTrue
- UnfilteredWallPrecheckFallbackTest: self-skip via Assumptions.assumeTrue when the DEBUG-only force-wall-start-failure hook is not armed, so the test no longer fails spuriously in release builds
- add BaseWallClock::isForceStartFailureForTest() getter + JNI accessor isForceWallStartFailureArmedForTest0 (returns JNI_FALSE outside DEBUG)
@jbachorik jbachorik added test:asan Run CI tests with AddressSanitizer configuration test:tsan Run CI tests with ThreadSanitizer configuration test:fuzz labels Aug 31, 2026
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from b331762 to cd2ad5b Compare September 28, 2026 16:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sphinx:critical Sphinx: critical — human review required test:asan Run CI tests with AddressSanitizer configuration test:fuzz test:tsan Run CI tests with ThreadSanitizer configuration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants