Skip to content

feat(trace-normalization): normalize v0.4 traces - #2509

Closed
Eldolfin wants to merge 1 commit into
oscarld/extract-trace-types-cratefrom
oscarld/normalize-trace-v04
Closed

Eldolfin wants to merge 1 commit into
oscarld/extract-trace-types-cratefrom
oscarld/normalize-trace-v04

Conversation

@Eldolfin

@Eldolfin Eldolfin commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Add a normalize_trace_v04 function, that performs normalization on our internal span representation.

Motivation

This will be used to normalize traces entering the data-pipeline, before stats computation happens, in the goal of having client side stats equivalent to agent side stats

Additional Notes

How to test the change?

Describe here in detail how the change can be validated.

[generated using Codex]

@Eldolfin
Eldolfin added this pull request to stack #2510 September 11, 2026 12:16
@dd-octo-sts

dd-octo-sts Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 95.57 MB 95.57 MB +0% (+1.52 KB) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.96 MB 8.96 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 12.10 MB 12.11 MB +0% (+976 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 106.92 MB 106.94 MB +.01% (+16.74 KB) 🔍
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 28.91 MB 28.91 MB -0% (-1.50 KB) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 96.08 KB 96.08 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 191.08 MB 191.08 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 810.01 MB 810.84 MB +.10% (+849.50 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 9.65 MB 9.65 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 96.08 KB 96.08 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 27.37 MB 27.37 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 55.31 MB 55.31 MB +0% (+2.74 KB) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 25.26 MB 25.26 MB +.01% (+3.00 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 97.58 KB 97.58 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 196.30 MB 196.32 MB +0% (+16.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 795.90 MB 796.59 MB +.08% (+707.91 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 7.46 MB 7.46 MB +.01% (+1.00 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 97.58 KB 97.58 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 29.44 MB 29.45 MB +.02% (+8.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 52.22 MB 52.22 MB +0% (+2.30 KB) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 85.61 MB 85.61 MB -0% (-624 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 9.99 MB 9.99 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 101.52 MB 101.51 MB -0% (-4.42 KB) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 12.20 MB 12.20 MB -0% (-32 B) 👌

@pr-commenter

pr-commenter Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-09-11 14:33:43

Comparing candidate commit 025655e in PR branch oscarld/normalize-trace-v04 with baseline commit c8f26cf in branch oscarld/extract-trace-types-crate.

📊 Benchmarking dashboard

Found 35 performance improvements and 3 performance regressions! Performance is the same for 128 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟩 execution_time [-5.052µs; -4.852µs] or [-6.291%; -6.043%]
  • 🟩 throughput [+802144.423op/s; +835226.156op/s] or [+6.441%; +6.707%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟩 execution_time [-7.029µs; -6.853µs] or [-9.420%; -9.184%]
  • 🟩 throughput [+1357798.201op/s; +1389375.527op/s] or [+10.131%; +10.367%]

scenario:credit_card/is_card_number/378282246310005

  • 🟩 execution_time [-6.392µs; -6.243µs] or [-9.008%; -8.797%]
  • 🟩 throughput [+1361462.841op/s; +1391268.881op/s] or [+9.661%; +9.872%]

scenario:credit_card/is_card_number/37828224631000521389798

  • 🟩 execution_time [-8.236µs; -8.203µs] or [-15.467%; -15.405%]
  • 🟩 throughput [+3420654.033op/s; +3435620.640op/s] or [+18.214%; +18.294%]

scenario:credit_card/is_card_number/x371413321323331

  • 🟩 execution_time [-473.004ns; -470.467ns] or [-7.329%; -7.289%]
  • 🟩 throughput [+12185097.243op/s; +12250632.945op/s] or [+7.864%; +7.907%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

  • 🟩 execution_time [-4.792µs; -4.632µs] or [-7.345%; -7.100%]
  • 🟩 throughput [+1173065.074op/s; +1215795.425op/s] or [+7.653%; +7.932%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

  • 🟩 execution_time [-5.128µs; -5.076µs] or [-8.716%; -8.628%]
  • 🟩 throughput [+1606194.563op/s; +1621829.490op/s] or [+9.449%; +9.541%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

  • 🟩 execution_time [-5.244µs; -5.178µs] or [-9.417%; -9.297%]
  • 🟩 throughput [+1842571.278op/s; +1864326.931op/s] or [+10.261%; +10.382%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

  • 🟩 execution_time [-8.231µs; -8.198µs] or [-15.457%; -15.395%]
  • 🟩 throughput [+3418051.781op/s; +3432706.088op/s] or [+18.201%; +18.279%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

  • 🟩 execution_time [-471.376ns; -468.897ns] or [-7.304%; -7.266%]
  • 🟩 throughput [+12143807.028op/s; +12208249.984op/s] or [+7.837%; +7.878%]

scenario:datadog_sample_span/complex_rule_partial_match/wall_time

  • 🟥 execution_time [+13.123ns; +13.387ns] or [+5.620%; +5.733%]

scenario:glob_matcher/unicode_pattern_wildcard_match/wall_time

  • 🟩 execution_time [-5.797ns; -5.719ns] or [-4.487%; -4.426%]

scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...

  • 🟩 execution_time [-19.375µs; -19.191µs] or [-9.433%; -9.343%]
  • 🟩 throughput [+502041.700op/s; +506754.049op/s] or [+10.312%; +10.409%]

scenario:normalization/normalize_name/normalize_name/bad-name

  • 🟩 execution_time [-1.313µs; -1.275µs] or [-7.009%; -6.805%]
  • 🟩 throughput [+3905388.366op/s; +4021248.386op/s] or [+7.315%; +7.532%]

scenario:normalization/normalize_name/normalize_name/good

  • 🟩 execution_time [-1.404µs; -1.379µs] or [-12.839%; -12.613%]
  • 🟩 throughput [+13222384.609op/s; +13450535.188op/s] or [+14.455%; +14.704%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

  • 🟩 execution_time [-34.964µs; -34.361µs] or [-6.532%; -6.420%]
  • 🟩 throughput [+128213.320op/s; +130574.185op/s] or [+6.863%; +6.989%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

  • 🟩 execution_time [-20.194µs; -20.058µs] or [-10.596%; -10.525%]
  • 🟩 throughput [+617511.848op/s; +621579.451op/s] or [+11.768%; +11.846%]

scenario:normalization/normalize_service/normalize_service/test_ASCII

  • 🟥 execution_time [+2.843µs; +2.904µs] or [+6.229%; +6.363%]
  • 🟥 throughput [-1311468.443op/s; -1284435.379op/s] or [-5.985%; -5.862%]

scenario:profiler_attached/fast_path_system/4096

  • 🟩 execution_time [-4.266ns; -4.113ns] or [-4.191%; -4.040%]

scenario:profiler_attached/slow_path_system/4096

  • 🟩 execution_time [-9.468ns; -9.302ns] or [-6.044%; -5.938%]

scenario:trace_buffer/4_senders/no_delay

  • 🟩 execution_time [-192.465µs; -180.212µs] or [-5.138%; -4.811%]
  • 🟩 throughput [+48679.869op/s; +52030.742op/s] or [+5.065%; +5.413%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:datadog_sample_span/parent_not_sampled_short_circuit/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+556.313%; -556.007%]

scenario:datadog_sample_span/parent_sampled_short_circuit/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+555.927%; -555.825%]

scenario:glob_matcher/ascii_case_insensitive_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+555.735%; -555.735%]

scenario:glob_matcher/ascii_exact_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+574.536%; -564.672%]

scenario:glob_matcher/ascii_exact_miss/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+560.661%; -558.060%]

scenario:glob_matcher/ascii_wildcard_backtrack_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+555.735%; -555.735%]

scenario:glob_matcher/ascii_wildcard_heavy_backtrack/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+576.409%; -565.571%]

scenario:glob_matcher/ascii_wildcard_question_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+554.967%; -555.373%]

scenario:glob_matcher/ascii_wildcard_star_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+474.297%; -519.511%]

scenario:glob_matcher/star_short_circuit/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+554.979%; -555.379%]

Candidate

Omitted due to size.

Baseline

Omitted due to size.

@Eldolfin
Eldolfin force-pushed the oscarld/normalize-trace-v04 branch from cfa88f9 to 025655e Compare September 11, 2026 13:56
@Eldolfin
Eldolfin marked this pull request as ready for review September 11, 2026 15:02
@Eldolfin
Eldolfin requested review from a team as code owners September 11, 2026 15:02
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-11T15:08:41.333974Z 025655e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 025655ea2e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +83 to +84
let mut normalized = <S as Borrow<str>>::borrow(field).to_owned();
normalizer(&mut normalized);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid allocating every text field during normalization

For every v0.4 span—even when all fields are already normalized—this helper copies each service, name, type, and optional env value into a new String; the resource path performs another unconditional copy. Since normalization runs per span on the trace ingestion hot path, this adds four or more transient heap allocations per span and defeats the zero-copy benefit of the generic span representation. Add borrowed fast paths or lazily allocate only after detecting that a field needs normalization.

AGENTS.md reference: AGENTS.md:L76-L76

Useful? React with 👍 / 👎.

Comment on lines +106 to +109
let first_trace_id = first_span.trace_id;
anyhow::ensure!(
first_trace_id != 0,
"TraceID is zero (reason:trace_id_zero)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject trace IDs whose wire ID is zero

When a v0.4 span has a nonzero upper half but a zero lower half, such as trace_id == 1 << 64, this full-width check accepts it. Both the native v0.4 MessagePack encoder and the agentless v0.4 encoder serialize only trace_id as u64, however, so the normalized trace is emitted with trace ID zero and is rejected downstream. Validate that the low 64-bit v0.4 wire ID is nonzero before accepting the trace.

Useful? React with 👍 / 👎.

Comment on lines +149 to +152
// Normalize parent_id using u128 trace_id comparison for Zipkin root span detection
let parent_id_as_u128 = u128::from(span.parent_id);
if parent_id_as_u128 == trace_id && trace_id == u128::from(span.span_id) {
span.parent_id = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compare Zipkin roots against the low trace ID

For a 128-bit Zipkin trace where low64(trace_id) == span_id == parent_id, the v0.4 wire representation carries the low half as its trace ID and the high half separately in _dd.p.tid, so the existing u64 normalizer would recognize this as the special Zipkin root. Comparing against the full u128 value leaves the self-parent intact, causing root-dependent processing such as stats and _trace_root injection to miss the span. Apply the existing normalize_parent_id logic to the low 64 bits.

Useful? React with 👍 / 👎.

@VianneyRuhlmann VianneyRuhlmann left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add more context about why you need this ?

// Normalize resource using helper: borrow span.name, allocate one String for resource, compare,
// replace if different
let name_ref = <T::Text as Borrow<str>>::borrow(&span.name);
let mut resource_str = <T::Text as Borrow<str>>::borrow(&span.resource).to_owned();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's worth implementing Into\<String> on SpanText to make it simpler


// Normalize resource using helper: borrow span.name, allocate one String for resource, compare,
// replace if different
let name_ref = <T::Text as Borrow<str>>::borrow(&span.name);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let name_ref = <T::Text as Borrow<str>>::borrow(&span.name);
let name_ref: &str = &span.name.borrow();

@Eldolfin

Copy link
Copy Markdown
Contributor Author

Closing for now after discussing about the tradeoffs IRL

@Eldolfin Eldolfin closed this Sep 15, 2026
@Eldolfin
Eldolfin removed this pull request from stack #2510 September 15, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants