Aggregate LLMObs span.finished telemetry counts per interval

[Yun-Kim]

What Does This Do

Makes LLMObsMetricCollector aggregate mlobs.span.finished counts per tag combination in-process and emit one metric carrying the summed value per metrics interval, instead of enqueuing one raw metric of value 1 per finished span.

Motivation

dd.instrumentation_telemetry_data.mlobs.span.finished underreports ~100x under-count on the spans that are received by Datadog LLM Observability backend.

recordSpanFinished enqueued one LLMObsMetric of value 1 per span, and prepareMetrics() was an explicit no-op, so no summation ever happened. Two independent failures result:

1. Same-second point collapse (the ~100x). MetricCollector.Metric timestamps at second granularity (System.currentTimeMillis() / 1000). Every point a series emits within the same second collapses to a single value at the metrics intake, so the reported rate is pinned at ~1/s per series no matter the real span rate. Confirmed on the affected org: max:dd.instrumentation_telemetry_data.mlobs.span.finished{org_id:...}.as_rate() sits flat at exactly 1.0/s, and the org-wide sum of ~1k/min is just the ~17 active tag combinations each contributing 1/s.

2. Raw queue overflow. The queue holds RAW_QUEUE_SIZE (1024) entries and is drained once per 10s metrics interval, capping reporting at ~102 spans/s per JVM. Overflow was dropped at debug level. This is currently masked by (1) but would bind on its own.

How It Works

• recordSpanFinished increments a LongAdder keyed by tag combination; no per-span allocation into the queue.
• prepareMetrics() drains each counter with sumThenReset() and stages one metric per tag combination carrying the true count.
• The queue now holds one entry per tag combination per interval rather than one per span, so 1024 slots is ample (bounded at 128 combinations x 6 intervals per heartbeat).
• If a metric cannot be staged, its count is returned to the counter and reported in a later interval rather than lost.
• Counter entries are deliberately never removed: a recorder thread may already hold a reference to a LongAdder, so removing one would drop a concurrent increment. Tag values come from bounded sets, so idle entries cost at most MAX_TAG_COMBINATIONS.

Claude session: 05402794-04c0-4abe-a984-4b7b4eeb71ec
Resume: claude --resume 05402794-04c0-4abe-a984-4b7b4eeb71ec

🤖 Generated with Claude Code

[datadog-datadog-prod-us1-2]

🎯 Code Coverage (details)
• Patch Coverage: 83.33%
• Overall Coverage: 57.60% (-0.31%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d8d5dad | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.91 s	13.89 s	[-0.5%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent	12.90 s	13.00 s	[-1.7%; +0.1%] (no difference)
startup:petclinic:appsec:Agent	16.88 s	16.11 s	[+0.4%; +9.2%] (maybe worse)
startup:petclinic:iast:Agent	16.99 s	16.91 s	[-0.4%; +1.3%] (no difference)
startup:petclinic:profiling:Agent	16.82 s	16.81 s	[-1.0%; +1.1%] (no difference)
startup:petclinic:sca:Agent	16.69 s	16.52 s	[+0.0%; +2.1%] (maybe worse)
startup:petclinic:tracing:Agent	16.09 s	16.16 s	[-1.3%; +0.3%] (no difference)

Commit: d8d5dadf · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.