[clifordshelton]

Jira: SVLS-10111, a sub-task of SVLS-10109

Overview

The tracer's feature flag provider downloads flag rules from Datadog's CDN and only reads DD_API_KEY. When customers keep the key in Secrets Manager (DD_API_KEY_SECRET_ARN), only the Extension can read it, so flags never load: every check returns PROVIDER_NOT_READY.

dd-trace-js 6.15+ (and dd-trace-py) can fetch flag rules from a custom address, and skip the API key when one is set. This adds a route to the Extension's trace agent that does that request for them:

• GET /api/v2/feature-flagging/config/rules-based/server on 127.0.0.1:8126 is forwarded right away to https://ufc-server.ff-cdn.<DD_SITE> with the API key the Extension resolved, and Datadog's reply is returned as-is (status, Content-Type, Content-Encoding, ETag, Cache-Control, body, including 304).
• Accept, Accept-Encoding and If-None-Match are passed through. dd_env is added from DD_ENV unless present, because the tracer drops it with a custom address.
• Replies 403 when the key can't be resolved (the provider logs it and doesn't retry), and 502 when Datadog is unreachable or the key lookup takes too long.
• Doesn't follow redirects, so the API key is only ever sent to the CDN host.
• The route lives in a new traces/passthrough module with a fixed route list. Nothing else is forwarded.

Customers point the tracer at the Extension with:

DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=http://127.0.0.1:8126

No tracer change is needed.

Likely concerns

Can this forward requests or the API key anywhere else?
No. There's one route, GET /api/v2/feature-flagging/config/rules-based/server, and it always goes to https://ufc-server.ff-cdn.<DD_SITE> on the same path. The host comes from the Extension's config, not from the request, and the pass-through uses an HTTP client that doesn't follow redirects (reqwest keeps custom headers like DD-API-KEY on a redirect to another host). A 3xx from Datadog goes back to the tracer as is. Every other path keeps returning 404.

Can the API key leak back to the tracer or into logs?
The key is only added as a request header to Datadog. Only Content-Type, Content-Encoding, ETag and Cache-Control are copied back to the tracer, so nothing Datadog echoes can return the key. Error logs include the request URL, which never contains the key.

What if Datadog is slow or down?
The whole request, including resolving the API key, times out after 4 seconds, below the tracer's default 5-second timeout, so the tracer gets our error instead of timing out itself. Unreachable or failed upstream requests, and a key lookup that takes longer than that, return 502, which the tracer retries a few times with backoff, as it would against the CDN.

Can a tracer disconnect break the Extension's key loading?
No. The key is resolved in its own task, so it finishes even if the tracer gives up and the request is dropped. A key load cancelled midway would leave ApiKeyFactory stuck, with every flusher waiting on it. SVLS-10129 tracks making ApiKeyFactory itself safe to cancel.

What if the Extension can't resolve the API key?
It returns 403 and logs the failure. The flag provider logs it and doesn't retry, instead of retrying a 5xx with backoff.

How big are the responses?
The body is read fully and returned as-is. Flag settings are typically small; the largest we've seen (RBI's production config) is about 470 KB, and the body stays compressed when the tracer asks for gzip.

What if a customer sets the env var but runs an older Extension?
The older Extension returns 404 for this path. The tracer then doesn't load flags, the same result as having no key today, and it doesn't fall back to the CDN even if DD_API_KEY is set. Docs and install tooling should only set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL together with an Extension version that has this route.

Does it work behind a proxy or in FIPS mode?
Yes. It uses the Extension's shared HTTP client, which already applies the configured HTTPS proxy and the FIPS-compliant TLS setup.

Does it add to cold starts?
No. Nothing runs at init. The Extension resolves the key lazily, so the first flag request may include one Secrets Manager call.

Does it conflict with #1507?
Both add to trace_agent.rs: #1507 adds the EVP flag routes and the /info change, this PR adds the pass-through hook. They're independent, but whichever merges second needs a quick rebase.

Testing

• Unit tests: forwarding with the key and dd_env, 304 pass-through, an existing dd_env kept, 403 without a key, 502 when unreachable, other paths not forwarded, a 302 returned without following it (the redirect target gets no request), and a slow key lookup that returns 502 and then succeeds on the next request once the background load finishes. The last two fail without the fix.
• cargo test --lib, plus apm_integration_test, lifecycle_contract_test and port_conflict_test.
• Real Lambda with only DD_API_KEY_SECRET_ARN (no DD_API_KEY anywhere in the function's environment), to confirm the flag settings calls work when only the Extension can read the key. Setup: serverless-simulation-lab, arm64, Node.js 22, a test flag in the DDServerless org, and an Extension layer built from this change combined with [SVLS-10110] fix(traces): bind the trace agent listener before registering #1506 and [SVLS-10112] feat(traces): relay feature flag events through the EVP proxy #1507. The code from this PR is unchanged in that build, and this route doesn't depend on the other two (it doesn't use /info).
  • Without DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL: every flag check returned PROVIDER_NOT_READY, and the tracer logged DD_API_KEY is required (today's behavior).
  • With it set to http://127.0.0.1:8126: the provider was ready and the flag returned its real value on cold and warm invocations, on dd-trace-js 6.15 and 6.20.1, including TARGETING_MATCH from a targeting rule. The Extension logged Forwarded flag config request: 200 OK, and 304 Not Modified on later polls.

🤖 Generated with Claude Code

[datadog-datadog-prod-us1-2]

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8f9573d | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-11T23:22:45.514842Z	8f9573d	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🟡 Changes recommended

Redirect handling can leak the API key to another host, and the timeout excludes key resolution, allowing stalled lookups to exceed the tracer deadline.

2 open issues

• Disable redirects to prevent API-key disclosure · New
• Include API-key resolution in the end-to-end timeout · New

What changed in this PR

Adds a feature-flag configuration route to the Lambda Extension’s trace agent so tracers can fetch rules using an API key resolved by the Extension, including keys stored in Secrets Manager.

Changes:

• Forwards flag configuration requests to the Datadog CDN with the resolved key and a default dd_env.
• Returns upstream status, selected headers, and body, with tests for forwarding and error handling.
• Integrates the passthrough router into trace-agent startup.

File	Description
bottlecap/​src/​traces/​trace_agent.rs	Adds optional passthrough configuration and router integration.
bottlecap/​src/​traces/​passthrough.rs	Implements flag configuration forwarding and unit tests.
bottlecap/​src/​traces/​mod.rs	Exposes the passthrough module.
bottlecap/​src/​startup.rs	Wires passthrough into the trace agent with shared configuration and clients.

_{Review effort: Balanced}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c9a535da7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[purple4reina]

Question.

I see about the DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL env var. Should we set this automatically in our lambda layers? So for example if someone is using python, they instrument their handler with datadog-lambda-python, and the first thing that happens (before dd-trace-py is imported) we set that env var for them.

There's precedent for this already in datadog-lambda-python. See https://github.com/DataDog/datadog-lambda-python/blob/a73361475e45ce71519efedfd647e04a129857b5/datadog_lambda/config.py#L155-L172

[lucaspimentel]

In most tracers, env, version, and service can be set by either DD_ENV/DD_VERSION/DD_SERVICE or DD_TAGS:

For example, these are equivalent:

DD_ENV=prod
DD_VERSION=1
DD_SERVICE=myapp

DD_TAGS=env:prod,version:1,service:myapp

When both are present, the individual configs take precedence over DD_TAGS.