[clifordshelton]

Jira: SVLS-10110, a sub-task of SVLS-10109

Overview

Tracers call /info early in their own startup. Lambda starts the runtime, and with it the tracer's init, as soon as every extension has registered; the extension's first /next only marks its own init as done. The trace agent used to bind its socket much later, after register and most of the extension's setup, so that first /info call could be refused. The tracer then skips the Extension for the life of the execution environment. For example, dd-trace-js 6.20 either sends feature flag events directly (if it has an API key) or turns delivery off and retries after 60 seconds.

This binds the trace agent's port in main() before the extension registers. Connections that arrive before the server starts are queued by the kernel and served once it starts.

• TraceAgent::bind_listener(port) binds the socket through tokio::net::TcpSocket (127.0.0.1, SO_REUSEADDR).
• main() calls it before register and passes the listener through extension_loop_active to startup::start_trace_agent, which serves it with TraceAgent::start_with_listener().
• Bind failures still log Error starting trace agent: ..., and no trace agent is served, as before. start() still binds and serves, so the test-mode binary is unchanged.

Based on Rey's change on rey.abolofia/llmobs-info-endpoint (33483fd8), and on Aaron's review finding and patch that moved the bind before register. The /evp_proxy/v2/ advertisement from Rey's branch ships with the EVP proxy work in #1507 (SVLS-10112).

Part of SVLS-10109 (feature flags through the Lambda Extension).

Likely concerns

Does this slow down cold starts?
No measurable change. 100 cold starts per layer, alternating main and this branch: p50 814 vs 812 ms, p90 1065 vs 1041 ms. Details in the comment below. That run bound inside start_trace_agent; moving the bind earlier does the same work at a different point.

Does it block the async runtime?
No. Binding a socket on 127.0.0.1 is a local syscall with no network wait. Serving still happens in the spawned task.

What if port 8126 is already taken?
Same as before: the error is logged as Error starting trace agent: ..., nothing panics, and the rest of the Extension keeps running. port_conflict_test (trace_agent_port_8126_held) still passes.

What happens to a request that arrives before the server is ready?
It waits in the kernel queue (backlog 1024, capped by the kernel) until the spawned task starts serving, then gets a normal reply. Before this change, it was refused.

What if the extension falls back to idle mode?
The listener is moved into extension_loop_active, so it's dropped if that loop fails, and nothing keeps 8126 open without serving it.

Does it affect Managed Instance mode or the test-mode binary?
Both modes go through the same main() path, so both get the fix. The test-mode binary calls start(), which binds and serves as before.

Testing

• New lifecycle tests: the fake Extensions API makes one non-retrying connection to 8126 while it handles register, and trace_agent_listens_before_register (plus a Managed Instance variant) asserts it was accepted. Both fail on the previous code and on main, and pass now.
• Unit test: a client can connect right after bind_listener(), before the agent serves.
• Full cargo test (lib and all integration test binaries).
• Cold start before/after with custom layers built from main and this branch (serverless-simulation-lab): no measurable change (see comment).

🤖 Generated with Claude Code

[clifordshelton]

Cold start check: no measurable impact. 100 cold starts per layer (arm64, Node 22 layer 143, 2048 MB, DD_API_KEY_SECRET_ARN), alternating main and this branch, using the serverless-simulation-lab feature-flag-extension-routes scenario with custom layers built from each.

Layer	Mean	p50	p90	p99
main	835 ms	814 ms	1065 ms	1099 ms
this branch	794 ms	812 ms	1041 ms	1117 ms

Both layers show the same slower group of ~10–15% of cold starts around 1040–1140 ms, which drives p90 for both.

[datadog-datadog-prod-us1-2]

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 757fc1a | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-12T17:12:02.667499Z	757fc1a	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🟡 Changes recommended

The unit test does not guard against reintroducing the startup binding race; the required process-level regression test must verify an immediate connection on the first /next and a bounded /info response.

1 open issue

• Add a process-level startup binding regression test · New

What changed in this PR

This PR addresses the trace-agent startup race by binding its listener before the Extension calls /next, allowing early tracer connections to queue instead of being refused. It prepares startup for the separate EVP proxy work without changing endpoint advertisement.

Changes:

• Separates synchronous socket binding from asynchronous serving.
• Binds during shared startup while preserving error logging and the test-mode start() interface.
• Adds a unit test for connections arriving before serving begins.

File	Description
bottlecap/​src/​traces/​trace_agent.rs	Adds separate binding and serving methods, an explicit backlog, and a pre-serving connection test.
bottlecap/​src/​startup.rs	Binds the trace listener before spawning the server task and returning to the caller.

_{Review effort: Balanced}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b47cb97c6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[shreyamalpani]

LGTM! Non-blocking suggestion: test the ordering through start_trace_agent

bind_accepts_connections_before_serving calls bind() directly, so it would still pass if the bind moved back into tokio::spawn. A check at the first /next, as suggested by Codex and Copilot, has the same problem, since the old code had already bound by then.

A current-thread test that calls start_trace_agent and immediately does a single non-retrying connect to 8126, with no .await in between, would exercise this better. Spawn-then-bind gets Connection refused; this PR connects.

[aarsilv]

🤖 Comment from Claude working with aarsilv:

Heads up: the other local listeners also start after register

Lambda starts the runtime once every extension has registered (see the comment on startup.rs line 256). These listeners also bind after the register response, so the same window applies to them. No action is needed in this PR.

• Lifecycle listener, 127.0.0.1:8124: spawned at main.rs:479; it binds in lifecycle/listener.rs:83. It keeps a deprecated /lambda/hello route, so older tracing libraries called it. I did not check whether any library calls it during init.
• Runtime API proxy, port 9000: binds inside a spawned task (proxy/interceptor.rs:79-82). When scripts/datadog_wrapper sets AWS_LAMBDA_RUNTIME_API to 127.0.0.1:9000, the runtime's own Runtime API calls go through it.
• DogStatsD, 8125/udp: created in start_dogstatsd (main.rs:1261). UDP datagrams sent before that are lost.

If any of these matter, the same fix applies: bind before register and pass the socket down.

[aarsilv]

🤖 Comment from Claude working with aarsilv:

Heads up: On-Demand mode does not act on a SHUTDOWN that answers the first /next (pre-existing, not from this PR)

I found this while I wrote a test for this PR. No change is needed here. I suggest a ticket for a follow-up.

At main.rs:737, handle_next_invocation(next_lambda_response, &invocation_processor_handle).await; drops its return value. Later /next calls keep the result as maybe_shutdown_event and act on it. So if Lambda answers the first /next with SHUTDOWN, the On-Demand loop does not start its shutdown path. With the end flush strategy, and with the default strategy until it has seen 20 invocations, the loop waits for a platform.runtimeDone that never comes, and the extension does not exit. Lambda ends an extension that does not finish within the 2,000 ms SHUTDOWN limit with SIGKILL, so the shutdown flush does not run.

This can happen when Lambda shuts an environment down before its first invocation. I did not check how often that happens. Managed Instance mode is not affected: its SHUTDOWN task handles the first event.

Reproduction: fails on main and on this branch

For tests/lifecycle_contract_test.rs:

/// Lambda can answer the first `/next` with SHUTDOWN, for example when it shuts
/// an environment down before its first invocation. The extension must exit
/// within the SHUTDOWN deadline.
#[tokio::test(flavor = "multi_thread")]
#[serial_test::serial(bottlecap_ports)]
#[ignore = "<ticket>: On-Demand drops a SHUTDOWN that answers the first /next and runs until killed"]
async fn shutdown_as_first_event_exits_cleanly() {
    let s = Scenario::start(FaultMode::Ok).await;
    let mut bc = s.bottlecap().spawn();
    let startup = bc.startup_deadline();
    assert!(
        s.wait_next_or_exit(&mut bc, 1, startup).await,
        "no first /next. {}",
        bc.log_tail(40)
    );
    s.shutdown_cleanly(&mut bc).await;
}

cargo nextest run --test lifecycle_contract_test --run-ignored only -E 'test(shutdown_as_first_event)' fails on main (ee19798) and on this branch (9b47cb9): the process is still running about 6 s after the 2 s SHUTDOWN deadline. The harness sets DD_SERVERLESS_FLUSH_STRATEGY=end.

[clifordshelton]

Thanks @shreyamalpani! You were right that a check at the first /next could not catch this. With @aarsilv's finding that the race is at register, the bind moved to before register (faf5d3b), so your idea now lives in the integration harness: the fake Extensions API makes a single non-retrying connect to 8126 while it handles register, and the new lifecycle tests assert it succeeded. They fail with the old ordering and pass now.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: faf5d3b858

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[clifordshelton]

Thanks @aarsilv, both are good finds. We'll handle them in follow-ups rather than in this PR:

Other listeners binding after register → SVLS-10125
This PR is scoped to the trace agent, the port tracers probe through /info, and its tests cover only that. The lifecycle listener (lifecycle/listener.rs) and the Runtime API proxy (proxy/interceptor.rs) are separate code paths. Each needs its own investigation and regression test, which would roughly double this PR. The Runtime API proxy on 9000 is the one we'll look at first: when datadog_wrapper points AWS_LAMBDA_RUNTIME_API at it, the runtime's own calls go through it, so we want to understand whether the runtime retries or fails before changing anything. The fix should follow the same pattern as here, with a register-time probe in the fake Extensions API.

SHUTDOWN on the first /next ignored in On-Demand mode → SVLS-10126
We hit this independently while writing the register-ordering test here: on main, the extension is still running ~6 s past the 2 s SHUTDOWN deadline, every time, and your main.rs:737 explanation matches. It's already on main and unrelated to startup ordering, so it's better as its own small fix with its own regression test. The new tests in this PR stop the process instead of depending on that shutdown path.

[lucaspimentel]

LGTM, thanks!

[lucaspimentel]

@DataDog review

[datadog-datadog-prod-us1-2]

More details

Binding port 8126 before registration lets early tracer connections queue until serving begins, while bind failures remain nonfatal. The completed static review identified no reportable defect.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 757fc1a}