[lym953]

Problem

With universal instrumentation on, the extension's memory grows on every warm invocation and is never freed. Universal instrumentation is the default for Go. A customer reported in #1496 that several 1024 MB Go functions reach their memory limit after an execution environment serves about 10,000 to 20,000 invocations. They saw the growth on v94 and v101. With DD_UNIVERSAL_INSTRUMENTATION=false, the memory stays flat.

Cause

With universal instrumentation, the tracer calls the extension's /lambda/start-invocation and /lambda/end-invocation endpoints once each per invocation. The extension's listener for these calls puts the work for each call into a Tokio JoinSet. A JoinSet keeps each finished task until something joins it, and the listener joins only at shutdown. So the set grows by two tasks per invocation for the life of the execution environment. #796 added this in August 2025.

This PR

Before each new task, the listener removes the tasks that have finished, with JoinSet::try_join_next(). The listener still joins the tasks that are running at shutdown.

The proxy in front of the Lambda Runtime API has the same pattern. A separate PR fixes it.

Testing

Unit test

Steps

1. Run cargo test --lib lifecycle::listener.
2. Remove the two new calls that remove finished tasks, and run the new test again.

Result

The new test sends 100 start and end pairs through the listener and counts the tasks left in the set.

Check	Before	After
Tasks in the set after 200 requests	200 (test fails)	2 or fewer (test passes)

Lambda

Steps

1. Deploy the customer's Go handler from Extension memory grows across warm invocations with universal instrumentation (v94, v101) #1496: datadog-lambda-go v1.32.0, provided.al2023, arm64, 1024 MB, reserved concurrency 1. Use their settings: payload capture on, X-Ray merge on, X-Ray active tracing.
2. Send 6,000 direct invocations with an API Gateway HTTP API (v2) event, one at a time, so one execution environment serves all of them.
3. Every 250 invocations, read the VmRSS of the datadog-agent process from /proc/<pid>/status in the handler.
4. Repeat for four extension builds.

Result

Extension build	Universal instrumentation	Extension RSS at invocation 250	Extension RSS at invocation 6,000
Public layer v101	on	69 MB	150 MB
Public layer v101	off	69 MB	81 MB
main without this PR	on	59 MB	117 MB
main with this PR	on	64 MB	81 MB

The two main builds are the same commit, so the difference between them comes from this PR. With this PR, the extension's RSS stays between 76 and 89 MB from invocation 2,000 to 6,000. With universal instrumentation off, it stays between 73 and 83 MB over the same range. Only the Go tracer was tested.

Open questions

• In a local test with a counting allocator, the set held about 2 KB of live heap per invocation. On Lambda, RSS grew 10 to 14 KB per invocation. Allocator fragmentation could explain the difference, but I did not check it. The Lambda runs show only that this PR removes the RSS growth.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

No active review issues are reported for the lifecycle-listener task cleanup change.

No open issues 🎉

What changed in this PR

Fixes lifecycle-listener memory growth during warm Lambda invocations with universal instrumentation, addressing the retained tasks described in #1496.

Changes:

• Removes completed tasks from the listener’s JoinSet before spawning start- and end-invocation work.
• Logs task errors during cleanup while preserving shutdown handling for running tasks.
• Adds a regression test covering task retention across 100 start/end request pairs.

File	Description
bottlecap/​src/​lifecycle/​listener.rs	Adds completed-task cleanup to both invocation handlers and tests that retained task counts remain bounded.

_{Review effort: Balanced}