feat(runners): make the scale-up SQS event source mapping optional

[bayramkeles61]

Motivation

CreateFleet consumes the account's ec2:RunInstances request bucket, which AWS documents as capacity 5, refill 2/sec, per account per Region and adjustable only through a Support case. The module's own docs/rate-limits-and-tuning.md notes that CreateFleet "empirically throttles at low single-digit TPS".

When that limit is hit the failure does not settle on its own: a throttled call creates no instance, so the SQS message is never deleted, so it is redelivered, so more calls are made. We measured 6,833 scale-up invocations against 122 queued messages with zero instances created — the retries manufacture the throttle that causes the retries.

The lever the module offers today is scale_up_reserved_concurrent_executions, but that bounds invocations in flight, not invocations per second. The two differ by the API's latency, so the same reservation of 5 yields ~2.5/sec at 2s per call and ~10/sec at 500ms. There is no setting that expresses a rate.

A rate limit needs a token bucket, and a token bucket needs a process that outlives a single event. Lambda invocations are independent processes with no shared state, so the counter has to live somewhere else — which means letting something else consume the build queue.

What this changes

Adds enable_scale_up_event_source_mapping to the root module and to modules/runners, wired to the enabled argument of aws_lambda_event_source_mapping.scale_up.

Default is true, so behaviour is unchanged for every existing configuration. Setting it to false lets an external consumer drain the build queue at a controlled rate and invoke the scale-up lambda itself; the handler's contract is already suitable for that, since it takes a standard SQSEvent and returns SQSBatchResponse.

Why a variable rather than doing it out of band

The resource does not set enabled, so the provider default (true) is what lands in state. Disabling the mapping with the AWS CLI therefore shows up as drift and is reverted by the next apply — silently re-enabling a second consumer of the queue.

Related

• visibility timeout is set to the same value as lambda timeout #5193 — visibility timeout tied to the lambda timeout; same area, not addressed here.
• The docs/rate-limits-and-tuning.md guidance on batch_size reduces CreateFleet calls per message but does not bound the call rate.

Testing

tofu fmt and tofu validate pass. The change is inert at the default, and terraform-docs output is left to the update-docs workflow.

🤖 Generated with Claude Code

[edersonbrilhante]

@bayramkeles61 Can you sign your commit?