Skip to content

refactor(multi-runner): decouple EC2 Terraform logic into provider modules - #5257

Merged
edersonbrilhante merged 27 commits into
experimental-multi-runner-config-v2-20260805from
refactor-multi-runner-provider-modules
Aug 7, 2026
Merged

refactor(multi-runner): decouple EC2 Terraform logic into provider modules#5257
edersonbrilhante merged 27 commits into
experimental-multi-runner-config-v2-20260805from
refactor-multi-runner-provider-modules

Conversation

@edersonbrilhante

@edersonbrilhante edersonbrilhante commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Description

  • Keep modules/runners and stable multi_runner_config dispatch unchanged. Stable configurations retain their historical module.runners["configuration"] addresses and flat runners_map fields.
  • Add explicit opt-in through experimental.multi_runner_config_v2. Stable and experimental configurations can coexist when their keys do not overlap; duplicate keys are rejected.
  • Normalize stable v1 once for shared queues, webhook matching, and runner-binary discovery while routing only v2 configurations through modules/runner-stack.
  • Make runner-stack the provider-neutral control plane for scale-up, scale-down, pool, job retry, SSM housekeeping, common Lambda IAM, and runner-role ownership.
  • Keep EC2-specific launch templates, instance profiles, security groups, AMI/bootstrap resources, runner log groups, IAM fragments, and Lambda environment fragments under modules/compute-providers/ec2.
  • Define provider-owned runner-role requirements in EC2 and attach them to the common runner role in runner-stack, allowing future compute providers to supply different policies without duplicating the role lifecycle.
  • Replace flat runner-stack inputs with ownership-based nested objects. Logging configuration is grouped under observability.logs, including level, retention, encryption, class, and tags.
  • Pass the canonical compute_provider.ec2 object and nested runner, github, ssm, and observability objects directly into the EC2 resource and runner-role policy modules instead of expanding them back into prefixed scalar inputs.
  • Layer module, shared-resource, component, subcomponent, and EC2 runtime tags with documented precedence; provider-required EC2 bootstrap tags retain final precedence.
  • Group experimental v2 outputs by ownership: runner.role, scale_up.{lambda,log_group,role}, scale_down.{lambda,log_group,role}, nullable pool.{lambda,log_group,role}, and provider-specific resources under provider.<type>.
  • Use caller-known optional wrappers for external AMI parameters and KMS keys. The wrapper determines Terraform graph shape while its arn leaf may remain unknown until apply.
  • Generate runner-stack, pool, job-retry, and EC2 IAM policies with aws_iam_policy_document and retain provider-policy merge behavior.
  • Document the experimental boundary, ownership model, plan-time wrapper pattern, phased migration, and nested output contract under the internal module documentation path.

This draft is stacked on #5251 because the provider boundary consumes the experimental v2 normalization introduced there.

Lambda/TypeScript terminology changes are tracked separately in #5258.

Test Plan

  • pre-commit run --all-files — Terraform fmt, TFLint, validation, and merge-conflict checks passed.
  • terraform test in modules/runner-stack — 11 passed.
  • terraform test in modules/multi-runner — 7 passed.
  • terraform test in modules/compute-providers/ec2 — 4 passed.
  • terraform test in modules/compute-providers/ec2/runner-role — 3 passed.
  • terraform test in modules/runner-stack/pool — 1 passed.
  • terraform test in modules/runner-stack/job-retry — 1 passed.
  • terraform validate in modules/lambda — passed.
  • Verified modules/runners has no diff from origin/main, stable v1 still dispatches only to module.runners, and only the experimental map dispatches to module.runner_stacks.
  • Verified computed external role, profile, AMI-parameter, managed-policy, and KMS ARN inputs plan successfully through the real wrapper fixture.

No live AWS apply was performed. Terraform tests use mocked providers, and state migration is intentionally deferred to the later migration phase.

Related Issues

Closes #5252

Depends on #5251

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@edersonbrilhante edersonbrilhante changed the title refactor(multi-runner): decouple EC2 Terraform logic refactor(multi-runner): decouple EC2 Terraform logic into provider modules Aug 7, 2026
@edersonbrilhante
edersonbrilhante marked this pull request as ready for review August 7, 2026 19:09
@edersonbrilhante
edersonbrilhante requested review from a team as code owners August 7, 2026 19:09
@edersonbrilhante
edersonbrilhante merged commit 7d94e7c into experimental-multi-runner-config-v2-20260805 Aug 7, 2026
60 checks passed
@edersonbrilhante
edersonbrilhante deleted the refactor-multi-runner-provider-modules branch August 7, 2026 19:12
edersonbrilhante added a commit that referenced this pull request Aug 12, 2026
…dules (#5257)

## Description

- Keep `modules/runners` and stable `multi_runner_config` dispatch
unchanged. Stable configurations retain their historical
`module.runners["configuration"]` addresses and flat `runners_map`
fields.
- Add explicit opt-in through `experimental.multi_runner_config_v2`.
Stable and experimental configurations can coexist when their keys do
not overlap; duplicate keys are rejected.
- Normalize stable v1 once for shared queues, webhook matching, and
runner-binary discovery while routing only v2 configurations through
`modules/runner-stack`.
- Make `runner-stack` the provider-neutral control plane for scale-up,
scale-down, pool, job retry, SSM housekeeping, common Lambda IAM, and
runner-role ownership.
- Keep EC2-specific launch templates, instance profiles, security
groups, AMI/bootstrap resources, runner log groups, IAM fragments, and
Lambda environment fragments under `modules/compute-providers/ec2`.
- Define provider-owned runner-role requirements in EC2 and attach them
to the common runner role in `runner-stack`, allowing future compute
providers to supply different policies without duplicating the role
lifecycle.
- Replace flat runner-stack inputs with ownership-based nested objects.
Logging configuration is grouped under `observability.logs`, including
`level`, retention, encryption, class, and tags.
- Pass the canonical `compute_provider.ec2` object and nested `runner`,
`github`, `ssm`, and `observability` objects directly into the EC2
resource and runner-role policy modules instead of expanding them back
into prefixed scalar inputs.
- Layer module, shared-resource, component, subcomponent, and EC2
runtime tags with documented precedence; provider-required EC2 bootstrap
tags retain final precedence.
- Group experimental v2 outputs by ownership: `runner.role`,
`scale_up.{lambda,log_group,role}`,
`scale_down.{lambda,log_group,role}`, nullable
`pool.{lambda,log_group,role}`, and provider-specific resources under
`provider.<type>`.
- Use caller-known optional wrappers for external AMI parameters and KMS
keys. The wrapper determines Terraform graph shape while its `arn` leaf
may remain unknown until apply.
- Generate runner-stack, pool, job-retry, and EC2 IAM policies with
`aws_iam_policy_document` and retain provider-policy merge behavior.
- Document the experimental boundary, ownership model, plan-time wrapper
pattern, phased migration, and nested output contract under the internal
module documentation path.

This draft is stacked on #5251 because the provider boundary consumes
the experimental v2 normalization introduced there.

Lambda/TypeScript terminology changes are tracked separately in #5258.

## Test Plan

- `pre-commit run --all-files` — Terraform fmt, TFLint, validation, and
merge-conflict checks passed.
- `terraform test` in `modules/runner-stack` — 11 passed.
- `terraform test` in `modules/multi-runner` — 7 passed.
- `terraform test` in `modules/compute-providers/ec2` — 4 passed.
- `terraform test` in `modules/compute-providers/ec2/runner-role` — 3
passed.
- `terraform test` in `modules/runner-stack/pool` — 1 passed.
- `terraform test` in `modules/runner-stack/job-retry` — 1 passed.
- `terraform validate` in `modules/lambda` — passed.
- Verified `modules/runners` has no diff from `origin/main`, stable v1
still dispatches only to `module.runners`, and only the experimental map
dispatches to `module.runner_stacks`.
- Verified computed external role, profile, AMI-parameter,
managed-policy, and KMS ARN inputs plan successfully through the real
wrapper fixture.

No live AWS apply was performed. Terraform tests use mocked providers,
and state migration is intentionally deferred to the later migration
phase.

## Related Issues

Closes #5252

Depends on #5251

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
edersonbrilhante added a commit that referenced this pull request Aug 12, 2026
…dules (#5257)

## Description

- Keep `modules/runners` and stable `multi_runner_config` dispatch
unchanged. Stable configurations retain their historical
`module.runners["configuration"]` addresses and flat `runners_map`
fields.
- Add explicit opt-in through `experimental.multi_runner_config_v2`.
Stable and experimental configurations can coexist when their keys do
not overlap; duplicate keys are rejected.
- Normalize stable v1 once for shared queues, webhook matching, and
runner-binary discovery while routing only v2 configurations through
`modules/runner-stack`.
- Make `runner-stack` the provider-neutral control plane for scale-up,
scale-down, pool, job retry, SSM housekeeping, common Lambda IAM, and
runner-role ownership.
- Keep EC2-specific launch templates, instance profiles, security
groups, AMI/bootstrap resources, runner log groups, IAM fragments, and
Lambda environment fragments under `modules/compute-providers/ec2`.
- Define provider-owned runner-role requirements in EC2 and attach them
to the common runner role in `runner-stack`, allowing future compute
providers to supply different policies without duplicating the role
lifecycle.
- Replace flat runner-stack inputs with ownership-based nested objects.
Logging configuration is grouped under `observability.logs`, including
`level`, retention, encryption, class, and tags.
- Pass the canonical `compute_provider.ec2` object and nested `runner`,
`github`, `ssm`, and `observability` objects directly into the EC2
resource and runner-role policy modules instead of expanding them back
into prefixed scalar inputs.
- Layer module, shared-resource, component, subcomponent, and EC2
runtime tags with documented precedence; provider-required EC2 bootstrap
tags retain final precedence.
- Group experimental v2 outputs by ownership: `runner.role`,
`scale_up.{lambda,log_group,role}`,
`scale_down.{lambda,log_group,role}`, nullable
`pool.{lambda,log_group,role}`, and provider-specific resources under
`provider.<type>`.
- Use caller-known optional wrappers for external AMI parameters and KMS
keys. The wrapper determines Terraform graph shape while its `arn` leaf
may remain unknown until apply.
- Generate runner-stack, pool, job-retry, and EC2 IAM policies with
`aws_iam_policy_document` and retain provider-policy merge behavior.
- Document the experimental boundary, ownership model, plan-time wrapper
pattern, phased migration, and nested output contract under the internal
module documentation path.

This draft is stacked on #5251 because the provider boundary consumes
the experimental v2 normalization introduced there.

Lambda/TypeScript terminology changes are tracked separately in #5258.

## Test Plan

- `pre-commit run --all-files` — Terraform fmt, TFLint, validation, and
merge-conflict checks passed.
- `terraform test` in `modules/runner-stack` — 11 passed.
- `terraform test` in `modules/multi-runner` — 7 passed.
- `terraform test` in `modules/compute-providers/ec2` — 4 passed.
- `terraform test` in `modules/compute-providers/ec2/runner-role` — 3
passed.
- `terraform test` in `modules/runner-stack/pool` — 1 passed.
- `terraform test` in `modules/runner-stack/job-retry` — 1 passed.
- `terraform validate` in `modules/lambda` — passed.
- Verified `modules/runners` has no diff from `origin/main`, stable v1
still dispatches only to `module.runners`, and only the experimental map
dispatches to `module.runner_stacks`.
- Verified computed external role, profile, AMI-parameter,
managed-policy, and KMS ARN inputs plan successfully through the real
wrapper fixture.

No live AWS apply was performed. Terraform tests use mocked providers,
and state migration is intentionally deferred to the later migration
phase.

## Related Issues

Closes #5252

Depends on #5251

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
edersonbrilhante added a commit that referenced this pull request Aug 12, 2026
…dules (#5257)

## Description

- Keep `modules/runners` and stable `multi_runner_config` dispatch
unchanged. Stable configurations retain their historical
`module.runners["configuration"]` addresses and flat `runners_map`
fields.
- Add explicit opt-in through `experimental.multi_runner_config_v2`.
Stable and experimental configurations can coexist when their keys do
not overlap; duplicate keys are rejected.
- Normalize stable v1 once for shared queues, webhook matching, and
runner-binary discovery while routing only v2 configurations through
`modules/runner-stack`.
- Make `runner-stack` the provider-neutral control plane for scale-up,
scale-down, pool, job retry, SSM housekeeping, common Lambda IAM, and
runner-role ownership.
- Keep EC2-specific launch templates, instance profiles, security
groups, AMI/bootstrap resources, runner log groups, IAM fragments, and
Lambda environment fragments under `modules/compute-providers/ec2`.
- Define provider-owned runner-role requirements in EC2 and attach them
to the common runner role in `runner-stack`, allowing future compute
providers to supply different policies without duplicating the role
lifecycle.
- Replace flat runner-stack inputs with ownership-based nested objects.
Logging configuration is grouped under `observability.logs`, including
`level`, retention, encryption, class, and tags.
- Pass the canonical `compute_provider.ec2` object and nested `runner`,
`github`, `ssm`, and `observability` objects directly into the EC2
resource and runner-role policy modules instead of expanding them back
into prefixed scalar inputs.
- Layer module, shared-resource, component, subcomponent, and EC2
runtime tags with documented precedence; provider-required EC2 bootstrap
tags retain final precedence.
- Group experimental v2 outputs by ownership: `runner.role`,
`scale_up.{lambda,log_group,role}`,
`scale_down.{lambda,log_group,role}`, nullable
`pool.{lambda,log_group,role}`, and provider-specific resources under
`provider.<type>`.
- Use caller-known optional wrappers for external AMI parameters and KMS
keys. The wrapper determines Terraform graph shape while its `arn` leaf
may remain unknown until apply.
- Generate runner-stack, pool, job-retry, and EC2 IAM policies with
`aws_iam_policy_document` and retain provider-policy merge behavior.
- Document the experimental boundary, ownership model, plan-time wrapper
pattern, phased migration, and nested output contract under the internal
module documentation path.

This draft is stacked on #5251 because the provider boundary consumes
the experimental v2 normalization introduced there.

Lambda/TypeScript terminology changes are tracked separately in #5258.

## Test Plan

- `pre-commit run --all-files` — Terraform fmt, TFLint, validation, and
merge-conflict checks passed.
- `terraform test` in `modules/runner-stack` — 11 passed.
- `terraform test` in `modules/multi-runner` — 7 passed.
- `terraform test` in `modules/compute-providers/ec2` — 4 passed.
- `terraform test` in `modules/compute-providers/ec2/runner-role` — 3
passed.
- `terraform test` in `modules/runner-stack/pool` — 1 passed.
- `terraform test` in `modules/runner-stack/job-retry` — 1 passed.
- `terraform validate` in `modules/lambda` — passed.
- Verified `modules/runners` has no diff from `origin/main`, stable v1
still dispatches only to `module.runners`, and only the experimental map
dispatches to `module.runner_stacks`.
- Verified computed external role, profile, AMI-parameter,
managed-policy, and KMS ARN inputs plan successfully through the real
wrapper fixture.

No live AWS apply was performed. Terraform tests use mocked providers,
and state migration is intentionally deferred to the later migration
phase.

## Related Issues

Closes #5252

Depends on #5251

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant