Skip to content

[FEATURE](ci) Dual-publish to legacy and MCD CodeArtifact accounts - #726

Open
John McCall (lowlydba) wants to merge 10 commits into
mainfrom
lowlydba-dual-publish-codeartifact-accounts
Open

[FEATURE](ci) Dual-publish to legacy and MCD CodeArtifact accounts#726
John McCall (lowlydba) wants to merge 10 commits into
mainfrom
lowlydba-dual-publish-codeartifact-accounts

Conversation

@lowlydba

@lowlydba John McCall (lowlydba) commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Context

Part of OvertureMaps/ops-team#466: schema, tf-data-platform, and places-quality-model need to dual-publish overture-pypi packages into both the legacy CodeArtifact account (505071440022) and the new MCD account (763944545891). OvertureMaps/omf-core-data-opentofu#91 already mirrors the domain into MCD and grants this repo a codeartifact-pypi-publish-oidc-overturemaps role there. Also supports #510 (Phase 4, vnext dev builds to CodeArtifact), which is blocked on the same underlying CodeArtifact migration and will need the same dual-publish wiring.

Change

main-publish.yaml's publish job splits into build and two independent publish-legacy/publish-mcd jobs, each calling OvertureMaps/workflows/.github/actions/uv-publish-to-codeartifact (OvertureMaps/workflows#97) against its own account. Neither blocks the other, and one package failing to build doesn't skip publishing the rest (if: success() || failure()).

build now uploads the wheel/sdist as an artifact for the publish jobs to download, and keeps the old inline check that the built filenames match the version compute-version computed, catching a stamp/build mismatch once instead of duplicating it per account.

A final are-we-good job (lowlydba/are-we-good) rolls both publish jobs into one required check, Main publish / are-we-good (create-check-run, to avoid colliding with other workflows' same-named jobs).

Version computation still queries the legacy account only, via setup-codeartifact (OvertureMaps/workflows#80). Both shared actions are pinned to @main.

Deletes the local .github/actions/code-artifact action (its other caller, reusable-check-python-package-versions.yaml, is migrated to setup-codeartifact too), and fixes every uses: ./... ref repo-wide to uses: $/..., since zizmor 1.30.0 now flags the old syntax.

Note

Branch protection tracking publish today should switch to Main publish / are-we-good.

Testing

The pull_request path exercises build's legacy read-only auth, version query, and filename check; the publish jobs and are-we-good are skipped there, same as before. zizmor is clean. The real dual-publish can only be verified once merged to main.

Fixes #725

Swaps the local .github/actions/code-artifact composite action for the
shared OvertureMaps/workflows/.github/actions/setup-codeartifact action
(format: pypi, from workflows#80), then calls it twice in main-publish's
publish job, once per account, so overture-schema packages land in both
the legacy Airflow CodeArtifact account (505071440022) and the new MCD
account (763944545891) during the ops-team#466 migration. The publish
step now attempts both accounts even if one fails, so an outage in one
does not block the other. The pull_request smoke test still stays
read-only against both.

reusable-check-python-package-versions.yaml (the only other caller of
the local action) is migrated to the shared action too, so nothing
references .github/actions/code-artifact anymore and it can be deleted.

Fixes #725

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

🗺️ Schema reference docs preview is live!

🌍 Preview https://staging.overturemaps.org/schema/pr/726/schema/index.html
🕐 Updated Sep 09, 2026 17:07 UTC
📝 Commit 9a154bb
🔧 env SCHEMA_PREVIEW true

Note

♻️ This preview updates automatically with each push to this PR.

Swaps the hand-rolled `uv publish` invocations for the new shared
OvertureMaps/workflows/.github/actions/uv-publish-to-codeartifact
composite action (workflows#97), which wraps setup-codeartifact + uv
publish in one step per account. Drops the now-unused MCD
setup-codeartifact auth step, since the new action authenticates for
itself.

Pinned to workflows#97's branch head SHA for now, since it hasn't
merged yet; will re-pin to main once it does.

Publishing is now two `uses:` steps (one per account) with
continue-on-error plus an explicit failure check after, replicating the
previous both-accounts-attempted-even-if-one-fails behavior without a
run: step wrapping two external actions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
Matches the repo's existing convention for OvertureMaps/workflows
actions (sync-issue-type-and-scope.yml, schema-pr-preview.yml):
setup-codeartifact is already merged there, so track @main with a
zizmor: ignore[unpinned-uses] comment instead of a SHA pin.
uv-publish-to-codeartifact stays SHA-pinned since workflows#97 hasn't
merged yet.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
…blishes

Splits the publish matrix job into build/publish-legacy/publish-mcd
jobs connected by upload-artifact/download-artifact, so are-we-good
can aggregate the two publish jobs' results the way it's designed to
(needs + toJSON(needs)) instead of us hand-rolling a step-outcome
check.

Signed-off-by: John McCall <john@overturemaps.org>
zizmor 1.30.0 (introduced the self-repository audit) is what the
org's required Overture Security Checks workflow now runs, and it
flags every workspace-relative './...' uses: clause repo-wide, not
just anything touched by this PR.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
OvertureMaps/workflows#97 merged, so the dual-publish steps can track
main instead of the PR's branch head SHA, same as setup-codeartifact.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
@lowlydba
John McCall (lowlydba) marked this pull request as ready for review September 8, 2026 18:11
@lowlydba
John McCall (lowlydba) requested a review from a team as a code owner September 8, 2026 18:11
Copilot AI lite review requested due to automatic review settings September 8, 2026 18:11

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Multiple workflows/actions use uses: $/.github/..., which is not valid GitHub Actions syntax for local references and will prevent the workflows from parsing/running.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the release/publish GitHub Actions workflows to support dual-publishing Python packages to both the legacy and MCD AWS CodeArtifact accounts during migration, while also refactoring the main publish flow into build + independent publish jobs.

Changes:

  • Split main-publish.yaml into build, publish-legacy, publish-mcd, and are-we-good jobs, and switch publishing to the shared uv-publish-to-codeartifact action.
  • Migrate the version-collision check workflow from the local CodeArtifact composite action to setup-codeartifact, and delete the local .github/actions/code-artifact action.
  • Replace local uses: ./.github/... references with uses: $/.github/... repo-wide.
File summaries
File Description
.github/workflows/main-publish.yaml Refactors main publish into build + dual publish jobs and adds an aggregator check.
.github/workflows/reusable-check-python-package-versions.yaml Switches CodeArtifact auth to setup-codeartifact for version existence checks.
.github/workflows/check-python-package-versions.yaml Updates caller to use the reusable workflow via the new local uses: syntax.
.github/workflows/release-trigger.yaml Updates local composite action references to the new local uses: syntax.
.github/actions/detect-version-bumps/action.yml Updates nested local action reference to the new local uses: syntax.
.github/actions/detect-affected-packages/action.yml Updates nested local action reference to the new local uses: syntax.
.github/actions/compute-version/action.yml Updates documentation to reference setup-codeartifact outputs for index URL.
.github/actions/code-artifact/action.yml Deletes the legacy local CodeArtifact credential helper action.
Review details

Suppressed comments (3)

.github/workflows/main-publish.yaml:123

  • uses: $/... is not valid GitHub Actions syntax for local actions/workflows; local references must use a relative path like ./.github/.... As written, this workflow will fail to parse/run.
      - name: Compute version
        id: compute
        uses: $/.github/actions/compute-version
        with:
          package: ${{ matrix.package }}

.github/workflows/release-trigger.yaml:122

  • uses: $/... is not valid GitHub Actions syntax for local actions/workflows; local references must use a relative path like ./.github/.... As written, this workflow will fail to parse/run.

      - name: Create release
        uses: $/.github/actions/create-package-release
        with:
          package: ${{ matrix.package }}

.github/workflows/main-publish.yaml:197

  • Because this job needs: build, it will be skipped entirely if any matrix leg in build fails (a matrix job's overall result is failure when any leg fails). That defeats the intent of letting one package failure not block publishing other packages. Consider making this job run regardless of build's conclusion so successful legs can still publish.
    name: Publish ${{ matrix.package }} to MCD CodeArtifact
    needs: [detect, build]
    if: github.event_name != 'pull_request'
    runs-on: ubuntu-latest
  • Files reviewed: 8/8 changed files
  • Comments generated: 7
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/actions/detect-affected-packages/action.yml
Comment thread .github/actions/detect-version-bumps/action.yml
Comment thread .github/workflows/check-python-package-versions.yaml
Comment thread .github/workflows/main-publish.yaml
Comment thread .github/workflows/release-trigger.yaml
Comment thread .github/workflows/reusable-check-python-package-versions.yaml
Comment thread .github/workflows/main-publish.yaml
needs: build alone skips publish-legacy/publish-mcd entirely whenever
any package in the build matrix fails, since a matrix job's overall
result is failure if any leg failed. (success() || failure()) lets
them run anyway; the per-package download-artifact step then fails
just the legs whose build didn't produce an artifact, leaving the
rest to publish.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
The native per-workflow job check ("Are we good?") would collide with
any other workflow using the same job name for the same pattern.
create-check-run registers a standalone check named "Main publish /
are-we-good" via the Checks API instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Moved from the old inline publish step to run once in build, right
after uv build, instead of being duplicated per publish-legacy/
publish-mcd account. Catches a stamp/build version mismatch before
either CodeArtifact publish, not just a generic dist/* glob miss.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
@lowlydba

Copy link
Copy Markdown
Contributor Author

Copilot resolve the merge conflicts in this pull request

…-codeartifact-accounts

# Conflicts:
#	.github/workflows/main-publish.yaml
#	.github/workflows/reusable-check-python-package-versions.yaml

Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>

Copilot AI commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Copilot resolve the merge conflicts in this pull request

Merged main and resolved the workflow conflicts in merge commit 9a154bb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dual-publish to legacy and MCD CodeArtifact accounts

5 participants