[FEATURE](ci) Dual-publish to legacy and MCD CodeArtifact accounts - #726
[FEATURE](ci) Dual-publish to legacy and MCD CodeArtifact accounts#726John McCall (lowlydba) wants to merge 10 commits into
Conversation
Swaps the local .github/actions/code-artifact composite action for the shared OvertureMaps/workflows/.github/actions/setup-codeartifact action (format: pypi, from workflows#80), then calls it twice in main-publish's publish job, once per account, so overture-schema packages land in both the legacy Airflow CodeArtifact account (505071440022) and the new MCD account (763944545891) during the ops-team#466 migration. The publish step now attempts both accounts even if one fails, so an outage in one does not block the other. The pull_request smoke test still stays read-only against both. reusable-check-python-package-versions.yaml (the only other caller of the local action) is migrated to the shared action too, so nothing references .github/actions/code-artifact anymore and it can be deleted. Fixes #725 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
Swaps the hand-rolled `uv publish` invocations for the new shared OvertureMaps/workflows/.github/actions/uv-publish-to-codeartifact composite action (workflows#97), which wraps setup-codeartifact + uv publish in one step per account. Drops the now-unused MCD setup-codeartifact auth step, since the new action authenticates for itself. Pinned to workflows#97's branch head SHA for now, since it hasn't merged yet; will re-pin to main once it does. Publishing is now two `uses:` steps (one per account) with continue-on-error plus an explicit failure check after, replicating the previous both-accounts-attempted-even-if-one-fails behavior without a run: step wrapping two external actions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
Matches the repo's existing convention for OvertureMaps/workflows actions (sync-issue-type-and-scope.yml, schema-pr-preview.yml): setup-codeartifact is already merged there, so track @main with a zizmor: ignore[unpinned-uses] comment instead of a SHA pin. uv-publish-to-codeartifact stays SHA-pinned since workflows#97 hasn't merged yet. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
…blishes Splits the publish matrix job into build/publish-legacy/publish-mcd jobs connected by upload-artifact/download-artifact, so are-we-good can aggregate the two publish jobs' results the way it's designed to (needs + toJSON(needs)) instead of us hand-rolling a step-outcome check. Signed-off-by: John McCall <john@overturemaps.org>
zizmor 1.30.0 (introduced the self-repository audit) is what the org's required Overture Security Checks workflow now runs, and it flags every workspace-relative './...' uses: clause repo-wide, not just anything touched by this PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
335f0b6 to
380271e
Compare
OvertureMaps/workflows#97 merged, so the dual-publish steps can track main instead of the PR's branch head SHA, same as setup-codeartifact. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
Alex Iannicelli (atiannicelli)
left a comment
There was a problem hiding this comment.
k
There was a problem hiding this comment.
🟡 Changes recommended
Multiple workflows/actions use uses: $/.github/..., which is not valid GitHub Actions syntax for local references and will prevent the workflows from parsing/running.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the release/publish GitHub Actions workflows to support dual-publishing Python packages to both the legacy and MCD AWS CodeArtifact accounts during migration, while also refactoring the main publish flow into build + independent publish jobs.
Changes:
- Split
main-publish.yamlintobuild,publish-legacy,publish-mcd, andare-we-goodjobs, and switch publishing to the shareduv-publish-to-codeartifactaction. - Migrate the version-collision check workflow from the local CodeArtifact composite action to
setup-codeartifact, and delete the local.github/actions/code-artifactaction. - Replace local
uses: ./.github/...references withuses: $/.github/...repo-wide.
File summaries
| File | Description |
|---|---|
| .github/workflows/main-publish.yaml | Refactors main publish into build + dual publish jobs and adds an aggregator check. |
| .github/workflows/reusable-check-python-package-versions.yaml | Switches CodeArtifact auth to setup-codeartifact for version existence checks. |
| .github/workflows/check-python-package-versions.yaml | Updates caller to use the reusable workflow via the new local uses: syntax. |
| .github/workflows/release-trigger.yaml | Updates local composite action references to the new local uses: syntax. |
| .github/actions/detect-version-bumps/action.yml | Updates nested local action reference to the new local uses: syntax. |
| .github/actions/detect-affected-packages/action.yml | Updates nested local action reference to the new local uses: syntax. |
| .github/actions/compute-version/action.yml | Updates documentation to reference setup-codeartifact outputs for index URL. |
| .github/actions/code-artifact/action.yml | Deletes the legacy local CodeArtifact credential helper action. |
Review details
Suppressed comments (3)
.github/workflows/main-publish.yaml:123
uses: $/...is not valid GitHub Actions syntax for local actions/workflows; local references must use a relative path like./.github/.... As written, this workflow will fail to parse/run.
- name: Compute version
id: compute
uses: $/.github/actions/compute-version
with:
package: ${{ matrix.package }}
.github/workflows/release-trigger.yaml:122
uses: $/...is not valid GitHub Actions syntax for local actions/workflows; local references must use a relative path like./.github/.... As written, this workflow will fail to parse/run.
- name: Create release
uses: $/.github/actions/create-package-release
with:
package: ${{ matrix.package }}
.github/workflows/main-publish.yaml:197
- Because this job
needs: build, it will be skipped entirely if any matrix leg inbuildfails (a matrix job's overall result is failure when any leg fails). That defeats the intent of letting one package failure not block publishing other packages. Consider making this job run regardless ofbuild's conclusion so successful legs can still publish.
name: Publish ${{ matrix.package }} to MCD CodeArtifact
needs: [detect, build]
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
- Files reviewed: 8/8 changed files
- Comments generated: 7
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
needs: build alone skips publish-legacy/publish-mcd entirely whenever any package in the build matrix fails, since a matrix job's overall result is failure if any leg failed. (success() || failure()) lets them run anyway; the per-package download-artifact step then fails just the legs whose build didn't produce an artifact, leaving the rest to publish. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
The native per-workflow job check ("Are we good?") would collide with
any other workflow using the same job name for the same pattern.
create-check-run registers a standalone check named "Main publish /
are-we-good" via the Checks API instead.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: John McCall <john@overturemaps.org>
Alex Iannicelli (atiannicelli)
left a comment
There was a problem hiding this comment.
lgtm
Moved from the old inline publish step to run once in build, right after uv build, instead of being duplicated per publish-legacy/ publish-mcd account. Catches a stamp/build version mismatch before either CodeArtifact publish, not just a generic dist/* glob miss. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
|
Copilot resolve the merge conflicts in this pull request |
…-codeartifact-accounts # Conflicts: # .github/workflows/main-publish.yaml # .github/workflows/reusable-check-python-package-versions.yaml Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>
Merged |
Context
Part of OvertureMaps/ops-team#466:
schema,tf-data-platform, andplaces-quality-modelneed to dual-publishoverture-pypipackages into both the legacy CodeArtifact account (505071440022) and the new MCD account (763944545891). OvertureMaps/omf-core-data-opentofu#91 already mirrors the domain into MCD and grants this repo acodeartifact-pypi-publish-oidc-overturemapsrole there. Also supports #510 (Phase 4,vnextdev builds to CodeArtifact), which is blocked on the same underlying CodeArtifact migration and will need the same dual-publish wiring.Change
main-publish.yaml'spublishjob splits intobuildand two independentpublish-legacy/publish-mcdjobs, each callingOvertureMaps/workflows/.github/actions/uv-publish-to-codeartifact(OvertureMaps/workflows#97) against its own account. Neither blocks the other, and one package failing to build doesn't skip publishing the rest (if: success() || failure()).buildnow uploads the wheel/sdist as an artifact for the publish jobs to download, and keeps the old inline check that the built filenames match the versioncompute-versioncomputed, catching a stamp/build mismatch once instead of duplicating it per account.A final
are-we-goodjob (lowlydba/are-we-good) rolls both publish jobs into one required check,Main publish / are-we-good(create-check-run, to avoid colliding with other workflows' same-named jobs).Version computation still queries the legacy account only, via
setup-codeartifact(OvertureMaps/workflows#80). Both shared actions are pinned to@main.Deletes the local
.github/actions/code-artifactaction (its other caller,reusable-check-python-package-versions.yaml, is migrated tosetup-codeartifacttoo), and fixes everyuses: ./...ref repo-wide touses: $/..., sincezizmor1.30.0 now flags the old syntax.Note
Branch protection tracking
publishtoday should switch toMain publish / are-we-good.Testing
The
pull_requestpath exercisesbuild's legacy read-only auth, version query, and filename check; the publish jobs andare-we-goodare skipped there, same as before.zizmoris clean. The real dual-publish can only be verified once merged tomain.Fixes #725