diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml index 93c602a..bf58610 100644 --- a/.github/workflows/python-publish.yml +++ b/.github/workflows/python-publish.yml @@ -6,7 +6,9 @@ # every wheel before publishing, so it needs no separate dry run; see RELEASE.md. # A published GitHub release (tag vX.Y.Z) publishes to PyPI after every build and test # passes, through PyPI trusted publishing and the protected `pypi` environment; so does -# a manual run with mode=release. mode=dry_run builds and tests only. +# a manual run with mode=release. mode=dry_run builds and tests only. Once PyPI has the +# release, the crates go to crates.io through crates.io trusted publishing and the +# protected `crates-io` environment. name: Python Publish on: @@ -212,3 +214,37 @@ jobs: - name: Publish to PyPI run: uv publish --trusted-publishing always dist/* + + publish-crates: + # After PyPI, so the two registries never disagree. If only this job fails, rerun it on + # its own; the wheels don't need rebuilding. The crates.io trusted publishers for both + # crates are registered for this workflow and environment `crates-io`. + needs: publish + runs-on: ubuntu-24.04 + timeout-minutes: 60 + environment: + name: crates-io + url: https://crates.io/crates/lance-csr + permissions: + id-token: write + contents: read + steps: + - uses: actions/checkout@v4 + + - name: Install build dependencies + run: | + sudo apt-get update + sudo apt-get install -y protobuf-compiler + + # Verifying compiles lance, so do it before getting the token, which lasts 30 minutes. + # Cargo packages lance-csr-catalog first, since lance-csr depends on it. + - name: Package and verify the crates + run: cargo package --locked -p lance-csr-catalog -p lance-csr + + - uses: rust-lang/crates-io-auth-action@v1 + id: auth + + - name: Publish to crates.io + run: cargo publish --locked --no-verify -p lance-csr-catalog -p lance-csr + env: + CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }} diff --git a/AGENTS.md b/AGENTS.md index e315629..98629e6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,5 +31,5 @@ - Follow the existing history style (`feat(graph):`, `docs:`, `refactor(query):`), using imperative, ≤72-character subjects. - Reference issues or discussions when relevant and include brief context in the body. - PRs should describe scope, list test commands run, mention benchmark deltas when applicable, and highlight impacts on bindings or examples. -- To cut a release, follow `RELEASE.md`: bump the version and lockfiles, publish a GitHub release (which builds, tests and publishes to PyPI in one run, with no separate dry run), then publish the crates by hand once PyPI has the new version. +- To cut a release, follow `RELEASE.md`: bump the version and lockfiles, publish a GitHub release, which builds, tests and publishes to PyPI and then crates.io in one run, with no separate dry run. - Before a PR claims a code change is faster, or doesn't slow anything down, follow `CONTRIBUTING.md`: explain the problem and why the change is needed, and measure the branch against `main` on both benchmark suites with its harness (release wheels loaded with `uv run --with`, both `memory` and `lazy` modes). Never edit benchmark queries, expected answers or runner settings, and report every query. diff --git a/RELEASE.md b/RELEASE.md index a4a983d..96aab8a 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -2,9 +2,9 @@ A release publishes the Python package to [PyPI](https://pypi.org/project/lance-csr/) and the `lance-csr-catalog` and `lance-csr` crates to [crates.io](https://crates.io/crates/lance-csr). PyPI publishing is -automated from a GitHub release. crates.io publishing is done by hand, afterwards. +automated from a GitHub release, and the crates follow once PyPI has the new version. -The whole release takes about 40 minutes, almost all of it the macOS wheel build. Don't add steps that build the +The whole release takes about 50 minutes: the macOS wheel build, then the crates. Don't add steps that build the release wheels a second time. ## Before you start @@ -68,10 +68,13 @@ Publishing it starts `.github/workflows/python-publish.yml`: | `build-macos` | about 35 min | macOS arm64 wheel, built natively on `macos-15` | | `test` | under 1 min each | each wheel on Python 3.12, 3.13 and 3.14 | | `publish` | seconds | checks the tag matches `python/pyproject.toml`, then uploads through PyPI trusted publishing | +| `publish-crates` | about 10 to 15 min | after `publish`: packages and verifies `lance-csr-catalog` and `lance-csr`, then uploads them through crates.io trusted publishing | -The build jobs run in parallel. Nothing reaches PyPI unless every build and test job passes. The `pypi` -environment only accepts `v*` tags, and the PyPI trusted publisher is registered for workflow -`python-publish.yml` and environment `pypi`, so don't rename either. +The build jobs run in parallel. Nothing reaches PyPI unless every build and test job passes, and nothing +reaches crates.io unless PyPI publishing succeeds. The `pypi` and `crates-io` environments only accept `v*` +tags. The PyPI trusted publisher is registered for workflow `python-publish.yml` and environment `pypi`, and +the crates.io trusted publishers for both crates for the same workflow and environment `crates-io`, so don't +rename any of them. **Don't run a separate `mode=dry_run` before a release.** The release run builds and tests the same wheels before it publishes, so a dry run only doubles the wait. Changes to how wheels are built (this workflow, @@ -79,26 +82,25 @@ before it publishes, so a dry run only doubles the wait. Changes to how wheels a ### If the release run fails -Nothing has been published, so nothing needs undoing on PyPI. +If a build or test job failed, nothing has been published, so nothing needs undoing on PyPI. - **A flaky test or a runner problem** (for example a GitHub runner image outage): rerun the failed jobs with `gh run rerun --failed`. - **A real bug:** fix it on `main` through a normal PR, then release the next patch version. Don't move or reuse a tag that has already been released. -## 4. Publish the crates +### If only `publish-crates` fails -Only after PyPI shows the new version (`https://pypi.org/pypi/lance-csr/json`), so the two registries never -disagree. A maintainer with a crates.io token runs this from the merged `main`: +PyPI already has the release, so don't release again. Rerun just that job with `gh run rerun --failed`; +the wheels aren't rebuilt. -```bash -cargo publish -p lance-csr-catalog -p lance-csr -``` - -Cargo publishes `lance-csr-catalog` first, since `lance-csr` depends on it. `lance-csr-python` and -`lance-csr-benches` are `publish = false`. +Cargo publishes `lance-csr-catalog` first, since `lance-csr` depends on it. If it published the catalog and +then failed on `lance-csr`, a rerun fails because the catalog version already exists. Publish the rest by hand +from the release tag: a crate owner runs `cargo publish -p lance-csr`. If **Require trusted publishing** is on +for the crate, crates.io rejects API tokens, so turn it off for the manual publish and back on afterwards. +`lance-csr-python` and `lance-csr-benches` are `publish = false`. -## 5. Check the release +## 4. Check the release ```bash uv run --no-project --with lance-csr==A.B.C python -c "import lance_csr, importlib.metadata as m; print(m.version('lance-csr'))"