Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 121 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,16 @@ on:
pull_request:
branches:
- main
# `labeled` lets reviewers opt a PR into integration / GPU coverage by
# adding `run-integration` or `run-gpu` to the PR. Per-job `if:` checks
# the label set, so unrelated jobs still skip.
types: [opened, synchronize, reopened, labeled]
push:
branches:
- main
- staging
- develop
tags:
- 'v*'
workflow_dispatch: # lets you run it manually from the Actions tab

concurrency:
Expand Down Expand Up @@ -58,12 +63,28 @@ jobs:
run: uv run pytest tests/x_vec/test_paillier_encryption.py

# ── Integration tests (require XTrace API credentials) ──────────
# Triggers:
# - PRs: only when explicitly labeled `run-integration` (avoids quota
# burn on every PR).
# - Push: main / staging / v* tags (catch regressions before release).
# - workflow_dispatch: manual.
# Fork PRs skip cleanly because GitHub does not pass secrets to forks —
# the test modules' `_REQUIRED` env-var check sees the empty values and
# skips. No global kill-switch variable needed.
integration:
name: Integration tests (meta search + hamming)
runs-on: ubuntu-latest
timeout-minutes: 30
needs: qa # only run if qa passes
if: vars.XTRACE_INTEGRATION_TESTS == 'true' # opt-in via repo variable
if: >-
github.event_name == 'workflow_dispatch'
|| (github.event_name == 'push' && (
github.ref == 'refs/heads/main'
|| github.ref == 'refs/heads/staging'
|| startsWith(github.ref, 'refs/tags/v')
))
|| (github.event_name == 'pull_request'
&& contains(github.event.pull_request.labels.*.name, 'run-integration'))

steps:
- name: Check out repository
Expand All @@ -82,14 +103,106 @@ jobs:

- name: Pytest (test_meta_search)
env:
XTRACE_API_KEY: ${{ secrets.XTRACE_API_KEY }}
XTRACE_ADMIN_KEY: ${{ secrets.XTRACE_ADMIN_KEY }}
XTRACE_ORG_ID: ${{ secrets.XTRACE_ORG_ID }}
XTRACE_API_KEY: ${{ secrets.XTRACE_API_KEY }}
XTRACE_ADMIN_KEY: ${{ secrets.XTRACE_ADMIN_KEY }}
XTRACE_ORG_ID: ${{ secrets.XTRACE_ORG_ID }}
XTRACE_API_URL: ${{ secrets.XTRACE_API_URL }}
XTRACE_ADMIN_API_URL: ${{ secrets.XTRACE_ADMIN_API_URL }}
run: uv run pytest tests/x_vec/test_meta_search.py -v

- name: Pytest (test_hamming_search)
env:
XTRACE_API_KEY: ${{ secrets.XTRACE_API_KEY }}
XTRACE_ADMIN_KEY: ${{ secrets.XTRACE_ADMIN_KEY }}
XTRACE_ORG_ID: ${{ secrets.XTRACE_ORG_ID }}
XTRACE_API_KEY: ${{ secrets.XTRACE_API_KEY }}
XTRACE_ADMIN_KEY: ${{ secrets.XTRACE_ADMIN_KEY }}
XTRACE_ORG_ID: ${{ secrets.XTRACE_ORG_ID }}
XTRACE_API_URL: ${{ secrets.XTRACE_API_URL }}
XTRACE_ADMIN_API_URL: ${{ secrets.XTRACE_ADMIN_API_URL }}
run: uv run pytest tests/x_vec/test_hamming_search.py -v

# ── Build GPU extensions on a free runner (Docker + nvcc, no GPU needed) ─
# The build script only needs Docker — nvcc runs in a pinned nvidia/cuda
# devel image. Splitting the build off the GPU runner keeps the paid GPU
# minutes for actual test execution. Same gating as gpu-tests so we only
# spend time building when the gpu-tests job is going to consume the .so.
build-gpu-extensions:
name: Build GPU extensions (.so via Docker)
runs-on: ubuntu-latest
timeout-minutes: 30
needs: qa
if: >-
github.event_name == 'workflow_dispatch'
|| (github.event_name == 'push' && (
github.ref == 'refs/heads/main'
|| github.ref == 'refs/heads/staging'
|| startsWith(github.ref, 'refs/tags/v')
))
|| (github.event_name == 'pull_request'
&& contains(github.event.pull_request.labels.*.name, 'run-gpu'))

steps:
- name: Check out repository
uses: actions/checkout@v4

- name: Build GPU extensions
run: ./build_gpu_binaries.sh

- name: Upload .so artifacts
uses: actions/upload-artifact@v4
with:
name: gpu-extensions
path: |
src/xtrace_sdk/x_vec/crypto/paillier_gpu_ext/*.so
src/xtrace_sdk/x_vec/crypto/paillier_lookup_gpu_ext/*.so
if-no-files-found: error
retention-days: 7

# ── GPU tests (run on GitHub-managed partner GPU runner; paid minutes) ─
# Triggers:
# - PRs: only when explicitly labeled `run-gpu` (paid runner — opt in).
# - Push: main / staging / v* tags.
# - workflow_dispatch: manual.
# Requires `gpu` environment with required reviewers; runner label is
# `gpu-runner`. Compiled .so files are produced by the build-gpu-extensions
# job above and downloaded into place before pytest.
gpu-tests:
name: GPU tests (paillier client)
runs-on: gpu-runner
timeout-minutes: 30
needs: build-gpu-extensions
if: >-
github.event_name == 'workflow_dispatch'
|| (github.event_name == 'push' && (
github.ref == 'refs/heads/main'
|| github.ref == 'refs/heads/staging'
|| startsWith(github.ref, 'refs/tags/v')
))
|| (github.event_name == 'pull_request'
&& contains(github.event.pull_request.labels.*.name, 'run-gpu'))
environment: gpu
concurrency:
group: gpu-tests
cancel-in-progress: true

steps:
- name: Check out repository
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Set up uv
uses: astral-sh/setup-uv@v4

- name: Install package + all dev dependencies
run: uv sync --all-groups

- name: Download GPU extension artifacts
uses: actions/download-artifact@v4
with:
name: gpu-extensions
path: src/xtrace_sdk/x_vec/crypto/

- name: Pytest (test_GPU_paillier_clients)
run: uv run pytest tests/x_vec/test_GPU_paillier_clients.py -v
110 changes: 96 additions & 14 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,75 @@ jobs:
- name: Pytest (paillier encryption)
run: uv run pytest tests/x_vec/test_paillier_encryption.py

build:
name: Build sdist + wheel
# ── Build matrix wheel: one cp3X wheel per supported Python ───────────
# Builds the GPU .so inside an Ubuntu 20.04 CUDA container, then `uv build`
# bundles it, and `auditwheel repair` retags as manylinux_2_31_x86_64 so
# PyPI accepts it and pip can pick it up on glibc>=2.31 (Ubuntu 20.04+,
# RHEL 8.4+, Debian 11+).
build-wheel:
name: Build wheel (py${{ matrix.python_version }})
runs-on: ubuntu-latest
timeout-minutes: 30
needs: qa
strategy:
fail-fast: false
matrix:
include:
- python_version: "3.11"
python_full_version: "3.11.15"
- python_version: "3.12"
python_full_version: "3.12.13"
- python_version: "3.13"
python_full_version: "3.13.13"
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true

- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python_version }}

- uses: astral-sh/setup-uv@v4
with:
enable-cache: true

- name: Build GPU .so (Ubuntu 20.04 / glibc 2.31 baseline)
env:
PYTHON_VERSION: ${{ matrix.python_version }}
PYTHON_FULL_VERSION: ${{ matrix.python_full_version }}
PYTHON_STANDALONE_TAG: "20260414"
CUDA_IMAGE: nvidia/cuda:12.4.1-devel-ubuntu20.04
run: ./build_gpu_binaries.sh

- name: Resolve version from git tag
run: |
if git describe --tags --exact-match HEAD 2>/dev/null; then
VER=$(git describe --tags --exact-match HEAD)
echo "SETUPTOOLS_SCM_PRETEND_VERSION=${VER#v}" >> "$GITHUB_ENV"
fi

- name: Build wheel
run: uv build --wheel

- name: Repair wheel (manylinux_2_31_x86_64)
run: |
uv pip install --system auditwheel patchelf
mkdir -p wheelhouse
auditwheel repair dist/*.whl --plat manylinux_2_31_x86_64 -w wheelhouse
rm dist/*.whl
mv wheelhouse/*.whl dist/
ls -lh dist/

- uses: actions/upload-artifact@v4
with:
name: dist-wheel-py${{ matrix.python_version }}
path: dist/*.whl

# ── Build sdist once (no Python ABI dependency) ───────────────────────
build-sdist:
name: Build sdist
runs-on: ubuntu-latest
needs: qa
steps:
Expand All @@ -73,27 +140,32 @@ jobs:
echo "SETUPTOOLS_SCM_PRETEND_VERSION=${VER#v}" >> "$GITHUB_ENV"
fi

- name: Build artifacts
run: uv build
- name: Build sdist
run: uv build --sdist

- uses: actions/upload-artifact@v4
with:
name: dist-artifacts
path: dist
name: dist-sdist
path: dist/*.tar.gz

# ── Verify each repaired wheel installs and imports on its target ABI ─
verify:
name: Verify wheel is installable
name: Verify wheel (py${{ matrix.python_version }})
runs-on: ubuntu-latest
needs: build
needs: build-wheel
strategy:
fail-fast: false
matrix:
python_version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/download-artifact@v4
with:
name: dist-artifacts
name: dist-wheel-py${{ matrix.python_version }}
path: dist

- uses: actions/setup-python@v5
with:
python-version: "3.12"
python-version: ${{ matrix.python_version }}

- uses: astral-sh/setup-uv@v4

Expand All @@ -102,18 +174,23 @@ jobs:
uv pip install --system dist/*.whl
python -c "import xtrace_sdk; print('Imported:', xtrace_sdk.__name__)"

# ── Publish to TestPyPI on staging branch pushes ──────────────────────
publish-testpypi:
name: Publish to TestPyPI
runs-on: ubuntu-latest
needs: verify
needs: [verify, build-sdist]
if: github.ref == 'refs/heads/staging'
permissions:
id-token: write
steps:
- uses: actions/download-artifact@v4
with:
name: dist-artifacts
pattern: dist-*
path: dist
merge-multiple: true

- name: List artifacts
run: ls -lh dist/

- name: Publish (TestPyPI via OIDC)
uses: pypa/gh-action-pypi-publish@release/v1
Expand All @@ -123,18 +200,23 @@ jobs:
packages-dir: dist
skip-existing: true

# ── Publish to PyPI on tag pushes (v*) ────────────────────────────────
publish-pypi:
name: Publish to PyPI
runs-on: ubuntu-latest
needs: verify
needs: [verify, build-sdist]
if: startsWith(github.ref, 'refs/tags/v')
permissions:
id-token: write
steps:
- uses: actions/download-artifact@v4
with:
name: dist-artifacts
pattern: dist-*
path: dist
merge-multiple: true

- name: List artifacts
run: ls -lh dist/

- name: Publish (PyPI via OIDC)
uses: pypa/gh-action-pypi-publish@release/v1
Expand Down
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,31 @@
All notable changes to this project will be documented in this file.

## [Unreleased]

### Changed

- **Device selection moved from the `DEVICE` environment variable to a `device=` constructor keyword** on `PaillierClient`, `PaillierLookupClient`, `ExecutionContext.create`, `ExecutionContext.load_from_disk`, and `ExecutionContext.load_from_remote`. The new default is `device="auto"`: clients probe for the GPU extension at construction time and fall back to CPU silently when no GPU is present. If a GPU is detected on the host (`/dev/nvidia0`, `nvidia-smi`, or `CUDA_VISIBLE_DEVICES`) but the extension fails to load, a warning is emitted instead of a silent CPU fallback so misconfigured GPU hosts don't quietly run on CPU. `device="gpu"` raises if the extension is unavailable; `device="cpu"` skips probing entirely.
- The `DEVICE` environment variable is no longer read. Existing callers must migrate to the `device=` kwarg.

### Added

- `xtrace_sdk.x_vec.crypto.device.resolve_device` — shared device-resolution helper used by both Paillier clients and intended for future homomorphic clients with optional GPU backends.
- `test_cross_device_interop` — parametrised test covering CPU↔GPU portability for both `PaillierClient` and `PaillierLookupClient`. Verifies that contexts saved on one device load correctly on the other (hash equality), and that ciphertexts produced on either backend round-trip correctly through the other for both encryption and server-side homomorphic add.

## [0.2.0] - 2026-04-18

### Added

- **CUDA backend for Paillier and Paillier-Lookup homomorphic encryption.** The `.cu` sources are now in-tree under `src/xtrace_sdk/x_vec/crypto/paillier-GPU-{,lookup-}client/`, and the new `build_gpu_binaries.sh` script compiles both pybind11 extensions inside a pinned `nvidia/cuda:12.4.0-devel-ubuntu22.04` Docker container. No local CUDA toolkit is required on the build host — only Docker and `nvcc` (provided by the image). Runtime requires NVIDIA driver ≥ 550. Opt-in via `DEVICE=gpu`; CPU remains the default.
- `PaillierGPUClient` and `PaillierLookupGPUClient` — standalone GPU-backed Hamming clients for `ExecutionContext`. Type dispatch in `ExecutionContext.create` now accepts these as first-class alternatives to their CPU counterparts.
- Documented `DEVICE` environment variable in the configuration reference.
- "GPU acceleration" section in the installation docs covering build script usage, runtime requirements, and per-build knobs (`PYTHON_VERSION`, `SMS`, `KEY_BITS`, `ALPHA_LEN`, `CUDA_IMAGE`).

### Changed

- `DEVICE` is now resolved at client instantiation time rather than at module import time. Previously, setting `os.environ["DEVICE"]` after importing a client (common in notebooks) had no effect; the module-level constant was already frozen to `cpu`. The env var is now read fresh inside `PaillierClient.__init__`, `PaillierLookupClient.__init__`, and `PaillierLookupClient.__setstate__`, so CPU and GPU clients can be instantiated in the same process and ordering of imports no longer matters.
- Error messages raised when the GPU extension cannot be loaded now point users at `./build_gpu_binaries.sh` and document runtime driver requirements, replacing the previous stale reference to `src/crypto/` and non-existent documentation.

## [0.1.1] - 2026-03-31

### Fixed
Expand Down
Loading