Skip to content

feat(cli): rework build / test environment handling #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mikonse wants to merge 1 commit into
base: main
Choose a base branch
from
+4,429 −1,001
Open

feat(cli): rework build / test environment handling #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,10 @@ jobs:
contents: read
# Push of the pre-built CI image is additionally gated to main below.
packages: write
env:
# Registry and Environment Host roots together, so the cached build
# tree stays registered under the same Environment id.
DEBMAGIC_DATA_DIR: /tmp/debmagic-data
steps:
- uses: actions/checkout@v5
- uses: Swatinem/rust-cache@v2
Expand All @@ -149,16 +153,18 @@ jobs:
- name: Cache the debmagic build tree
uses: actions/cache@v4
with:
path: /tmp/debmagic
key: debmagic-build-${{ hashFiles('Cargo.lock', 'debian/changelog', 'debian/control', 'debian/rules') }}
restore-keys: debmagic-build-
- name: Prune stale build trees from the restored cache
path: /tmp/debmagic-data
key: debmagic-data-${{ hashFiles('Cargo.lock', 'debian/changelog', 'debian/control', 'debian/rules') }}
restore-keys: debmagic-data-
- name: Prune Environments of other package versions from the restored cache
run: |
mkdir -p /tmp/debmagic
current="$(dpkg-parsechangelog -SSource)-$(dpkg-parsechangelog -SVersion)"
find /tmp/debmagic -mindepth 1 -maxdepth 1 -type d \
! -name "$current" ! -name "${current}-test" \
-exec rm -rf {} +
cargo run --locked -q -p debmagic -- env list \
| awk -v current="$current" 'NR > 1 && $4 != current { print $1 }' \
| while read -r id; do
cargo run --locked -q -p debmagic -- env clean "$id" \
|| echo "::warning::could not prune Environment $id"
done
- name: Resolve CI image tag
id: image
run: |
Expand Down
98 changes: 96 additions & 2 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ ignore = { version = ">=0.4.25" }
md-5 = { version = ">=0.11" }
sha1 = { version = ">=0.11" }
sha2 = { version = ">=0.11" }
rusqlite = { version = ">=0.32" }

# dev dependencies
test-case = { version = ">=3.3.1" }
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Modern, robust & easy [Debian](https://debian.org)/[Ubuntu](https://ubuntu.com)
Debmagic unifies the the packaging experience as a single, streamlined tool. Additionally, `debmagic-pkg` allows you to define package build recipes in Python.

- **Isolated builds without the setup**: `debmagic build binary` builds a `debian/`-packaged source tree in a temporary container (using LXD, Incus, Docker)
- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; `--shell-on-failure` and `debmagic shell` drop you right where the build broke
- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; a failed build prints `debmagic env shell <id>`
- **Test & sign integrated**: `debmagic test` runs the package's autopkgtest tests in a fresh environment, `debmagic sign` GPG-signs `.changes`/`.dsc`/`.buildinfo` on the host
- **Python packaging API**: replace complicated `debian/rules` Makefiles with typed Python `debian/rules.py`, with optional `dh` compatibility

Expand All @@ -30,7 +30,7 @@ uvx debmagic build binary --driver docker
| `debmagic build binary` | Build a binary package (`.deb`) in an isolated environment |
| `debmagic build source` | Create a source package (`.dsc`) for upload (incl signing) |
| `debmagic test` | Run the package's autopkgtest tests (`debian/tests/`) against a prior build |
| `debmagic shell` | Attach an interactive shell to the build environment |
| `debmagic env` | List, clean, or open a shell in a build or test environment |
| `debmagic sign` | GPG-sign a `.changes` file (and its `.dsc`/`.buildinfo`) on the host |
| `debmagic config` | Inspect and edit the effective `debmagic.toml` configuration |
| `debmagic check` | Lint the package *(in progress)* |
Expand Down
3 changes: 3 additions & 0 deletions debian/control
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ Build-Depends:
librust-md-5-dev (>= 0.11),
librust-sha1-dev (>= 0.11),
librust-sha2-dev (>= 0.11),
librust-rusqlite-dev (>= 0.32),
libsqlite3-dev,
pkg-config,
Rules-Requires-Root: no
X-Style: black
Standards-Version: 4.7.2
Expand Down
56 changes: 47 additions & 9 deletions debian/rules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,43 @@
)

# TODO: add this section to our rust debmagic module to make setting this up less annoying
os.environ.update({
"PATH": f"/usr/share/cargo/bin:{os.environ['PATH']}",
"CARGO": "/usr/share/cargo/bin/cargo",
"CARGO_HOME": f"{pkg.base_dir}/debian/cargo_home",
"CARGO_REGISTRY": f"{pkg.base_dir}/debian/cargo_registry",
"DEB_CARGO_CRATE": f"{pkg.build_env.DEB_SOURCE}_{pkg.build_env.DEB_VERSION_UPSTREAM}"
})
os.environ.update(
{
"PATH": f"/usr/share/cargo/bin:{os.environ['PATH']}",
"CARGO": "/usr/share/cargo/bin/cargo",
"CARGO_HOME": f"{pkg.base_dir}/debian/cargo_home",
"CARGO_REGISTRY": f"{pkg.base_dir}/debian/cargo_registry",
"DEB_CARGO_CRATE": f"{pkg.build_env.DEB_SOURCE}_{pkg.build_env.DEB_VERSION_UPSTREAM}",
# debmagic-pkg is setuptools and ignores this. The rewrite below is what
# makes resolution succeed; this only keeps maturin from enabling the
# (now empty) default feature.
"MATURIN_PEP517_ARGS": "--no-default-features",
}
)

# Cargo resolves every feature name in the table, including ones that are not
# enabled, so `--no-default-features` still requires rusqlite to advertise
# `bundled`. Debian's librust-rusqlite-dev deletes that feature and links the
# system libsqlite3. Drop the reference for this build, then put the manifest
# back so a host or incremental tree is not left edited.
cargo_toml = Path("packages/debmagic/Cargo.toml")
bundled_sqlite = 'bundled-sqlite = ["rusqlite/bundled"]'
system_sqlite = "bundled-sqlite = []"


def use_system_sqlite() -> str:

@TheJJ TheJJ Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uuh there must be a better way, since we only want to change a feature flag. maybe feature flags can be set somehow differentlly (env? cli?)

original = cargo_toml.read_text()
if bundled_sqlite in original:
cargo_toml.write_text(original.replace(bundled_sqlite, system_sqlite, 1))
return original
if system_sqlite in original:
restored = original.replace(system_sqlite, bundled_sqlite, 1)
return restored
raise SystemExit(
f"{cargo_toml}: expected {bundled_sqlite!r} so the Debian build can drop "
"the rusqlite bundled feature that librust-rusqlite-dev removes"
)


packages = {
"debmagic": ("packages/debmagic", "debmagic"),
Expand All @@ -33,10 +63,14 @@
cargo_lock = Path("Cargo.lock")
cargo_lock_saved = Path("Cargo.lock.saved")


def dh_auto(build: Build, stage: str, use_destdir: bool = False):
for pkg_name, (path, python_pkg_name) in packages.items():
destdir = f" --destdir debian/{pkg_name} " if use_destdir else ""
build.cmd(f"{stage} -p {pkg_name} --sourcedirectory {path} --buildsystem=pybuild {destdir} -- --name {python_pkg_name}")
build.cmd(
f"{stage} -p {pkg_name} --sourcedirectory {path} --buildsystem=pybuild {destdir} -- --name {python_pkg_name}"
)


@dhp.override
def dh_auto_configure(build: Build):
Expand Down Expand Up @@ -68,4 +102,8 @@ def dh_auto_clean(build: Build):
dh_auto(build, "dh_auto_clean")


pkg.pack()
cargo_toml_original = use_system_sqlite()
try:
pkg.pack()
finally:
cargo_toml.write_text(cargo_toml_original)
3 changes: 2 additions & 1 deletion docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Working with Debian packages without `debmagic` means juggling with several inde
Debmagic unifies this into one CLI with useful defaults:

- **Isolated builds without the setup**: `debmagic build binary` builds a `debian/`-packaged source tree in a temporary container (using LXD, Incus, Docker)
- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; `--shell-on-failure` and `debmagic shell` drop you right where the build broke
- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; a failed build prints `debmagic env shell <id>`

@TheJJ TheJJ Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can analyze failures with devmagic env shell

- **Test & sign integrated**: `debmagic test` runs the package's autopkgtest tests in a fresh environment, `debmagic sign` GPG-signs `.changes`/`.dsc`/`.buildinfo` on the host
- **Package uploading**: `debmagic upload` (to `debian`, `ubuntu`, `ppa:user/repo`) pushes signed builds to archives and PPAs via scp/sftp, with configurable pre-upload checks
- **Upstream tracking**: `debmagic upstream list`/`switch` replace uscan/uupdate, reading the package's existing `debian/watch` and `debian/copyright`
Expand Down Expand Up @@ -37,6 +37,7 @@ Then dive into the pages below — start with [Getting started](usage/getting-st
usage/getting-started.md
usage/build.md
usage/test.md
usage/env.md
usage/source.md
usage/upload.md
usage/upstream.md
Expand Down
Loading
Loading