package: Add missing openssl and decompression utilities to KVM agent dependencies by weizhouapache · Pull Request #14307 · apache/cloudstack

weizhouapache · 2026-10-05T08:58:33Z

Description

Add bzip2, gzip, unzip, and openssl as explicit dependencies for the KVM agent. These utilities are required for:

Template decompression (bunzip2 for .bz2 files, gzip for .gz, unzip for .zip)
SSL/TLS operations
Direct download template handling on Ceph/RBD storage

Fixes direct download failures when template files are compressed (e.g. .qcow2.bz2).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

weizhouapache · 2026-10-05T08:58:48Z

@blueorangutan package

codecov · 2026-10-05T09:08:05Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 20.21%. Comparing base (f03cffc) to head (265fe9d).

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #14307   +/-   ##
=========================================
  Coverage     20.21%   20.21%           
- Complexity    20747    20750    +3     
=========================================
  Files          6426     6426           
  Lines        580181   580181           
  Branches      71033    71033           
=========================================
+ Hits         117282   117288    +6     
+ Misses       450184   450180    -4     
+ Partials      12715    12713    -2

Flag	Coverage Δ
uitests	`3.69% <ø> (ø)`
unittests	`21.51% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Match the KVM agent dependencies added in apache/cloudstack#14307: openssl, bzip2, gzip and unzip are needed for template decompression (.bz2/.gz/.zip) and direct download handling. Install them via apt on Debian/Ubuntu and dnf on CentOS 8/EL9.

… dependencies Add bzip2, gzip, unzip, and openssl as explicit dependencies for the KVM agent. These utilities are required for: - Template decompression (bunzip2 for .bz2 files, gzip for .gz, unzip for .zip) - SSL/TLS operations - Direct download template handling on Ceph/RBD storage Fixes direct download failures when template files are compressed (e.g. .qcow2.bz2).

weizhouapache · 2026-10-08T19:26:44Z

@blueorangutan package

blueorangutan · 2026-10-08T19:28:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot

🟡 Changes recommended

The agent still lacks explicit file and xz dependencies required by its supported extraction path.

3 open findings

Declare Debian file and xz-utils runtime packages · New
Declare runtime dependencies for file and xz executables · New
Declare runtime dependencies for file and xz executables · New

What changed in this PR

Adds explicit SSL and decompression runtime dependencies to KVM agent packages.

Changes:

Adds OpenSSL, bzip2, gzip, and unzip dependencies across supported distributions.

File	Description
`packaging/suse15/cloud.spec`	Updates SUSE agent dependencies.
`packaging/el8/cloud.spec`	Updates EL8 agent dependencies.
`debian/control`	Updates Debian agent dependencies.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

 Package: cloudstack-agent
 Architecture: all
-Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat
+Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat, openssl, bzip2, gzip, unzip


+Requires: openssl
+Requires: bzip2
+Requires: gzip
+Requires: unzip


+Requires: openssl
+Requires: bzip2
+Requires: gzip
+Requires: unzip


blueorangutan · 2026-10-08T20:17:40Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19489

Damans227 · 2026-10-09T18:54:36Z

+Requires: openssl
+Requires: bzip2
+Requires: gzip
+Requires: unzip


should we add xz here too? the direct download code also unpacks .xz templates, so those would still fail

Damans227 · 2026-10-09T18:54:37Z

 Package: cloudstack-agent
 Architecture: all
-Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat
+Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat, openssl, bzip2, gzip, unzip


does the agent also need the file package? it uses that to tell if a downloaded template is compressed at all

…ide NFS and iSCSI (#237) * Add Ceph (RBD) as a primary storage option alongside NFS and iSCSI Ceph nodes are provisioned out-of-band by a chained "ceph-deployer" Jenkins job; this adds the Ansible side: a cloudstack-cephstorage role that carves two 30GB RBD pools (with per-pool cephx credentials) out of the cluster, registers them with CloudStack via cloudmonkey, and points the generated Marvin config at the RBD pools instead of NFS. NFS mount points are still created but no longer added to CloudStack when env_priprot=ceph. * Fix Jenkins failure: Ceph group_vars templating relied on vars only in group_vars/all group_vars/all is gitignored, so the Jenkins box's copy predates the new def_env_ceph_*/def_pri_password_ceph defaults. Referencing an undefined def_ var inside default() aborts the whole Jinja render, leaving a stale unrendered group_vars file and tripping the "curly braces found" check. Chain a literal fallback so these vars work even before the production group_vars/all is updated. Also guard the new primary_storage_hosts_ceph inventory group so it doesn't emit blank-hostname entries on non-Ceph runs where no Ceph IPs are supplied. * Ceph primary storage: SSH to Ceph nodes with the same password as other storage hosts The cloudstack-cephstorage role was failing SSH auth against the Ceph nodes because pri_password_ceph fell back to an empty-string default (def_pri_password_ceph isn't set on existing group_vars/all files). Drop the separate Ceph-only SSH credential and just reuse pri_password/def_pri_password, matching how the other primary storage hosts authenticate. * Fix Ceph RBD storage pool URL using wrong cephx username addpools.yml creates a cephx identity scoped to each pool (client.<poolname>), but the rbd:// URLs built for cloudmonkey and the Marvin config hardcoded env_ceph_username ("admin") as the username while supplying that pool-scoped key - a cephx identity/key mismatch that would fail authentication regardless of any client library version issue. Use the pool name as the username, matching the identity addpools.yml actually created, and drop the now-unused env_ceph_username/def_env_ceph_username plumbing. * Use pri to size the number of Ceph RBD pools instead of env_ceph_poolcount pri already controls how many primary storage pools get created for NFS/iSCSI/ESXi/XenServer/KVM - reuse it for Ceph too instead of a separate env_ceph_poolcount knob, so one extra-var consistently controls pool count across all primary storage protocols. * Fix Marvin test_data.py: internal nfs/nfs2 pools always use NFS, not env_priprot marvin_pri1/nfs2 are Marvin's own internal test primary storage pools - plain NFS directories always created on env_prihost by cloudstack-nfsshares regardless of the zone's primary storage protocol. The template interpolated env_priprot for their URL scheme, so with env_priprot=ceph the generated test_data.py claimed "Ceph://..." for directories that are actually NFS exports. Hardcode nfs:// for these two. * Install ceph-common on KVM hosts for Ceph RBD volume encryption support CloudStack's RBD volume encryption on KVM needs the rbd CLI/config from ceph-common on the hypervisor. Install it via the generic package module (works across all supported distros) when env_priprot=ceph. * marvin: fetch ipmisim.py fix from GitHub to patch installed package Replace the old hardcoded wget-based fix (which assumed a fixed python3.6 site-packages path) with a pip-show-based lookup of the installed ipmisim package location, then fetch the updated ipmisim.py directly via get_url from shapeblue/ipmisim@a8e56bd (tip of shapeblue/ipmisim#18), so it works regardless of Python version/site-packages layout and doesn't need to be vendored in this repo. * Move ceph-common install into per-distro KVM task files The generic package: module task in roles/kvm/tasks/main.yml wasn't valid there. Install ceph-common via the distro's own module instead: apt in ubuntu.yml/debian.yml, dnf in centos8.yml and el9.yml, each gated on env_priprot=ceph. * Do not percent-encode the Ceph RBD secret in storage pool URLs Traced both the cloudstack-cloudmonkey and apache/cloudstack source: cloudmonkey's encodeRequestParams() already percent-encodes the whole url= value exactly once via Go's url.QueryEscape before sending the API request, and CloudStack's UriUtils.getRbdUrlInfo() expects to receive the raw literal secret (it does its own "+"->"-" and "/"->"_" substitution before handing it to a lenient base64 decoder that accepts either alphabet). Pre-encoding the secret ourselves with Jinja's urlencode double-encoded it, leaving literal %2B/%2F/%3D text embedded in the credentials instead of the real characters. Pass the raw secret as-is. * Add SharedMountPoint primary storage support (env_priprot=sharedmountpoint) NFS shares are created on the primary storage host as for NFS, but are not registered as NFS pools. The new cloudstack-sharedmountpoint role mounts each KVM share on every KVM host at the same local path (env_prismp_path, default /mnt/sharedmp/), and the pools are added to CloudStack and the Marvin config as SharedMountPoint primary storage. Teardown unmounts them before the NFS shares are removed. * Install openssl and decompression utilities on KVM hosts Match the KVM agent dependencies added in apache/cloudstack#14307: openssl, bzip2, gzip and unzip are needed for template decompression (.bz2/.gz/.zip) and direct download handling. Install them via apt on Debian/Ubuntu and dnf on CentOS 8/EL9. * removeproject: skip project deletion when no matching project is found cmk deleteproject id= cleanup=true fails with "Invalid value provided for API arg: id" when the jq lookup finds no project matching build_project (e.g. already removed in a prior partial run). Only run the delete when a projectid was actually found. * Drop removeprimary for Ceph and SharedMountPoint storage The Ceph cluster and its RBD pools are torn down by the separate ceph-deployer Jenkins job, so the cloudstack-cephstorage role (and the destroyvms.yml play using it) no longer needs a removeprimary action. SharedMountPoint only reuses the NFS shares, which are still removed by the cloudstack-nfsshares removeprimary step, so the cloudstack-sharedmountpoint role does not need one either.

NuxRo

LGTM

boring-cyborg Bot added component:packaging distro:ubuntu labels Oct 5, 2026

weizhouapache requested a review from NuxRo October 5, 2026 08:58

weizhouapache mentioned this pull request Oct 5, 2026

[4.22] keystore-setup script does not create agent.properties on fresh KVM hosts #13079

Open

weizhouapache force-pushed the 24-package-add-openssl branch from 7d69608 to 265fe9d Compare October 8, 2026 19:26

weizhouapache marked this pull request as ready for review October 8, 2026 19:26

Copilot AI balanced review requested due to automatic review settings October 8, 2026 19:26

Copilot started reviewing on behalf of weizhouapache October 8, 2026 19:27

View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Damans227 reviewed Oct 9, 2026

View reviewed changes

NuxRo approved these changes Oct 16, 2026

View reviewed changes

Navigation Menu

Repository navigation

package: Add missing openssl and decompression utilities to KVM agent dependencies - #14307

package: Add missing openssl and decompression utilities to KVM agent dependencies#14307
weizhouapache wants to merge 1 commit into
apache:mainfrom
weizhouapache:24-package-add-openssl

weizhouapache commented Oct 5, 2026

Uh oh!

weizhouapache commented Oct 5, 2026

Uh oh!

codecov Bot commented Oct 5, 2026 •

edited

Loading

Uh oh!

weizhouapache commented Oct 8, 2026

Uh oh!

blueorangutan commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

blueorangutan commented Oct 8, 2026

Uh oh!

Damans227 Oct 9, 2026

Uh oh!

Damans227 Oct 9, 2026

Uh oh!

NuxRo left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

weizhouapache commented Oct 5, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

weizhouapache commented Oct 5, 2026

Uh oh!

codecov Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Oct 8, 2026

Uh oh!

blueorangutan commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

blueorangutan commented Oct 8, 2026

Uh oh!

Damans227 Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

NuxRo left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Oct 5, 2026 •

edited

Loading