Repository navigation
Conversation
… VM's disk is never attached or deleted Cloning a VM, or making a template of it, ran qemu-img convert without -f: a raw disk is the guest's to write, and a qcow2 header written there naming a host file had qemu-img, as root, read that file into the copy. The copy now names the format the definition gives (a template's disk is always qcow2, so a raw disk is copied, never moved). attach_disk and volume deletion only looked at the disks VMs name: the base image other VMs read through, a template's disk and the disks of a VM in the recycle bin could be attached read-write or deleted. They are refused. A template is made from a VM with one disk only (the others were left behind, and every deployment got its one disk under each of them); a move to another filesystem checks the room before the VM is undefined.
…; /host/network follows ?node= A peer ticket, and a task ticket bound to a request's task, named only the account: a delegated (scoped) API token sent with ?node= to another member, or a task it started, acted there with all of its owner's rights. The ticket now carries the token's perimeter, and the member puts it back (role delegated, its one role on its one perimeter). /host/network had no forwarding: the dashboard's ?node=B reached the node it was connected to, which read and rewrote its own interfaces as B's.
…e checks the image; a copy keeps its own backups - A remote's export named each backup's VM and time, and the sync used them as the directory it wrote into, as root: a hostile remote could write files anywhere on the node. A backup's time is now one directory name, its VM a UUID, its disks and files names in it; a bad entry is skipped, not the whole job. - A live restore read through the backup's image without the check every other read of a foreign image makes (a backing file or a data file outside the backup). - A store recorded as encrypted whose directory says it is not is refused: whoever can write the share could have every later backup written in clear. - A copy's backups are deleted only by its policy over what it holds; a source's entries dated in the future never rank, so a hostile source cannot push the copy's own backups out.
…; a changed plan acts with its editor's - A patch plan changed by someone else (a pool or a tag as its target, a reboot) kept its first author, and ran with that author's rights: whoever changes a plan now becomes its author. - A scheduled run checked vm.exec only when it was asked for: at its window, root updated and rebooted the guests even if the account had lost its rights, ended or been deleted. The rights are read again when the run opens (vm.exec), and before a reboot (vm.power); a plan's run acts for the plan's author.
…a group's members are the VMs it made - The freeze and keep protections were checked on the URLs about a VM only: a restore over it, a move to another site, and the deletions of VM groups and autoscaling went through. Deleting, restoring over and moving a VM check them. Autoscaling never picks a protected member. - A VM group's members were every VM named <prefix>-<n>: someone's VM named so was counted, and stopped and deleted when the count went down. A group records the VMs it makes, a group made before adopts those named so once, and a prefix an existing VM already has is refused. - Autoscaling picked a stopped member and asked it to shut down, which was refused: nothing was ever removed. A stopped member is drained as it is.
… status is known - The definition kept for recovery set every VM's machine to pc (i440fx): a q35 VM (every UEFI VM, Windows 11) could not start on another node. The machine's family is kept, its version left to the target; the CPU's topology too. - A member with no status at all counted as lost, and was powered off through its BMC at once: hyperlite-cfs restarting on the leader (no status could be read) or a leader that just took over did that to healthy members. A round where no status is read decides nothing, and a member without status is lost only once the leader waited for it as long as for any other.
…a waiting change keeps its timer - Each node applied the saved firewall only when its ruleset differed from what it had applied last: after a reboot the kernel's table is gone, nothing differed, and every port stayed open with no warning. What is applied is now also checked in the kernel. - The rollback timer of a change waiting for its confirmation does not survive a reboot: a change within its deadline was then never undone. At start-up it gets its timer back for the time left (host firewall and host network).
… creating a cluster works again The SSH helpers imported a function the job engine no longer has since its steps moved to hyperlite-daemon: every cluster creation waited ten minutes for SSH and failed, its VMs left behind (the tests patched the helper, hiding it). The API runs unprivileged and cannot read the automation key: the steps now go to two administrator operations of the daemon, job.vm_script (the join token on standard input, never on a command line) and job.vm_copy (only files of the Kubernetes downloads).
… pools and security keys hyperlite-daemon checks every operation itself, whatever the API says (docs/design/privilege-separation.md). Three of its rules were looser than the routes: - access.pool_member_add checked vm.allocate on the pool only: a pool holder could put anyone's VM in the pool and get their role on it. It now needs vm.allocate on the VM too, or that the caller just made it and it is in no pool. - What an account does to itself (password, second factors, API tokens, its Terminal sessions) accepted any proof: an API token, an administrator's included, could mint a lasting token or change the password without the current one. It needs a signed-in session; a token revokes only itself. - access.key_used took any key: any account could set another's security-key counter and refuse its next sign-in. It records only a key of the caller's account.
…s relay cannot be opened The relay and the tmux clients run outside the box, as the box's own uid. With no PID namespace, a shell in the box saw them, and could write into the relay's memory (/proc/<pid>/mem, ptrace allowed on such hosts) to use its way out of the box's network, to the loopback ports (the VMs' consoles among them). The box gets its own PID namespace (systemd 257 and later; checked on Debian 13: three processes visible), and the relay makes itself not dumpable.
…they belong; delegated tokens read no secret
- The assistant's API key, never shown again once saved, was sent to whatever address an administrator set next: a
new provider or address now needs the key again. A link-local endpoint (a metadata service) is refused, and an
endpoint's error body goes to the log, not to whoever asked.
- A tool's arguments, which the model may take from a VM's name or notes, went into API paths as they were
("x/identity"): a name or a node is one plain name, a kind one the tool offers.
- A delegated token could list and reveal the vault's secrets its owner may read: it reads none.
- A replicated row's column names went into the SQL as they came from the tree: one this node's table lacks fails the
apply, as SQLite itself would, without being written into the statement.
…oval rechecks; a deleted account leaves nothing - On-demand rights asked the TOTP code only of accounts with TOTP, with no limit on wrong codes: an account protected by a security key, or by its password alone, took rights with its session only. Without TOTP the password is asked (the directory's for an LDAP account), and five failures lock it for five minutes, as every re-authentication. - Approving a pending request did not check that its account still exists and is still eligible. - Deleting an account left its eligibilities, its vault read grants and its open requests to the next account created under that name.
…kup directory; a shared disk is not wiped - vm.attach_disk, vm.attach_interface and vm.set_interface checked vm.hardware only: the daemon now also needs storage.allocate on the volume's pool and network.assign on the network, as the routes do. - backup.run wrote into any directory an operator named, as root: another directory than the default one, or the one an administrator set in the VM's schedule, is an administrator's. - Deleting a ZFS pool ran wipefs on the whole disk of each of its partitions: a pool built on one partition took the disk's other partitions with it. A disk is wiped only when the pool held all of it.
…de's address; idle sessions end - Every session's X server took display :1 with an abstract socket, which lives in the host's network namespace: a second session's client drew on the first session's screen (or did not start). The X server listens on a file socket of the unit's own /tmp only (checked on Debian 13: two sessions side by side, no abstract socket). - The guest's address can be its agent's word: an address of this node is refused, so a session never connects to the host's own services. - A session whose viewer left (a closed tab, a restarted API) ran until its 8-hour limit: the display ends ten minutes after its last viewer, and the session with it.
…on names nothing of this host by a detour - For a VM arriving running, hyperlite-daemon handed the API an open connection to the root libvirt socket, and the migration's call filter ran in the API: an API not to be trusted held an unfiltered libvirt. The daemon now hands a socket whose other end goes through the filter, run in the daemon. - The destination definition's check let through a firmware path with .., a UEFI variables template, a console socket or render node, and a character device connecting to another VM's socket (its monitor). They are refused. - A prepare must name both channels as this incoming VM's sockets (without them libvirt listens on TCP for the migration's time); listen_address and disks_port are no longer accepted. - The incoming VM is defined only if no VM of its name or UUID appeared since it was prepared (libvirt would redefine it silently).
…ntact; hyperlite-admin never sends a ticket in clear - hyperlite login with a server already pinned met another certificate as it meets a first contact, and offered to trust it: one Enter replaced the pin. It now refuses, shows both fingerprints, and says how to trust a renewed one on purpose (hyperlite forget, then sign in again). - hyperlite-admin --server took any URL: the root ticket or the token went over plain HTTP to another host. https only, or http to this computer.
…revious version is kept; checks and runs guarded - Uploading or deleting an update file while an update ran removed the packages and lists it installed from (dpkg left half-configured): refused while an update runs. - An update from a file read the candidate version from the node's own stale lists, kept nothing, and left an older package as "the previous version" for the watchdog and the return: the candidate is read from the file's lists. - A rolling update started this node's own update without the guard /update/apply takes: one started by hand meanwhile raced it. The guard holds there too; and a start that fails no longer leaves the marker that takes the node out of maintenance at its next restart. - A check by someone who may view but not apply (its refresh refused) replaced the last good result with an error: only a check that reached the repository is remembered.
…s; a stopped publication can be resumed
- From 1.14 the node's keys live in hyperlite-cfs's database: after apt remove (not purge) the tree is unmounted, the
package's check found no key, and the reinstall generated a new secret key and SSH key that no longer matched the
other members'. A node whose data is there is installed again as an upgrade.
- A publication that stopped after its push to the mirror (the release, the ISO or the verification failed) could not
be run again ("already published"): the same version from the same commit resumes after the push.
…names are not written; inodes are indexed - Every lookup or stat through /etc/hyperlite listed the whole subtree of the path's parent to find it, on the event loop the cluster's messages share: any local account stat-ing names that do not exist could stall it. It is two point lookups (cfs_store_stat). - A tree entry named like the mount's own names at the root (local, .members, .quorum, .version) could not be read through the mount and was listed twice: such a name is refused when written, and never listed twice. - Finding a path's inode scanned every path the mount had seen: an open-addressing index makes it constant time. Checked with the C tests under ASan and UBSan (7/7) and GCC's analyzer (no warning).
…lones, pools and withdrawals - A task's detail carried its target's audit trail to any account: only to who may read the audit log. - The runs of guest scripts showed their parameters and outputs to any node.view holder: only to who started a run or may run commands in each of its VMs. - The Terminal's file download and upload checked the path in one process and used it in another: a link swapped in between led out of the home. The path is walked and the file opened in one process, never through a link. - A sign-in rule requiring a second factor did not hold for SSO sign-ins: it does. - Clones skipped the cluster's guardrails (name pattern, size, networks, required tags): they are checked. - A directory pool could sit on libvirt's or the system's own state (its secrets, the VMs' NVRAM, /var/spool): refused. - A node's withdrawal from a join spent its secret even when it could not be taken out (no quorum for a moment): the secret stays until it is. - The Terminal's task type had no label in the task list.
…or speaks for the current one A console connection awaiting its ticket or its module when the mode or the VM changed went on, mounted a second session on the same screen, and its errors and its end set the current session's state. Each attempt has a number; a late result, error or event of an older one is dropped.
…ut a gap; one-time codes count cluster-wide - Renaming a VM left its HA policy and monitoring, its load-balancing behaviour, its schedules, its keep protection and its place in DRS rules under the old name: the VM lost them (a kept VM could be deleted), and a new VM of the old name would have got them. They follow it. - Every start dropped and recreated all the replication triggers one statement at a time while the other services wrote: a write in between was never sent to the cluster. Only the triggers that differ are replaced, in one savepoint. - A TOTP code's step and a security key's counter were kept per node: a code used on one node was accepted on another. They are replicated.
…y; paused guests and late watchdogs respected - The BMC was asked within the round: an unreachable one took minutes, the leader lost the HA lead (its health is the round's age) and nothing was ever recovered. It is asked on a thread; a quick answer still comes in the same round. - The placement map of a round was overwritten by the node where the VM might already run: "must" rules (apart, not on these nodes) were never applied at recovery. Kept under its own name. - A member that answered again before it was proven stopped had its paused (or shutting down) guests counted as stopped and started elsewhere: a guest whose QEMU still exists stays. - The leader waited the watchdog's reset time from the member's last status, but a member may feed its watchdog until its heartbeat is 40 s stale: the wait counts those 40 s. - Admission control counted members in maintenance as room for a failure; recovery does not use them.
…uster budgets count every node's VMs - A VM schedule stepped from day to day on a time with a fixed offset: on the day summer time ended it ran twice (a VM stopped at 02:00 and again at 03:00), and an hour late in spring. Each day's time is placed with its own offset. - A budget on the cluster, a pool or a tag was checked by every node against its own VMs only: three nodes at 50 % each never alerted at 150 %. Each node publishes its VMs' spend per budget (cost_spend, replicated) and, in a cluster, the leader alone alerts on the sum. - A test of the scheduler's start counted a call another loop may make first.
… one unreadable backup holds nothing up - Backups, replication points and disk moves copied a VM's disk with qemu-img guessing its format, as clones did: a raw disk carrying a guest-written qcow2 header naming a host file had it read in, as root. Every copy of a VM's disk now names the format its definition gives. - One backup that could not be verified (a disk with I/O errors, a store whose key is gone) was asked again at every tick and stopped every other verification, the stores' garbage collection and their copies: it is recorded as unreadable (shown as such) and the scheduler's steps run each on its own. - A file-restore request that timed out left its answer to the next request, every later one off by one: the session ends. A session that failed to start no longer leaves its helper and its directory; two opens at once no longer both pass the limit; libguestfs installed afterwards is found without a restart. - A disk move whose definition could not be read right after the pivot deleted the file the VM was running on: any failure after the pivot keeps both files. - A store backup that could not reach libvirt left its task running for ever: it fails, said and audited. - A key added to authorized_keys without a final newline was glued onto the last line. - Two membership changes through two members at once could lose a member or give a node id twice: the cluster's configuration is written only over the version it was read at.
…lf-built, renames all or nothing, tunnels drained - node_files.write() failed with a TypeError whenever /etc/hyperlite was not the tree's mount (and could not replace a file anyway): it writes, replacing the file atomically. - A rule iptables refused half-way through a network's firewall left its chain without its last rules, its default DROP among them: the rules in place are put back. - A VM whose files could not all be renamed kept its new name in libvirt while Hyperlite's records stayed on the old one (its pools, rights, backups lost): the files moved go back and the VM takes its old name back. - The tunnel to a registered node's libvirt never read what ssh printed: after a few hundred lines the pipe filled and the tunnel stopped forwarding. It is read on a thread; a stopped tunnel is waited for (no zombie).
… said instead of success - POST /host/power took a libvirt that did not answer for "nothing runs": the node rebooted or powered off with its VMs and containers running, and the task said success. It now answers 503 until libvirt answers (no LXC driver still means no containers). - The Terminal's file upload read the whole body into memory before checking its size: refused at once over 20 MB. - A network renamed whose firewall could not be applied again was reported as a clean rename: the answer says so. - A migration whose replication checkpoints could not be forgotten left its task running and skipped moving its HA protection, tags and start at boot: that step is one of the others, and any unexpected error closes the task. - A node registered while the shared storage pools could not be created there looked like a node with none: said in the answer and the journal. - A metric server changed to InfluxDB without a token was saved and silently sent nothing.
…ge never undone; stale answers dropped - A cluster with an IPv6 member address could not turn its firewall on: nft refused it in the IPv4 set of the members. Members go in a set of their family each, and IPv6 members are let in too (checked with nft -c on Debian 13). - Between a change's confirmation and its save for the cluster, the follow loop saw no change waiting and applied the old document again: the confirmed document is kept on the node until the cluster's setting holds it. - The host shell tab: a Connect right after a Disconnect could leave two root shells, the old one's end marking the new one as not connected. The previous shell ends first, and only the current socket's end counts. - A VM's change history, a node's hardware and a VM's identity showed the answer that arrived last, not the one asked last: a late answer is dropped.
…all and VLANs; a firewall follows its VM - A live migration without shared storage deleted every ISO the VM had in a drive from the source node, the library's included (other VMs used them), and overwrote one of the same name on the target. Only the VM's own drives (its cloud-init ISO) leave the source; a library image is copied only where the target lacks it. - Clones and deployed templates referenced the source's own firewall filter: they were filtered by its rules and changed with them. Each gets a copy of its own. A renamed VM's filter follows its name, and a deleted VM's goes, so a new VM of that name never inherits it. - A deployed template kept VLAN settings recorded under the template's MAC address, and its interface fell on the untagged VLAN: the new MAC is chosen at deployment and its VLANs follow it.
…lates held while made; no reservation left - Anyone with vm.snapshot on a VM could direct its backups, or its schedule, to any backup store (another team's, the off-site one), written as root: a store must be given to the account (backup_store.read on it, or backup.manage), or be an administrator's choice; the daemon checks it too. - A store backup path naming a parent (..) made its "time" the VM's whole directory, deleted with it: refused. - Making a template took no hold on its name: two at once wrote the same disk. Making, renaming and deleting a template hold its name, and renaming or deleting is refused while a VM is deployed from it. - A creation that failed kept its address reservation and its MAC for ever: released. - Hook-script histories of VMs whose names share a stem (a.b, a.c) went through one temporary file.
…ailed transfers tested A confirmation queued behind a busy Corosync and then dropped left the node's "confirmation in flight" flag set: it never confirmed again, and the other nodes' changes waited for it until a membership change. A dropped confirmation is now sent again with the next one; a dropped state or change, which the members wait for, makes the node leave and join again. Tests: the transport retrying then failing (both cases), and a state transfer corrupted on its way, whose receiver leaves with its store as it was. Checked under ASan and UBSan.
…ithout the privilege; cluster faults stricter - A parametrised test takes each hyperlite-daemon operation that needs a privilege on a container, a storage pool, a backup store, a list of VMs or an area, and checks it refuses an account without it and lets in one that holds it on that very object. - The cluster fault tests now also require every node, the faulted one included, to take writes again after the fault; asking for the cluster tests (HYPERLITE_CFS_CLUSTER=1) where no lab exists is an error, not a quiet skip.
…ccess request's password
| conn = open_conn(readonly=True) | ||
| try: | ||
| return {**_network_summary(conn.networkLookupByName(new)), "guests_updated": moved} | ||
| return {**_network_summary(conn.networkLookupByName(new)), "guests_updated": moved, "warnings": warnings} |
| "INSERT INTO vault_grants (id, secret_id, subject_type, subject_id) VALUES ('g', 's', 'user', 'bob')" | ||
| ) | ||
| db.commit() | ||
| assert client.delete("/auth/users/bob", headers=admin).status_code == 200 |
| assert started[0][1:] == ("root", True, True) # (task, user, system, offline) | ||
| # While it runs, its file is what it installs from: neither replaced nor removed. | ||
| assert client.post("/update/offline", files={"file": ("u.tar", b"junk")}, headers=admin).status_code == 409 | ||
| assert client.delete("/update/offline", headers=admin).status_code == 409 |
Comment on lines
+139
to
+140
| monkeypatch.setattr(backups, "_store", lambda: type("S", (), {"get": lambda self, i: { | ||
| "status": "succeeded", "vm_name": "db1"}})()) # fmt: skip |
|
|
||
| import pytest | ||
|
|
||
| import app.daemon.ops # noqa: F401 (registers the operations) |
| upstream.sendall(flt.feed(data)) | ||
| except site_relay.Refused as e: | ||
| logger.warning("Incoming %s: a call refused by the relay: %s", incoming[:8], e) | ||
| except OSError: |
| try: | ||
| while data := upstream.recv(65536): | ||
| ours.sendall(data) | ||
| except OSError: |
| if not cluster_lead.in_cluster() or not cluster_lead.is_leader() or cluster_lead.read_only_reason(): | ||
| with _lock: | ||
| _lost.clear() # another node acts, or none may: start again from what is seen when this one may | ||
| _leading_since = None |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Security and fixes release from
master(CONTRIBUTING.md: urgent production fixes use ahotfix/...branch frommaster, merged back intotestafterwards).Fixes from a full code review of 1.28.1. The security advisories stay private until 1.28.2 is published; the CHANGELOG section lists them by id.
Tests
Then
release: 1.28.2(VERSION, CHANGELOG from the fragments, French release notes).