vmm: fix migration sender hang on worker failure - #193
Merged
Coffeeri merged 2 commits intoAug 19, 2026
Conversation
Coffeeri
force-pushed
the
fix/migration-sender-hang
branch
from
August 18, 2026 15:27
847d8d8 to
630f7d5
Compare
phip1611
requested changes
Aug 18, 2026
Coffeeri
force-pushed
the
fix/migration-sender-hang
branch
from
August 19, 2026 06:40
630f7d5 to
da54a33
Compare
amphi
approved these changes
Aug 19, 2026
A single failing parallel migration connection can deadlock the migration sender. When the bounded send-channel is full, `SendAdditionalConnections::cleanup()` silently drops the Disconnect messages via the non-blocking `try_send`. The surviving workers drain the channel and then block forever in `recv()`, while the main thread blocks forever in `join()`. The VM stays stuck in the "migrating" state. We fix this by sending the `Disconnect` messages with a blocking send, ensuring the messages reach the migration workers. On-behalf-of: SAP leander.kohler@sap.com Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
When a memory-sending worker fails, the migration thread calls `SendAdditionalConnections::cleanup()`. Cleanup tries to enqueue one `Disconnect` message per worker, but the bounded channel may still contain `SendMemoryThreadMessage::Memory` values. The surviving workers currently call `send_memory_ranges()` for each such value, even though the migration has already failed. Skip that call when `worker_error` is set. This consumes the queued values without processing them and lets cleanup enqueue the `Disconnect` messages. On-behalf-of: SAP leander.kohler@sap.com Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Coffeeri
force-pushed
the
fix/migration-sender-hang
branch
from
August 19, 2026 08:19
da54a33 to
07ed6a3
Compare
arctic-alpaca
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A single failing parallel migration connection can deadlock the migration sender.
When the bounded send-channel is full,
SendAdditionalConnections::cleanup()silently drops the Disconnect messages via the non-blockingtry_send. The surviving workers drain the channel and then block forever inrecv(), while the main thread blocks forever injoin(). The VM stays stuck in the "migrating" state.We fix this by sending the
Disconnectmessages with a blocking send. In addition, once one of the workers setsworker_error, the other workers stop sending the queued memory chunks and drain the channel, so the Disconnects are queued and all workers exit.Pipeline
cc @tpressure @phip1611