Skip to content

fix: sweep crash leftovers at startup, free a cancelled zip, give each click render its own temp - #900

Merged
thcp merged 1 commit into
big-releasefrom
fix/b41-crash-leftovers-sweep
Nov 3, 2026
Merged

thcp merged 1 commit into
big-releasefrom
fix/b41-crash-leftovers-sweep

Conversation

@thcp

@thcp thcp commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

What changes for the user

  • Disk space after a crash comes back on the next start. A stem fill, a vocal split decode, a stem written beside its final name, an export render and an all-stems zip each write to a temp name first. A force quit, a power loss or an OOM kill in the middle used to leave that temp on disk for good, often hundreds of MB, in a dot-named file nobody could find. The startup sweep now removes them.
  • A cancelled zip export no longer leaves its zip behind. Stopping the server during an all-stems zip skipped the cleanup (CancelledError is not an Exception), and the build thread went on writing.
  • Two exports with the same click no longer lose one click. They shared one temp name, so the first rename could take the other's file away.

What the sweep may remove, and what it never touches

  • Song folders: only those registry.job_dirs and holds_job_files accept (the rule from fix(reset): remove only StemDeck's own entries in a shared jobs folder #886), and never through a link or junction (is_real_entry).
    • .fill-<8 hex>/ (stem fill work folder)
    • in stems/: .decode-*/ (vocal split decode), and dot names ending in .tmp or .tmp.mp3, file or folder (stem writes)
  • Export caches (cache/mixdown, cache/click): only the exact temp names (.<name>.<32 hex>.tmp[.ext], .stemdeck_zip_*.zip, .zipwork-*/), and only when nothing has written them for 10 minutes, because a second server can share the cache folder.
  • It runs once, at startup, before the queue starts anything, like the existing section workspace and render temp sweeps.
  • A song's stems, MP3s, peaks, beats, source, metadata and takes, finished or stopped, never match any of these names.

Other changes that make the sweep exact

  • The vocal split decode moved from the system temp folder to a dot folder beside the stem.
  • The zip, and the folder its stems are transcoded in, moved from the system temp folder to the export cache.
  • The sweep therefore never deletes in a folder that other programs share.
  • The zip temp now goes once the request has given it up and the build thread has ended, whichever comes second. On Windows, removing it sooner fails while the thread holds the file.
  • Each click render gets its own uuid dot name, removed in a finally.
  • tests/conftest.py now points the export caches at a per-test folder. Every TestClient lifespan runs the sweep, and it must never reach a developer's real cache.

How it was verified

  • New tests in tests/test_crash_leftovers.py. All 4 fail on big-release and pass with this change:
    • Startup removes every planted leftover and keeps every file of a finished song and of a stopped song, plus the cache entries, a recent cache temp, a .DS_Store, and folders that are not StemDeck's.
    • The sweep never follows a junction: neither a linked song folder nor a linked stems/.
    • A zip request cancelled mid-build: the file stays while the thread writes, and is gone once the thread ends. It is built in the export cache.
    • Two renders of one click key: on big-release the second one fails with FileNotFoundError; with this change both finish.
  • Full Python suite, ruff check and ruff format --check pass.
  • End to end on a throwaway server (Windows, uvicorn, throwaway jobs, data and cache folders):
    • 8 planted leftovers were gone after startup, and none of the 22 song files was lost.
    • WAV and MP3 zips downloaded correctly, with nothing left in the cache.
    • A client that disconnected after the first bytes left nothing either: uvicorn still runs the response's background cleanup.
  • Merges cleanly with the open atomic stem writes branch, and its .vocal_split.<hex>.tmp work folder is covered.

Update path

  • Does not touch uv.lock or the desktop shell.

…h click render its own temp

A stem fill, a vocal split decode, a stem written beside its final name, an
export render and an all-stems zip each write to a temp name first. A kill or
a power loss in the middle left that temp on disk for good, and each can be
hundreds of MB. The startup sweep now removes them: only in StemDeck's own
song folders, never through a link or junction, never a file a song keeps,
and in the export caches only by the exact temp names, and only when nothing
has written them for ten minutes (a second server may share the cache).

The decode and the zip, with its transcode folder, moved out of the system
temp folder (beside the stem, and into the export cache) so the sweep never
has to delete in a folder other programs share.

A cancelled zip request (the server stopping) skipped the cleanup, because
CancelledError is not an Exception, and the build thread went on writing.
The zip now goes once the request has given it up and the thread has ended.

Two exports rendering the same click at once shared one temp name, so one
could rename the other's file away and lose its click. Each render now has
its own uuid dot name, removed in a finally.
@thcp
thcp merged commit c46a028 into big-release Nov 3, 2026
19 of 20 checks passed
thcp added a commit that referenced this pull request Nov 3, 2026
…ed (#901)

## What changes for the user

A song whose separation was cut off part way (the app killed, the PC
losing power) no longer comes back after a restart as a finished song
with stems, the Original track or the mix missing.

- A job that was running comes back in the queue and runs again from the
start, as a job with no stems on disk already did.
- If registry.json was lost as well, an upload comes back as a stopped
song, so Extract runs it again from the file it kept. A link's folder
stays on disk, untouched, and is not shown as finished.
- A song rebuilt from its folder counts as split into Lead + Backing
only when both files are there. A kill between the two renames used to
bring back a lead vocal without its backing; the lone file stays on disk
and the next split replaces it.

## Why

restore() rebuilds a song from its folder when the registry says the job
was still running, or has no entry for it. It took any folder holding
one stem for a finished song. collect() moves the stems in one at a
time, and original.wav, mix.wav and the MP3 storage come after it, so a
kill at any of those steps returned an incomplete song as "done". #898
made each file arrive whole; this makes the set of files whole.

## How

- The last thing a separation writes is `stems/.complete`, after every
stem file and the folder are fsynced (`registry.mark_stems_complete`,
called at the end of both pipelines). The marker goes through the
existing atomic writer: fsynced temp file, rename, folder fsync.
- `_rebuild_from_folder` rebuilds a folder as done only when it has the
marker.
- Songs made before this change: on startup, every song the registry
holds as done gets the marker if its folder has none (only the marker is
written, the stems are left alone). A song whose registry record was
already gone has metadata.json, which StemDeck writes once a song is
done (and now only after the marker), and is rebuilt exactly as before.
Nothing is hidden, moved or deleted. Songs made by 0.20.0 after a
downgrade get the marker on the way back, and 0.20.0 ignores the file.
- If the marker cannot be written (a full disk), the job still finishes
and the error is logged: failing the job would delete stems that are
whole, and the song then rebuilds by its metadata.json like an older
song.
- Since the marker is the last write, a kill during the lyrics or
section analysis after separation now runs the job again, where the old
code kept the stems. That is the cost of "never done unless finished".

## Verified

- `tests/test_stem_completeness.py` runs the real pipeline steps that
write `stems/` (collect, original.wav, mix.wav, MP3 storage) and kills
the process at eight points: one stem moved, before original.wav, before
mix.wav, after mix.wav, before stem storage, one MP3 moved in, one WAV
deleted, after stem storage. Each is checked with the registry kept (job
queued again) and lost (the upload comes back stopped with its source).
On the old code every one of these came back "done", and the split case
listed a lone lead vocal. The finished run is rebuilt as done both ways,
and a song from before the marker gets it from the registry.
- A real separation on a throwaway server (MP3 storage on):
`stems/.complete` is there, and after deleting registry.json a restart
brings the song back done with every stem and the mix.
- Marker cost, measured on this machine's SSD: 0.14 s for eight 40 MB
stems written just before it, 0.004 s when they are already on disk. Not
measured on a hard disk or a network share.
- Tests whose folders stand for a finished song made by this version now
carry the marker. ruff check and ruff format pass. Full Python suite:
3819 passed, 12 skipped; after the rebase onto #899 and #900 the 247
tests of the affected files pass again.

Does not touch `uv.lock` or the desktop shell.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant