A killed separation can come back as a finished song with a truncated or missing file #969

Problem

ffmpeg wrote original.wav and mix.wav straight to their final names, and the Lead + Backing split did the same for lead_vocals.wav and backing_vocals.wav. A failed split also left a lone lead_vocals.wav behind. When the app was killed part way and the song was rebuilt from its folder (the registry said it was still running, or had lost it), any folder with one stem counted as a finished song. The result was a "done" song with a truncated original, a missing mix, or a lead vocal without its backing.

Who is affected: everyone, after a kill or a power loss during a separation. Narrow, but the song looks finished when it is not.

Where

• app/pipeline/collect.py:83 write to a dot temp name, then os.replace (:91)
• app/pipeline/vocal_split.py:103 the split pair moved in, removed whole on failure (:107)
• app/core/registry.py:796 STEMS_COMPLETE, :799 mark_stems_complete, :859 the rebuild check

Current state

Fixed in PR #898 and PR #901 (merged into big-release, ships with 0.21.0).

Proposed fix

What the merged fixes do:

• PR fix(pipeline): write stem audio whole, and keep WAVs until MP3s are on disk #898: original.wav, mix.wav and the split pair are written to a dot temp name in the same folder and moved in with os.replace only when ffmpeg succeeded. A failed split removes the pair whole.
• PR fix(registry): rebuild a song as done only when its separation finished #901: the last thing a separation writes is stems/.complete, after every stem and the folder are fsynced. A folder without it is not rebuilt as done: a running job goes back in the queue, and an upload whose registry entry is lost comes back as a stopped song that Extract can run again. A split counts only when both files are there.

Verification

• pytest tests/test_pipeline_vocal_split.py -k test_a_failed_split_leaves_no_lead_vocals
• pytest tests/test_pipeline_collect_mix.py
• pytest tests/test_stem_completeness.py, in particular test_a_song_killed_part_way_is_never_rebuilt_as_done