fix: prevent stuck distributed job completion

[robertvoy]

Summary

• propagate ComfyUI processing interrupts through the sync-to-async bridge instead of swallowing BaseException subclasses
• cancel server-loop coroutines when the synchronous bridge times out
• retain short-lived terminal job IDs so delayed worker callbacks receive a typed 410 job_closed response while genuinely unknown jobs remain 404
• treat job_closed as a terminal acknowledgement on workers, preventing a late callback from becoming a node exception
• cap consecutive busy-worker grace periods so a stale /prompt response cannot keep the master collector alive indefinitely

Why

Issue #89 reports a worker receiving 404 from /distributed/job_complete while the master remains stuck in Running and cannot be cancelled.

Current ComfyUI defines InterruptProcessingException as a direct BaseException subclass. The bridge in utils/async_helpers.py caught only Exception, so the interrupt could terminate the wrapper without being returned to the calling execution thread. In parallel, the collector allowed unlimited timeout extensions whenever a worker continued to report queue_remaining > 0.

Behaviour

• Active job callback: unchanged (200, result queued).
• Known closed job callback: 410 with code: job_closed; worker logs and stops sending remaining payloads.
• Unknown job callback: unchanged (404) so configuration and routing mistakes remain visible.
• Busy worker with no results: at most 10 additional configured worker-timeout periods by default. Override with COMFYUI_MAX_COLLECTOR_BUSY_GRACE_PERIODS.
• Closed job IDs are retained for one hour by default. Override with COMFYUI_CLOSED_JOB_TTL_SECONDS.

Verification

• python -m pytest -q — 263 passed
• python -m unittest discover -s tests — 194 passed
• python -m compileall -q api nodes utils tests — passed
• git diff --check — passed
• confirmed the installed ComfyUI InterruptProcessingException is a BaseException, not an Exception
• confirmed the live dev instance remained healthy with an empty queue; this branch was tested from an isolated worktree and was not installed into the running custom-node checkout

Reporter validation requested

@maxnbk, could you test this branch on both the master and runners?

git fetch origin
git switch fix/issue-89-job-completion-lifecycle

Please check:

1. the workflow that intermittently produced the late /distributed/job_complete failure;
2. cancelling while the master is waiting for a runner;
3. whether both master and runner leave Running without restarting the master;
4. any job_closed, busy-grace, or callback errors from both logs.

Fixes #89