fix: resolve result_future on inference error instead of tearing down the model - #157
Conversation
… the model QueueWorker.queue_worker_llm (and the identical loop in every other worker: vlm, whisper, qwen3_asr, kokoro, qwen3_tts, emb, rr) broke out of its processing loop on an inference error before ever calling packet.result_future.set_result(...). Non-streaming callers such as WorkerRegistry.generate() await that future, so a failed generation hung the HTTP caller forever. The same branch also fired an unconditional registry.register_unload(model_name), whose destructor can touch a corrupted device context after certain errors and SIGABRT the whole process, taking every other loaded model down with it. Each worker now resolves result_future with the error via set_exception(RuntimeError(...)) and continues its loop instead of breaking, so a failed request surfaces as an error to its caller and the model/worker stay alive for the next request. The EMB and RR workers use a different failure condition (an empty response, since InferWorker.infer_emb tags a raised exception as a truthy "Error: ..." string) but had the exact same break-before-resolve bug. Fixes SearchSavior#153
|
@AmirF194 Thanks for the PR! RE:
Does this mean the proposed fix has not been tested on hardware? |
|
Correct, not on real Intel GPU/OpenCL hardware, this session doesn't have one. What's verified: the hung future (the caller never gets a response) and that all 8 worker functions share the same break-before-resolve pattern, both confirmed by running the workers directly. The SIGABRT itself I can't reproduce without a genuine device error on real hardware, so the fix removes the crash-triggering call ( |
Root cause
QueueWorker.queue_worker_llmand the identical loop repeated in every otherworker (
vlm,whisper,qwen3_asr,kokoro,qwen3_tts,emb,rr)broke out of its
while Trueloop on an inference error before ever callingpacket.result_future.set_result(...). Non-streaming callers such asWorkerRegistry.generate()awaitthat future, so a failed generation hungthe caller forever. The same branch also fired an unconditional
registry.register_unload(model_name); that unload's pipeline destructor cantouch a corrupted device context after certain errors and abort the whole
process, taking every other loaded model down with it. This matches the issue
exactly for the LLM worker; I confirmed by reading each of the other seven
worker functions that they share the identical structural bug (
grep/astenumeration in
src/server/worker_registry.py), not only the ones named inthe issue as "likely" affected.
embandrrtrigger on a different condition (not completed_packet.response,since
InferWorker.infer_emb/infer_rerankalready tag a raised exception asa truthy
"Error: ..."string, so that branch is actually reached only whenthe pipeline returns no data with no exception), but the same break-before-resolve
bug is there too.
Fix
Each worker now reports the failure through
result_future.set_exception(...)and
continues its loop instead ofbreaking, so a failed request surfaces asan error to its caller and the worker/model stay alive to serve the next
request, per the issue's suggested design.
Verification
tests/unit/test_worker_registry_error_handling.pydrives the real(unmocked)
queue_worker_*loop for all 8 workers with a failing requestfollowed by a healthy one. Confirmed fails on unmodified
main(eachtest times out waiting on
result_future, matching the reported hang) andpasses on this branch, in a clean
python:3.12-slimcontainer.uv run pytest -W ignore tests/unit: 119 passed (was 111), noregressions.
coverage run --source=src.server.worker_registry: every line this diffchanges is exercised by the new tests.
SIGABRTitself, since that needs a real IntelGPU/OpenCL context corrupted by a genuine device error, which this session
doesn't have. This PR removes the
register_unloadcall on the error pathentirely (the mechanism the issue names as the crash trigger), but I can't
reproduce the abort to confirm the crash is gone, only that the code no
longer calls the destructor on that path.
Fixes #153