Give a timed-out solver a chance to clean up after itself - #218
Merged
Conversation
Thrust gives up on a solver that exceeds its timeout by killing the wrapper with SIGKILL, which runs no trap and leaves the container behind: Docker keeps a container going after the client attached to it is gone, so the solver ran on in the background for as long as it took, or forever on an instance it cannot solve. Removing it has to be left to a process that outlives the wrapper. That process waits for the write end of a pipe, which the kernel closes however the wrapper dies, and then force-removes the container. It needs the container's id before the solver starts, hence `docker create` followed by `docker start --attach`, which reports the container's output and exit status just as `docker run` did. Only the wrapper may hold the pipe open; the subshell that starts the container closes its copy, or the removal would wait for the solver it is meant to stop. A container is still left behind if the wrapper is killed in the moment between `docker create` being asked for a container and the removal being set up, but it is one that never started, and Thrust's timeout does not expire that early. Closes #49 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qpacd6KyiRsKFexkrgfX6Q
Thrust ended a solver that exceeded its timeout with SIGKILL, a signal the process cannot act on, and `tests/thrust-pcsat-wrapper` was therefore left no way to stop the Docker container it had started. Docker keeps a container running once the client attached to it is gone, so the solver went on in the background for as long as the instance took, or forever on one it cannot solve. Send SIGTERM instead, and kill only what has not exited two seconds later, so a solver command holding resources of its own can release them. The process is never reaped in between, so the system cannot hand its identifier to an unrelated process that the SIGKILL would then reach. The wrapper removes its container from a SIGTERM trap. It starts the container in the background and waits on it, because bash runs a trap only once the command in the foreground has finished -- which would be the very solver the signal is meant to stop. Closes #49 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qpacd6KyiRsKFexkrgfX6Q
The step-by-step account of `terminate` and of the wrapper's trap restated the lines they sat above; what is left is why a solver is signalled rather than killed, why its identifier is safe to signal, and why the container is removed rather than stopped. The grace period loses its doc comment along the same lines, matching the other constants in the crate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qpacd6KyiRsKFexkrgfX6Q
Waiting out a grace period cost every timeout two seconds of doing nothing, and the SIGKILL that followed it was sent whether or not anything was still there to receive it. A solver command is expected to exit on the signal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qpacd6KyiRsKFexkrgfX6Q
coord-e
force-pushed
the
claude/docker-container-persistence-dddom9
branch
from
August 16, 2026 02:45
e08ac22 to
ca1b2bc
Compare
coord-e
marked this pull request as ready for review
August 16, 2026 02:46
Contributor
There was a problem hiding this comment.
Pull request overview
Updates solver timeout handling so processes can clean up resources before exiting, addressing issue #49.
Changes:
- Sends SIGTERM instead of immediately killing timed-out solvers.
- Adds Docker container cleanup to the PCSAT wrapper.
- Adds the
nixsignal dependency.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
tests/thrust-pcsat-wrapper |
Traps SIGTERM and removes the solver container. |
src/chc/solver.rs |
Sends SIGTERM when solver execution times out. |
Cargo.toml |
Adds nix for signal handling. |
Cargo.lock |
Records updated dependencies. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #49.
The cause
Two things together, of which only the first is described in the issue.
Thrust ended a solver that exceeds its timeout with SIGKILL (
terminate_for_timeout), which the process cannot act on. But even a wrapper that could run a trap would not have been enough:docker runis only a client of the daemon, so killing it does not touch the container, and--rmremoves a container once it stops, which a running solver never does on its own. Observed after killing the wrapper:Nothing tells the container to stop, so the solver runs until it finishes — forever on an instance it cannot solve, which is how the issue was hit.
The change
Thrust sends SIGTERM and leaves the solver to exit on it, so a solver command that holds resources of its own can release them.
terminate_for_timeoutis dropped, which is also what makes the signal safe to send: without it the timed-out child is neither killed nor reaped, so its identifier stays allocated and cannot be handed to an unrelated process in the meantime.The wrapper removes its container from a SIGTERM trap. It starts the container in the background and waits on it, because bash runs a trap only once the command in the foreground has finished — which would be the very solver the signal is meant to stop.
docker create+docker start --attachreplacesdocker runso the trap has the container's id before the solver starts; it reports the same output and exit status.nix(signalfeature only) is added forkill;libcdirectly would have been the firstunsafeinsrc/.Verification
Against the pinned
COAR_IMAGE, withTHRUST_SOLVER_TIMEOUT_SECS=1ontests/ui/pass/iterators/annot_range_loop.rs:error: verification error: Timeout(1s), the running container is gone, and the run takes 1.26s, so nothing is spent waiting on the solver after the timeout.docker createwas asked for a container, and the removal at 150-200ms after the signal.cargo test: 308 UI tests and 2 doc tests pass, with no container and no stray.smt2left behind. The stray temp files are incidentally fixed too — the wrapper'sEXITtrap could not run under SIGKILL either.Notes
A solver command that does not exit on SIGTERM is left running; nothing kills it afterwards. So is anything it started of its own, as the signal reaches only the command itself.
A container is also left behind if the timeout expires in the moment between
docker createbeing asked for a container and the trap being set, measured at ~55ms against a warm daemon. It takes a timeout far shorter than the 30s default to land there.