Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions changelog.d/1096-handoff-failed-child.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Fixed
The handoff wrapper no longer reports a run that never started as a successful
handoff: when the wait window closes with the child gone and no new
`status.json` where the wrapper can see it, it writes `waiting.json` with
`status: failed` and `state: exited`, and exits nonzero with the child's own
reason left in `worker.log`. A child that is still running and has not written
its status yet is unchanged: `waiting`, `found: false`, exit 0. A store the
caller owns as the run directory is still reported from its own rewritten
`status.json`. The verdict reads the store again after the liveness probe, so a
status written while the probe ran is still this handoff's. The page says all
three (#1096).
23 changes: 19 additions & 4 deletions docs/dsh-super-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,9 +152,11 @@ the pair is what makes the run stop itself and write why.

Two stores, one variable: the wrapper's `--store` is where the report goes, and
the run's store is the CLI's own `--store`, so pass the same value on both. The
wrapper cannot see the child's store — give it only to the wrapper and `evolve`
writes to the default store, the wrapper waits out its deadline and reports
`found: false`, and the read-back below cannot find the run.
wrapper only sees the store it was given: give it only to the wrapper and
`evolve` writes to the default store, so the wrapper waits out its deadline,
finds no status of its own where it can look, and reports `status: failed` with
exit 2 — the child did start, but its status sits in a store this handoff cannot
read, and the read-back below cannot find the run.

It starts the command in a detached `tmux` session (`setsid` when tmux is
absent), waits for the run's own `status.json`, writes `waiting.json` beside it
Expand All @@ -167,13 +169,26 @@ arguments rather than splicing them into a shell string, so a `--store`
containing a quote, a space, `$` or `;` is used literally instead of being read
as shell syntax (`#1109`).

A child that exits without leaving a `status.json` where the wrapper can see it
handed nothing off, and the wrapper says so: when the window closes with the
run's session gone and no new status file, `waiting.json` carries
`status: failed` and `state: exited`, the wrapper exits nonzero, and `log` names
the `worker.log` where the child's own reason sits (`#1096`). That includes a
run that wrote its status into a different store — the child started, this
handoff did not receive it. The window itself is unchanged for the case it
exists for — a run that is alive and has not written its status yet is still
`waiting`, with exit 0.

A store is reused for every Ticket, so it already holds earlier runs. The
wrapper only reports a `status.json` that appears after it launches, or one that
changes with a `started` no older than the launch, so a handoff into a used store
reports `found: false` until its own run writes one. A file whose `started`
predates the launch is never reported: that is a run this handoff did not start
— possibly still going and rewriting its own file every round — and naming it
would send the read-back to the wrong run. The tmux session carries a
would send the read-back to the wrong run. A caller-owned store, where the store
itself is the run directory, must be the run this handoff starts: the wrapper
takes its changed top-level `status.json` as this launch's own, so a second live
writer in that directory can be reported as this handoff. The tmux session carries a
per-launch suffix for the same reason: two handoffs into stores both called
`runs` get two sessions, not one shared, refused name. One live handoff per
store: two launched into one store inside the same wait window both read that
Expand Down
146 changes: 122 additions & 24 deletions scripts/dsh-superagent-run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@
# dsh-superagent-run.sh --store DIR -- agentdescent --store DIR evolve spec.json --detach
#
# The wrapper's --store is where the report goes; the run's store is the CLI's
# own --store. Give both the same value, or the run lands in the default store
# and the report says found: false.
# own --store. Give both the same value: the wrapper decides "failed" from the
# store it was given, so a child that writes its status into a different store
# is reported as a failed handoff even when the run did start (#1096).
#
# tmux is preferred because dsh-superagent already depends on it and a session
# is not in the worker's process group, so the kill that ends the worker's
Expand All @@ -16,7 +17,10 @@
# worker hands on.
#
# Exit 0 once the run is detached and DIR/waiting.json is written. It never
# waits for the run: a handoff that blocks is the failure this replaces.
# waits for the run: a handoff that blocks is the failure this replaces. Exit 2
# when the child was already gone at the end of the wait window and wrote no
# status.json -- nothing was handed off, and DIR/worker.log holds its reason
# (#1096).
set -euo pipefail

WAIT="${DSH_RUN_HANDOFF_WAIT:-10}" # seconds to wait for the run's status.json
Expand Down Expand Up @@ -78,26 +82,57 @@ with open(out, "w", encoding="utf-8") as fh:
PY
if command -v tmux >/dev/null 2>&1; then
DRIVER=tmux
RUN_PID=""
tmux new-session -d -s "$SESSION" -c "$PWD" "${LAUNCH[@]}" "${CMD[@]}" \
|| die "tmux refused the session $SESSION"
elif command -v setsid >/dev/null 2>&1; then
DRIVER=setsid
setsid "${LAUNCH[@]}" "${CMD[@]}" >/dev/null 2>&1 &
RUN_PID=$!
else
die "neither tmux nor setsid is available; this host cannot hand off a run"
fi

# The run owns its status.json (`<store>/<run_id>/status.json`, or the store
# itself when the caller owns the run directory). Absent is reported as such:
# an invented run id is worse than a gap a Trigger can see.
python3 - "$STORE" "$WAIT" "$DRIVER" "$SESSION" "$LOG" "$QUOTED" "$SNAP" <<'PY'
import glob, json, os, sys, time
store, wait, driver, session, log, command, snap_path = sys.argv[1:8]
python3 - "$STORE" "$WAIT" "$DRIVER" "$SESSION" "$LOG" "$QUOTED" "$SNAP" "$RUN_PID" <<'PY'
import glob, json, os, subprocess, sys, time
store, wait, driver, session, log, command, snap_path, run_pid = sys.argv[1:9]
before = json.load(open(snap_path, encoding="utf-8"))
epoch = float(before.pop("epoch"))
deadline, found = time.time() + float(wait), None


def still_running():
"""Whether the command this handoff started is alive at the window's end.

`found: false` alone cannot tell "alive, no status yet" from "gone" (#1096).
The driver answers it: a tmux session ends with its command, and the setsid
pid is this shell's own child, so `kill(pid, 0)` says whether it is still
there. Unknown is reported as alive -- a false "failed" would be worse than
a missed one.
"""
if driver == "tmux":
try:
return subprocess.run(["tmux", "has-session", "-t", "=" + session],
capture_output=True).returncode == 0
except OSError:
# The probe could not run at all (exec pressure, binary gone,
# permission): it did not answer, and "cannot judge" is alive. Letting
# this raise would kill the report before it is written (#1096 review).
return True
if not run_pid:
return True
try:
os.kill(int(run_pid), 0)
except ProcessLookupError:
return False
except (PermissionError, ValueError):
return True
return True


def started_at(path):
"""A run's launch time, or None when the file does not carry a numeric one."""
try:
Expand All @@ -110,11 +145,18 @@ def started_at(path):
return float(started)


while time.time() < deadline:
def fresh():
"""This launch's own status.json, read from the store now, or None.

The scan is a function so the decision comes from a read taken at decision
time: the old loop exited on a scan up to 200 ms old, and a status.json
written during that last sleep was reported as a failed handoff (#1096
review).
"""
paths = [p for p in [os.path.join(store, "status.json")]
+ sorted(glob.glob(os.path.join(store, "*", "status.json")))
if os.path.isfile(p)]
fresh = []
found_paths = []
for path in paths:
try:
st = os.stat(path)
Expand All @@ -123,33 +165,89 @@ while time.time() < deadline:
seen = before.get(path)
if seen is None:
# Not there when the snapshot was taken: this launch made it.
fresh.append(path)
found_paths.append(path)
continue
if [st.st_mtime, st.st_size] == seen:
continue
if path == os.path.join(store, "status.json"):
# The store *is* the run directory when the caller owns it, so this
# file is this handoff's own run and its `started` predates the
# launch by construction. The epoch filter below is for another run
# in a shared store and must not reject it (#1096 review).
found_paths.append(path)
continue
# A reused store can hold a run that is still going, and that run
# rewrites its own status.json every round (#1101). Only a file whose
# `started` is no older than this launch is this handoff's.
started = started_at(path)
if started is None or started < epoch:
continue
fresh.append(path)
if fresh:
found = max(fresh, key=os.path.getmtime)
found_paths.append(path)
return max(found_paths, key=os.path.getmtime) if found_paths else None


def make_report(found):
"""The report for the status file this handoff is, or a bare `waiting`."""
status = json.load(open(found)) if found else {}
return {"status": "waiting", "found": bool(found),
"run_id": status.get("run_id", "") if found else "",
"round": status.get("round", 0),
"state": status.get("state", "unknown"),
"run_dir": os.path.dirname(found) if found else store,
"driver": driver,
"session": session if driver == "tmux" else "", "log": log,
"command": command.strip()}


while True:
found = fresh()
if found is not None or time.time() >= deadline:
break
time.sleep(0.2)
run_dir = os.path.dirname(found) if found else store
status = json.load(open(found)) if found else {}
report = {"status": "waiting", "found": bool(found),
"run_id": status.get("run_id", "") if found else "",
"round": status.get("round", 0), "state": status.get("state", "unknown"),
"run_dir": run_dir, "driver": driver,
"session": session if driver == "tmux" else "", "log": log,
"command": command.strip()}
report = make_report(found)
path = os.path.join(store, "waiting.json")
with open(path, "w", encoding="utf-8") as fh:
json.dump(report, fh, indent=2, sort_keys=True)
fh.write("\n")
print(f"waiting {report['run_id'] or '(no run yet)'} round {report['round']} {run_dir}")


def write_report():
with open(path, "w", encoding="utf-8") as fh:
json.dump(report, fh, indent=2, sort_keys=True)
fh.write("\n")


# The report lands before the liveness question is asked: if the probe cannot
# run -- or the wrapper is killed while it waits -- a `waiting` report still
# exists, which is the direction #1096 calls for. Nothing this launch could
# report and the child gone means nothing was handed off; the wrapper used to
# print `waiting` and exit 0 there, which is how a child that refused to start
# read as a successful handoff (#1096).
write_report()
expired = found is None and not still_running()
if expired:
# The probe spawns a tmux client -- 10-30 ms, unbounded on a loaded box --
# and nothing reads the store while it runs, so a status.json written in
# that window is on disk when the verdict says it is not. Read once more at
# the moment of the decision and believe the file over the probe: a run
# that wrote and exited 20 ms after the window is a handoff, and calling it
# failed costs a duplicate run (#1096 review). This is the probe's window
# only; the deadline is not extended, so a child that wrote nothing is
# still `failed`.
late = fresh()
if late is not None:
found = late
report = make_report(found)
expired = False
# The provisional `waiting` report is already on disk and names no run;
# replace it so a reader sees the handoff this launch actually made.
write_report()
if expired:
report["status"] = "failed"
report["state"] = "exited"
write_report()
print(f"{report['status']} {report['run_id'] or '(no run yet)'} "
f"round {report['round']} {report['run_dir']}")
print(f"report {path}")
if expired:
print(f"error: the run exited before writing a status; its reason, if it "
f"wrote one, is in {log}", file=sys.stderr)
sys.exit(2)
PY
Loading
Loading