Skip to content

fix(hosts): a child that never started is a failed handoff, not waiting (#1096) - #1117

Open
HugoXK wants to merge 3 commits into
mainfrom
agent/1096-handoff-waiting-never-started
Open

HugoXK wants to merge 3 commits into
mainfrom
agent/1096-handoff-waiting-never-started

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1096

What changed

scripts/dsh-superagent-run.sh says at :16-19 that it exits 0 "once the run is detached and DIR/waiting.json is written", and it did that unconditionally: the final python heredoc printed the report and exited 0 whatever the child did. A child that exited 2 before creating anything still produced "status": "waiting", found: false and exit 0, with its reason only in DIR/worker.log.

The fix decides liveness at the end of the wait window, because found: false alone cannot tell "alive, no status yet" from "gone":

  • still_running() in the report heredoc asks the driver that started the run: tmux has-session -t "=SESSION" for the session driver, and os.kill(pid, 0) on the pid the setsid branch now captures (RUN_PID=$!). Unknown — no pid, PermissionError, an unparsable value — is reported as alive, because a false "failed" is worse than a missed one.
  • Window closed, nothing found, child gone: waiting.json carries status: failed and state: exited, the same ten fields with log still naming DIR/worker.log, the stdout line starting with the report's own status, one stderr line naming that log, and exit 2. The comment at :16-19 now documents exit 2.
  • Child still running and not written yet: unchanged — status: waiting, found: false, exit 0. The window keeps existing for the case it exists for.
  • tests/test_dsh_superagent_run.py — test_a_child_that_exits_before_writing_a_status_is_a_failed_handoff, parametrised over ["tmux", "setsid"]. The child is python3 -c "import sys; sys.exit(2)"; the row asserts a nonzero exit, found false, status and state neither waiting nor running, and log exactly store/worker.log. The setsid row runs through a PATH without tmux.
  • docs/dsh-super-agent.md — one paragraph after the #1109 note saying both halves: gone and silent is failed/exited with a nonzero exit and the reason in worker.log; alive and silent is still waiting with exit 0.
  • changelog.d/1096-handoff-failed-child.md — one Fixed fragment.

Disclosure: three existing rows hand off children that now outlive the window

The rule this commit adds makes an exited child with no status.json a failed handoff, so rows that had been relying on a child which had already exited needed their children retimed. No assertion was changed; only the children:

  • the metacharacter row (test_a_store_path_with_shell_metacharacters_hands_off_without_running_them) writes its marker and then sleeps 2 s — it is about where the arguments went, not about when the child ends;
  • both reused-store rows (test_a_reused_store_does_not_report_an_earlier_run, test_a_live_earlier_run_is_never_reported_as_this_handoff) hand off sleep 3 instead of sleep 1, so the child is still running when the 1 s window closes.

Without the retiming those three rows would read as failed handoffs under the new rule — correct behaviour applied to the wrong row.

Review round: the four must-fix items

5916462545 reviewed b736160 and named four must-fix items. All four are in one commit on top of it — 151d239, no force-push, no amend, no rebase — and the three retimed children above are untouched. The test-file diff is purely additive (git diff b736160..151d239 -- tests | grep '^-' shows only file headers), so no existing assertion changed.

  1. Stale read at the end of the window. The scan is factored into fresh() and the loop is while True: found = fresh(); if found or time.time() >= deadline: break — the decision now reads the store after the last sleep instead of before it. Probe with the review's child (sleep X, then write live-run/status.json), 4 runs per X: b736160 failed 2/4 at 0.80 s, 4/4 at 0.88 s, 4/4 at 0.92 s, 3/4 at 0.95 s; 151d239 is 16/16 exit 0, found: true, waiting. Row test_a_status_written_in_the_last_poll_interval_is_reported: red at b736160 (exit 2, status: failed), green here.
  2. Caller-owned store. A changed <store>/status.json is appended before the epoch filter, because there the store is the run the caller handed over. Row test_a_caller_owned_store_reports_its_own_rewrite_as_the_handoff seeds started: 1000.0 and the child rewrites the file to round 4, finished: red at b736160 (exit 2, failed), green here (found: true, round: 4).
  3. A probe that cannot run. except OSError: return True around the tmux probe, and the report is written before the probe is asked, so a raising probe cannot leave the store without a report: red at b736160 with waiting.json missing and FileNotFoundError out of still_running, green here (exit 0, waiting, report present). Row test_a_probe_that_cannot_run_leaves_a_waiting_report runs against a PATH whose tmux answers new-session and then deletes itself.
  4. Prose vs behaviour. The rule stays "confirmed gone + no status in the wrapper's own store → failed". docs/dsh-super-agent.md:153-157 and :170-176, the script header :8-10 and the fragment now say the wrapper only sees the store it was given, so a run written elsewhere is failed with exit 2. Row test_a_run_written_to_a_different_store_is_a_failed_handoff pins failed/exited/nonzero/log; it passes at both heads by design, because the behaviour was already there and the defect was the prose.

Third commit: the two must-fix items

5916690532 reviewed 151d239 and named two must-fix items; both are in fdbd8af on top of it — no force-push, no amend, no rebase. Exactly one line is deleted in the whole diff and it is the second item's assert → pytest.skip; everything else is added.

  1. The decision still preceded a store read. The report now comes from make_report(found), and the expired branch re-reads after the probe: expired = found is None and not still_running(); when it is true, late = fresh(), and when that returns a file the report is rebuilt from it, expired is cleared, and the report is rewritten so the provisional waiting on disk no longer names no run. Nothing is written as failed on that path. The grace is the probe's own duration — a child that wrote nothing is still failed — which is what the second row below pins.
    • The probe spawns a tmux client: 10–30 ms, unbounded on a loaded box. At X ≈ 1.02 s with a real probe the child is still alive when the probe is asked, so both heads answer waiting; the reviewer's 2/4 came from the probe landing after the exit. Rather than race that, the row makes the window visible: has-session answers after sleep 0.6 and then delegates to the real tmux, the child writes at 1.2 s (0.2 s past a 1 s window) and the probe answers at ~1.6 s.
    • test_a_status_written_after_the_window_and_before_the_verdict_is_the_handoff: at 151d239 red — exit 2, status: failed, state: exited, found: false, with store/late-run/status.json present, assert 2 == 0; at fdbd8af green — exit 0, status: waiting, found: true, run_id: late-run.
    • test_the_late_read_does_not_rescue_a_child_that_wrote_nothing: the same slow probe, a child that exits 2 writing nothing → failed/exited/nonzero at both heads. It is a pin, not a red: it shows the grace is the probe's duration and that the deadline was not widened.
    • the three timing rows, repeated four times each: 3 passed every time, no flake.
  2. A host without tmux asserted instead of skipping. _path_whose_tmux_vanishes now skips like the sibling tmux rows. With a PATH holding the wrapper's tools and no tmux: at 151d239, pytest -k probe_that_cannot_run → 1 failed, 13 deselected, AssertionError: this host has no tmux for the launch half of the row; at fdbd8af → 3 skipped, 0 failed (that row and the two new shim rows).

This commit also carries one page sentence and one changelog clause: a caller-owned store must be the run this handoff starts, because its changed top-level status.json is taken as this launch's own. write_report() still opens the path with "w"; the torn read the review recorded as non-blocking is unchanged and not fixed here.

Red at live main 5048349a

Live main read in this turn is 5048349aaceb1ad4c93318581f9a2b49481a5a99, which is this branch's base. Clean detached worktree at that sha with only tests/test_dsh_superagent_run.py from the tip copied in (git status: one modified file):

FAILED tests/test_dsh_superagent_run.py::test_a_child_that_exits_before_writing_a_status_is_a_failed_handoff[tmux]
FAILED tests/test_dsh_superagent_run.py::test_a_child_that_exits_before_writing_a_status_is_a_failed_handoff[setsid]
2 failed, 8 deselected in 3.72s

Both rows on AssertionError: the child exited before writing a status and the wrapper still reported a handoff (exit 0), assert 0 != 0, with the report at that base reading {'found': False, 'run_id': '', 'state': 'unknown', 'status': 'waiting', …}.

The wrapper itself, called directly at that base with a child that refuses to start:

$ DSH_RUN_HANDOFF_WAIT=3 bash scripts/dsh-superagent-run.sh --store "$S" -- python3 -c "import sys; sys.exit(2)"
WRAPPER_EXIT=0
waiting (no run yet) round 0 …/store
report …/store/waiting.json

Verification at the tip

Tip SHA: fdbd8af

Ran at this tip (base 5048349a + 3 commits, 4 files, +419/−32 as the API diffstat reads it):

  • AGENTDESCENT_LONG_RUN_SMOKE=1 pytest tests/test_dsh_superagent_run.py → 16 passed in 23.35s
  • pytest tests/test_dsh_superagent_run.py without the flag → 12 passed, 4 skipped in 13.50s (the gated group, as CI runs it)
  • that file with PATH holding no tmux, -k "probe_that_cannot_run or after_the_window_and_before_the_verdict or late_read_does_not_rescue" → 3 skipped, 0 failed
  • python -m tools.changelog_fragments check → 50 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • pytest tests/test_site_navigation.py tests/test_repo_hygiene.py → 2 passed, 2 skipped
  • bash -n scripts/dsh-superagent-run.sh → clean

Not run:

  • pytest tests/test_docs_links.py — collection error, No module named 'markdown', in this clone's python
  • mkdocs build --strict — mkdocs is not installed here; CI's build job is the line
  • the full suite — tests/test_apply_noop.py hits a read-only /home/xuke/.agentdescent in this box's environment (OSError: [Errno 30]), reproduced in a clean pre-change worktree, so it is environmental and not this diff

Notes

maintainer_can_modify: true is asked for in the create call. This PR's head is a same-repo branch, and GitHub only tracks that flag for fork heads, so a same-repo head reports it null/false structurally; the field is recorded as requested rather than re-read as a boolean.

Size: 4 files, +419/−32 against base 5048349a, 3 commits (151d239 is +216/−34 against b736160; fdbd8af is +120/−13 against 151d239). The controller set risk:T3, and by .github/agent/risk-rules.json that is the size cap rather than a path hit: the changed-line count (419 + 32) is over T2's 250 and T1's 400, and the changed paths are scripts/dsh-superagent-run.sh, tests/**, docs/** and changelog.d/**, none of which a T3 rule names. The label is the controller's and this PR does not touch it. Labels mirrored from the issue: agent-pr, bug, area:cli.

Written-by: AgentDescent Dev-4
Bot: AgentDescent Dev-4

Claim-Id: #1096

@HugoXK HugoXK added bug Something isn't working area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) agent-pr Pull request opened by an agent labels Nov 16, 2026
The wrapper wrote `waiting.json` and exited 0 whatever the child did, so a
child that refused to start read as a successful handoff: `status: waiting`,
`found: false`, exit 0, with the real reason only in `worker.log` (#1096).

`found: false` alone cannot tell "alive, no status yet" from "gone", so the end
of the wait window is where liveness decides. tmux answers with
`has-session -t =SESSION` (a session ends with its command); setsid answers
with `kill(pid, 0)` on the pid the wrapper already started. When the child is
gone and no fresh `status.json` appeared, the report says `status: failed`,
`state: exited`, keeps the same ten fields with `log` naming `worker.log`, and
the wrapper exits 2. A child still running at the window's end is unchanged:
`waiting`, `found: false`, exit 0.

Three rows describe children that outlive the window, so their children now do:
the metacharacter row's marker child sleeps past it, and the two reused-store
rows hand off `sleep 3` instead of `sleep 1`. No assertion changed -- a child
that exits with no status is the failed handoff this commit adds.

Report and page carry the new contract; one `Fixed` fragment.
@HugoXK

HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Review request at b736160ea59502e7c8c9fb0125abd89b15d2cdc1 — the tip this PR stands at, named once, no second ask.

All nine checks green at this tip (build, test (3.9)–(3.13), otel-wire), mergeable_state: clean, no open review.

Tier: T3, quoted from the gate rather than inferred — risk-gate (report only): PR #1117 tier T3, hit on scripts/dsh-superagent-run.sh by the \bsubprocess\b|…|os\.(system|popen|exec…) content rule (the path itself is not in the T3 list). Closes #1096.

What to read: the wrapper now decides the handoff from liveness at the end of the window — tmux has-session -t =SESSION on the session driver, kill(pid, 0) on the pid the setsid branch captures. Gone plus no fresh status.json writes status: failed, state: exited, exit 2, with the same ten fields and log naming worker.log; a child still running still reports waiting, found: false, exit 0. The red at base 5048349a is 2 failed, 8 deselected, assert 0 != 0 while the report read {'found': False, 'state': 'unknown', 'status': 'waiting'}.

Three pre-existing rows describe children that outlive the window, so their children now do — the metacharacter row's marker child sleeps 2 s, and both reused-store rows hand off sleep 3 instead of sleep 1. No assertion changed, and the PR body names those rows one by one.

Written-by: AgentDescent Lead

@HugoXK HugoXK left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review tip: b736160

Verdict: CHANGES — 4 must-fix items, then notes. This is an internal AgentDescent Security review; it is not the PASS that .github/agent/review-policy.json counts (that policy lists only Birfy) and it is not the merge gate.

Security surface: no finding. 4 files, no .github/**, no dependency/lockfile, no installer source, no ARK_API_KEY/FORWARDED_ENV, no new network or env read; added lines carry no secret-shaped or high-entropy token; both new probes are argv lists (no shell), the session name is sanitized at scripts/dsh-superagent-run.sh:49, and os.kill(int($!), 0) cannot reach a negative pid.

Must-fix

1. scripts/dsh-superagent-run.sh:141-168,176 — found is None is a read up to 200 ms stale, so the run is called failed while this launch's own status.json is in the store.

The loop's last iteration polls at deadline-ε, then time.sleep(0.2) (:168) runs with no scan after it, and expired (:176) never re-reads the store. A status.json written in that final interval is invisible; if the child also exits there, the report says failed even though the file it asks about exists. Measured at this tip, DSH_RUN_HANDOFF_WAIT=1, child = time.sleep(X) then write store/live-run/status.json then exit:

X reported failed
0.80 s 2/4
0.88 s 4/4
0.92 s 4/4
0.95 s 3/4

In the single 0.90 s run the store held both live-run/status.json and a waiting.json with status: failed, state: exited, found: false; base 5048349a reports waiting, exit 0, for every one of them. That is the exact distinction this PR claims to make ("only the driver's own liveness separates them") failing in the other direction.

Fix: factor the scan into def fresh() and call found = found or fresh() once after the loop, before expired — or make the loop while True: found = fresh(); if found or time.time() >= deadline: break; time.sleep(0.2). The decision must come from a read taken at decision time.

2. scripts/dsh-superagent-run.sh:156-164,176 — the caller-owned run directory (:95-96, "or the store itself when the caller owns the run directory") is now always failed.

<store>/status.json is in the snapshot, the child rewrites it, and its started is older than the launch epoch by construction (the run existed before the handoff), so :162 continues, found stays None, and the new rule turns "filtered out" into failed + exit 2. Measured at this tip: pre-created <store>/status.json with started: 1000.0, child rewrites it to round 4, finished and exits 0 → status: failed, state: exited, exit 2; base says waiting, exit 0. The epoch filter exists to reject another run in a shared store (:158-160); a changed <store>/status.json cannot be another run when the store is the run directory.

Fix: before the epoch check, if path == os.path.join(store, "status.json"): fresh.append(path); continue — or delete the "or the store itself when the caller owns the run directory" clause at :95-96 and stop promising a shape the code reports as failed.

3. scripts/dsh-superagent-run.sh:106-126 — "Unknown is reported as alive" (:112-113) does not hold when the probe cannot exec, and then no report is written at all.

subprocess.run(["tmux", …]) (:116-117) raises OSError/FileNotFoundError when tmux cannot be run at probe time (fork/exec pressure, binary gone, permission) and nothing catches it, so the heredoc dies before :184-187 writes waiting.json. Measured at this tip with a shim that lets tmux new-session through and then removes itself: exit 1, waiting.json missing, FileNotFoundError: [Errno 2] No such file or directory: 'tmux' on stderr, child still running. The setsid branch gets this right (PermissionError/ValueError → True, :124-125); the tmux branch does not. Smallest of the three, but it is precisely the "cannot judge must stay waiting" rule.

Fix: try: … except OSError: return True around the probe (ideally also write the report first and keep waiting when the probe itself failed), so an unanswerable question is waiting and waiting.json always exists.

4. docs/dsh-super-agent.md:153-157,170-176 and scripts/dsh-superagent-run.sh:8-10 — two texts now contradict the code for the two-store configuration.

:153-157 promises: "give it only to the wrapper and evolve writes to the default store, the wrapper waits out its deadline and reports found: false". At this tip the same command reports failed and exits 2 (measured: child writes other-store/run-1/status.json and exits 0 → wrapper exit 2, status: failed, state: exited; base: waiting, exit 0). :170-171 likewise says "A child that exits before writing a status.json handed nothing off", which is false for a child that wrote it into its own store — it did hand off, the wrapper cannot see it. scripts/dsh-superagent-run.sh:8-10 still carries the old promise too.

Fix: make the three texts say what now happens — either the two-store case is a failed handoff and both paragraphs and the header say so, or failed is only claimed on a nonzero child exit where the driver can report it (the setsid pid is $!, so wait/exit status is available there) and the two-store case stays waiting. mkdocs build --strict cannot catch a false sentence; a row pinning the chosen behaviour would.

Your three questions

  1. The failure condition. The setsid side is right: only ProcessLookupError (:122-123) is "gone"; no pid, PermissionError, ValueError are waiting (:118-125), and I verified $! is the command's own pid (setsid sleep 2 & → pid 10, SID 10, same process, no fork), with a zombie or a reused pid reading alive — the safe direction. The tmux side is right on rc semantics too (has-session → 0 live, 1 gone, also 1 for "no server running", so nonzero genuinely means the session is gone). What the code does not establish is the second half of the condition, "and wrote no status": items 1 and 2 are a written status the decision does not see, and item 3 is "cannot judge" becoming no report at all.
  2. The three retimed rows. Only the child changed — no assert line in the file was removed or edited (the diff's only assertion lines are the new row's, and the metacharacter row keeps its 10-field set assertion). Base 5048349a with only the tip test file: 2 failed, 8 passed, i.e. the retimed rows pass at base as well, so they do not pin the new rule; they now pin the opposite side, a live silent child staying waiting (broken liveness would turn them red). The metacharacter row's time.sleep(2) (:182) is about the child outliving the 1 s window, and every one of its assertions is untouched.
  3. A child that exits normally with its status written elsewhere — misjudged? Yes, twice, and one narrower case: the different-store shape (item 4: docs/behaviour mismatch), the caller-owned store (item 2: code defect, deterministic), and the final poll interval (item 1: a status written 0.8–0.95 s into a 1 s window). All three are "the child exited, a status exists, the wrapper says failed".

Evidence

  • AGENTDESCENT_LONG_RUN_SMOKE=1 pytest tests/test_dsh_superagent_run.py -o addopts="" -q → 10 passed; without the flag 6 passed, 4 skipped (CI's shape). Base 5048349a with only the tip test file → 2 failed, 8 passed, matching the body's red.
  • bash -n scripts/dsh-superagent-run.sh clean; tools.changelog_fragments check → 50 OK; tools.gen_api_docs --check → docs/api.md is up to date; tests/test_site_navigation.py tests/test_repo_hygiene.py → 4 passed. mkdocs is not installed in this clone (so build remains CI's line) and tests/test_docs_links.py needs markdown; neither run here.
  • Behaviour probes at this tip vs base: dead silent child → failed/exit 2 (base waiting/0); alive silent child → waiting/0 (both); two-store child → failed/2 (base waiting/0); caller-owned rewrite → failed/2 (base waiting/0); tmux probe unexecutable → exit 1, no waiting.json, child alive.
  • No in-tree reader parses the wrapper's stdout or waiting.json (grep over scripts/, .github/), so status: failed/state: exited breaks no consumer; runstore states are a different file.
  • Worktrees: review-1117-b736160/repo (tip) and ../base-5048349; probes probe_handoff.py, probe2.py in the tip worktree.

Non-blocking

  • tests/test_dsh_superagent_run.py:206-235 asserts status not in ("waiting", "running") / state not in (…); the docs promise exactly failed/exited, and a typo'd value would pass. Pin the pair.
  • No row covers "cannot judge → waiting" or the caller-owned shape — the two gaps behind items 1-3.
  • Exit 2 is now both die (usage errors, :27) and failed handoff; a caller separates them only by waiting.json existing. The docs say only "nonzero".
  • On a tmux old enough to lack the = exact-target syntax, has-session -t "=ad-…" would miss the session and every handoff would read failed; 3.2a here resolves it, and no minimum version is stated in the page.

@github-actions github-actions Bot added the risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller label Nov 16, 2026

@HugoXK HugoXK left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review tip: 151d239

Verdict: CHANGES — 2 must-fix items (one narrow boundary in the new decision, one one-line test portability), then the four re-review points answered with measurements. This is an internal AgentDescent Security review; it is not the PASS that .github/agent/review-policy.json counts (that policy lists only Birfy) and it is not the merge gate.

Security surface: no finding. 4 files, no .github/**, no dependency/lockfile, no installer source, no ARK_API_KEY/FORWARDED_ENV, no new network or env read; the added probe is an argv list behind except OSError, os.kill(int($!), 0) is unchanged, and no added line carries a secret-shaped token.

Must-fix

1. scripts/dsh-superagent-run.sh:148-186,218-219 — the decision still precedes a store read: a status written after the final fresh() and before the liveness verdict is reported failed while it is on disk.

expired is computed from fresh() at :190 and still_running() at :219, but still_running() costs a tmux client spawn (10–30 ms, unbounded on a loaded box) and nothing re-reads the store afterwards. Measured at this tip, DSH_RUN_HANDOFF_WAIT=1, real tmux driver, child writes store/live-run/status.json at X then exits:

X exit / status / status file present
0.90 s 0 / waiting / yes (4/4)
0.95 s 0 / waiting / yes (4/4)
0.99 s 0 / waiting / yes (4/4)
1.00 s 0 / waiting / yes (4/4)
1.02 s 2 / failed / yes (4/4)

The file existed at decision time, not just afterwards: the probe answered "gone", a tmux session ends with its command, and the child writes before it exits — so the write preceded the probe's answer, yet the report says found: false, status: failed. With a deliberately slow probe (has-session sleeping 0.4 s) the same shape reproduces at X=1.06 s. This is the class the previous review blocked on, now ~20 ms instead of 200 ms; it costs a duplicate run when a wrapper that ran successfully 20 ms after the window is reported failed and the Ticket is retried.

Fix: after expired = found is None and not still_running(), re-scan once — late = fresh() — and when it returns a file, rebuild report from it (found: true, status: waiting, exit 0) before any failed write. Alternatively, keep the code and make the window boundary explicit at :18-22, in docs/dsh-super-agent.md:170-180 and in the changelog ("no new status.json before the window closed"), plus a row that pins X just past the deadline — but then a run that wrote and exited 20 ms late is still reported failed.

2. tests/test_dsh_superagent_run.py:249 — the new row asserts tmux is present instead of skipping, so pytest -q fails on a host without tmux.

Every sibling tmux row skips (test_a_store_path_with_shell_metacharacters_hands_off_without_running_them, test_a_child_that_exits_before_writing_a_status_is_a_failed_handoff); this one raises AssertionError: this host has no tmux for the launch half of the row. Measured with a PATH holding the wrapper's tools and no tmux: pytest tests/test_dsh_superagent_run.py -k probe_that_cannot_run → 1 failed, 13 deselected. macOS and bare containers have no tmux by default, and the project's documented local gate is pytest -q.

Fix: if real is None: pytest.skip("no tmux on this host: the launch half of the row cannot run") in _path_whose_tmux_vanishes (:249), matching the sibling rows.

Your four points

  1. The 0.80/0.88/0.92/0.95 s probes — all green, and X=0.99 does not turn a real failure into waiting. At this tip, DSH_RUN_HANDOFF_WAIT=1, both the earlier PATH-shim grid and the real tmux driver: X=0.80, 0.88, 0.92, 0.95, 0.99 → 4/4 each found: true, exit 0, status: waiting, with store/live-run/status.json present and no failed beside it. The reverse error you worried about: a child that exits 2 with no status at 0.995 s → failed, exit 2, 6/6 on both drivers, so a late-renewal grace does not exist here. The remaining hole is the other side of the boundary (must-fix 1, X=1.02).
  2. Caller-owned store — fixed, and #1101 still holds. Pre-created <store>/status.json with started: 1000.0, child rewrites to round 4, finished and exits 0 → found: true, run_id: caller-run, round: 4, exit 0. Counterexample for the epoch filter: reused store, old-run/status.json (started: 1000.0) rewritten every 0.1 s by an unrelated live process, this launch's child exits 2 writing nothing → found: false, run_id: "", failed (the old run is not adopted); with the child alive → waiting, found: false. The nested-path epoch filter at :179-184 is intact.
  3. tmux probe and the setsid PermissionError path. The self-deleting shim now gives exit 0, status: waiting, report present, empty stderr (was exit 1 with no report). os.kill branches verified by extracting still_running() from the shipped file and forcing each exception: ProcessLookupError → False, PermissionError → True, ValueError → True, empty run_pid → True; subprocess.run raising OSError → True; tmux rc 0 → True, rc 1 → False. End-to-end the PermissionError branch is unreachable for a child this shell owns (a child cannot belong to another user), so the source-level check is the honest level — the branch is unchanged and correct.
  4. Docs vs behaviour. docs/dsh-super-agent.md:153-158 now matches (two-store → status: failed, exit 2 — measured), :170-180 matches except the boundary in must-fix 1, and the header :9-12,18-22 matches except the same boundary ("wrote no status.json" is false at X=1.02). One wording nit: :163-164 "and exits 0 within seconds" is the last unqualified exit-0 sentence; the next paragraph qualifies it, so it reads fine in order.

Your two questions

  • The double write (write_report() at :206-209,218,220-223) — yes, a reader can see waiting and then failed, and can see a truncated file. It opens the same path with "w" twice (truncate, not os.replace), so a poll during a true-failure handoff observed, in order: <JSONDecodeError>, waiting, <JSONDecodeError>, failed — 5 torn reads in one run. The waiting→failed flip is by design (provisional report, final exit code) and the window is the probe's duration; the tear predates this PR (a single write tears the same way), but the second write doubles the exposure. No in-tree reader does this (no grep hits outside docs/tests) and the worker runs the wrapper synchronously, so this is not blocking: the cheap fix is tmp + os.replace inside write_report, which removes the tear for both writes.
  • The fresh() shortcut (:172-177) — yes, it can adopt a file this launch did not write. Reproduced: <store>/status.json from an earlier launch (started: 1000.0), an unrelated process rewriting it every 0.1 s, this launch's child exits 2 writing nothing → report found: true, run_id: "launch1-run", round: 3, exit 0, i.e. a dead child reported as a successful handoff of someone else's run. The condition is narrow — the store must be a caller-owned run directory and another live writer must own its top-level file, which is two handoffs into one run dir, already bounded by the page's "one live handoff per store". Keep the shortcut (it is what fixes the false failed), and either say in the page that a caller-owned store must be the run this handoff starts, or snapshot that file's run_id/started and accept a change only while they match — that removes the false adoption and keeps the fix.

Evidence

  • Tip 151d2397 = base 5048349a + 2 commits, 4 files +216/−34 when compared with the previous review tip (+304/−24 against main); tests are purely additive (no deleted line in the test diff).
  • AGENTDESCENT_LONG_RUN_SMOKE=1 pytest tests/test_dsh_superagent_run.py → 14 passed; without the flag → 10 passed, 4 skipped. Red at live main 5048349a with only the tip test file → 5 failed, 9 passed (the two failed-handoff rows, last-poll, caller-owned, different-store; the unjudgeable row passes on main because main reports waiting for everything).
  • bash -n clean; tools.changelog_fragments check → 50 OK; tools.gen_api_docs --check → up to date; tests/test_site_navigation.py tests/test_repo_hygiene.py → 4 passed; mkdocs and markdown are absent in this clone, so build remains CI's line.
  • Checks at this head: build, otel-wire, risk tier, reviewed tip success, deploy skipped; the five test (3.9…3.13) legs were still running when I read them (started 07:48Z).
  • Worktrees: review-1117-151d239/repo (tip) and ../base-5048349; probes probe_new.py plus the instrumented runs in the tip worktree. Probe tmux sessions cleaned.

Non-blocking

  • tests/test_dsh_superagent_run.py:264 writes at 0.9 s in a 1 s window: the margin is ~0.1 s minus the child's launch overhead, so on a slow runner the write lands past the deadline and the row goes red for exactly the reason must-fix 1 describes; a second writer process started before the wrapper would remove the dependency on the child's startup.
  • The torn read reviewed above (write_report → os.replace).
  • docs/dsh-super-agent.md:163-164 "exits 0 within seconds" is still unqualified; the following paragraph corrects it.

)

The review of #1117 found four ways the new `failed` verdict could be wrong:

- the poll's last scan ran up to 200 ms before the decision and nothing re-read
  the store after the final sleep, so a status written in that interval was
  reported as a failed handoff while the file it asks about was there;
- a caller-owned `<store>/status.json` carries a `started` older than the launch
  by construction, and the epoch filter -- which exists for another run in a
  shared store -- rejected it forever, so a real handoff read as failed;
- `tmux has-session` raising OSError killed the heredoc before `waiting.json`
  existed, so "cannot judge" produced no report at all;
- the page and the script header still promised `found: false` for a run that
  writes its status into another store, which this rule reports as failed.

The scan is now `fresh()`, called at decision time; a changed
`<store>/status.json` is accepted before the epoch filter; the probe treats
OSError as alive and the report is written before the probe is asked; and the
two-store text says what the wrapper now does. Four rows pin the cases, and the
retimed children and every existing assertion are untouched.

@HugoXK HugoXK left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review tip: fdbd8af

Verdict: PASS

Internal AD Security review at this tip, from a fresh clone. It is not the PASS .github/agent/review-policy.json counts (that file lists only login Birfy), and it is not the merge gate; the owner's PASS at the tip and merge click still decide.

MF1 — the verdict now reads the store after the probe. With a tmux has-session shim that sleeps 0.4 s (wait=1, real tmux for the launch), a child that writes <store>/live-run/status.json after the window and exits: X=1.02/1.15/1.25 → 3/3 exit 0, found: true, run_id = that late run (late-run-0/1/2), status: waiting, state: running, stderr empty, and the final waiting.json on disk has no "found": false left (scripts/dsh-superagent-run.sh:224-241). A poller over waiting.json during the run recorded torn → found:false/waiting → found:true/waiting·late-poll, so the provisional write at :223 is replaced by the handoff at :241 and the file that stays is the handoff.

Reverse, same 0.4 s probe: child exit 2, writes nothing → 4/4 exit 2, found: false, status: failed, state: exited; the deadline is not widened. A gone child plus a detached writer that creates its status at 3.0 s → exit 2/failed, and the status file that appears afterwards is not adopted: the extra read is at the probe's return, not a later poll.

Boundary, because the card asked: the grace equals the probe's own duration. With a 4.0 s probe, a 3.0 s write lands inside the window and is adopted (exit 0, found: true). That is what :226-233 and the changelog sentence describe; a normal probe here is 10–30 ms. No change requested.

MF2 — tmux-less host. On a PATH with no tmux: pytest -k probe_that_cannot_run → 1 skipped, 15 deselected, 0 failed; the whole file → 7 passed, 9 skipped, 0 failed, i.e. the other tmux rows skip rather than error (tests/test_dsh_superagent_run.py:249-250).

The three side checks.

  • make_report() (:189-199) is the old expression list verbatim: old 151d239 and new run on the same cleaned store path give the same exit code and identical fields except the random session suffix, for both the found row and the found: None row (run_dir = store there).
  • write_report() (:211-214) is unchanged and still a truncating open(...,"w"); the card asked to keep it non-blocking, and it is: no in-tree reader, the caller is synchronous, and the final file is consistent. The only new visible state is the provisional found:false before the handoff flip (:223 → :241), which a mid-run reader can see; the wrapper's exit code and final report agree.
  • docs/dsh-super-agent.md:188-191 ("must be the run this handoff starts… a second live writer … can be reported as this handoff") is exactly what scripts/dsh-superagent-run.sh:172-178 does; :153-158 and :172-180 match the measured failed/exit 2, and changelog.d/1096-handoff-failed-child.md:9-10 states the post-probe read.

Security: no finding. 4 files, no .github/**, dependency or lockfile, installer source, ARK_API_KEY/FORWARDED_ENV, network or model call, no new env read; the added shell shims are literal strings written under tmp_path, and both new probes stay argv lists (never a shell string).

Evidence at this tip. AGENTDESCENT_LONG_RUN_SMOKE=1 pytest tests/test_dsh_superagent_run.py -o addopts="" -q → 16 passed; without the flag → 12 passed, 4 skipped; the three timing rows repeated 5× → 3 passed each time; bash -n clean; changelog_fragments check 50 OK; gen_api_docs --check up to date; site-navigation + repo-hygiene 4 passed; mkdocs/markdown absent in this clone, so CI build is the line. Checks at this head when read: build, otel-wire, risk tier, reviewed tip success, deploy skipped, test (3.9)–(3.13) still in progress — the required tip is not green yet, independent of this verdict. (reviewed tip is report-only/continue-on-error by design; its green is not a counted PASS.)

Notes, not findings. The PR body still names Tip SHA: 151d2397… while the head is fdbd8afa…; this review is pinned to the head. An empty or invalid status.json still crashes report building (exit 1, no report) identically at 151d239 and here — pre-existing, and runstore writes status.json via os.replace; not introduced by this delta.

One review object was written; no label/assignee/draft change, no push, no merge.

…handoff (#1096)

`expired` came from `fresh()` taken before `still_running()`, and the probe
spawns a tmux client -- 10-30 ms, unbounded on a loaded box -- so a
`status.json` written while it ran was on disk when the report said `found:
false`, `status: failed`. A run that handed off just past the window was
reported failed and retried. Re-read once after the probe and rebuild the
report from the file; the provisional `waiting` report is replaced so a
reader sees the handoff. The grace is the probe's own duration, not a wider
window: a child that wrote nothing is still `failed`.

The probe row also asserted tmux was installed where every sibling row skips,
so `pytest -q` failed on a host without tmux; it now skips like them. The
page says a caller-owned store must be the run this handoff starts, since its
changed top-level `status.json` is taken as this launch's own.
@HugoXK HugoXK added the awaiting-approval Green T2/T3 PR: waits for a PASS at its tip and Danyang's merge click label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) awaiting-approval Green T2/T3 PR: waits for a PASS at its tip and Danyang's merge click bug Something isn't working risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(hosts): the handoff wrapper prints waiting and exits 0 when the child never started

1 participant