fbuild-daemon leaks the serial fd, breaking the next run; 'daemon stop' cannot see it

[zackees]

Summary

fbuild-daemon keeps an open fd on the serial port after a run finishes, so the next run fails with Could not open /dev/ttyACM0, the port is busy or doesn't exist. There is no supported way to clear it: fbuild daemon stop reports "daemon is not running" while the daemon is demonstrably alive and holding the fd.

On an unattended bench this reads as a wedged board. It is not — the device stays enumerated the whole time.

Evidence

After a failed autoresearch rp2350w --net-peer --ota run, the ESP32-C6 port would not open:

RuntimeError: attach failed: open_port(/dev/ttyACM0) exceeded 3s; serial driver may be wedged

The device was still enumerated (303a:1001), the node existed with correct ownership, and lsof reported nothing. Walking /proc/*/fd found the holder:

holder pid=414394 comm=fbuild-daemon

open() returned EBUSY while held. After killing the daemon it opened in 0.00 s (it had been taking 13.3 s while contended).

It reproduces. The next autoresearch invocation spawned a new daemon that did the same thing, breaking the run after it:

holder pid=434691 comm=fbuild-daemon age=179s
ERROR: A fatal error occurred: Could not open /dev/ttyACM0, the port is busy or doesn't exist.

Killing pid 434691 freed the port, and the following run deployed and passed RPC smoke normally.

The recovery path is missing

$ fbuild daemon stop
daemon is not running

$ ps -eo pid,etimes,comm | grep fbuild
 434691  180 fbuild-daemon        # holding /dev/ttyACM0

stop cannot see the process it needs to stop, so kill is the only way out. That is the same instance-visibility problem soldr has with a version-mismatched daemon, and it turns a transient leak into a hard stop for anyone who does not think to inspect /proc.

Impact

1. Any run following a failed peer/OTA run fails on a port that is actually fine.
2. The error message points at the serial driver and the device ("serial driver may be wedged"), not at the daemon holding the fd, so the natural next step is to power-cycle or reflash a healthy board.
3. Unattended, it stops a bench until someone intervenes.

Suggested fixes

• Release the serial handle when the monitor/deploy session ends, including on the error paths — a failed OTA run is exactly when this leaked.
• Make fbuild daemon stop find and stop a running daemon regardless of how it was started, or say which process it could not match instead of "not running".
• When an open fails with EBUSY, name the holding pid rather than blaming the serial driver. The information is in /proc/*/fd and would have turned a long investigation into one line.

Environment

NixOS, Linux 6.18.48, x86_64, fbuild 2.5.23. ESP32-C6 (303A:1001) and RP2350W (2E8A:F00F) on the same host.

Related: #1424 (a healthy port reported unusable for a host-side reason — same family).

Generated with Claude Code

https://claude.ai/code/session_01KkufoNxfnNRU9psT3R9F51

[zackees]

Stronger evidence: four distinct daemons, and it breaks a real test path

Since filing, this reproduced on every deploy today. Four separate fbuild-daemon pids each grabbed /dev/ttyACM0 and held it:

holder pid=414394 comm=fbuild-daemon
holder pid=434691 comm=fbuild-daemon   age=179s
holder pid=544820 comm=fbuild-daemon   age=128s
holder pid=551628 comm=fbuild-daemon

Each one leaves the next run failing with Could not open /dev/ttyACM0, the port is busy or doesn't exist on a board that stays enumerated (303A:1001) and is perfectly healthy — after kill, the same board deploys and passes RPC smoke immediately.

open() timing makes the contention explicit: 13.3 s while a daemon held it, 0.00 s after killing it.

It is not just noise — it breaks a test that otherwise passes

FastLED's autoresearch rp2350w --net-peer --ota fails at its first RPC because the C6's port is still held by the daemon from that run's own deploy. The failure presents as the device being unresponsive, and it took a per-link ping probe to establish that the port, not the board, was the problem.

The duration correlation is the useful clue:

Mode	C6 deploy time	Outcome
--net-peer	96.5 s	works — 6 full cycles
--net-peer --ota	25.7 s	first RPC to the C6 times out

A longer deploy apparently gives the leaked handle time to go away; a shorter one loses the race. So this is intermittent in a way that tracks deploy duration, which makes it easy to misattribute to whatever test happened to be running.

Diagnostic note worth acting on

lsof /dev/ttyACM0 returns nothing for these holders — only walking /proc/*/fd finds them. Combined with an error message that says "serial driver may be wedged", the natural conclusion is a hardware fault. I power-cycled reasoning around a healthy board more than once before checking /proc.

Naming the holding pid on EBUSY would collapse this from a multi-step investigation into one line.

Generated with Claude Code

https://claude.ai/code/session_01KkufoNxfnNRU9psT3R9F51