diff --git a/docs/architecture/delegation-capacity.md b/docs/architecture/delegation-capacity.md new file mode 100644 index 000000000..2e3fd6cea --- /dev/null +++ b/docs/architecture/delegation-capacity.md @@ -0,0 +1,72 @@ +# 委派并发上限：超出时排队 + +Host 可以限制本机同时运行的顶层 Thread 数量。开启后，`delegate start` 遇到上限时不立即创建 Thread，而是排队：命令照常立即返回 Thread 标识，状态为 `queued`，并给出排队位置；有空位时按先后自动开始。默认关闭，关闭时 `delegate start` 的行为与以前相同。 + +## 设置 + +- 位置：Codex Desktop 设置 → codexhost“通用”页，“资源管理”分组下方的“委派”分组。开关“限制同时运行的委派”默认关闭；开启后才显示“同时运行上限”，取值为 1～32 的整数，默认 4。 +- 存储与下发沿用空闲释放：Renderer 写入 localStorage（`codexhost.delegation-capacity.v1`），连接本机 Host 后以及每次修改时，通过 `codexhost/settings/delegation-capacity/set` 下发。请求只发给本机 Host，不随当前远程会话切换。Host 只把设置保存在内存中，不写配置文件。设置的校验规则由 `shared-contracts` 提供，Renderer 与 Host 共用。 +- 旧 Host 不认识该请求，或 Host 没有挂接委派控制时，返回“方法不支持”，设置页显示“当前 Host 不支持”，不会当作已生效。 + +## 计数 + +占用名额的是本 Host Runtime 中正在运行（有活跃 Turn）的顶层 Thread： + +- 用户手动打开的顶层 Thread、以及外部客户端（`client:*`）通过委派创建的顶层 Thread 都算，外部 Harness 与原生 Codex 都算； +- 全部三类子会话均不计入顶层运行数： + 1. 原生与外部 Harness 的 Subagent（外部 `record.subagent`，原生 `thread/started` 中的子代理来源或 `spawnAgent` 的接收方）； + 2. Fork 派生的子会话（外部 `record.forkSource`，原生 `forkedFromId` / `history_base.thread_id`）； + 3. 由已有会话通过 `delegate start` 派生的子会话（存在 `DelegationRecord` 且 `parentHostThreadId` 不是 `client:*`）； +- 已放行、正在创建、尚未报告为运行的顶层启动先占一个名额，避免多个并发启动一起越过上限。 + +原生 Codex Thread 依据 Host 观察到的 Turn 开始与结束事件计数。Host Runtime 重启前已经在运行的原生 Turn，要等到新的事件才会重新计入；Host 没有观察到子代理或 Fork 身份的原生 Thread，按顶层 Thread 计入。 + +## 哪些请求会排队 + +- 只有外部客户端（`client:*`）发起的顶层 `delegate start` 会受顶层并发上限约束并排队（外部 Harness 与 `--harness codex` 都一样，CLI、MCP `delegate_start` 以及带 `--watch true` 的启动也都一样）；由已有会话派生的子会话属于该父会话的内部子任务，直接放行，不进入顶层队列。 +- `thread send`、用户在 Desktop 中发送的消息，以及对已有 Thread 的后续消息，从不排队，也不会因上限失败，即使运行数已经超过上限。 +- 参数、父 Thread、工作区与 Model 配置的校验，以及幂等去重，都在排队前完成：无效请求直接报错，不进入队列。相同 Request ID，或 30 秒内同一父 Thread、目标与任务的重复请求，返回同一个排队中的 Thread；Request ID 相同而配置不同时返回 `INVALID_ARGUMENT`。 +- 已经有启动在排队时，新的启动排在后面，即使此刻有空位；空位总是先给队首。 + +## 排队中的 Thread + +排队时 Thread 的标识就已分配，开始运行后沿用同一个标识。排队期间拿到的链接、注册的 watch 与后续命令，在开始后继续有效。 + +| 命令 | 行为 | +| --- | --- | +| `delegate start` | 立即返回 Thread 链接、`status: "queued"` 与 `queue.position`（compact 输出为 `queuePosition`），以及工作区、标题和父 Thread | +| `thread read` | 返回 `status: "queued"` 与 `queue.position`，没有 Turn；开始创建后为 `creating`，之后与普通 Thread 相同 | +| `thread wait` | 排队时间计入超时；到期时 Thread 仍在排队，则返回排队快照与 `timedOut: true` | +| `thread watch` | 照常注册。排队与创建期间既不算停下，也不算卡住提醒的静默；Thread 开始后真正停下才通知。被移出队列时通知 `interrupted` | +| `thread list` | 排队中的 Thread 按队列顺序排在第一页最前面，不占 `--limit`，续页不再出现；同样按 `--parent` 或工作区过滤 | +| `thread cancel` | 把 Thread 移出队列并返回 `cancelled: true`。之后读取为 `interrupted`，并说明它在开始前已被移出队列，不会再启动；原生 Thread 已有 Turn 时按已开始的委派取消 | +| `thread send` | 返回 `THREAD_BUSY`，`details.status` 为 `queued` 或 `creating`，排队时附 `queuePosition`；移出队列或启动失败后返回 `THREAD_NOT_FOUND` | + +移出队列、启动失败或被重启中断的外部 Harness 启动不会生成 Thread；原生 Codex 的排队 Thread 见下文。远程 Host 上排队中的 Thread 同样可以用 `thread read --host` 读取。Host 在内存中保留最近 50 条这类记录，使 `read`、`wait`、`cancel` 与 watch 能给出终态；Host Runtime 重启后不再保留。 + +## 自动开始 + +队列每 2 秒检查一次正在运行的 Thread（与 `thread wait`、watch 一样基于轮询），有空位就按先后启动队首，直到名额用完。上限调高或关闭后，等待中的启动随之开始。启动沿用排队时记录的参数（任务、任务名、工作区、父 Thread、Model 与 Thinking 选择）和已分配的 Thread 标识。启动失败时，该 Thread 读取为 `failed` 并附失败原因，同时写入 Host Runtime 日志。 + +## 原生 Codex 的排队 + +官方后端不接受调用方指定的 Thread 标识，所以 `--harness codex` 需要排队时，Host 先用官方 `thread/start` 与 `thread/name/set` 建好 Thread，只把第一条 Turn 排队。返回的标识就是真实的原生 Thread，深链接、父子关系和子进程环境不需要别名。没有排队、直接放行的启动仍按原流程创建。 + +- 没有 Turn 的原生 Thread 会写入 rollout，官方后端重启后仍能 `thread/resume` 并开始 Turn。官方 `thread/list` 不列出没有 Turn 的 Thread，因此 Desktop 重启后，排队中的原生 Thread 可能不在线程列表中出现，直到第一条 Turn 开始；`codexhost thread list` 仍由队列给出。空闲释放只作用于外部 Harness，不会卸载它。 +- 开始时，Host 先 `thread/resume`，再读取最新 Turn；已经有 Turn 时沿用它补建委派记录，不再发送任务。 +- 用户在 Desktop 中直接给排队中的原生 Thread 发消息时，Host 把排队的任务放在这条消息之前一起发出。在后端答复之前，这一项留在原位、标为正在认领：队列不会同时开始它，取消要等认领结束。后端开始了 Turn，才移出队列并建立委派记录；后端拒绝时取消标记，仍在原位排队。任务只执行一次，这条 Turn 不受上限限制，Model 与 Thinking 以 Desktop 的请求为准。 +- 排队相关的代码从不删除 Thread：后端对有进行中 Turn 的 Thread 也会执行 `thread/delete` 并删掉 rollout。`thread cancel` 先读取 Thread 的 Turn：没有 Turn 时移出队列，之后读取为 `interrupted`，空 Thread 留在后端（官方 `thread/list` 不列出它）；已有 Turn 时按已开始的委派补建记录并中断这条 Turn。取消与用户消息交错时，最坏结果是用户的消息照常进入这个 Thread、委派记为取消，内容不丢。 +- 重启前正在开始或正在认领的原生记录，恢复后标为“可能已开始”。之后无论是自动开始、取消、用户发消息还是 `thread read`，都先 `thread/resume` 并读取 Turn：已有 Turn 时按已开始的委派收编，补建委派记录，watch 随之观察这个 Thread，不再发送任务，也不再把任务拼进用户消息；没有 Turn 才按排队处理。Host Runtime 关闭时仍把启动结果写盘；强杀时写不了，靠这项检查保障。 + +## 重启 + +- 队列保存在 `<数据目录>/delegation-queue/`（默认 `~/.codexhost`），每个 Runtime 进程一个仅当前用户可读写的文件。与 watch 记录不同，排队记录必须包含任务文本，因为 Thread 在开始前并不存在；启动后记录随即删除。 +- 多个 Runtime 进程可以共用数据目录。每个进程只写自己的文件；启动时接管已退出进程（或上次开机遗留）的队列，每个文件的接管通过跨进程文件锁串行处理，并以原子重命名认领，同一记录只被一个进程接管，接管后按原排队时间合并。无法解析的记录改名为 `.invalid` 留存并写诊断，不阻止 Runtime 启动。 +- 上限设置只在内存中，重启后要等 Renderer 重新下发。在此之前，恢复的启动保持排队，新的启动排在它们后面；最多等待 5 分钟，仍未收到设置就按关闭处理，依次启动。 +- 重启时正在启动的外部 Harness 记录无法确认 Thread 是否已经创建，按 `interrupted` 结束并说明原因，不自动重试；如果 Thread 实际已经创建，读取时以持有它的会话为准。 + +## 实现与边界 + +- `packages/host-runtime/src/delegation-capacity-queue.ts` 实现计数、排队、恢复与自动开始，由 `DelegationControlRegistry` 持有，位于各 Host 会话之上；`delegation-capacity-store.ts` 负责队列文件。各会话通过注册提供正在运行的顶层 Thread，并把收到的设置转交给队列。 +- 外部 Harness 的名额检查位于创建 Thread 之前；原生 Codex 的在 `AppServerHost` 中，需要排队时才通过 `prepare` 建好 Thread，去重先于建 Thread，重试不会留下多余的 Thread。排队的启动开始时，以已分配的标识重新进入同一启动流程，跳过名额检查与去重。 +- 只有收到设置的 Host Runtime 计数并限制。同一台机器上的其他 Runtime（例如远程监听）既不计入，也不受限制；它们只会接管已退出 Runtime 留下的队列文件。 diff --git a/docs/architecture/delegation-thread-list.md b/docs/architecture/delegation-thread-list.md index c931a987c..86d6ec68a 100644 --- a/docs/architecture/delegation-thread-list.md +++ b/docs/architecture/delegation-thread-list.md @@ -9,6 +9,8 @@ - 没有 catalog 且只有一个注册时，直接透传该注册的列表和 cursor。 - 没有 catalog 且有多个独立注册时，Registry 按请求的 created、updated 或 recency 时间升／降序归并各来源。各来源内部的同时间戳顺序由来源维护，不建立新的全局同时间戳次序契约。 +开启[委派并发上限](delegation-capacity.md)时，排队中和正在创建的委派 Thread 只存在于 Registry 持有的队列中。不带 cursor 的请求把它们按队列顺序放在第一页最前面，同样按 `--parent` 或 cwd 过滤；它们不占 `limit`，不进入任何 cursor，续页不再出现。 + 全局列表不加载外部 Harness Session 来枚举历史。发送、读取等 Thread 操作仍走既有所有权路由，不因列表 catalog 转移执行归属。 ## 独立来源的复合 cursor diff --git a/docs/architecture/external-clients.md b/docs/architecture/external-clients.md index 3b7d215be..61931638c 100644 --- a/docs/architecture/external-clients.md +++ b/docs/architecture/external-clients.md @@ -34,6 +34,7 @@ - 这类委派写入普通的 v1 Delegation 关系，父标识为 `client:`。幂等重试、`thread list --client `（即按该父标识列举）沿用既有路径；旧版本读取时只是一个不存在的父 Thread，不会丢弃记录。 - 客户端没有自己的工作目录，所以无父委派必须给出 `--cwd`；Registry 把它交给共享 catalog 或唯一的 Host 会话，无法唯一确定时返回 `PARENT_THREAD_AMBIGUOUS`。 - 经外部监听且未给 `--cwd` 的 `thread list` 不限定工作目录。 +- 开启[委派并发上限](delegation-capacity.md)时，客户端发起的委派同样受限：超出时 `delegate start`（MCP `delegate_start`）返回 `queued` 与排队位置，watch 照常等到 Thread 开始后真正停下。 ## 引用 @@ -43,7 +44,7 @@ ## 事件 -- 被通知方为客户端的 watch 停下时，Host 不启动 Turn，而是在 `<数据目录>/client-events//.jsonl` 追加事件，写入并 fsync 后才算投递。事件只含标识、时间、结果、来源 Turn、Harness、深度链接与引用，不含会话内容。 +- 被通知方为客户端的 watch 触发通知时，Host 不启动 Turn，而是在 `<数据目录>/client-events//.jsonl` 追加事件，写入并 fsync 后才算投递。事件只含标识、时间、结果、来源 Turn、Harness、深度链接、引用与结果原因，不含会话内容。可选 `--stall-after-ms`（MCP 中为 `stallAfterMs`）产生 `outcome: "stalled"`，`note` 说明 Thread 仍在运行、Host 未观察到事件和静默阈值；提醒不结束观察，新活动后可再次触发。默认关闭，恢复后的静默重新计时，见 [Thread watch](thread-watch.md)。 - 事件标识由时钟（微秒）派生，在进程内严格递增，重启后从已有最大值之上继续。多个 Runtime 共用数据目录时标识可能交错，但不会重复使用同一进程内的值。 - 整日超过保留期（默认 7 天，`CODEXHOST_CLIENT_EVENT_RETENTION_DAYS`）的文件被删除，删除范围记在 `pruned.json`；早于该范围的游标返回 `CURSOR_EXPIRED` 与当前最早事件标识，客户端用 `thread list --client` 对账。 - 崩溃留下的半行被跳过，下一次追加先补换行，不会吞掉新事件。 diff --git a/docs/architecture/thread-watch.md b/docs/architecture/thread-watch.md index c39838a5b..af0119c96 100644 --- a/docs/architecture/thread-watch.md +++ b/docs/architecture/thread-watch.md @@ -1,4 +1,4 @@ -# Thread watch：一次性的 Thread 停下通知 +# Thread watch：停下通知与可选卡住提醒 `codexhost thread watch` 让一个 Thread 在另一个 Thread **停下（不再运行）时收到一次通知**。调用方注册后立即返回，可以继续工作或结束自己的 Turn，不需要等待或轮询。 @@ -7,16 +7,16 @@ ## 模型 - 只有两个 Thread：被观察的 Thread 和被通知的 Thread。与委派的父子关系无关，任意两个不同的 Thread 都可以。 -- 一次性。被观察 Thread 停下，或 watch 到期，哪个先到就通知一次，随后 watch 消失。换了一个仍在运行的 Turn 不会通知。没有取消或退订；想继续等就再注册一次。 -- 每次注册各自兑现。同一对 Thread 仍在观察中时重复注册，返回已有的 watch；该 watch 已停下、通知仍在待送达时再次注册（例如 Thread 又开始了新一轮），会新建一个 watch 观察下一次停下，旧通知照常送达，不会被替换或丢弃。 +- 停下通知一次性。被观察 Thread 停下，或 watch 到期，哪个先到就通知一次，随后 watch 消失。可选卡住提醒不结束观察。换了一个仍在运行的 Turn 不会通知。没有取消或退订；想继续等就再注册一次。 +- 每次注册各自兑现。同一对 Thread 仍在观察中时重复注册，返回已有的 watch，并保留原有期限和卡住阈值；该 watch 已停下、通知仍在待送达时再次注册（例如 Thread 又开始了新一轮），会新建一个 watch 观察下一次停下，旧通知照常送达，不会被替换或丢弃。 - 通知只报告执行状态：Thread 链接和结果；终态结果注明来自哪个 Turn，以便区分同一 Thread 不同轮次的通知。通知不包含会话正文或摘要，也不代表工作被验收。接收方应自行 `thread read`。 入口： | 命令 | 用途 | | --- | --- | -| `thread watch [--notify ] [--timeout-ms ]` | 观察一个已有 Thread | -| `delegate start ... --watch true` | 创建委派后顺手观察，通知 Host 解析出的发起方 | +| `thread watch [--notify ] [--timeout-ms ] [--stall-after-ms ]` | 观察一个已有 Thread | +| `delegate start ... --watch true [--stall-after-ms ]` | 创建委派后顺手观察，通知 Host 解析出的发起方 | | `thread watches` | 列出尚未送达的 watch | watch 首次持久化失败时不保留内存注册，`thread watch` 返回错误，调用方可稍后重试。`delegate start` 先完成委派；watch 注册失败不会让命令失败，而是在返回的 `watch` 字段里报告 `notRegistered` 和原因。`thread send` 不提供 `--watch`，发送后需要通知时单独执行 `thread watch`。 @@ -29,11 +29,18 @@ watch 首次持久化失败时不保留内存注册，`thread watch` 返回错 | `timedOut` | 到期时 Thread 仍未停下；也覆盖 Harness 停止但没有报告的情况 | | `unreadable` | 连续 60 秒读取失败，状态未知 | | `notFound` | Thread 已不存在 | +| `stalled` | Thread 仍在运行，但在配置时长内没有观察到 Harness 事件；只是静默提醒，不代表失败 | 默认观察 29 分钟，`--timeout-ms` 可调整；到期报告 `timedOut`，需要继续等待时再注册一次。 +卡住提醒默认关闭。传入正整数 `--stall-after-ms` 后，从注册时刻开始记录静默时间；`delegate start` 只有同时提供 `--watch true` 才接受该参数。Host 收到的任何 Harness 事件都刷新活动时间，包括消息、工具调用和输出、推理、文件变化、交互，以及 Harness 子代理事件；子代理活动也刷新其祖先 Thread。轮询读取和 Host 自己的投影通知不算活动。 + +运行中的 Thread 静默达到阈值时，以 `stalled` 提醒一次并继续观察；持续静默不会重复提醒。提醒写明 Thread 仍在运行、至少多长时间没有观察到 Harness 事件，时长照阈值原值表述（例如 90 秒写作 `90 s`），不取整；与其他通知一样不附带进度或会话内容，接收方用 `thread read` 查看。新的 Harness 活动重新计时，之后再次静默才会再次提醒。提醒是触发时刻的事实，即使被通知 Thread 正忙、期间活动恢复，也保留该待投递提醒；它会与同一接收方的其他提醒或停下通知合并。观察期限仍独立生效，到期后结束观察。 + 注册时 Thread 已是终态则返回 `alreadyTerminal`：不注册、不通知，调用方直接读取即可。 +被观察 Thread 还在[委派队列](delegation-capacity.md)中时照常注册：排队与创建都不算停下，也不计入卡住提醒的静默；Thread 开始后真正停下才通知，开始前被移出队列则通知 `interrupted`。排队时间计入观察期限。 + ## 送达 - 通知通过普通的 `send` 在被通知 Thread 中启动一个新 Turn，外部 Harness 与原生 Codex 使用同一路径；不做同轮注入。 @@ -43,6 +50,7 @@ watch 首次持久化失败时不保留内存注册，`thread watch` 返回错 - 被通知 Thread 不存在或只读：立即标记 `undeliverable`； - 其他失败都视为结果未知，包括 Harness 启动确认超时后包装成的 `DELEGATION_FAILED`：原生可能已经启动了 Turn，为避免重复唤醒不再重试，标记 `undeliverable` 并在原因中说明。 - 每次投递将同一个被通知 Thread 当前所有待送达通知合并为一条消息，包括此前轮询积累的通知，只启动一个 Turn。 +- 接收方为外部客户端时，同批通知写入该客户端的事件日志，不启动 Turn。卡住事件使用 `outcome: "stalled"`，`note` 与发给 Thread 的提醒文字相同，说明 Thread 仍在运行、至少多长时间没有观察到事件以及观察会继续；事件通过同一游标、SSE 和轮询接口读取。 - 被通知 Thread 不存在、只读，或超过 6 小时仍无法送达时，watch 标记为 `undeliverable` 并保留原因，可由 `thread watches` 查看；最多保留最近 50 条无法送达记录。 ## 与 Stop 的关系 @@ -56,16 +64,17 @@ watch 首次持久化失败时不保留内存注册，`thread watch` 返回错 身份只确定一次，watch 不做自己的推断： 1. `delegate start --watch`：使用委派已经解析出的父 Thread；没有父 Thread 时报告 `notRegistered`。 -2. `thread watch`：显式 `--notify`，否则使用 Host 提供给外部 Harness 的 `CODEXHOST_THREAD_ID`。 -3. 两者都没有（原生 Codex）时返回 `INVALID_ARGUMENT`，要求显式 `--notify`。原生 Codex 可使用 `delegate start` 响应中返回的 parent 作为自己的 Thread。 -4. 经外部客户端监听调用时，未给 `--notify` 即通知调用客户端自身（`client:`）。此时“投递”是向该客户端的事件日志追加一条事件，不启动 Turn；追加失败在投递期限内重试。见 [外部客户端](external-clients.md)。 +2. `thread watch`：显式 `--notify`，否则使用 Host 提供给外部 Harness 的 `CODEXHOST_THREAD_ID`，再取原生 Codex 的 `CODEX_THREAD_ID`。 +3. 经外部客户端监听调用且没有调用方 Thread 标识时，未给 `--notify` 即通知调用客户端自身（`client:`）。此时“投递”是向该客户端的事件日志追加一条事件，不启动 Turn；追加失败在投递期限内重试。见 [外部客户端](external-clients.md)。MCP 的 `thread_watch` 和 `delegate_start`（`watch: true`）以 `stallAfterMs` 提供相同的可选阈值。 +4. 均没有时返回 `INVALID_ARGUMENT`，要求显式 `--notify`。 ## 实现与边界 -- `packages/host-runtime/src/delegation-watch.ts` 只依赖公开的 `read` 与 `send`，每轮完成后间隔 2 秒继续轮询（现有 `thread wait` 同样基于轮询）。它不依赖具体 Harness、Desktop、Renderer 或委派血缘。 +- `packages/host-runtime/src/delegation-watch.ts` 通过公开的 `read` 与 `send` 观察状态和投递，活动从 Host 已有事件流订阅，每轮完成后间隔 2 秒继续轮询（现有 `thread wait` 同样基于轮询）。它不依赖具体 Harness、Desktop、Renderer 或委派血缘。 +- 活动时间只放在 Host 内存中，不增加定时器、后台进程或 rollout 扫描。原生 Codex 使用已有 app-server 通知和请求中的 Thread 标识、`thread/started` 的子代理来源与 `spawnAgent` 接收方建立祖先关系；外部 Harness 在输出消费入口记录全部事件与交互，并使用已知子代理身份传播活动。原生祖先关系只在 Thread 启动时获知，因此原生 Thread 收到 `thread/closed` 后，只要还有未关闭的后代，通往祖先的整条链就保留，整棵子树都关闭后才清除；外部 Harness 的关系随每个事件重新记录，会话输出结束（包括空闲释放）即清除；官方 app-server 停止时全部清除，不随 Host 运行时间增长。未接入当前 Host 事件流的 Thread、没有上报活动的 Adapter，或未公布身份关系的子代理无法提供完整活动证据；静默提醒只表示“Host 没有观察到事件”，长工具执行也可能触发。 - 服务由 `DelegationControlRegistry` 持有，位于各 Host 会话之上，因此两端可以属于不同的 Host 会话。 -- watch 持久保存在 `<数据目录>/delegation-watches/`（默认 `~/.codexhost`），每个 watch 一个仅当前用户可读写的记录，只含 Thread 标识、状态、期限与原因。Runtime 重启后恢复观察与投递，调用方无需重新注册；期限按绝对时间计算，停机时间计入。 -- 多个 Runtime 进程可共用数据目录（例如 Desktop 与远程监听同时运行）。每个进程只写自己的记录；启动时接管已退出进程（或上次开机遗留）的记录，以原子重命名认领，同一记录只被一个进程接管。 +- watch 持久保存在 `<数据目录>/delegation-watches/`（默认 `~/.codexhost`），每个 watch 一个仅当前用户可读写的记录，只含 Thread 标识、状态、期限与原因。Runtime 重启后恢复观察与投递，调用方无需重新注册；观察与投递期限按绝对时间计算，停机时间计入。卡住阈值随 watch 保存；卡住提醒、活动时间与“本段静默已提醒”状态只在内存中，恢复观察后从 Runtime 恢复时刻重新计时，停机时长不计入静默，重启前未送达的卡住提醒不再投递，静默持续时恢复后会再次提醒。卡住提醒是独立于主 watch 的待投递项，送达或投递失败不会结束主 watch。 +- 多个 Runtime 进程可共用数据目录（例如 Desktop 与远程监听同时运行）。每个进程只写自己的记录；启动时接管已退出进程（或上次开机遗留）的记录，每个记录的接管通过跨进程文件锁串行处理，并以原子重命名认领，同一记录只被一个进程接管。 - 恢复后的 watch 最多等待 5 分钟让持有 Thread 的会话重新注册：这期间读取失败不计入 60 秒 `unreadable` 宽限，也不报告 `notFound` 或 `timedOut`；待投递通知要等被通知 Thread 可读后再发送。 - Host Runtime 重启会结束它持有的 Harness 进程，正在运行的 Turn 随之中断。之后读取时它可能显示为已完成（有的 Harness 恢复后还会换一个 Turn 标识），也可能根本无法重新打开。因此，重启前处于观察中的 watch，恢复后第一次成功读取若已是终态，或宽限结束后仍报告 Thread 不存在，都按 `interrupted` 通知并说明原因；原因随记录保存，再次重启也不丢失。代价是：由长期运行的远程监听持有、在停机期间真正完成的 Thread 也会被报告为 `interrupted`；被通知方本来就要先读取 Thread 再行动。 - 发送前先把通知记为“投递中”。Runtime 在发送途中退出时，恢复后再投一次，因此重启可能重复通知，但不会丢失。运行期间“投递结果未知”仍标记为 `undeliverable`，不重试。 diff --git a/docs/index.md b/docs/index.md index 3b08cc876..9ccdd8da1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -29,7 +29,8 @@ | [`architecture/external-thread-media.md`](architecture/external-thread-media.md) | 外部 Thread 中本地图片的 Desktop 加载条件、Agent 消息媒体路径改写和用户消息图片附件还原；修改媒体显示或排查图片不显示时阅读。 | | [`architecture/delegation-titles.md`](architecture/delegation-titles.md) | 委派子会话标题、子任务显示名、创建快照与重试边界。 | | [`architecture/delegation-thread-list.md`](architecture/delegation-thread-list.md) | 委派 CLI 列表的共享 catalog、独立来源分页、查询绑定与活列表一致性边界。 | -| [`architecture/thread-watch.md`](architecture/thread-watch.md) | 一次性 Thread 停下通知（`thread watch`）的模型、结果、送达与边界；修改委派跟进或通知时阅读。 | +| [`architecture/thread-watch.md`](architecture/thread-watch.md) | Thread 停下通知与可选卡住提醒（`thread watch`）的模型、结果、送达与边界；修改委派跟进或通知时阅读。 | +| [`architecture/delegation-capacity.md`](architecture/delegation-capacity.md) | 本机同时运行的委派上限：设置下发、计数口径、超出时排队、各命令对排队 Thread 的行为与重启恢复；修改委派启动或该设置时阅读。 | | [`architecture/external-clients.md`](architecture/external-clients.md) | 同用户 CLI 自动认证、命名客户端登记、本机监听、Thread 引用、事件日志与 MCP；修改外部接入时阅读。 | | [`architecture/official-traffic-ownership.md`](architecture/official-traffic-ownership.md) | Host 只截获自身流量、其余原样转发官方的归属判别与例外；修改请求路由或参数校验时阅读。 | | [`architecture/app-server-transport.md`](architecture/app-server-transport.md) | 原生大历史响应的 WebSocket 与 JSONL 传输边界；排查任务加载、消息大小和转发性能时阅读。 | diff --git a/openspec/changes/add-delegation-capacity/.openspec.yaml b/openspec/changes/add-delegation-capacity/.openspec.yaml new file mode 100644 index 000000000..1913c7d99 --- /dev/null +++ b/openspec/changes/add-delegation-capacity/.openspec.yaml @@ -0,0 +1,2 @@ +schema: codexhost-lite +created: 2026-11-06 diff --git a/openspec/changes/add-delegation-capacity/proposal.md b/openspec/changes/add-delegation-capacity/proposal.md new file mode 100644 index 000000000..a29be554a --- /dev/null +++ b/openspec/changes/add-delegation-capacity/proposal.md @@ -0,0 +1,38 @@ +# Proposal + +## Why + +编排类工作常一次派出多个委派，同时运行的 Harness 过多会占满本机资源与 Provider 并发，目前只能由调用方或用户自己控制派发节奏。需要一个默认关闭的本机上限：超出时委派排队，有空位再按先后自动开始，而不是失败或要求调用方重试。 + +## What Changes + +- Codex Desktop 设置页“通用”中新增“委派”分组：开关“限制同时运行的委派”（默认关闭）与“同时运行上限”（1～32，默认 4）。存储与下发沿用空闲释放：Renderer 本地偏好，只下发给本机 Host，Host 保存在内存。 +- 开启后，本机正在运行的顶层 Thread（含用户手动打开的，不含子代理）达到上限时，`delegate start`（外部 Harness 与原生 Codex 相同）立即返回 Thread 标识、`status: "queued"` 与排队位置，有空位时按先后自动开始。`thread send` 与用户消息不受限制。 +- `thread read`、`thread list` 显示排队状态与位置；`thread wait`、`thread watch` 等到 Thread 开始后真正停下；`thread cancel` 把它移出队列；`thread send` 对排队中的 Thread 返回 `THREAD_BUSY`。 +- 队列持久保存在 Host 私有数据目录，Host Runtime 重启后按原顺序恢复。 +- CLI 帮助、MCP `delegate_start` 说明与架构文档同步更新。 + +## Decisions + +- **排队而非拒绝。** 拒绝要求每个调用方自己重试并维持顺序；排队保持 `delegate start`“立即返回、之后观察”的语义，调用方用同一套 read、wait、watch 跟进。 +- **排队时即分配 Thread 标识。** 排队期间拿到的链接、watch 与后续命令在开始后继续有效。 +- **只限制新委派。** 后续消息属于已有工作，拦截会让进行中的协作卡住，用户消息也不应因 Agent 的派发被拒绝。上限因此只是开始新委派的门槛，运行数可能因后续消息超过上限。 +- **原生 Codex 委派先建 Thread，只排队第一条 Turn。** 主要的编排场景使用 `--harness codex`，不排队上限就不起作用。官方后端不接受调用方指定的 Thread 标识，由 Host 分配别名则深链接、父子关系与子进程环境都要换算；先用官方 `thread/start` 建好 Thread，排队时返回的就是真实标识。没有 Turn 的 Thread 会持久保存，重启后可恢复并开始 Turn，但官方 `thread/list` 不列出它。用户直接向它发消息时，排队的任务并入这条消息，只执行一次。 +- **设置与队列分开保存。** 设置是用户偏好，沿用空闲释放由 Renderer 在连接后下发；队列是待执行的工作，必须跨重启保存。重启后设置到达前，恢复的委派保持排队，最多等待 5 分钟，避免在上限未知时一次全部开始。 +- **重启时正在开始的外部 Harness 委派不自动重试。** 无法确认 Thread 是否已经创建，重试可能重复创建；按 `interrupted` 结束并说明原因。原生 Codex 的 Thread 已经存在，恢复为“可能已开始”，之后的开始、取消、用户消息与读取都先检查是否已有 Turn。 +- **排队的原生 Thread 从不删除。** 官方后端对有进行中 Turn 的 Thread 也会执行 `thread/delete` 并删掉 rollout，而重启或用户消息都可能让 Turn 在 Host 不知情时开始。取消只移出队列，空 Thread 留在后端（官方 `thread/list` 不列出它）。后端有 `thread/archive`，但确认没有 Turn 之后用户消息仍可能到达，本次不归档。 +- **用户消息分两步认领排队的任务。** 后端答复前任务留在原位，答复开始了 Turn 才移出队列；拒绝时任务不丢。 + +## Capabilities + +### New Capabilities + +无。 + +### Modified Capabilities + +- `cross-harness-delegation`：`delegate start` 在本机委派上限已满时排队后立即返回（为使该需求描述不超过 500 字符，描述不再逐一列出观察命令，由已有场景覆盖）；新增“本机同时运行的委派可选上限”和“排队中的 Thread 可观察、可等待、可取消”两条需求。 + +## Impact + +`packages/shared-contracts` 新增设置契约；`packages/host-runtime` 的委派 Registry、协调器、CLI 输出与帮助、MCP 说明，以及 `<数据目录>/delegation-queue/` 下的私有队列文件；`packages/renderer-extension` 的设置页、偏好同步与本机 Host 请求。新增 `docs/architecture/delegation-capacity.md`，`thread-watch.md`、`delegation-thread-list.md`、`external-clients.md` 补充排队行为。 diff --git a/openspec/changes/add-delegation-capacity/specs/cross-harness-delegation/spec.md b/openspec/changes/add-delegation-capacity/specs/cross-harness-delegation/spec.md new file mode 100644 index 000000000..77e4ffd29 --- /dev/null +++ b/openspec/changes/add-delegation-capacity/specs/cross-harness-delegation/spec.md @@ -0,0 +1,175 @@ +## MODIFIED Requirements + +### Requirement: 委派创建与结果观察解耦且不主动注入父 Session +`codexhost delegate start --harness --task [--parent-thread ] [--request-id ] [--watch true|false] [--watch-timeout-ms ]` SHALL 在创建目标 Session/Thread 并投递任务后立即返回；本机委派上限已满时 SHALL 排队后立即返回。`--harness` 与 `--task` SHALL 为必填参数；`--parent-thread` SHALL 显式覆盖 Host 推断的调用方 Thread；`--request-id` SHALL 承载调用方提供的幂等标识。发起方 Agent SHALL 可以自主选择读取、有界等待、注册一次性停下通知、稍后再次观察，或不再跟踪。除调用方显式注册的 watch 外，Host MUST NOT 在子任务完成后向父 Session 注入结果、唤醒父 Agent 或为此创建自主 Turn。 + +#### Scenario: 委派创建成功返回 +- **WHEN** 调用方执行有效的 `codexhost delegate start --harness --task ` +- **THEN** CLI SHALL 返回包含 `delegationId`、`threadId`、`harnessId`、`deepLink` 与当前 `status` 的 JSON 对象 +- **AND** MUST NOT 等待被委派工作完成 +- **AND** SHALL 在响应中给出可用于 `thread read` 与 `thread wait` 的下一步命令提示 + +#### Scenario: 本机委派上限已满 +- **WHEN** 本机委派上限已开启且已满，调用方执行有效的 `delegate start` +- **THEN** CLI SHALL 立即返回 Thread 标识、`status: "queued"` 与排队位置 +- **AND** Host MUST NOT 在排到之前投递任务；外部 Harness MUST NOT 在排到之前创建目标 Session/Thread +- **AND** 该委派开始后 SHALL 沿用已返回的 Thread 标识 + +#### Scenario: 运行期间打开外部 Harness 委派 Thread +- **WHEN** 外部 Harness 的委派 Turn 仍在运行且用户打开其子 Thread +- **THEN** Host 的实时 Turn 投影 SHALL 包含本次委派任务对应的 `userMessage` +- **AND** SHALL 在同一 Turn 中继续投影 Agent 的可见进度与最终消息 +- **AND** 完成后的实时 Turn 与原生历史恢复结果 MUST NOT 重复该 `userMessage` + +#### Scenario: 显式指定父 Thread 与 Request ID +- **WHEN** 调用方同时提供 `--parent-thread ` 与 `--request-id ` +- **THEN** Host SHALL 使用规范化后的父 Thread 标识记录 Delegation 关系 +- **AND** SHALL 使用该 Request ID 执行幂等创建 +- **AND** MUST NOT 以环境中的 Host Thread 标识或活跃 Turn 推断覆盖显式参数 + +#### Scenario: 发起方选择等待 +- **WHEN** 发起方对一个运行中的子 Thread 调用有界 `thread wait` +- **THEN** Host SHALL 等待到子任务达到终态或期限到期 +- **AND** 目标在期限内完成时 SHALL 返回结构化结果 + +#### Scenario: 等待超时 +- **WHEN** 一次有界等待在被委派工作结束前到期 +- **THEN** 命令 SHALL 以成功退出并报告状态为运行中 +- **AND** 被委派的工作 SHALL 继续执行 +- **AND** 该结果 MUST NOT 被表述为失败 + +#### Scenario: 发起方选择后台运行或不再跟踪 +- **WHEN** 发起方未注册 watch，并在创建后不调用 `thread wait` 或结束当前 Turn +- **THEN** 被委派工作 SHALL 独立继续执行 +- **AND** Host MUST NOT 为发起方建立完成通知、输入注入或续写义务 +- **AND** 子 Thread SHALL 继续可由用户或后续 Agent 调用通过标识读取 + +#### Scenario: 发起方在创建时注册 watch +- **WHEN** 调用方执行 `delegate start ... --watch true` +- **THEN** Host SHALL 先完成委派或排队，再以委派已解析的父 Thread 作为被通知方为子 Thread 注册 watch +- **AND** 响应 SHALL 在 `watch` 字段中报告注册结果 +- **AND** watch 注册失败或无父 Thread 时 SHALL 报告 `notRegistered` 与原因，MUST NOT 使委派本身失败 + +#### Scenario: 子任务完成 +- **WHEN** 被委派的子 Thread 达到完成、失败或取消终态 +- **THEN** Host SHALL 更新该子 Thread 与 Delegation 关系的状态，并保留可读取的结构化结果 +- **AND** 没有针对该子 Thread 的已注册 watch 时，MUST NOT 因该终态向父 Session 提交新的输入 + +#### Scenario: 结果被结构化描述 +- **WHEN** 委派结果通过 `thread read` 或 `thread wait` 返回 +- **THEN** 它 SHALL 包含子 Thread 标识、当前状态，以及由 `availability` 和可选 `text` 构成的结果判定 +- **AND** `availability` SHALL 为 `pending`、`available` 或 `unavailable` +- **AND** 它 MUST NOT 仅以自由文本表述成败 + +## ADDED Requirements + +### Requirement: 本机同时运行的委派可选上限 +Host SHALL 提供本机同时运行的委派上限设置，默认关闭，上限为 1～32 的整数。设置 SHALL 保存在 Desktop 本地偏好中并只下发给本机 Host，Host 只在内存中保存。开启时，占用名额的 SHALL 是本 Host Runtime 中有活跃 Turn 的顶层 Thread，包括用户手动打开的 Thread，不含子代理。只有 `delegate start` SHALL 受限，外部 Harness 与原生 Codex 相同；后续消息与用户消息 MUST NOT 被拦截、排队或因上限失败。队列 SHALL 持久保存并在 Host Runtime 重启后恢复。 + +#### Scenario: 默认关闭 +- **WHEN** 用户没有开启上限 +- **THEN** `delegate start` SHALL 与没有该设置时行为一致，立即创建 Thread，MUST NOT 排队 + +#### Scenario: 超过上限时排队 +- **WHEN** 上限已开启，正在运行的顶层 Thread 数已达到上限，或已有委派在排队 +- **THEN** `delegate start` SHALL 排队，并立即返回 Thread 标识、`queued` 状态与从 1 开始的排队位置 +- **AND** 参数、父 Thread 与 Model 配置的校验 SHALL 在排队前完成，无效请求 MUST NOT 进入队列 +- **AND** 相同 Request ID 或近期相同任务的重复请求 SHALL 返回同一个排队中的 Thread + +#### Scenario: 有空位时按先后开始 +- **WHEN** 运行中的 Turn 结束、上限调高或上限关闭，使名额空出 +- **THEN** Host SHALL 按排队先后开始排队的委派，直到名额用完 +- **AND** 新的启动 MUST NOT 越过已在排队的委派 +- **AND** 开始时 SHALL 沿用排队时的任务参数与已分配的 Thread 标识 +- **AND** 开始失败时该 Thread SHALL 读取为 `failed` 并附原因 + +#### Scenario: 计数口径 +- **WHEN** 用户手动打开的 Thread、原生 Codex 委派或子代理正在运行 +- **THEN** 手动打开的 Thread 与原生 Codex 委派 SHALL 占用名额 +- **AND** 外部 Harness 与原生 Codex 的子代理 MUST NOT 占用名额 + +#### Scenario: 原生 Codex 委派排队 +- **WHEN** `delegate start --harness codex` 需要排队 +- **THEN** Host SHALL 在官方后端先创建没有 Turn 的 Thread，并返回该 Thread 的标识 +- **AND** 只有第一条 Turn SHALL 排队，开始时 SHALL 在同一 Thread 中投递任务 +- **AND** 排队相关的处理 MUST NOT 删除该 Thread + +#### Scenario: 取消排队中的原生 Thread +- **WHEN** 调用方取消一个排队中的原生 Codex Thread +- **THEN** Host SHALL 先读取它的 Turn +- **AND** 没有 Turn 时 SHALL 将它移出队列并记为 `interrupted`，保留该 Thread +- **AND** 已有 Turn 时 SHALL 按已开始的委派建立委派关系并取消该 Turn + +#### Scenario: 用户直接向排队中的原生 Thread 发消息 +- **WHEN** 用户在 Desktop 中向排队中的原生 Codex Thread 发送消息 +- **THEN** Host SHALL 把排队的任务与该消息在同一 Turn 中投递 +- **AND** 后端答复前，该委派 SHALL 留在队列原位，MUST NOT 被队列开始，取消 SHALL 等到答复之后 +- **AND** 后端开始该 Turn 后 SHALL 将它移出队列并建立委派关系，排队的任务 MUST NOT 之后再次开始 +- **AND** 后端拒绝该 Turn 时，该委派 SHALL 留在原位继续排队 + +#### Scenario: 取消与用户消息交错 +- **WHEN** 对排队中原生 Thread 的取消与用户在 Desktop 中发送的消息同时发生 +- **THEN** 用户的消息 SHALL 照常进入该 Thread，MUST NOT 丢失 +- **AND** 该委派 SHALL 记为取消 + +#### Scenario: 后续消息不受上限限制 +- **WHEN** 运行数已达到或超过上限，调用方执行 `thread send` 或用户在已有 Thread 中发送消息 +- **THEN** Host SHALL 照常启动 Turn +- **AND** MUST NOT 排队或因上限失败 + +#### Scenario: 设置下发 +- **WHEN** Desktop 连接本机 Host 或用户修改设置 +- **THEN** Desktop SHALL 把设置下发给本机 Host，MUST NOT 发给当前远程 Host +- **AND** Host 不支持该设置时，设置页 SHALL 提示不支持，MUST NOT 显示为已生效 + +#### Scenario: Host Runtime 重启 +- **WHEN** Host Runtime 在排队的委派开始前重启 +- **THEN** 重启后队列 SHALL 按原顺序与原 Thread 标识恢复 +- **AND** 设置重新下发前，恢复的委派 SHALL 保持排队，新的启动排在其后；5 分钟内仍未收到设置时 SHALL 按关闭处理 +- **AND** 重启时正在开始的外部 Harness 委派 SHALL 以 `interrupted` 结束并说明原因，MUST NOT 自动重试 +- **AND** 重启时正在开始或正在被用户消息认领的原生 Codex 委派 SHALL 恢复为可能已开始；其后的自动开始、取消、用户消息与读取 SHALL 先读取 Thread 的 Turn +- **AND** 已有 Turn 时 SHALL 按已开始的委派建立委派关系，MUST NOT 再次投递任务或把任务并入用户消息；没有 Turn 时才 SHALL 按排队处理 +- **AND** Host Runtime 正常关闭时 SHALL 把已结束的启动结果写入存储 + +#### Scenario: 队列存储 +- **WHEN** 排队记录写入持久存储 +- **THEN** 存储 SHALL 仅当前 OS 用户可读写 +- **AND** 记录 SHALL 只包含开始委派所需的任务参数、标识与排队时间，并在委派开始后删除 +- **AND** 多个 Host Runtime 共用数据目录时，每条记录 SHALL 只由一个 Runtime 开始；已退出 Runtime 的队列 SHALL 只被接管一次 + +### Requirement: 排队中的 Thread 可观察、可等待、可取消 +排队中的 Thread SHALL 使用开始后沿用的同一标识，并接受既有 Thread 命令。`thread read` 与 `thread list` SHALL 显示 `queued` 状态与排队位置；`thread wait` 与 `thread watch` SHALL 等到 Thread 开始后真正停下；`thread cancel` SHALL 将其移出队列且不再开始；`thread send` SHALL 以 `THREAD_BUSY` 拒绝。 + +#### Scenario: 读取排队中的 Thread +- **WHEN** 调用方 `thread read` 一个排队中的 Thread +- **THEN** 结果 SHALL 为 `status: "queued"`、排队位置与 `pending` 结果，不含 Turn +- **AND** 开始创建后 SHALL 报告 `creating`，之后与普通 Thread 相同 + +#### Scenario: 列举排队中的 Thread +- **WHEN** 调用方不带 cursor 执行 `thread list`，且有符合 `--parent` 或工作区过滤的排队 Thread +- **THEN** 它们 SHALL 按队列顺序排在第一页最前面，并带排队位置 +- **AND** 它们 MUST NOT 占用 `--limit`，也 MUST NOT 出现在续页中 + +#### Scenario: 等待排队中的 Thread +- **WHEN** 调用方对排队中的 Thread 执行有界 `thread wait` +- **THEN** 排队时间 SHALL 计入期限 +- **AND** 期限内开始的 Thread SHALL 在剩余期限内按普通 Thread 等待 +- **AND** 到期时仍在排队 SHALL 返回排队快照与 `timedOut: true` + +#### Scenario: watch 排队中的 Thread +- **WHEN** 调用方对排队中的 Thread 注册 `thread watch` +- **THEN** watch SHALL 注册为观察中，排队与创建 MUST NOT 被视为停下 +- **AND** 排队与创建期间 MUST NOT 计为卡住提醒的静默 +- **AND** Thread 开始后停下时 SHALL 按实际终态通知；开始前被移出队列时 SHALL 通知 `interrupted` + +#### Scenario: 取消排队中的 Thread +- **WHEN** 调用方对排队中的 Thread 执行 `thread cancel` +- **THEN** Host SHALL 将它移出队列并返回 `cancelled: true`，其后的排队位置依次前移 +- **AND** 该 Thread MUST NOT 再开始，之后读取 SHALL 为 `interrupted` 并说明开始前已移出队列 +- **AND** 再次取消 SHALL 返回 `cancelled: false` + +#### Scenario: 向排队中的 Thread 发送消息 +- **WHEN** 调用方对排队中或正在创建的 Thread 执行 `thread send` +- **THEN** 命令 SHALL 以 `THREAD_BUSY` 失败，并在详情中给出状态与排队位置 +- **AND** MUST NOT 排队该消息 +- **AND** 已移出队列或开始失败的 Thread SHALL 返回 `THREAD_NOT_FOUND` diff --git a/openspec/changes/add-delegation-capacity/tasks.md b/openspec/changes/add-delegation-capacity/tasks.md new file mode 100644 index 000000000..57042819a --- /dev/null +++ b/openspec/changes/add-delegation-capacity/tasks.md @@ -0,0 +1,24 @@ +# Tasks + +依赖 `add-external-client-access`（MCP `delegate_start`）与 `add-thread-watch-stall-alerts`（卡住提醒）。行为验收见 [spec](specs/cross-harness-delegation/spec.md)。 + +## 1. 设置 + +- [x] 1.1 共享设置契约（开关与 1～32 的上限，默认关闭、4）；Renderer 本地偏好，连接本机 Host 后与修改时下发，Host 不支持时提示；设置页“委派”分组。验证：`renderer-delegation-capacity-preference.test.ts`、`renderer-host-routing.test.ts`、`versioned-renderer-adapter.test.ts`，以及 `tests/e2e/renderer-delegation-capacity.spec.ts`。 +- [x] 1.2 Host 接收设置：没有委派控制时返回“方法不支持”，非法值被拒绝。验证：`app-server-host.delegation-capacity.test.ts`。 + +## 2. 计数与排队 + +- [x] 2.1 计数：正在运行的顶层 Thread（含手动打开的，不含外部与原生子代理），加上已放行、尚未报告运行的启动。验证：`app-server-host.delegation-capacity.test.ts`、`thread-activity.test.ts`、`delegation-capacity-queue.test.ts`。 +- [x] 2.2 `delegate start` 在校验与去重后排队，立即返回标识与位置；按先后自动开始，新启动不越过等待者。验证：`delegation-capacity-queue.test.ts`、`harness-delegation-coordinator.test.ts`、`delegation-capacity-registry.test.ts`。 +- [x] 2.3 排队中 Thread 的 read、list、wait、watch、cancel、send 行为与 CLI 输出；远程 `thread read --host` 读取排队中的 Thread。验证：`delegation-capacity-registry.test.ts`、`delegation-cli-output.test.ts`、`remote-delegation-read.test.ts`。 +- [x] 2.4 原生 Codex：排队时先建 Thread，只排队第一条 Turn，排队相关代码不删除 Thread；取消先读 Turn，已有 Turn 时按已开始取消；用户直接发消息时分两步认领，后端开始 Turn 后才移出队列，拒绝时留在原位；取消与用户消息交错时不丢内容；重启后“可能已开始”的记录先读 Turn 再决定收编或排队；关闭时写入启动结果。验证：`app-server-host.native-delegation-capacity.test.ts`、`delegation-capacity-queue.test.ts`。 + +## 3. 持久化与恢复 + +- [x] 3.1 每个 Runtime 一个私有队列文件，接管已退出 Runtime 的队列；设置到达前保持排队，5 分钟后按关闭处理；重启时正在开始的外部 Harness 委派以 `interrupted` 结束。验证：`delegation-capacity-store.test.ts`、`delegation-capacity-queue.test.ts`、`delegation-capacity-registry.test.ts`。 + +## 4. 文档与验收 + +- [x] 4.1 CLI 帮助、MCP 说明、`docs/architecture/delegation-capacity.md` 及相关架构文档。验证：帮助、文档与规格描述一致。 +- [ ] 4.2 真实 Desktop 验收：上限设为 1，手动会话运行时分别委派一个外部 Harness 与一个 `--harness codex` 任务，确认排队并在会话结束后自动开始；排队中重启 codexhost 后恢复；取消排队中的任务；在 Desktop 中直接给排队中的原生 Thread 发消息，确认任务只执行一次，并记录它在线程列表中的样子。记录平台与版本。 diff --git a/openspec/changes/add-thread-watch-stall-alerts/.openspec.yaml b/openspec/changes/add-thread-watch-stall-alerts/.openspec.yaml new file mode 100644 index 000000000..1913c7d99 --- /dev/null +++ b/openspec/changes/add-thread-watch-stall-alerts/.openspec.yaml @@ -0,0 +1,2 @@ +schema: codexhost-lite +created: 2026-11-06 diff --git a/openspec/changes/add-thread-watch-stall-alerts/proposal.md b/openspec/changes/add-thread-watch-stall-alerts/proposal.md new file mode 100644 index 000000000..f936f1ce4 --- /dev/null +++ b/openspec/changes/add-thread-watch-stall-alerts/proposal.md @@ -0,0 +1,33 @@ +# Proposal + +## Why + +长程委派常跑数小时。被观察 Thread 卡在某个操作上时，停下通知要等它停下或 watch 到期才会到达，发起方中途无从得知它已经很久没有动静。 + +## What Changes + +- `thread watch` 与 `delegate start --watch true` 增加可选的 `--stall-after-ms `（MCP 中为 `stallAfterMs`），默认关闭。 +- 被观察 Thread 仍在运行、但 Host 在阈值内没有观察到它或其 Harness 子代理的任何事件时，以 `stalled` 提醒一次；观察继续，新的活动重新计时，下一段静默可再次提醒。 +- 提醒沿用已有的待投递、合并与持久化机制；接收方为外部客户端时写入其事件日志。 +- 不加参数时 watch 的行为不变。 + +## Decisions + +- **提醒不带进度。** 与其他 watch 通知一样只报告执行状态与 Thread 链接；静默时长写作“至少 <阈值>”，照原值表述、不取整。接收方自己 `thread read` 查看进度。 +- **活动只来自已有事件流。** 只用 Host 已收到的原生 app-server 通知与外部 Harness 输出，不新增定时器、进程或 rollout 扫描。代价是 Host 看不到的活动会被当作静默（例如 Desktop 取消订阅仍在运行的原生 Thread、不上报事件的 Adapter），所以提醒只表示“没有观察到事件”。 +- **静默不跨重启。** 停机不是 Harness 静默：恢复后从恢复时刻重新计时，只有阈值随 watch 保存。卡住提醒不是终态，不写入持久记录：静默在恢复后重新测量，旧版 Runtime 也无法读取 `stalled` 结果。 +- **作为新增需求，不修改既有 watch 需求。** `persist-thread-watches` 与 `add-external-client-access` 都以 MODIFIED 整体替换“调用方可显式注册一次性 Thread 停下通知”。卡住提醒独立成一条 ADDED 需求，无论哪个变更先归档都不会覆盖它。 + +## Capabilities + +### New Capabilities + +无。 + +### Modified Capabilities + +- `cross-harness-delegation`：新增“调用方可选择 Thread 静默卡住提醒”。 + +## Impact + +`packages/host-runtime` 的 watch 服务、活动收集、CLI 与 MCP 参数；watch 持久记录增加可选的静默阈值字段。`codexhost delegate --help`、`docs/architecture/thread-watch.md` 与 `external-clients.md` 同步说明。依赖 `persist-thread-watches`（watch 持久化）与 `add-external-client-access`（客户端事件日志与 MCP）。 diff --git a/openspec/changes/add-thread-watch-stall-alerts/specs/cross-harness-delegation/spec.md b/openspec/changes/add-thread-watch-stall-alerts/specs/cross-harness-delegation/spec.md new file mode 100644 index 000000000..99c3a9ca2 --- /dev/null +++ b/openspec/changes/add-thread-watch-stall-alerts/specs/cross-harness-delegation/spec.md @@ -0,0 +1,47 @@ +## ADDED Requirements + +### Requirement: 调用方可选择 Thread 静默卡住提醒 +`thread watch` 与 `delegate start --watch true` SHALL 接受可选正整数 `--stall-after-ms `（MCP 的 `thread_watch` 与 `delegate_start` 中为 `stallAfterMs`），默认关闭；未启用 watch 的 `delegate start` MUST NOT 接受该参数。启用后，被观察 Thread 仍在运行而 Host 在阈值内没有观察到它的 Harness 活动时，SHALL 以 `stalled` 结果提醒；提醒 MUST NOT 结束观察，也 MUST NOT 被表述为失败。同一对 Thread 的重复注册 SHALL 保留原观察期限与静默阈值。 + +#### Scenario: 静默达到阈值只提醒一次 +- **WHEN** 启用卡住提醒的被观察 Thread 仍在运行，且从注册或最后一次活动起静默达到阈值 +- **THEN** Host SHALL 生成一条 `stalled` 通知，含 Thread 链接与可用的 Turn 标识，并说明至少多长时间没有观察到 Harness 事件 +- **AND** 该时长 SHALL 按调用方给出的阈值精确表述，MUST NOT 取整（例如 90 秒不得写成 2 分钟） +- **AND** 同一段持续静默 MUST NOT 重复提醒，原 watch SHALL 继续观察停下与到期 + +#### Scenario: 提醒只报告执行状态 +- **WHEN** 接收方收到 `stalled` 通知或事件 +- **THEN** 通知 SHALL 说明 Thread 仍在运行、静默不证明失败，并让接收方读取 Thread 查看进度 +- **AND** 通知 MUST NOT 携带或摘要会话内容与进度 + +#### Scenario: 任意 Harness 活动与子代理活动 +- **WHEN** Host 收到被观察 Thread 或其 Harness 子代理的任何事件或交互，包括工具输出、推理与文件变化 +- **THEN** Host SHALL 刷新该 Thread 及已知子代理祖先 Thread 的活动时间，并重新允许下一次静默提醒 +- **AND** 轮询读取与 Host 自己的投影通知 MUST NOT 被当作活动 +- **AND** 活动收集 MUST 使用已有原生 Codex app-server 与外部 Harness 事件流；MUST NOT 新建定时器、进程或 rollout 扫描 + +#### Scenario: 中间 Thread 先关闭 +- **WHEN** 原生子代理链中的中间 Thread 或祖先 Thread 已关闭，而其后代仍在运行并产生事件 +- **THEN** Host SHALL 继续把后代活动计入整条链上的祖先 Thread，MUST NOT 因中间 Thread 关闭而对祖先误报静默 +- **AND** 该子树全部关闭后，Host SHALL 清除其祖先关系 + +#### Scenario: 合并投递且不结束观察 +- **WHEN** 卡住提醒触发时接收方正忙，或同一接收方还有其他待投递通知 +- **THEN** 卡住提醒 SHALL 沿已有待投递机制重试并合并；活动恢复后，待投递提醒仍保留触发时刻的事实 +- **AND** 卡住提醒送达或投递失败 MUST NOT 结束主 watch 的停下观察 + +#### Scenario: 外部客户端接收卡住提醒 +- **WHEN** 启用卡住提醒的 watch 以外部客户端为接收方，运行中的 Thread 静默达到阈值 +- **THEN** Host SHALL 向该客户端事件日志写入 `outcome: "stalled"` 事件，含 Thread 链接、可用的 Turn 标识与说明静默的 `note`，不启动 Turn +- **AND** 事件 MUST NOT 表示 Thread 已失败或已停止；后续 Harness 活动 SHALL 允许下一段静默再次提醒 + +#### Scenario: 默认关闭 +- **WHEN** 调用方未提供 `--stall-after-ms` +- **THEN** watch SHALL 保持原有的一次性停下、到期与读取故障通知行为，MUST NOT 因静默生成通知 + +#### Scenario: 重启后重新计时 +- **WHEN** Host Runtime 重启 +- **THEN** 静默阈值 SHALL 随 watch 保存并恢复，主 watch SHALL 继续观察 +- **AND** 卡住提醒不是终态，MUST NOT 写入 watch 持久记录；重启前未送达的卡住提醒不再投递 +- **AND** 恢复观察 SHALL 从恢复时刻重新计时，MUST NOT 将停机时长算作静默 +- **AND** 重启前已经提醒过的静默，恢复后再次达到阈值时 SHALL 重新提醒 diff --git a/openspec/changes/add-thread-watch-stall-alerts/tasks.md b/openspec/changes/add-thread-watch-stall-alerts/tasks.md new file mode 100644 index 000000000..bd525da8b --- /dev/null +++ b/openspec/changes/add-thread-watch-stall-alerts/tasks.md @@ -0,0 +1,15 @@ +# Tasks + +依赖 `persist-thread-watches` 与 `add-external-client-access`。行为验收见 [spec](specs/cross-harness-delegation/spec.md)。 + +## 1. 静默检测与提醒 + +- [x] 1.1 `thread watch`、`delegate start --watch true` 与 MCP 接受 `--stall-after-ms`/`stallAfterMs`，未启用 watch 时拒绝，重复注册保留原阈值。验证：CLI、MCP 与 watch 服务测试。 +- [x] 1.2 从原生 app-server 通知与外部 Harness 输出收集活动，沿子代理祖先关系传播；原生 Thread 关闭后只要还有未关闭的后代就保留整条祖先链，子树全部关闭、外部会话释放或后端停止后清除。验证：活动收集单元测试与 Host 集成测试。 +- [x] 1.3 静默达到阈值时提醒一次并继续观察；提醒以“至少 <阈值>”精确表述、不带进度，经待投递机制合并投递，外部客户端写入事件日志。验证：watch 服务与客户端事件测试。 +- [x] 1.4 阈值随 watch 持久化，卡住提醒、活动时间与已提醒状态不落盘，重启后重新计时。验证：存储与重启测试。 + +## 2. 文档与验收 + +- [x] 2.1 更新 `codexhost delegate --help`（`thread watch` 结果列表与 `events`）、`docs/architecture/thread-watch.md`、`external-clients.md` 与 `docs/index.md`。验证：帮助测试。 +- [ ] 2.2 真实 Desktop 验收：委派长任务并 `--watch true --stall-after-ms `，子 Thread 长时间没有事件时父 Thread 收到一次卡住提醒，之后子 Thread 停下时再收到停下通知。记录平台与版本。 diff --git a/packages/host-runtime/src/app-server-host.ts b/packages/host-runtime/src/app-server-host.ts index 081732ae0..10b4a6613 100644 --- a/packages/host-runtime/src/app-server-host.ts +++ b/packages/host-runtime/src/app-server-host.ts @@ -16,6 +16,7 @@ import { harnessInstallationRequestTimeoutMs, THREAD_CATALOG_READ_METHOD, threadCatalogReadParamsSchema, + type HostThreadId, } from "@codexhost/shared-contracts"; import { requestDesktopRemoteConnections } from "@codexhost/desktop-control"; import type { RuntimeMaintenance } from "./runtime-maintenance.js"; @@ -40,6 +41,8 @@ import { webOpenResultSchema, } from "@codexhost/shared-contracts"; import { + DELEGATION_CAPACITY_SETTINGS_METHOD, + delegationCapacitySettingsSchema, DELEGATION_MENTION_PATH_PREFIX, IDLE_RELEASE_SETTINGS_METHOD, restoreHarnessCommandMentions, @@ -55,6 +58,7 @@ import { import { managedDelegationSkillReference } from "./delegation-skill.js"; import { AccountRateLimits } from "./codex-runtime/account-rate-limits.js"; import { NativeAccountObserver } from "./native-account-observer.js"; +import { ThreadActivity } from "./thread-activity.js"; import { nativeThreadSupportsReferences } from "./native-thread-reference-capability.js"; import { THREAD_REFERENCE_SOURCE_HOST_PARAM } from "@codexhost/shared-contracts"; import { HarnessAccountInspectionCache } from "./harness-accounts.js"; @@ -151,6 +155,7 @@ import { listExternalItems, listExternalTurns, } from "./external-thread-history.js"; +import { originClientName } from "./external-client-origin.js"; import { executeExternalThreadRollback } from "./external-thread-rollback.js"; import { createExternalThreadRecordInput, @@ -201,6 +206,11 @@ import { loadHarnessPlugins } from "./harness-plugin-loader.js"; import type { HarnessPluginRegistry } from "./harness-plugin-registry.js"; import { DesktopRequestQueue } from "./desktop-request-queue.js"; import type { + DelegationAdmission, + DelegationCapacityGate, + DelegationQueueClaim, + DelegationQueuedStart, + DelegationStartIdentity, DelegationControlRegistration, DelegationStartInput, DelegationStartResult, @@ -396,6 +406,13 @@ export interface AppServerHostOptions { usageStatistics?: UsageStatistics; } +/** A native delegation Thread from thread/start, or from thread/resume once it was queued. */ +interface OpenedOfficialThread { + threadId: string; + thread: JsonObject | null; + result: JsonObject | null; +} + interface TurnProjectionGate { promise: Promise; resolve(): void; @@ -658,14 +675,18 @@ export class AppServerHost { #delegationCoordinator: HarnessDelegationCoordinator; #sessionImportRequests: SessionImportRequests | undefined; #unregisterDelegationApi: (() => void) | undefined; + /** Set by the delegation registry this Host joined; absent means delegated starts are never queued. */ + #delegationCapacity: DelegationCapacityGate | undefined; #unsubscribeAccountState: (() => void) | undefined; #activeOfficialTurns = new Map(); /** Active reply guards by request ID; each records whether its request was answered. */ readonly #replyGuards = new Map>(); #pendingOfficialTurnStarts = new Map(); + /** Desktop Turn starts carrying a queued native task, by request ID, until the backend answers. */ + readonly #pendingQueuedClaims = new Map(); #activeWorkDrainWaiters = new Set<() => void>(); #pendingOfficialDelegationThreads = new Set(); - #pendingOfficialTerminalStatuses = new Map(); + #pendingOfficialTerminalStatuses = new Map(); readonly #officialRateLimits = new AccountRateLimits(); #routeObservationTracker = new RequestRouteObservationTracker(); #officialServerRequests = new Map(); @@ -684,6 +705,8 @@ export class AppServerHost { readonly #consoleReplies = new Map void>(); #nextConsoleRequest = 0; #subagentThreadStatuses = new Map(); + readonly #threadActivity = new ThreadActivity(); + readonly #officialForkChildren = new Set(); #runningSubagentsByParent = new Map>(); #pendingExternalCommandRequests = new Set(); readonly #externalQueue = new ExternalThreadQueue(); @@ -827,9 +850,12 @@ export class AppServerHost { scope: this.#officialRuntimeScope, onBackendStopped: () => { this.#pendingOfficialTurnStarts.clear(); + this.#abandonQueuedClaims(); this.#activeOfficialTurns.clear(); this.#client.resetNativeSession(); for (const listener of this.#nativeSettingsNotifications) listener(null); + this.#threadActivity.clear(); + this.#officialForkChildren.clear(); this.#signalActiveWorkChanged(); }, output: async (output) => @@ -902,31 +928,46 @@ export class AppServerHost { readOfficial: (input) => this.#readOfficialDelegationThread(input), sendOfficial: (input) => this.#sendOfficialDelegationThread(input), cancelOfficial: (input) => this.#cancelOfficialDelegationThread(input), - startOfficial: (input) => this.#startOfficialDelegation(input), + startOfficial: (input, admitted) => this.#startOfficialDelegation(input, admitted), listOfficial: (input) => this.#listDelegationThreads(input), officialThreadMetadata: (threadId) => this.#readOfficialThreadMetadata(threadId), activeOfficialParents: () => [...this.#activeOfficialTurns.keys()], externalThreadBusy: (thread) => this.#externalThreadBusy(thread), + admitDelegation: (request) => this.#delegationCapacity?.admit(request), }); const maintenanceOperation = (run: () => Promise): Promise => options.runtimeMaintenance ? options.runtimeMaintenance.operation(run) : run(); const unregisterDelegationApi = options.onDelegationApi?.({ + subscribeActivity: (listener) => this.#threadActivity.subscribe(listener), + runningThreadIds: () => this.#runningTopLevelThreadIds(), + useCapacity: (capacity) => { + this.#delegationCapacity = capacity; + }, + adoptQueued: async (queued) => { + if (queued.input.harnessId !== "codex" || this.#options.externalOnly) return undefined; + return this.#adoptQueuedOfficialStart(queued); + }, listHarnesses: () => this.#waitForPlugins().then(() => this.#delegationCoordinator.listHarnesses()), inspect: (input) => this.#waitForPlugins().then(() => this.#delegationCoordinator.inspect(input)), - start: (input) => + start: (input, admitted) => maintenanceOperation(async () => { if (options.sharedDelegation && input.harnessId !== "codex") { const active = [...this.#activeOfficialTurns.keys()]; const parentThreadId = input.parentThreadId ?? (active.length === 1 ? active[0] : undefined); - return options.sharedDelegation.start({ - ...input, - ...(parentThreadId ? { parentThreadId } : {}), - }); + return options.sharedDelegation.start( + { + ...input, + ...(parentThreadId ? { parentThreadId } : {}), + }, + admitted, + ); } - return this.#waitForPlugins().then(() => this.#delegationCoordinator.start(input)); + return this.#waitForPlugins().then(() => + this.#delegationCoordinator.start(input, admitted), + ); }), send: (input) => maintenanceOperation(() => @@ -1042,6 +1083,10 @@ export class AppServerHost { this.#rollbackExternalThread(request, derived, rollback), listExternalHistory: (request, thread, params, historyFresh) => this.#listExternalHistory(request, thread, params, historyFresh), + listQueuedOfficialHistory: (request, params) => + this.#listQueuedOfficialHistory(request, params), + readOrResumeQueuedOfficialThread: (request, params) => + this.#readOrResumeQueuedOfficialThread(request, params), startExternalTurn: (request, thread) => this.#startExternalTurn(request, thread), steerExternalTurn: (request, thread) => this.#steerExternalTurn(request, thread), interruptExternalTurn: (request, thread, requestedTurnId) => @@ -1278,6 +1323,7 @@ export class AppServerHost { this.#externalRuntime.clear(); this.#externalPrewarms.clear(); this.#pendingOfficialTurnStarts.clear(); + this.#abandonQueuedClaims(); this.#routeObservationTracker.clear(); this.#unregisterDelegationApi?.(); this.#unregisterDelegationApi = undefined; @@ -1324,6 +1370,40 @@ export class AppServerHost { for (const resolve of waiters) resolve(); } + /** The backend's answer to a Turn start that took over a queued native task. */ + #settleQueuedClaim(value: JsonValue): void { + if (!isRecord(value) || !("id" in value) || "method" in value) return; + const claim = this.#pendingQueuedClaims.get(value.id); + if (!claim) return; + this.#pendingQueuedClaims.delete(value.id); + const result = isRecord(value.result) ? value.result : null; + const turn = result && isRecord(result.turn) ? result.turn : null; + if (!turn || typeof turn.id !== "string") { + void claim.settle(isRecord(value.error) ? "rejected" : "unknown"); + return; + } + const turnId = turn.id; + void claim + .settle("started") + .then(() => + this.#adoptBegunOfficialDelegation({ + input: claim.input, + delegationId: claim.identity.delegationId, + threadId: claim.identity.threadId, + turnId, + taskDigest: claim.taskDigest, + }), + ) + .catch((error: unknown) => this.#diagnose(error)); + } + + /** Takeovers whose answer will never arrive; each Thread is checked for a Turn before reuse. */ + #abandonQueuedClaims(): void { + const claims = [...this.#pendingQueuedClaims.values()]; + this.#pendingQueuedClaims.clear(); + for (const claim of claims) void claim.settle("unknown"); + } + #observeOfficialTurnStartResponse(value: JsonValue): void { if (!isRecord(value) || !("id" in value)) return; const threadId = this.#pendingOfficialTurnStarts.get(value.id); @@ -1657,6 +1737,7 @@ export class AppServerHost { request.method === LOADED_SESSIONS_METHOD || request.method === IDLE_RELEASE_SETTINGS_METHOD || request.method === USAGE_STATISTICS_METHOD || + request.method === DELEGATION_CAPACITY_SETTINGS_METHOD || // Settings requests from Web clients; the shared service owns these handlers too. request.method.startsWith("codexhost/account/") || request.method.startsWith("codexhost/update/") || @@ -1696,6 +1777,20 @@ export class AppServerHost { } return; } + if (request.method === DELEGATION_CAPACITY_SETTINGS_METHOD) { + const parsed = delegationCapacitySettingsSchema.safeParse(request.params); + if (!this.#delegationCapacity) { + await this.#writer.json( + rpcError(request, -32601, "Delegation capacity is unavailable on this Host"), + ); + } else if (!parsed.success) { + await this.#writer.json(rpcError(request, -32602, "Invalid delegation capacity settings")); + } else { + const settings = this.#delegationCapacity.configure(parsed.data); + await this.#writer.json(rpcEnvelope(request, { result: settings })); + } + return; + } if (await this.#desktopControl.handle(request)) return; if ( request.method === "codexhost/thread/pins/list" || @@ -1707,10 +1802,53 @@ export class AppServerHost { if (await this.#threadRouting.handle(request, frame, origin)) return; await this.#forwardOfficialRequest( request, - await this.#rewriteOfficialDelegationMentions(request, frame), + await this.#rewriteOfficialTurnInput(request, frame), ); } + async #rewriteOfficialTurnInput( + request: JsonRpcRequest, + frame: Buffer, + ): Promise> { + const claimed = await this.#claimQueuedOfficialStart(request); + if (!claimed) return this.#rewriteOfficialDelegationMentions(request, frame); + try { + return await this.#rewriteOfficialDelegationMentions( + claimed, + encodeJsonFrame(claimed as unknown as JsonObject), + ); + } catch (error) { + // The backend never saw this Turn start, so the queued task must stay startable. + const claim = this.#pendingQueuedClaims.get(request.id); + this.#pendingQueuedClaims.delete(request.id); + void claim?.settle("rejected"); + throw error; + } + } + + /** + * A user message to a native Thread still waiting in the delegation queue starts + * the queued task now, ahead of the message in the same Turn, so it runs exactly once. + * The start stays held in the queue until the backend answers this Turn start. + */ + async #claimQueuedOfficialStart(request: JsonRpcRequest): Promise { + if (request.method !== "turn/start" || !this.#delegationCapacity) return undefined; + if (this.#options.externalOnly) return undefined; + if (!isRecord(request.params) || !Array.isArray(request.params.input)) return undefined; + const { threadId } = request.params; + if (typeof threadId !== "string") return undefined; + const claim = await this.#delegationCapacity.claim(threadId, "codex"); + if (!claim) return undefined; + this.#pendingQueuedClaims.set(request.id, claim); + return { + ...(request as unknown as JsonObject), + params: { + ...request.params, + input: [{ type: "text", text: claim.input.task }, ...request.params.input], + }, + } as unknown as JsonRpcRequest; + } + /** Native Codex Turns: turn `#` delegation chips into an explicit Skill-backed request. */ async #rewriteOfficialDelegationMentions( request: JsonRpcRequest, @@ -1788,6 +1926,10 @@ export class AppServerHost { if (request.method === "turn/start") { this.#pendingOfficialTurnStarts.delete(request.id); this.#signalActiveWorkChanged(); + const claim = this.#pendingQueuedClaims.get(request.id); + this.#pendingQueuedClaims.delete(request.id); + // The frame may have reached the backend before the failure. + void claim?.settle("unknown"); } await this.#writer.json( rpcError(request, -32001, "Official request failed; retry explicitly"), @@ -1801,6 +1943,8 @@ export class AppServerHost { value: JsonValue; }): Promise { for (const listener of this.#nativeSettingsNotifications) listener(input.value); + this.#threadActivity.observeOfficial(input.value); + this.#observeOfficialForkLineage(input.value); if (this.#options.externalOnly) { this.#client.observeNativeMessage(input.value); // Native children created through Delegation still have a native owner @@ -1811,6 +1955,7 @@ export class AppServerHost { const parsed = input.value; const observedAtMs = Date.now(); this.#observeOfficialTurnStartResponse(parsed); + this.#settleQueuedClaim(parsed); let forwarded: JsonValue = parsed; if (isRecord(parsed) && typeof parsed.method === "string" && "id" in parsed) { const originalId = parsed.id; @@ -1993,6 +2138,50 @@ export class AppServerHost { } } + #observeOfficialForkLineage(value: JsonValue): void { + if (!isRecord(value)) return; + const noteThread = (thread: unknown): void => { + if (!isRecord(thread) || typeof thread.id !== "string" || !thread.id) return; + const forkedFromId = + typeof thread.forkedFromId === "string" && thread.forkedFromId + ? thread.forkedFromId + : isRecord(thread.history_base) && typeof thread.history_base.thread_id === "string" + ? thread.history_base.thread_id + : undefined; + if (forkedFromId && forkedFromId !== thread.id) { + this.#officialForkChildren.add(thread.id); + } + }; + if (isRecord(value.params)) noteThread(value.params.thread); + if (isRecord(value.result)) { + noteThread(value.result.thread); + if (Array.isArray(value.result.data)) { + for (const entry of value.result.data) noteThread(entry); + } + } + } + + /** Running top-level Threads; Subagents, Fork children, and session-parented Delegation children are excluded. */ + #runningTopLevelThreadIds(): string[] { + const external = this.#externalRuntime + .values() + .filter( + (thread) => + !thread.record.subagent && + !thread.record.forkSource && + !this.#repository.hasSessionDelegationParent(thread.id) && + (this.#externalThreadBusy(thread) || thread.activeTurnId !== null), + ) + .map((thread) => thread.id); + const native = [...this.#activeOfficialTurns.keys()].filter( + (threadId) => + !this.#threadActivity.isSubagent(threadId) && + !this.#officialForkChildren.has(threadId) && + !this.#repository.hasSessionDelegationParent(threadId), + ); + return [...external, ...native]; + } + async #canHandleDelegationStart(input: DelegationStartInput): Promise { if (input.parentThreadId) return this.#ownsDelegationThread(input.parentThreadId); const externalActive = this.#externalRuntime.values().some((thread) => thread.running); @@ -2114,8 +2303,10 @@ export class AppServerHost { }; } + /** `admitted` starts a queued delegation in the Thread created when it was queued. */ async #startOfficialDelegation( input: DelegationStartInput & { parentThreadId: string; cwd: string; title: string }, + admitted?: DelegationStartIdentity, ): Promise { let requestedModel: HarnessModelRef | undefined; try { @@ -2136,14 +2327,17 @@ export class AppServerHost { }), ) .digest("hex"); - const existing = input.requestId - ? await this.#repository.findDelegationByRequest(input.requestId) - : await this.#repository.findRecentDelegation({ - parentHostThreadId: hostThreadIdSchema.parse(input.parentThreadId), - targetHarnessId: harnessIdSchema.parse("codex"), - taskDigest: digest, - since: new Date(Date.now() - 30_000), - }); + // A queued start was deduplicated and validated when it was queued. + const existing = admitted + ? null + : input.requestId + ? await this.#repository.findDelegationByRequest(input.requestId) + : await this.#repository.findRecentDelegation({ + parentHostThreadId: hostThreadIdSchema.parse(input.parentThreadId), + targetHarnessId: harnessIdSchema.parse("codex"), + taskDigest: digest, + since: new Date(Date.now() - 30_000), + }); if ( existing && input.requestId && @@ -2154,7 +2348,10 @@ export class AppServerHost { "Request ID is already associated with another Delegation configuration", ); } - if (existing) { + const existingQueued = existing + ? await this.#delegationCapacity?.peek?.(existing.childHostThreadId) + : undefined; + if (existing && !existingQueued) { const turnId = this.#activeOfficialTurns.get(existing.childHostThreadId) ?? "pending"; return { delegationId: existing.delegationId, @@ -2169,7 +2366,7 @@ export class AppServerHost { }, }; } - if (requestedModel || input.thinkingOptionId) { + if (!admitted && (requestedModel || input.thinkingOptionId)) { const inspected = await this.#inspectOfficialDelegationTarget({ harnessId: "codex", cwd: input.cwd, @@ -2198,6 +2395,72 @@ export class AppServerHost { } } } + const delegationId = hostThreadIdSchema.parse(admitted?.delegationId ?? randomUUID()); + let opened: OpenedOfficialThread | undefined; + if (admitted) { + opened = await this.#resumeQueuedOfficialThread(admitted.threadId); + // A restart or a user message may have begun the Turn without the queue. + const begun = await this.#latestOfficialTurnId(admitted.threadId); + if (begun) { + return this.#adoptBegunOfficialDelegation({ + input, + delegationId, + threadId: admitted.threadId, + turnId: begun, + taskDigest: digest, + }); + } + } + // A queued native start waits in a Thread without Turns, so its identity is final. + const admission = + admitted || !originClientName(input.parentThreadId) + ? undefined + : await this.#delegationCapacity?.admit({ + input, + title: input.title, + taskDigest: digest, + identity: { delegationId, threadId: delegationId, turnId: "pending" }, + prepare: async () => { + opened = await this.#openOfficialDelegationThread({ + input, + delegationId, + digest, + nativeSelection, + }); + return { delegationId, threadId: opened.threadId, turnId: "pending" }; + }, + }); + if (admission?.state === "queued") return this.#queuedOfficialResult(admission); + try { + return await this.#beginOfficialDelegation({ + input, + delegationId, + digest, + nativeSelection, + requestedModel, + opened: + opened ?? + (await this.#openOfficialDelegationThread({ + input, + delegationId, + digest, + nativeSelection, + })), + queued: admitted !== undefined, + }); + } finally { + admission?.release(); + } + } + + /** Creates, records, and names the native Thread of a delegation, without starting a Turn. */ + async #openOfficialDelegationThread(context: { + input: DelegationStartInput & { parentThreadId: string; cwd: string; title: string }; + delegationId: HostThreadId; + digest: string; + nativeSelection: ReturnType | undefined; + }): Promise { + const { input, delegationId, digest, nativeSelection } = context; const started = await this.#officialRuntime.request("thread/start", { cwd: input.cwd, ...nativeSelection, @@ -2206,17 +2469,167 @@ export class AppServerHost { ephemeral: false, historyMode: "paginated", }); - const startedResult = isRecord(started.result) ? started.result : null; - const thread = startedResult && isRecord(startedResult.thread) ? startedResult.thread : null; + const result = isRecord(started.result) ? started.result : null; + const thread = result && isRecord(result.thread) ? result.thread : null; const threadId = thread && typeof thread.id === "string" ? thread.id : null; if (!threadId) throw new Error("Official thread/start returned no Thread identity"); + try { + const source = await this.#repository.find(input.parentThreadId); + await this.#repository.createDelegation({ + delegationId, + parentHostThreadId: hostThreadIdSchema.parse(input.parentThreadId), + childHostThreadId: hostThreadIdSchema.parse(threadId), + sourceHarnessId: source?.harnessId ?? harnessIdSchema.parse("codex"), + targetHarnessId: harnessIdSchema.parse("codex"), + status: "creating", + ...(input.requestId ? { requestId: input.requestId } : {}), + taskDigest: digest, + }); + } catch (error) { + await this.#requestOfficial("thread/delete", { threadId }).catch(() => undefined); + throw error; + } + // The title is presentation only: a failure keeps the default title and still delivers the task. + await this.#requestOfficial("thread/name/set", { threadId, name: input.title }).catch( + () => undefined, + ); + return { threadId, thread, result }; + } + + /** Loads a queued native Thread again; Host and backend restarts unload it. */ + async #resumeQueuedOfficialThread(threadId: string): Promise { + const resumed = await this.#requestOfficial("thread/resume", { threadId, excludeTurns: true }); + if (isRecord(resumed.error) || !isRecord(resumed.result)) { + throw new DelegationControlError( + "DELEGATION_FAILED", + isRecord(resumed.error) && typeof resumed.error.message === "string" + ? resumed.error.message + : "Official queued Thread could not be resumed", + ); + } + const thread = isRecord(resumed.result.thread) ? resumed.result.thread : null; + return { threadId, thread, result: resumed.result }; + } + + /** Null when the Thread has no Turn; a Thread that cannot be read throws instead. */ + async #latestOfficialTurnId(threadId: string): Promise { + const response = await this.#requestOfficial("thread/read", { threadId, includeTurns: true }); + const result = isRecord(response.result) ? response.result : null; + const thread = result && isRecord(result.thread) ? result.thread : null; + if (isRecord(response.error) || !thread || !Array.isArray(thread.turns)) { + throw new DelegationControlError( + "DELEGATION_FAILED", + "Official queued Thread could not be read", + ); + } + const latest = thread.turns.at(-1); + if (latest === undefined) return null; + if (!isRecord(latest) || typeof latest.id !== "string") { + throw new DelegationControlError("DELEGATION_FAILED", "Official Turn has no identity"); + } + return latest.id; + } + + /** Whether a queued native Thread already has a Turn; one that does is recorded as the delegation. */ + async #adoptQueuedOfficialStart(queued: DelegationQueuedStart): Promise { + const { threadId, delegationId } = queued.identity; + let turnId: string | null; + try { + await this.#resumeQueuedOfficialThread(threadId); + turnId = await this.#latestOfficialTurnId(threadId); + } catch { + return undefined; + } + if (turnId === null) return false; + await this.#adoptBegunOfficialDelegation({ + input: queued.input, + delegationId, + threadId, + turnId, + taskDigest: queued.taskDigest, + }); + return true; + } + + #queuedOfficialResult( + admission: Extract, + ): DelegationStartResult { + const { identity } = admission; + return { + delegationId: identity.delegationId, + threadId: identity.threadId, + turnId: identity.turnId, + harnessId: "codex", + deepLink: `codex://threads/${identity.threadId}`, + status: admission.status, + ...(admission.position ? { queue: { position: admission.position } } : {}), + cwd: admission.cwd, + title: admission.title, + parentThreadId: admission.parentThreadId, + next: { + read: `codexhost thread read ${identity.threadId}`, + wait: `codexhost thread wait ${identity.threadId} --timeout-ms 30000`, + }, + }; + } + + /** Records a queued native start whose first Turn began without the queue, after a restart or by the user. */ + async #adoptBegunOfficialDelegation(input: { + input: DelegationStartInput & { parentThreadId: string }; + delegationId: string; + threadId: string; + turnId: string; + taskDigest: string; + }): Promise { + const { threadId } = input; + const existing = await this.#repository.getDelegationByChild( + hostThreadIdSchema.parse(threadId), + ); + if (!existing) { + const source = await this.#repository.find(input.input.parentThreadId); + await this.#repository.createDelegation({ + delegationId: hostThreadIdSchema.parse(input.delegationId), + parentHostThreadId: hostThreadIdSchema.parse(input.input.parentThreadId), + childHostThreadId: hostThreadIdSchema.parse(threadId), + sourceHarnessId: source?.harnessId ?? harnessIdSchema.parse("codex"), + targetHarnessId: harnessIdSchema.parse("codex"), + status: "running", + ...(input.input.requestId ? { requestId: input.input.requestId } : {}), + taskDigest: input.taskDigest, + }); + } else if (existing.status === "creating") { + await this.#repository.setDelegationStatus(existing.delegationId, "running"); + } + const status = existing && existing.status !== "creating" ? existing.status : "running"; + return { + delegationId: existing?.delegationId ?? input.delegationId, + threadId, + turnId: input.turnId, + harnessId: "codex", + deepLink: `codex://threads/${threadId}`, + status, + next: { + read: `codexhost thread read ${threadId}`, + wait: `codexhost thread wait ${threadId} --timeout-ms 30000`, + }, + }; + } + + async #beginOfficialDelegation(context: { + input: DelegationStartInput & { parentThreadId: string; cwd: string }; + delegationId: HostThreadId; + digest: string; + nativeSelection: ReturnType | undefined; + requestedModel: HarnessModelRef | undefined; + opened: OpenedOfficialThread; + /** A queued Thread is never deleted: a user message may have reached it. */ + queued: boolean; + }): Promise { + const { input, delegationId, digest, nativeSelection, requestedModel } = context; + const { threadId, thread, result: startedResult } = context.opened; this.#pendingOfficialDelegationThreads.add(threadId); let turnId: string; try { - // The title is presentation only: a failure keeps the default title and still delivers the task. - await this.#requestOfficial("thread/name/set", { threadId, name: input.title }).catch( - () => undefined, - ); const turn = await this.#requestOfficial("turn/start", { threadId, input: [{ type: "text", text: input.task }], @@ -2231,31 +2644,43 @@ export class AppServerHost { } catch (error) { this.#pendingOfficialDelegationThreads.delete(threadId); this.#pendingOfficialTerminalStatuses.delete(threadId); - await this.#requestOfficial("thread/delete", { threadId }).catch(() => undefined); + if (!context.queued) { + await this.#repository.removeDelegation(delegationId).catch(() => undefined); + await this.#requestOfficial("thread/delete", { threadId }).catch(() => undefined); + } else { + await this.#repository.setDelegationStatus(delegationId, "failed").catch(() => undefined); + } throw error; } this.#activeOfficialTurns.set(threadId, turnId); - const delegationId = hostThreadIdSchema.parse(randomUUID()); try { - const source = await this.#repository.find(input.parentThreadId); const pendingTerminal = this.#pendingOfficialTerminalStatuses.get(threadId); - await this.#repository.createDelegation({ - delegationId, - parentHostThreadId: hostThreadIdSchema.parse(input.parentThreadId), - childHostThreadId: hostThreadIdSchema.parse(threadId), - sourceHarnessId: source?.harnessId ?? harnessIdSchema.parse("codex"), - targetHarnessId: harnessIdSchema.parse("codex"), - status: pendingTerminal ?? "running", - ...(input.requestId ? { requestId: input.requestId } : {}), - taskDigest: digest, - }); + const status = pendingTerminal ?? "running"; + const existing = await this.#repository.getDelegationByChild( + hostThreadIdSchema.parse(threadId), + ); + if (existing) { + await this.#repository.setDelegationStatus(existing.delegationId, status); + } else { + const source = await this.#repository.find(input.parentThreadId); + await this.#repository.createDelegation({ + delegationId, + parentHostThreadId: hostThreadIdSchema.parse(input.parentThreadId), + childHostThreadId: hostThreadIdSchema.parse(threadId), + sourceHarnessId: source?.harnessId ?? harnessIdSchema.parse("codex"), + targetHarnessId: harnessIdSchema.parse("codex"), + status, + ...(input.requestId ? { requestId: input.requestId } : {}), + taskDigest: digest, + }); + } return { - delegationId, + delegationId: existing?.delegationId ?? delegationId, threadId, turnId, harnessId: "codex", deepLink: `codex://threads/${threadId}`, - status: pendingTerminal ?? "running", + status, cwd: thread && typeof thread.cwd === "string" ? thread.cwd : input.cwd, ...(requestedModel || input.thinkingOptionId ? { @@ -2285,9 +2710,13 @@ export class AppServerHost { }, }; } catch (error) { - this.#activeOfficialTurns.delete(threadId); - this.#signalActiveWorkChanged(); - await this.#requestOfficial("thread/delete", { threadId }).catch(() => undefined); + // A queued Turn keeps running: its Thread may already hold a user message. + if (!context.queued) { + this.#activeOfficialTurns.delete(threadId); + this.#signalActiveWorkChanged(); + await this.#repository.removeDelegation(delegationId).catch(() => undefined); + await this.#requestOfficial("thread/delete", { threadId }).catch(() => undefined); + } throw error; } finally { this.#pendingOfficialDelegationThreads.delete(threadId); @@ -2430,6 +2859,18 @@ export class AppServerHost { const running = this.#activeOfficialTurns.has(input.threadId) || (isRecord(thread.status) && thread.status.type === "active"); + const delegation = await this.#repository.getDelegationByChild( + hostThreadIdSchema.parse(input.threadId), + ); + if (turns.length === 0 && !running) { + const finished = await this.#delegationCapacity?.finished?.(input); + if (finished) { + if (delegation && finished.status !== "queued" && delegation.status !== finished.status) { + await this.#repository.setDelegationStatus(delegation.delegationId, finished.status); + } + return finished; + } + } const snapshot = projectDelegationThreadSnapshot({ threadId: input.threadId, harnessId: "codex", @@ -2440,15 +2881,131 @@ export class AppServerHost { ...(input.cursor ? { cursor: input.cursor } : {}), ...(input.limit !== undefined ? { limit: input.limit } : {}), }); - const delegation = await this.#repository.getDelegationByChild( - hostThreadIdSchema.parse(input.threadId), - ); - if (delegation && delegation.status !== snapshot.status) { + if (delegation && snapshot.status !== "queued" && delegation.status !== snapshot.status) { await this.#repository.setDelegationStatus(delegation.delegationId, snapshot.status); } return snapshot; } + #queuedOfficialPreviewTurn(threadId: string): JsonObject | undefined { + if (this.#options.externalOnly || this.#activeOfficialTurns.has(threadId)) return undefined; + return this.#delegationCapacity?.previewTurn?.(threadId); + } + + async #listQueuedOfficialHistory(request: JsonRpcRequest, params: JsonObject): Promise { + const threadId = typeof params.threadId === "string" ? params.threadId : null; + if (!threadId) return false; + const preview = this.#queuedOfficialPreviewTurn(threadId); + if (!preview) return false; + if ((await this.#latestOfficialTurnId(threadId).catch(() => null)) !== null) return false; + try { + const result = + request.method === "thread/turns/list" + ? listExternalTurns([preview], params) + : listExternalItems([preview], params); + await this.#writer.json(rpcEnvelope(request, { result })); + } catch (error) { + await this.#writer.json( + rpcError( + request, + error instanceof ExternalHistoryRequestError ? -32602 : -32076, + error instanceof ExternalHistoryRequestError + ? error.message + : "Official queued Thread history projection failed", + ), + ); + } + return true; + } + + async #readOrResumeQueuedOfficialThread( + request: JsonRpcRequest, + params: JsonObject, + ): Promise { + const threadId = typeof params.threadId === "string" ? params.threadId : null; + if (!threadId) return false; + const preview = this.#queuedOfficialPreviewTurn(threadId); + if (!preview) return false; + const options = this.#observedSessionUsage.requestOptions(request.method, params); + const response = await this.#requestOfficial( + request.method, + options ? { ...params, ...options } : params, + ); + if (isRecord(response.error) || !isRecord(response.result)) { + await this.#writer.json({ ...response, id: request.id }); + return true; + } + const result = response.result; + const thread = isRecord(result.thread) ? result.thread : null; + if (!thread) { + await this.#writer.json({ ...response, id: request.id }); + return true; + } + const nativeTurns = Array.isArray(thread.turns) + ? thread.turns.length + : (await this.#latestOfficialTurnId(threadId).catch(() => null)) !== null + ? 1 + : 0; + if (nativeTurns > 0) { + await this.#writer.json({ ...response, id: request.id }); + return true; + } + const turns = [preview]; + if (request.method === "thread/read") { + await this.#writer.json( + rpcEnvelope(request, { + result: { ...result, thread: { ...thread, turns } }, + }), + ); + return true; + } + try { + if ( + params.initialTurnsPage !== undefined && + params.initialTurnsPage !== null && + !isRecord(params.initialTurnsPage) + ) { + throw new ExternalHistoryRequestError("initialTurnsPage must be an object"); + } + const initialPageParams = isRecord(params.initialTurnsPage) + ? (params.initialTurnsPage as JsonObject) + : null; + const initialTurnsPage = initialPageParams + ? listExternalTurns(turns, initialPageParams) + : (result.initialTurnsPage ?? null); + const turnsBackwardsCursor = listExternalTurns(turns, { + limit: 1, + itemsView: "notLoaded", + }).backwardsCursor; + const itemsBackwardsCursor = listExternalItems(turns, { + limit: 1, + sortDirection: "desc", + }).backwardsCursor; + await this.#writer.json( + rpcEnvelope(request, { + result: { + ...result, + thread: { ...thread, turns: params.excludeTurns === true ? [] : turns }, + initialTurnsPage, + turnsBackwardsCursor, + itemsBackwardsCursor, + }, + }), + ); + } catch (error) { + await this.#writer.json( + rpcError( + request, + error instanceof ExternalHistoryRequestError ? -32602 : -32076, + error instanceof ExternalHistoryRequestError + ? error.message + : "Official queued Thread history projection failed", + ), + ); + } + return true; + } + async #listDelegationThreads(input: ThreadListInput): Promise { const [sortKey, sortDirection] = input.sort.split("-") as [string, "asc" | "desc"]; const request: JsonRpcRequest = { @@ -5310,6 +5867,7 @@ export class AppServerHost { async #consumeHarnessOutputs(thread: ExternalThread): Promise { try { for await (const output of thread.session.outputs) { + this.#threadActivity.note(thread.id, thread.record.subagent?.parentHostThreadId); await this.#externalRuntime.idleRelease.consumeOutput(thread, () => this.#projectHarnessOutput(thread, output), ); @@ -5337,6 +5895,7 @@ export class AppServerHost { this.#queuedTurnStarts.delete(thread.id); await this.#publishExternalQueue(thread.id); } + this.#threadActivity.forget(thread.id); } } @@ -5466,7 +6025,10 @@ export class AppServerHost { candidate.nativeSessionRef?.nativeSessionId === thread.record.nativeSessionRef?.nativeSessionId, ); - if (record) await this.#refreshOpenSubagentThread(record.hostThreadId, false); + if (record) { + this.#threadActivity.note(record.hostThreadId, thread.id); + await this.#refreshOpenSubagentThread(record.hostThreadId, false); + } return; } if (event.type === "subagent.state.changed") { @@ -5479,6 +6041,7 @@ export class AppServerHost { thread.record.nativeSessionRef?.nativeSessionId, ); if (!record) return; + this.#threadActivity.note(record.hostThreadId, thread.id); const status = event.status === "pending" || event.status === "running" ? "active" : "idle"; this.#trackRunningSubagent(thread.id, record.hostThreadId, status); await this.#setSubagentThreadStatus(record.hostThreadId, status); @@ -5522,6 +6085,7 @@ export class AppServerHost { subagent.status === "pending" || subagent.status === "running" ? "active" : "idle"; const record = await this.#repository.materializeSubagent(parent.record, subagent); if (!record) return subagent; + this.#threadActivity.note(record.hostThreadId, parent.id); if (this.#subagentThreadStatuses.has(record.hostThreadId)) { this.#trackRunningSubagent(parent.id, record.hostThreadId, status); await this.#setSubagentThreadStatus(record.hostThreadId, status); diff --git a/packages/host-runtime/src/delegation-capacity-queue.ts b/packages/host-runtime/src/delegation-capacity-queue.ts new file mode 100644 index 000000000..29e6d3779 --- /dev/null +++ b/packages/host-runtime/src/delegation-capacity-queue.ts @@ -0,0 +1,681 @@ +import path from "node:path"; + +import type { JsonObject } from "@codexhost/protocol-core"; +import { + DEFAULT_DELEGATION_CAPACITY_SETTINGS, + delegationCapacitySettingsSchema, + type DelegationCapacitySettings, +} from "@codexhost/shared-contracts"; + +import type { + DelegationQueueStore, + PersistedQueuedDelegation, +} from "./delegation-capacity-store.js"; +import { projectDelegationThreadSnapshot } from "./delegation-snapshot.js"; +import { + DelegationControlError, + type DelegationAdmission, + type DelegationAdmissionRequest, + type DelegationCapacityGate, + type DelegationQueueClaim, + type DelegationQueuedStart, + type DelegationStartIdentity, + type DelegationStartInput, + type DelegationThreadListItem, + type DelegationThreadSnapshot, + type ThreadCancelResult, + type ThreadReadInput, +} from "./delegation-types.js"; + +const DEFAULT_POLL_INTERVAL_MS = 2_000; +const DEFAULT_SETTINGS_GRACE_MS = 5 * 60_000; +const IMPLICIT_DEDUPLICATION_MS = 30_000; +const MAX_FINISHED_ENTRIES = 50; +const MAX_TIMER_MS = 2_147_483_647; +const CANCELLED_NOTE = "Removed from the queue before it started"; +const RESTART_NOTE = + "The Host Runtime restarted while this queued delegation was starting; start it again if the Thread does not exist"; + +interface QueueEntry { + record: PersistedQueuedDelegation; + /** `claiming`: a user message is taking the start over; it keeps its place until that settles. */ + state: "queued" | "starting" | "claiming"; + /** A start or takeover with an unknown outcome, so its Thread may already have a Turn. */ + uncertain?: boolean; + started?: Promise; + /** A Turn check or takeover in progress; nothing else happens to the entry until it settles. */ + busy?: Promise | undefined; +} + +/** A queued start that never became a Thread: cancelled, failed to start, or cut off by a restart. */ +interface FinishedEntry { + record: PersistedQueuedDelegation; + status: "failed" | "interrupted"; + message: string; +} + +/** What the queue needs from the Host sessions it sits above. */ +export interface DelegationCapacityRuntime { + /** Top-level Threads with an active Turn in any session; Harness subagents excluded. */ + runningThreadIds(): Iterable; + /** False while no session could take a start. */ + canStart(): boolean; + start(input: DelegationStartInput, identity: DelegationStartIdentity): Promise; + /** + * Whether the Thread a waiting start was given already has a Turn, adopting it as + * the start when it does; undefined when no session can tell. + */ + adopt?(queued: DelegationQueuedStart): Promise; +} + +function identityOf(record: PersistedQueuedDelegation): DelegationStartIdentity { + return { delegationId: record.delegationId, threadId: record.threadId, turnId: record.turnId }; +} + +function queuedStartOf(record: PersistedQueuedDelegation): DelegationQueuedStart { + return { + identity: identityOf(record), + input: { ...inputOf(record), parentThreadId: record.parentThreadId, cwd: record.cwd }, + taskDigest: record.taskDigest, + }; +} + +function inputOf(record: PersistedQueuedDelegation): DelegationStartInput { + return { + harnessId: record.harnessId, + task: record.task, + ...(record.taskName !== undefined ? { taskName: record.taskName } : {}), + cwd: record.cwd, + parentThreadId: record.parentThreadId, + ...(record.requestId !== undefined ? { requestId: record.requestId } : {}), + ...(record.model ? { model: record.model } : {}), + ...(record.thinkingOptionId ? { thinkingOptionId: record.thinkingOptionId } : {}), + }; +} + +function syntheticQueueTurn( + record: PersistedQueuedDelegation, + status: "inProgress" | "interrupted" | "failed", + message: string, +): JsonObject { + const turnId = `queued-${record.delegationId}`; + const seconds = Math.floor(record.queuedAt / 1000); + return { + id: turnId, + status, + items: [ + { + id: `${turnId}-user`, + type: "userMessage", + clientId: null, + content: [{ type: "text", text: record.task, text_elements: [] }], + }, + { + id: `${turnId}-agent`, + type: "agentMessage", + text: message, + phase: "commentary", + }, + ], + error: status === "failed" ? { message } : null, + startedAt: seconds, + completedAt: status === "inProgress" ? null : seconds, + durationMs: status === "inProgress" ? null : 0, + }; +} + +function recordFor( + request: DelegationAdmissionRequest, + identity: DelegationStartIdentity, +): PersistedQueuedDelegation { + const { input } = request; + return { + ...identity, + harnessId: input.harnessId, + task: input.task, + ...(input.taskName !== undefined ? { taskName: input.taskName } : {}), + cwd: input.cwd, + parentThreadId: input.parentThreadId, + ...(input.requestId !== undefined ? { requestId: input.requestId } : {}), + ...(input.model ? { model: input.model } : {}), + ...(input.thinkingOptionId ? { thinkingOptionId: input.thinkingOptionId } : {}), + title: request.title, + taskDigest: request.taskDigest, + queuedAt: Date.now(), + }; +} + +/** + * Local limit on concurrently running top-level Threads. Only delegated starts + * wait; a start that finds the limit reached, or anything already waiting, is + * queued and started in order once a slot frees. The setting lives in memory + * like idle release; the queue itself is stored and resumes after a restart. + */ +export class DelegationCapacityQueue implements DelegationCapacityGate { + readonly #runtime: DelegationCapacityRuntime; + readonly #store: DelegationQueueStore | undefined; + readonly #diagnose: (error: unknown) => void; + readonly #pollIntervalMs: number; + readonly #settingsGraceMs: number; + readonly #entries: QueueEntry[] = []; + /** Admitted starts whose Thread may not report as running yet. */ + readonly #reserved = new Set(); + readonly #finished = new Map(); + readonly #listeners = new Set<() => void>(); + readonly #ready: Promise; + /** Unknown after a restart until the Renderer delivers it again. */ + #settings: DelegationCapacitySettings | undefined; + #writes: Promise = Promise.resolve(); + #admissions: Promise = Promise.resolve(); + #timer: ReturnType | undefined; + #graceTimer: ReturnType | undefined; + #closed = false; + + constructor( + runtime: DelegationCapacityRuntime, + options: { + store?: DelegationQueueStore; + diagnose?: (error: unknown) => void; + pollIntervalMs?: number; + settingsGraceMs?: number; + } = {}, + ) { + this.#runtime = runtime; + this.#store = options.store; + this.#diagnose = options.diagnose ?? (() => undefined); + this.#pollIntervalMs = options.pollIntervalMs ?? DEFAULT_POLL_INTERVAL_MS; + this.#settingsGraceMs = options.settingsGraceMs ?? DEFAULT_SETTINGS_GRACE_MS; + this.#ready = this.#store ? this.#restore(this.#store) : Promise.resolve(); + } + + configure(value: DelegationCapacitySettings): DelegationCapacitySettings { + const settings = delegationCapacitySettingsSchema.parse(value); + this.#settings = settings; + if (this.#graceTimer) clearTimeout(this.#graceTimer); + this.#graceTimer = undefined; + this.#pump(); + return { ...settings }; + } + + admit(request: DelegationAdmissionRequest): Promise { + // One at a time, so a start whose Thread is still being prepared keeps its place. + const admitted = this.#admissions.then(() => this.#admit(request)); + this.#admissions = admitted.then( + () => undefined, + () => undefined, + ); + return admitted; + } + + async #admit(request: DelegationAdmissionRequest): Promise { + await this.#ready; + if (this.#closed) throw new DelegationControlError("INTERNAL_ERROR", "Host Runtime is closing"); + const existing = this.#duplicate(request); + if (existing) return this.#queued(existing); + if (!this.#mustQueue()) return this.#reserve(request.identity.threadId); + const identity = request.prepare ? await request.prepare() : request.identity; + const entry: QueueEntry = { record: recordFor(request, identity), state: "queued" }; + this.#entries.push(entry); + await this.#persist(); + this.#schedule(); + return this.#queued(entry); + } + + async claim(threadId: string, harnessId: string): Promise { + await this.#ready; + for (;;) { + const entry = this.#entry(threadId); + if (!entry || entry.state !== "queued" || entry.record.harnessId !== harnessId) + return undefined; + if (entry.busy) { + await entry.busy; + continue; + } + if (!entry.uncertain) return this.#hold(entry); + // A Turn that already began is the start itself, and the message joins it as an ordinary one. + if ((await this.#check(entry)) || entry.uncertain) return undefined; + } + } + + async #hold(entry: QueueEntry): Promise { + entry.state = "claiming"; + let release = (): void => undefined; + entry.busy = new Promise((resolve) => (release = resolve)); + this.#notify(); + // Stored as starting first, so a crash during the takeover leaves the Turn to be checked. + await this.#persist(); + let settled = false; + return { + ...queuedStartOf(entry.record), + settle: async (outcome) => { + if (settled) return; + settled = true; + entry.busy = undefined; + if (outcome === "started") { + this.#remove(entry); + } else { + entry.state = "queued"; + if (outcome === "unknown") entry.uncertain = true; + this.#notify(); + } + release(); + await this.#persist(); + this.#pump(); + }, + }; + } + + /** + * Asks the sessions whether the entry's Thread already has a Turn. One that does + * has been adopted as the start and leaves the queue. + */ + #check(entry: QueueEntry): Promise { + if (!this.#runtime.adopt) return Promise.resolve(false); + const check = (async () => { + let begun: boolean | undefined; + try { + begun = await this.#runtime.adopt?.(queuedStartOf(entry.record)); + } catch (error) { + this.#diagnose(error); + } + if (begun === undefined || (!begun && !entry.uncertain)) return false; + if (begun) this.#remove(entry); + else entry.uncertain = false; + await this.#persist(); + return begun; + })(); + entry.busy = check; + return check.finally(() => { + if (entry.busy === check) entry.busy = undefined; + this.#pump(); + }); + } + + /** The entry of a Thread, after checking a start that may have begun; undefined once it left. */ + async #current(threadId: string): Promise { + const entry = this.#entry(threadId); + if (entry?.uncertain && entry.state === "queued") await (entry.busy ?? this.#check(entry)); + return this.#entry(threadId); + } + + /** Re-evaluates the queue, for example after a session registers. */ + refresh(): void { + this.#pump(); + } + + async peek( + threadId: string, + ): Promise<{ status: "queued" | "creating"; position?: number } | undefined> { + await this.#ready; + const entry = await this.#current(threadId); + if (!entry) return undefined; + const position = this.#position(entry); + return { status: this.#status(entry), ...(position ? { position } : {}) }; + } + + /** The snapshot of a Thread that is waiting or being started by the queue. */ + async read(input: ThreadReadInput): Promise { + await this.#ready; + const entry = await this.#current(input.threadId); + if (!entry) return undefined; + const position = this.#position(entry); + return { + ...this.#snapshot(entry.record, input, true), + status: this.#status(entry), + ...(position ? { queue: { position } } : {}), + }; + } + + /** The terminal snapshot of a queued start that never became a Thread. */ + async finished(input: ThreadReadInput): Promise { + await this.#ready; + const finished = this.#finished.get(input.threadId); + if (!finished || this.#entry(input.threadId)) return undefined; + return { + ...this.#snapshot(finished.record, input, false), + status: finished.status, + result: { availability: "unavailable", message: finished.message }, + }; + } + + /** Synthetic Turn for a queued or unstarted native Thread so Desktop does not render an empty page. */ + previewTurn(threadId: string): JsonObject | undefined { + const entry = this.#entry(threadId); + if (entry) { + const position = this.#position(entry); + const message = + entry.state === "starting" + ? "Starting delegated session from the capacity queue…" + : position + ? `Queued waiting for delegation capacity (position ${position}).` + : "Queued waiting for delegation capacity."; + return syntheticQueueTurn(entry.record, "inProgress", message); + } + const finished = this.#finished.get(threadId); + if (!finished) return undefined; + return syntheticQueueTurn(finished.record, finished.status, finished.message); + } + + /** Resolves true once the Thread has left the queue, or false when the timeout passes first. */ + async settled(threadId: string, timeoutMs: number): Promise { + await this.#ready; + await this.#current(threadId); + if (!this.#entry(threadId)) return true; + return new Promise((resolve) => { + const finish = (value: boolean): void => { + clearTimeout(timer); + this.#listeners.delete(check); + resolve(value); + }; + const check = (): void => { + if (!this.#entry(threadId)) finish(true); + }; + const timer = setTimeout(() => finish(false), Math.min(timeoutMs, MAX_TIMER_MS)); + this.#listeners.add(check); + }); + } + + /** + * Removes a waiting start. A start already in progress, or whose Thread turns out + * to have a Turn, is left to the session instead, which owns the Thread from then on. + * A Thread created for the start stays: a user message may have reached it meanwhile. + */ + async cancel(threadId: string): Promise { + await this.#ready; + for (;;) { + const entry = this.#entry(threadId); + if (!entry) return undefined; + if (entry.state === "starting") { + await entry.started; + return undefined; + } + if (entry.busy) { + await entry.busy; + continue; + } + if (await this.#check(entry)) return undefined; + if (this.#entry(threadId) !== entry || entry.state !== "queued" || entry.busy) continue; + this.#remove(entry); + this.#finish(entry.record, "interrupted", CANCELLED_NOTE); + await this.#persist(); + this.#schedule(); + return { threadId, turnId: null, harnessId: entry.record.harnessId, cancelled: true }; + } + } + + /** Waiting and starting Threads for a list by parent, or by workspace when no parent is given. */ + async list(input: { + parentThreadId?: string; + cwd?: string; + }): Promise { + await this.#ready; + return this.#entries + .filter(({ record }) => + input.parentThreadId + ? record.parentThreadId === input.parentThreadId + : input.cwd === undefined || path.resolve(record.cwd) === path.resolve(input.cwd), + ) + .map((entry) => { + const position = this.#position(entry); + const queuedAt = new Date(entry.record.queuedAt).toISOString(); + return { + threadId: entry.record.threadId, + harnessId: entry.record.harnessId, + deepLink: `codex://threads/${entry.record.threadId}`, + status: this.#status(entry), + cwd: entry.record.cwd, + title: entry.record.title, + createdAt: queuedAt, + updatedAt: queuedAt, + ...(position ? { queue: { position } } : {}), + }; + }); + } + + /** + * Stops starting work in this process. The stored queue stays on disk for the next + * Runtime, and starts still settling keep updating it; resolves once pending writes finish. + */ + close(): Promise { + this.#closed = true; + if (this.#timer) clearInterval(this.#timer); + if (this.#graceTimer) clearTimeout(this.#graceTimer); + this.#timer = undefined; + this.#graceTimer = undefined; + return this.#writes; + } + + async #restore(store: DelegationQueueStore): Promise { + let records: PersistedQueuedDelegation[]; + try { + records = await store.adopt(); + } catch (error) { + this.#diagnose(error); + return; + } + if (records.length === 0 || this.#closed) return; + let interrupted = 0; + for (const record of records) { + if (record.starting && record.harnessId === "codex") { + // The native Thread was created when queued; whether its Turn began is checked before use. + const waiting = { ...record }; + delete waiting.starting; + this.#entries.push({ record: waiting, state: "queued", uncertain: true }); + } else if (record.starting) { + // The restart ended the Harness process the start was talking to. + this.#finish(record, "interrupted", RESTART_NOTE); + interrupted += 1; + } else { + this.#entries.push({ record, state: "queued" }); + } + } + if (this.#entries.length > 0) { + this.#diagnose(`restored ${this.#entries.length} queued delegation(s) after a restart`); + } + if (interrupted > 0) { + this.#diagnose(`${interrupted} queued delegation(s) were starting at the restart`); + void this.#persist(); + } + if (this.#settings === undefined && this.#entries.length > 0) { + // The setting arrives from the Renderer; without one, the queue must not wait forever. + this.#graceTimer = setTimeout(() => { + this.#graceTimer = undefined; + if (this.#settings !== undefined) return; + this.#diagnose("no delegation capacity setting arrived; starting the restored queue"); + this.#settings = { ...DEFAULT_DELEGATION_CAPACITY_SETTINGS }; + this.#pump(); + }, this.#settingsGraceMs); + this.#graceTimer.unref?.(); + } + this.#pump(); + } + + #duplicate(request: DelegationAdmissionRequest): QueueEntry | undefined { + const { input, taskDigest } = request; + if (input.requestId) { + const entry = this.#entries.find(({ record }) => record.requestId === input.requestId); + if ( + entry && + (entry.record.harnessId !== input.harnessId || entry.record.taskDigest !== taskDigest) + ) { + throw new DelegationControlError( + "INVALID_ARGUMENT", + "Request ID is already associated with another Delegation configuration", + ); + } + return entry; + } + const since = Date.now() - IMPLICIT_DEDUPLICATION_MS; + return this.#entries.find( + ({ record }) => + record.parentThreadId === input.parentThreadId && + record.harnessId === input.harnessId && + record.taskDigest === taskDigest && + record.queuedAt >= since, + ); + } + + #mustQueue(): boolean { + // Starts never overtake waiting ones, including restored ones waiting for the setting. + if (this.#entries.some((entry) => entry.state === "queued")) return true; + const settings = this.#settings; + return settings?.enabled === true && this.#occupied() >= settings.maxRunning; + } + + #occupied(): number { + const threads = new Set(this.#runtime.runningThreadIds()); + for (const threadId of this.#reserved) threads.add(threadId); + return threads.size; + } + + #reserve(threadId: string): DelegationAdmission { + this.#reserved.add(threadId); + let released = false; + return { + state: "admitted", + release: () => { + if (released) return; + released = true; + this.#reserved.delete(threadId); + this.#pump(); + }, + }; + } + + #queued(entry: QueueEntry): DelegationAdmission { + const position = this.#position(entry); + return { + state: "queued", + identity: identityOf(entry.record), + status: this.#status(entry), + ...(position ? { position } : {}), + parentThreadId: entry.record.parentThreadId, + cwd: entry.record.cwd, + title: entry.record.title, + }; + } + + #pump(): void { + if (this.#closed) return; + try { + for (;;) { + const next = this.#entries.find((entry) => entry.state === "queued"); + const settings = this.#settings; + if (!next || next.busy || settings === undefined || !this.#runtime.canStart()) break; + if (settings.enabled && this.#occupied() >= settings.maxRunning) break; + this.#launch(next); + } + } catch (error) { + this.#diagnose(error); + } + this.#schedule(); + } + + #launch(entry: QueueEntry): void { + const { record } = entry; + entry.state = "starting"; + this.#reserved.add(record.threadId); + void this.#persist(); + this.#notify(); + entry.started = this.#runtime + .start(inputOf(record), identityOf(record)) + .then( + () => this.#remove(entry), + (error: unknown) => { + if (this.#closed) { + // Shutdown can fail a start whose Turn already began; the next Runtime checks it. + entry.state = "queued"; + entry.uncertain = true; + return; + } + this.#remove(entry); + this.#finish(record, "failed", error instanceof Error ? error.message : String(error)); + this.#diagnose(error); + }, + ) + .finally(() => { + this.#reserved.delete(record.threadId); + void this.#persist(); + this.#pump(); + }); + } + + #schedule(): void { + const waiting = !this.#closed && this.#entries.some((entry) => entry.state === "queued"); + if (!waiting) { + if (this.#timer) clearInterval(this.#timer); + this.#timer = undefined; + return; + } + if (this.#timer) return; + // Slots free when Turns end in any session; polling the running set keeps one rule for all. + this.#timer = setInterval(() => this.#pump(), this.#pollIntervalMs); + this.#timer.unref?.(); + } + + #entry(threadId: string): QueueEntry | undefined { + return this.#entries.find((entry) => entry.record.threadId === threadId); + } + + #status(entry: QueueEntry): "queued" | "creating" { + return entry.state === "queued" ? "queued" : "creating"; + } + + #position(entry: QueueEntry): number | undefined { + if (entry.state !== "queued") return undefined; + return this.#entries.filter((candidate) => candidate.state === "queued").indexOf(entry) + 1; + } + + #snapshot( + record: PersistedQueuedDelegation, + input: ThreadReadInput, + running: boolean, + ): DelegationThreadSnapshot { + return projectDelegationThreadSnapshot({ + threadId: record.threadId, + harnessId: record.harnessId, + thread: { name: record.title }, + turns: [], + running, + view: input.view, + ...(input.cursor ? { cursor: input.cursor } : {}), + ...(input.limit !== undefined ? { limit: input.limit } : {}), + }); + } + + #remove(entry: QueueEntry): void { + const index = this.#entries.indexOf(entry); + if (index >= 0) this.#entries.splice(index, 1); + this.#notify(); + } + + #finish(record: PersistedQueuedDelegation, status: FinishedEntry["status"], message: string) { + this.#finished.delete(record.threadId); + this.#finished.set(record.threadId, { record, status, message }); + for (const threadId of [...this.#finished.keys()].slice(0, -MAX_FINISHED_ENTRIES)) { + this.#finished.delete(threadId); + } + } + + #notify(): void { + for (const listener of [...this.#listeners]) listener(); + } + + #persist(): Promise { + const store = this.#store; + if (!store) return Promise.resolve(); + // Each write stores the queue as it is when the write runs, in order. The file + // belongs to this process, so writes after close still reach only the next Runtime. + const next = this.#writes + .then(() => + store.save( + this.#entries.map(({ record, state, uncertain }) => + state !== "queued" || uncertain ? { ...record, starting: true } : record, + ), + ), + ) + .catch((error: unknown) => this.#diagnose(error)); + this.#writes = next; + return next; + } +} diff --git a/packages/host-runtime/src/delegation-capacity-store.ts b/packages/host-runtime/src/delegation-capacity-store.ts new file mode 100644 index 000000000..fb790a280 --- /dev/null +++ b/packages/host-runtime/src/delegation-capacity-store.ts @@ -0,0 +1,242 @@ +import { randomBytes } from "node:crypto"; +import { mkdir, readFile, readdir, rename, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; + +import type { HarnessModelRef, HarnessThinkingOptionId } from "@codexhost/harness-adapter"; +import type { RoutedHarnessId } from "@codexhost/protocol-core"; +import { harnessModelRefSchema, harnessThinkingOptionIdSchema } from "@codexhost/shared-contracts"; + +import { codexhostDataDirectory } from "./external-clients.js"; +import { withRemoteAppServerSocketInitializationLock as withFileMutationLock } from "./remote-socket-lock.js"; + +/** + * A start waiting for a free slot, as kept on disk. Unlike a watch it holds the + * task text, because the Thread does not exist until the start runs. + */ +export interface PersistedQueuedDelegation { + delegationId: string; + threadId: string; + turnId: string; + harnessId: RoutedHarnessId; + task: string; + taskName?: string; + cwd: string; + parentThreadId: string; + requestId?: string; + model?: HarnessModelRef; + thinkingOptionId?: HarnessThinkingOptionId; + title: string; + taskDigest: string; + /** Epoch milliseconds; orders the queue. */ + queuedAt: number; + /** The start had begun when this record was written. */ + starting?: boolean; +} + +export interface DelegationQueueStore { + /** Takes over the queues of Runtimes that have exited and returns their entries. */ + adopt(): Promise; + /** Replaces this Runtime's queue; an empty queue removes its record. */ + save(entries: readonly PersistedQueuedDelegation[]): Promise; +} + +interface Owner { + pid: number; + /** Boot time in seconds, so a pid reused after a reboot is not mistaken for the owner. */ + boot: number; + /** Tells this process from an earlier one that had the same pid. */ + instance: string; +} + +interface QueueFile { + formatVersion: 1; + owner: Owner; + queue: PersistedQueuedDelegation[]; +} + +const RECORD_SUFFIX = ".json"; +const CLAIM_MARKER = ".claim-"; +const BOOT_TOLERANCE_SECONDS = 5; +const REQUIRED_TEXT = [ + "delegationId", + "threadId", + "turnId", + "harnessId", + "task", + "cwd", + "parentThreadId", + "title", + "taskDigest", +] as const; + +export function defaultDelegationQueueDirectory(environment: NodeJS.ProcessEnv): string { + return path.join(codexhostDataDirectory(environment), "delegation-queue"); +} + +function processAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + // EPERM: the process exists but belongs to another user. + return (error as NodeJS.ErrnoException).code === "EPERM"; + } +} + +function ownerAlive(owner: Owner, self: Owner): boolean { + if (owner.instance === self.instance) return true; + if (Math.abs(owner.boot - self.boot) > BOOT_TOLERANCE_SECONDS) return false; + // This process holds the pid now, so the record's writer has exited. + if (owner.pid === self.pid) return false; + return processAlive(owner.pid); +} + +function text(value: unknown): value is string { + return typeof value === "string" && value.length > 0; +} + +function parseEntry(value: unknown): PersistedQueuedDelegation { + const entry = value as Partial> | null; + if ( + typeof entry !== "object" || + entry === null || + REQUIRED_TEXT.some((key) => !text(entry[key])) || + !Number.isFinite(entry.queuedAt) || + (entry.taskName !== undefined && !text(entry.taskName)) || + (entry.requestId !== undefined && !text(entry.requestId)) || + (entry.model !== undefined && !harnessModelRefSchema.safeParse(entry.model).success) || + (entry.thinkingOptionId !== undefined && + !harnessThinkingOptionIdSchema.safeParse(entry.thinkingOptionId).success) || + (entry.starting !== undefined && typeof entry.starting !== "boolean") + ) { + throw new Error("unrecognized queued delegation"); + } + return entry as PersistedQueuedDelegation; +} + +function parseQueueFile(content: string): QueueFile { + const value = JSON.parse(content) as Partial | null; + const owner = value?.owner; + if ( + value?.formatVersion !== 1 || + !owner || + !Number.isSafeInteger(owner.pid) || + !Number.isFinite(owner.boot) || + !text(owner.instance) || + !Array.isArray(value.queue) + ) { + throw new Error("unrecognized delegation queue record"); + } + return { formatVersion: 1, owner, queue: value.queue.map(parseEntry) }; +} + +/** + * One file per Runtime process holding its whole queue. Several Runtimes may + * share the data directory; each writes only its own file, and a starting + * Runtime adopts the files of Runtimes that have exited. Adoption claims a file + * with an atomic rename, so only one Runtime takes it over. + */ +export class FileDelegationQueueStore implements DelegationQueueStore { + readonly #directory: string; + readonly #diagnose: (error: unknown) => void; + readonly #owner: Owner = { + pid: process.pid, + boot: Math.round(Date.now() / 1_000 - os.uptime()), + instance: randomBytes(6).toString("hex"), + }; + readonly #path: string; + + constructor(directory: string, diagnose: (error: unknown) => void = () => undefined) { + this.#directory = directory; + this.#diagnose = diagnose; + this.#path = path.join(directory, `${this.#owner.pid}-${this.#owner.instance}${RECORD_SUFFIX}`); + } + + async adopt(): Promise { + let names: string[]; + try { + names = await readdir(this.#directory); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return []; + throw error; + } + const adopted = new Map(); + const claims: string[] = []; + for (const name of names) { + const claimAt = name.indexOf(CLAIM_MARKER); + // A claim left by an adopter that exited before finishing is adopted again. + const claimer = + claimAt >= 0 ? Number(name.slice(claimAt + CLAIM_MARKER.length).split(".")[0]) : NaN; + if (claimAt < 0 ? !name.endsWith(RECORD_SUFFIX) : !this.#abandonedClaim(claimer)) continue; + const claimed = await withFileMutationLock(`${this.#directory}.adoption`, () => + this.#claim(name, claimAt < 0), + ).catch((error: unknown) => { + this.#diagnose(error); + return undefined; + }); + if (!claimed) continue; + claims.push(claimed.path); + for (const entry of claimed.queue) { + if (!adopted.has(entry.threadId)) adopted.set(entry.threadId, entry); + } + } + const queue = [...adopted.values()].sort((left, right) => left.queuedAt - right.queuedAt); + if (claims.length > 0) { + await this.save(queue); + // Saved under this Runtime first, so a crash here repeats entries instead of losing them. + await Promise.all(claims.map((claim) => rm(claim, { force: true }))); + } + return queue; + } + + async save(entries: readonly PersistedQueuedDelegation[]): Promise { + if (entries.length === 0) { + await rm(this.#path, { force: true }); + return; + } + await mkdir(this.#directory, { recursive: true, mode: 0o700 }); + const file: QueueFile = { formatVersion: 1, owner: this.#owner, queue: [...entries] }; + const temporary = `${this.#path}.${randomBytes(6).toString("hex")}.tmp`; + try { + await writeFile(temporary, `${JSON.stringify(file)}\n`, { mode: 0o600, flag: "wx" }); + await rename(temporary, this.#path); + } finally { + await rm(temporary, { force: true }); + } + } + + #abandonedClaim(claimer: number): boolean { + return Number.isSafeInteger(claimer) && claimer !== process.pid && !processAlive(claimer); + } + + async #claim( + name: string, + record: boolean, + ): Promise<{ path: string; queue: PersistedQueuedDelegation[] } | undefined> { + const source = path.join(this.#directory, name); + let parsed: QueueFile; + try { + parsed = parseQueueFile(await readFile(source, "utf8")); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined; + await rename(source, `${source}.${Date.now()}.invalid`).catch(() => undefined); + throw new Error(`Set aside unreadable delegation queue record ${name}: ${String(error)}`); + } + if (record && ownerAlive(parsed.owner, this.#owner)) return undefined; + // Only the writer changes its file, and it has exited, so the content read above is current. + const base = record ? name : name.slice(0, name.indexOf(CLAIM_MARKER)); + const claim = path.join( + this.#directory, + `${base}${CLAIM_MARKER}${this.#owner.pid}.${randomBytes(4).toString("hex")}`, + ); + try { + await rename(source, claim); + } catch (error) { + // Another Runtime claimed it first. + if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined; + throw error; + } + return { path: claim, queue: parsed.queue }; + } +} diff --git a/packages/host-runtime/src/delegation-cli-help.ts b/packages/host-runtime/src/delegation-cli-help.ts index 1c4d0f59c..02be6eb0a 100644 --- a/packages/host-runtime/src/delegation-cli-help.ts +++ b/packages/host-runtime/src/delegation-cli-help.ts @@ -6,7 +6,7 @@ Register a named external client for HTTP/MCP or a separate CLI identity. Normal The client sets CODEXHOST_CLIENT_TOKEN (or CODEXHOST_CLIENT_TOKEN_FILE) and then uses the same delegate, thread and harness commands, or the MCP endpoint, against the external client listener (127.0.0.1:4398 unless CODEXHOST_EXTERNAL_CLIENT_PORT says otherwise). Names use lowercase letters, digits and hyphens; local-cli is reserved.`, events: `codexhost events [--after ] [--limit ] [--follow true|false] -For external clients: read the client's event log. Without --after, every retained event is returned, oldest first; keep the cursor to read only new ones. An event reports that a Thread the client watches stopped (outcome, Thread, Turn, Harness, deep link, references); it never carries Thread content, so read the Thread for results. +For external clients: read the client's event log. Without --after, every retained event is returned, oldest first; keep the cursor to read only new ones. An event reports that a Thread the client watches stopped (outcome, Thread, Turn, Harness, deep link, references), or, for a watch registered with --stall-after-ms, that the Thread is still running with no Harness event observed for at least that long (outcome stalled, explained in note; the watch continues). Events never carry Thread content, so read the Thread for results and progress. Without --follow, prints {events, cursor} once; pass cursor back as --after. With --follow true, prints one JSON line per event until interrupted. Events are kept for 7 days (CODEXHOST_CLIENT_EVENT_RETENTION_DAYS). CURSOR_EXPIRED means older events were removed: reconcile with thread list --client . The listener also serves GET /v1/events as server-sent events (resume with Last-Event-ID) and POST /v1/events/poll for long-polling.`, @@ -19,7 +19,7 @@ List Harnesses available in the active Runtime.`, "harness inspect": `codexhost harness inspect [--cwd ] [--refresh true|false] [--format json|compact] Read available Models, native defaults, Thinking options, and configuration capabilities. Use the returned Model and Thinking IDs for explicit selections.`, - "delegate start": `codexhost delegate start --harness --task [--task-name ] [--cwd ] [--model ] [--thinking ] [--parent-thread ] [--request-id ] [--ref ]... [--watch true|false] [--watch-timeout-ms ] [--format json|compact] + "delegate start": `codexhost delegate start --harness --task [--task-name ] [--cwd ] [--model ] [--thinking ] [--parent-thread ] [--request-id ] [--ref ]... [--watch true|false] [--watch-timeout-ms ] [--stall-after-ms ] [--format json|compact] Create an independent child Thread, submit the task, and return immediately. --task-name supplies a short subtask name (1-120 characters); otherwise the first task line is used. The child title is "subtask name · parent title", fixed at creation; renaming the parent does not rename existing children. @@ -27,43 +27,48 @@ Omit --model and --thinking to use the Harness native defaults. --cwd overrides the child workspace; a relative path resolves against the caller process cwd. Otherwise use the resolved parent Thread workspace, then the Host Runtime process cwd. --parent-thread overrides caller inference. PARENT_THREAD_AMBIGUOUS requires an explicit parent. Reuse --request-id for an idempotent retry. Identical recent parent/target/task/configuration requests are also deduplicated briefly. ---watch true also registers thread watch on the child with the resolved parent as the notified Thread; the response's watch field reports whether it was registered. +--watch true also registers thread watch on the child with the resolved parent as the notified Thread; the response's watch field reports whether it was registered. --stall-after-ms requires --watch true and enables the optional silence notification described by thread watch. --ref records where the work came from, such as an issue URL; repeat it for several. Absolute URIs only, at most 16 per Thread; credentials in a URI are removed. A child inherits its parent Thread's references. An external client that omits --parent-thread starts the work as itself (parent client:) and must pass --cwd. +When the local limit on running delegations is on (Codex Desktop settings; off by default) and reached, or other starts are already waiting, the start is queued: the response still returns at once with the Thread reference, status queued and queue.position (compact: queuePosition). Queued starts begin in order as running top-level Threads finish, including manually opened ones; the queue survives a Host Runtime restart. This applies to every Harness, --harness codex included. Wait and watch timeouts include time spent queued. The response confirms cwd and parent. Use its Thread reference with thread read, wait, send, cancel, or watch.`, "thread send": `codexhost thread send --message [--format json|compact] Start a new Turn in an idle writable Thread and return immediately. -THREAD_BUSY means the current Turn is still active: wait or cancel before sending. Messages are not queued. +THREAD_BUSY means the current Turn is still active, or the Thread is still queued: wait or cancel before sending. Messages are not queued. To be notified when that Turn ends, run thread watch after sending.`, "thread cancel": `codexhost thread cancel [--format json|compact] Request cancellation of the active Turn while preserving the Thread and history. -cancelled=true (compact: cancelRequested=true) means the cancellation request was accepted. Read or wait to confirm the terminal state. An idle Thread returns false.`, +cancelled=true (compact: cancelRequested=true) means the cancellation request was accepted. Read or wait to confirm the terminal state. An idle Thread returns false. +A queued Thread is removed from the queue instead and never starts; it then reads as interrupted.`, "thread read": `codexhost thread read [--host ] [--view result|messages] [--cursor ] [--limit ] [--format json|compact] Read immediately without starting a Turn. Accepts thread://?hostId= or --host with a bare ID. Explicit remote Hosts use the existing Desktop connection, never local fallback. Requires updated local and remote codexhost; remote send/wait/watch are not supported. -The default result view reports the latest Turn's status and result. +The default result view reports the latest Turn's status and result. A queued Thread reports status queued with queue.position (compact: queuePosition) and no Turn, then creating while its start runs. The messages view pages visible user/Agent messages, oldest first. Default limit 25, maximum 100; --cursor and --limit require --view messages. hasMore describes remaining messages now. Save nextCursor for later incremental reads even when hasMore=false. Compact messages output contains only the message page and status; compact result output includes the latest nonempty progress while running. Full JSON retains the complete snapshot. Tool calls/output, file activity, and reasoning are not included in either format.`, "thread wait": `codexhost thread wait [--timeout-ms ] [--view result|messages] [--cursor ] [--limit ] [--format json|compact] Wait until the Thread is terminal or the timeout expires (default 30000 ms), then return the same snapshot as thread read plus timedOut. -timedOut=true is a running checkpoint: the child keeps running. The response already includes the result when available; another read is unnecessary unless more information is needed. +timedOut=true is a running checkpoint: the child keeps running. The response already includes the result when available; another read is unnecessary unless more information is needed. A queued Thread is waited for through the queue; queued time counts toward the timeout. Message pagination uses --view messages, default limit 25, maximum 100. hasMore is for current pages; nextCursor also supports future incremental reads.`, - "thread watch": `codexhost thread watch [--notify ] [--timeout-ms ] [--format json|compact] + "thread watch": `codexhost thread watch [--notify ] [--timeout-ms ] [--stall-after-ms ] [--format json|compact] Ask the Host to notify one Thread, once, when the watched Thread stops. Returns immediately; no waiting or polling by the caller is needed, and the caller may end its Turn. --notify defaults to the calling Thread when it is identified (CODEXHOST_THREAD_ID, or CODEX_THREAD_ID in native Codex); for an external client it defaults to the client, and the notification becomes an event in its log (see codexhost events). Otherwise it is required. delegate start reports the caller as its parent. -The notification starts a new Turn in the notified Thread with the watched Thread's link and outcome: completed, failed, interrupted, timedOut, unreadable, or notFound; a terminal outcome names the Turn it came from. It reports execution state only; read the Thread to judge the work. +The notification starts a new Turn in the notified Thread with the watched Thread's link and outcome: completed, failed, interrupted, timedOut, unreadable, notFound, or stalled (only with --stall-after-ms; the watch continues); a terminal outcome names the Turn it came from. It reports execution state only; read the Thread to judge the work. --timeout-ms defaults to 1740000 (29 min). timedOut means the Thread had not reached a terminal state, which also covers a Harness that stopped without reporting it; watch again to keep waiting. unreadable means reads failed for 60 s, so the state is unknown. +--stall-after-ms is disabled by default. A positive integer enables stalled notifications when a running Thread has no observed Harness event for that many milliseconds. Tool output, reasoning, file changes, interactions, and Harness subagent events all count as activity; polling reads do not. Each silence episode notifies once; new activity rearms it. A stalled alert does not stop the Thread or end the watch, and does not prove a failure. The silence clock starts at registration and restarts after a Host Runtime restart, which drops stalled alerts not yet delivered; only live events already received by this Host are observable. state=watching means registered. state=alreadyTerminal means the Thread was not running: nothing was registered and nothing will be sent. +A queued Thread is watched through the queue until it stops after starting; queued time counts toward --timeout-ms but is not silence for --stall-after-ms. Removing it from the queue reports interrupted. A busy notified Thread is notified after its Turn ends (retried for up to 6 hours); notifications due together arrive as one message. THREAD_BUSY is never treated as delivered. -A watch is one-shot and cannot be cancelled. Watching the same pair again while it is still watched returns that watch; once it has stopped, a new watch covers the next stop and any pending notification is still delivered. Watches are stored and resume after a Host Runtime restart; a notification whose delivery was in progress when the Runtime stopped is sent again, so a restart may repeat one. A Turn that was running when the Runtime restarted is reported as interrupted, because the restart ended its Harness process.`, +The final stop/expiry notification is one-shot; optional stalled alerts keep the watch active. Watches cannot be cancelled. Watching the same pair again while it is still watched returns that watch with its original timeout and silence threshold; once it has stopped, a new watch covers the next stop and any pending notification is still delivered. Watches are stored and resume after a Host Runtime restart; a notification whose delivery was in progress when the Runtime stopped is sent again, so a restart may repeat one. A Turn that was running when the Runtime restarted is reported as interrupted, because the restart ended its Harness process.`, "thread watches": `codexhost thread watches [--format json|compact] List watches that have not been delivered: watching, pendingDelivery (waiting for the notified Thread), or undeliverable (with the reason). Delivered watches are removed.`, "thread list": `codexhost thread list [--cwd ] [--parent | --client ] [--ref ] [--limit ] [--cursor ] [--sort created-asc|created-desc|updated-asc|updated-desc|recency-asc|recency-desc] [--format json|compact] Find existing Threads by workspace, or use --parent to list a Thread's delegated children. Workspace listing defaults to the caller process cwd. Default limit 25 (maximum 100), sorted created-desc. --parent uses Delegation relationships. --client lists what an external client started without a parent Thread. --ref lists Threads carrying that reference, newest first, across workspaces. An external client lists all workspaces unless --cwd is given. A null nextCursor ends the list. -Compact output keeps task links, Harness, status, title, and workspace.`, +Queued Threads lead the first page in queue order, beyond --limit, with status queued and queue.position. +Compact output keeps task links, Harness, status, queue position, title, and workspace.`, } as const; export type DelegationCliCommand = keyof typeof COMMAND_HELP; diff --git a/packages/host-runtime/src/delegation-cli-output.ts b/packages/host-runtime/src/delegation-cli-output.ts index 5e3a41a51..dbd839a12 100644 --- a/packages/host-runtime/src/delegation-cli-output.ts +++ b/packages/host-runtime/src/delegation-cli-output.ts @@ -33,6 +33,7 @@ function watchOutput(watch: DelegationStartResult["watch"]): { watch?: unknown } state: watch.state, notify: targetLink(watch.notifyThreadId), timeoutMs: watch.timeoutMs, + ...(watch.stallAfterMs !== undefined ? { stallAfterMs: watch.stallAfterMs } : {}), }, }; } @@ -67,6 +68,7 @@ function snapshotOutput( : threadLink(snapshot.threadId), harnessId: snapshot.harnessId, status: snapshot.status, + ...(snapshot.queue ? { queuePosition: snapshot.queue.position } : {}), ...(snapshot.timedOut !== undefined ? { timedOut: snapshot.timedOut } : {}), }; if (view === "messages") { @@ -115,6 +117,7 @@ export function compactDelegationOutput( thread: result.deepLink, harnessId: result.harnessId, status: result.status, + ...(result.queue ? { queuePosition: result.queue.position } : {}), ...(result.cwd ? { cwd: result.cwd } : {}), ...(result.parentThreadId ? { @@ -155,27 +158,32 @@ export function compactDelegationOutput( state: result.state, status: result.status, timeoutMs: result.timeoutMs, + ...(result.stallAfterMs !== undefined ? { stallAfterMs: result.stallAfterMs } : {}), }; } case "thread watches": { const result = body as ThreadWatchListResult; return { - watches: result.watches.map(({ threadId, notifyThreadId, state, outcome, reason }) => ({ - thread: threadLink(threadId), - notify: targetLink(notifyThreadId), - state, - ...(outcome ? { outcome } : {}), - ...(reason ? { reason } : {}), - })), + watches: result.watches.map( + ({ threadId, notifyThreadId, state, outcome, reason, stallAfterMs }) => ({ + thread: threadLink(threadId), + notify: targetLink(notifyThreadId), + state, + ...(outcome ? { outcome } : {}), + ...(reason ? { reason } : {}), + ...(stallAfterMs !== undefined ? { stallAfterMs } : {}), + }), + ), }; } case "thread list": { const result = body as DelegationThreadListResult; return { - threads: result.threads.map(({ deepLink, harnessId, status, title, cwd }) => ({ + threads: result.threads.map(({ deepLink, harnessId, status, queue, title, cwd }) => ({ thread: deepLink, harnessId, status, + ...(queue ? { queuePosition: queue.position } : {}), ...(title ? { title } : {}), ...(cwd ? { cwd } : {}), })), diff --git a/packages/host-runtime/src/delegation-cli.ts b/packages/host-runtime/src/delegation-cli.ts index a9f2dee34..b8f0f91d3 100644 --- a/packages/host-runtime/src/delegation-cli.ts +++ b/packages/host-runtime/src/delegation-cli.ts @@ -310,6 +310,7 @@ export async function runDelegationCli(input: { "--watch", "--watch-timeout-ms", "--ref", + "--stall-after-ms", ]); if (parsed.positionals.length > 0) throw new DelegationControlError( @@ -329,6 +330,11 @@ export async function runDelegationCli(input: { "INVALID_ARGUMENT", "--watch-timeout-ms requires --watch true", ); + if (watch !== "true" && value(parsed, "--stall-after-ms") !== undefined) + throw new DelegationControlError( + "INVALID_ARGUMENT", + "--stall-after-ms requires --watch true", + ); writeResult( "delegate start", await requestRuntime({ @@ -353,6 +359,14 @@ export async function runDelegationCli(input: { watchTimeoutMs: value(parsed, "--watch-timeout-ms") ? positiveInteger(value(parsed, "--watch-timeout-ms"), "--watch-timeout-ms") : DEFAULT_WATCH_TIMEOUT_MS, + ...(value(parsed, "--stall-after-ms") !== undefined + ? { + stallAfterMs: positiveInteger( + value(parsed, "--stall-after-ms"), + "--stall-after-ms", + ), + } + : {}), } : {}), }, @@ -483,7 +497,7 @@ export async function runDelegationCli(input: { return 0; } if (group === "thread" && command === "watch") { - rejectUnknown(parsed, ["--notify", "--timeout-ms"]); + rejectUnknown(parsed, ["--notify", "--timeout-ms", "--stall-after-ms"]); if (parsed.positionals.length !== 1) throw new DelegationControlError( "INVALID_ARGUMENT", @@ -511,6 +525,14 @@ export async function runDelegationCli(input: { timeoutMs: value(parsed, "--timeout-ms") ? positiveInteger(value(parsed, "--timeout-ms"), "--timeout-ms") : DEFAULT_WATCH_TIMEOUT_MS, + ...(value(parsed, "--stall-after-ms") !== undefined + ? { + stallAfterMs: positiveInteger( + value(parsed, "--stall-after-ms"), + "--stall-after-ms", + ), + } + : {}), }, ...(input.fetchImpl ? { fetchImpl: input.fetchImpl } : {}), }), diff --git a/packages/host-runtime/src/delegation-control-registry.ts b/packages/host-runtime/src/delegation-control-registry.ts index 103c25c7b..5f3976f1b 100644 --- a/packages/host-runtime/src/delegation-control-registry.ts +++ b/packages/host-runtime/src/delegation-control-registry.ts @@ -1,9 +1,13 @@ import { randomUUID } from "node:crypto"; +import { DelegationCapacityQueue } from "./delegation-capacity-queue.js"; +import type { DelegationQueueStore } from "./delegation-capacity-store.js"; import { DelegationControlError, + type DelegationCapacityGate, type DelegationControlApi, type DelegationControlRegistration, + type DelegationStartIdentity, type DelegationStartInput, type DelegationWatchApi, type HarnessInspectInput, @@ -49,6 +53,8 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati #harnessCatalog: DelegationControlRegistration | undefined; // Watches sit above the sessions so either end may belong to any registered session. readonly #watchService: DelegationWatchService; + // So does the capacity limit: it counts the running Threads of every session. + readonly #capacity: DelegationCapacityQueue; readonly #remoteRead: ((input: ThreadReadInput) => ReturnType) | undefined; @@ -61,6 +67,12 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati watchStore?: DelegationWatchStore; refs?: ThreadRefsStore; clientEvents?: ClientEventSink; + capacity?: { + store?: DelegationQueueStore; + diagnose?: (error: unknown) => void; + pollIntervalMs?: number; + settingsGraceMs?: number; + }; } = {}, ) { this.#remoteRead = options.remoteRead; @@ -70,20 +82,50 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati ...(options.watchStore ? { store: options.watchStore } : {}), ...(options.clientEvents ? { clientEvents: options.clientEvents } : {}), }); + this.#capacity = new DelegationCapacityQueue( + { + runningThreadIds: () => + [...this.#registrations].flatMap( + (registration) => registration.runningThreadIds?.() ?? [], + ), + canStart: () => this.#registrations.size > 0, + start: (input, identity) => this.#startQueued(input, identity), + // A waiting Thread has no Turn or Delegation yet, so no session claims to own it. + adopt: async (queued) => { + for (const registration of [...this.#registrations]) { + const begun = await registration.adoptQueued?.(queued); + if (begun !== undefined) return begun; + } + return undefined; + }, + }, + options.capacity, + ); } get size(): number { return this.#registrations.size; } + /** The capacity limit sessions apply to delegated starts and the settings request configures. */ + get capacity(): DelegationCapacityGate { + return this.#capacity; + } + register( registration: DelegationControlRegistration, options: { harnessCatalog?: boolean } = {}, ): () => void { this.#registrations.add(registration); + const unsubscribeActivity = registration.subscribeActivity?.((threadIds) => + this.#watchService.noteActivity(threadIds), + ); + registration.useCapacity?.(this.#capacity); this.#registrationGeneration += 1; if (options.harnessCatalog) this.#harnessCatalog = registration; + this.#capacity.refresh(); return () => { + unsubscribeActivity?.(); if (this.#registrations.delete(registration)) this.#registrationGeneration += 1; if (this.#harnessCatalog === registration) this.#harnessCatalog = undefined; }; @@ -120,10 +162,30 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati } async send(input: Parameters[0]) { + const queued = await this.#capacity.peek(input.threadId); + if (queued) { + throw new DelegationControlError( + "THREAD_BUSY", + queued.status === "queued" ? "Thread is queued and has not started" : "Thread is starting", + { status: queued.status, ...(queued.position ? { queuePosition: queued.position } : {}) }, + ); + } + if (await this.#neverStarted({ threadId: input.threadId, view: "result" })) + throw new DelegationControlError("THREAD_NOT_FOUND", "Thread was never started"); return (await this.#registrationForThread(input.threadId)).send(input); } async cancel(input: Parameters[0]) { + const removed = await this.#capacity.cancel(input.threadId); + if (removed) return removed; + const finished = await this.#neverStarted({ threadId: input.threadId, view: "result" }); + if (finished) + return { + threadId: input.threadId, + turnId: null, + harnessId: finished.harnessId, + cancelled: false, + }; return (await this.#registrationForThread(input.threadId)).cancel(input); } @@ -156,16 +218,39 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati ); return this.#remoteRead(read); } + const queued = (await this.#capacity.read(read)) ?? (await this.#neverStarted(read)); + if (queued) return queued; return (await this.#registrationForThread(read.threadId)).read(read); } async wait(input: ThreadWaitInput) { if (input.hostId !== undefined && input.hostId !== "local") throw new DelegationControlError("INVALID_ARGUMENT", "Remote Hosts support thread read only"); - const snapshot = await (await this.#registrationForThread(input.threadId)).wait(input); + const snapshot = await this.#wait(input); return withRefs(snapshot, (await this.#refs?.get(snapshot.threadId)) ?? []); } + /** A queued Thread is waited for through the queue first, then through its session. */ + async #wait(input: ThreadWaitInput) { + let timeoutMs = input.timeoutMs; + if (await this.#capacity.peek(input.threadId)) { + if (!Number.isSafeInteger(input.timeoutMs) || input.timeoutMs <= 0) + throw new DelegationControlError( + "INVALID_ARGUMENT", + "timeoutMs must be a positive integer", + ); + const deadline = Date.now() + input.timeoutMs; + if (!(await this.#capacity.settled(input.threadId, input.timeoutMs))) { + const queued = await this.#capacity.read(input); + if (queued) return { ...queued, timedOut: true }; + } + timeoutMs = Math.max(1, deadline - Date.now()); + } + const finished = await this.#neverStarted(input); + if (finished) return { ...finished, timedOut: false }; + return (await this.#registrationForThread(input.threadId)).wait({ ...input, timeoutMs }); + } + async watch(input: ThreadWatchInput) { return this.#watchService.watch(input); } @@ -174,9 +259,10 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati return this.#watchService.watches(); } - /** Stops watching in this process; stored watches resume in the next Runtime. */ - close(): void { + /** Stops watching and starting queued work in this process; both resume in the next Runtime. */ + close(): Promise { this.#watchService.close(); + return this.#capacity.close(); } async list(input: ThreadListInput) { @@ -187,9 +273,22 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati "--client and --parent cannot be combined", ); if (ref !== undefined) return this.#listByRef(ref, rest); - const page = await this.#list( - client !== undefined ? { ...rest, parentThreadId: clientOrigin(client) } : rest, - ); + const scoped = client !== undefined ? { ...rest, parentThreadId: clientOrigin(client) } : rest; + const listed = await this.#list(scoped); + // Queued Threads exist only in the queue; they lead the first page, beyond its limit. + const queued = scoped.cursor ? [] : await this.#capacity.list(scoped); + const page = + queued.length > 0 + ? { + ...listed, + threads: [ + ...queued, + ...listed.threads.filter( + (thread) => !queued.some((entry) => entry.threadId === thread.threadId), + ), + ], + } + : listed; if (!this.#refs) return page; const refs = this.#refs; return { @@ -304,6 +403,34 @@ export class DelegationControlRegistry implements DelegationControlApi, Delegati ); } + /** + * A queued start may run after its parent's Turn ended, so a session that no + * longer recognizes the parent still takes it as the catalog or the only one. + */ + async #startQueued(input: DelegationStartInput, identity: DelegationStartIdentity) { + const matches = originClientName(input.parentThreadId) + ? [] + : await this.#matching((registration) => registration.canHandleStart(input)); + const registration = + matches.length === 1 + ? (matches[0] as DelegationControlRegistration) + : (this.#harnessCatalog ?? + only( + [...this.#registrations], + "A queued delegation needs exactly one active Host Runtime session", + )); + return registration.start(input, identity); + } + + /** A queued start that never became a Thread, unless a session holds the Thread after all. */ + async #neverStarted(input: ThreadReadInput) { + const finished = await this.#capacity.finished(input); + if (!finished) return undefined; + // A start cut off by a restart may have created its Thread before the Runtime stopped. + const owners = await this.#matching((registration) => registration.ownsThread(input.threadId)); + return owners.length === 0 ? finished : undefined; + } + async #registrationForThread(threadId: string): Promise { const registrations = [...this.#registrations]; const matches = await this.#matching((registration) => registration.ownsThread(threadId)); diff --git a/packages/host-runtime/src/delegation-control-server.ts b/packages/host-runtime/src/delegation-control-server.ts index 2eba3bbf7..adb90d865 100644 --- a/packages/host-runtime/src/delegation-control-server.ts +++ b/packages/host-runtime/src/delegation-control-server.ts @@ -95,12 +95,13 @@ async function startWithWatch( watches: DelegationWatchApi, body: Record, ): Promise { - const { watchTimeoutMs, ...start } = body; + const { watchTimeoutMs, stallAfterMs, ...start } = body; const result = await api.start(start as unknown as DelegationStartInput); const watch = await watchStartedTurn(watches, { threadId: result.threadId, notifyThreadId: result.parentThreadId ?? "", timeoutMs: watchTimeoutMs as number, + ...(stallAfterMs !== undefined ? { stallAfterMs: stallAfterMs as number } : {}), }); return { ...result, watch }; } @@ -117,6 +118,8 @@ export async function routeDelegationRequest( case "/v1/harness/inspect": return handlers.api.inspect(body as unknown as HarnessInspectInput); case "/v1/delegate/start": + if (body.stallAfterMs !== undefined && body.watchTimeoutMs === undefined) + throw new DelegationControlError("INVALID_ARGUMENT", "stallAfterMs requires a watch"); return body.watchTimeoutMs === undefined ? handlers.api.start(body as unknown as DelegationStartInput) : startWithWatch(handlers.api, watchApi(handlers.watchApi), body); diff --git a/packages/host-runtime/src/delegation-types.ts b/packages/host-runtime/src/delegation-types.ts index 7da49e677..008ffc9f1 100644 --- a/packages/host-runtime/src/delegation-types.ts +++ b/packages/host-runtime/src/delegation-types.ts @@ -1,10 +1,11 @@ -import type { RoutedHarnessId } from "@codexhost/protocol-core"; +import type { JsonObject, RoutedHarnessId } from "@codexhost/protocol-core"; import type { HarnessInspection, HarnessModelRef, HarnessSessionState, HarnessThinkingOptionId, } from "@codexhost/harness-adapter"; +import type { DelegationCapacitySettings } from "@codexhost/shared-contracts"; export const DELEGATION_RUNTIME_ENDPOINT_ENV = "CODEXHOST_RUNTIME_ENDPOINT"; export const DELEGATION_RUNTIME_TOKEN_ENV = "CODEXHOST_RUNTIME_TOKEN"; @@ -25,8 +26,14 @@ export const NATIVE_CODEX_THREAD_ID_ENV = "CODEX_THREAD_ID"; /** Default watch expiry; callers adjust it with --timeout-ms. */ export const DEFAULT_WATCH_TIMEOUT_MS = 29 * 60_000; +/** `queued`: waiting for the local capacity limit; the Thread does not exist yet. */ export type DelegationThreadStatus = - "creating" | "running" | "completed" | "failed" | "interrupted"; + "queued" | "creating" | "running" | "completed" | "failed" | "interrupted"; + +/** 1-based place among the queued starts; present only while queued. */ +export interface DelegationQueuePlace { + position: number; +} export type DelegationResultAvailability = "pending" | "available" | "unavailable"; @@ -62,6 +69,7 @@ export interface DelegationThreadSnapshot { nextCursor: string | null; /** Omitted when the Thread carries no references. */ refs?: string[]; + queue?: DelegationQueuePlace; } export interface DelegationStartInput { @@ -123,9 +131,75 @@ export interface DelegationStartResult { refs?: string[]; /** Present only when the caller asked `delegate start` to also watch the child. */ watch?: ThreadWatchResult | { state: "notRegistered"; reason: string }; + queue?: DelegationQueuePlace; next: { read: string; wait: string }; } +/** Identifiers a queued start keeps until its Thread is created. */ +export interface DelegationStartIdentity { + delegationId: string; + threadId: string; + turnId: string; +} + +/** A validated start with its parent and workspace resolved, offered to the capacity limit. */ +export interface DelegationAdmissionRequest { + input: DelegationStartInput & { parentThreadId: string; cwd: string }; + title: string; + taskDigest: string; + identity: DelegationStartIdentity; + /** + * Creates the Thread a queued start waits in, when its identity comes from the + * Harness; called only once the start must queue, and its identity replaces `identity`. + */ + prepare?(): Promise; +} + +/** A start waiting in the queue, with what its session needs to start or adopt it. */ +export interface DelegationQueuedStart { + identity: DelegationStartIdentity; + input: DelegationStartInput & { parentThreadId: string; cwd: string }; + taskDigest: string; +} + +/** + * A queued start a user message is taking over. It stays in its place, and is + * neither started nor cancelled, until the message's Turn start settles. + */ +export interface DelegationQueueClaim extends DelegationQueuedStart { + /** + * `started` removes it from the queue; `rejected` puts it back as it was; + * `unknown` puts it back to be checked for a Turn before anything else happens to it. + */ + settle(outcome: "started" | "rejected" | "unknown"): Promise; +} + +export type DelegationAdmission = + /** Start now; release once the start has settled. */ + | { state: "admitted"; release(): void } + /** Waiting or starting under this identity, possibly queued by an earlier identical request. */ + | { + state: "queued"; + identity: DelegationStartIdentity; + status: "queued" | "creating"; + position?: number; + parentThreadId: string; + cwd: string; + title: string; + }; + +export interface DelegationCapacityGate { + admit(request: DelegationAdmissionRequest): Promise; + configure(settings: DelegationCapacitySettings): DelegationCapacitySettings; + /** Holds a waiting start of this Harness for its session to start now; undefined when none waits. */ + claim(threadId: string, harnessId: string): Promise; + peek?( + threadId: string, + ): Promise<{ status: "queued" | "creating"; position?: number } | undefined>; + finished?(input: ThreadReadInput): Promise; + previewTurn?(threadId: string): JsonObject | undefined; +} + export interface ThreadSendInput { threadId: string; message: string; @@ -195,6 +269,7 @@ export interface DelegationThreadListItem { /** Set by native rows; recency-* sorts order by it, falling back to updatedAt. */ recencyAt?: string; refs?: string[]; + queue?: DelegationQueuePlace; } export interface DelegationThreadListResult { @@ -205,9 +280,11 @@ export interface DelegationThreadListResult { export interface ThreadWatchInput { /** Thread observed until it stops. */ threadId: string; - /** Thread that receives the single notification. */ + /** Thread that receives stop/expiry and any enabled silence notifications. */ notifyThreadId: string; timeoutMs: number; + /** Optional silence threshold; omitted disables stalled notifications. */ + stallAfterMs?: number; } export type ThreadWatchOutcome = @@ -216,6 +293,8 @@ export type ThreadWatchOutcome = | "interrupted" /** The Thread was still running when the watch expired. */ | "timedOut" + /** Still running, with no observed Harness event for the configured threshold. */ + | "stalled" /** Reads kept failing, so the state of the Thread is unknown. */ | "unreadable" | "notFound"; @@ -227,6 +306,7 @@ export interface ThreadWatchResult { state: "watching" | "alreadyTerminal"; status: DelegationThreadStatus; timeoutMs: number; + stallAfterMs?: number; } export interface ThreadWatchEntry { @@ -234,11 +314,12 @@ export interface ThreadWatchEntry { notifyThreadId: string; state: "watching" | "pendingDelivery" | "undeliverable"; outcome?: ThreadWatchOutcome; - /** Turn that reached the terminal outcome, when the Thread reported one. */ + /** Turn associated with this outcome, when the Thread reported one. */ turnId?: string; /** Present for `undeliverable`. */ reason?: string; registeredAt: string; + stallAfterMs?: number; } export interface ThreadWatchListResult { @@ -267,6 +348,22 @@ export interface DelegationControlApi { } export interface DelegationControlRegistration extends DelegationControlApi { + /** `admitted` starts a queued delegation under its identity, bypassing the limit and deduplication. */ + start( + input: DelegationStartInput, + admitted?: DelegationStartIdentity, + ): Promise; + /** Thread and Harness subagent ancestors active in an incoming Harness event. */ + subscribeActivity?(listener: (threadIds: readonly string[]) => void): () => void; + /** Top-level Threads with an active Turn, counted against the capacity limit. */ + runningThreadIds?(): readonly string[]; + /** Receives the capacity limit of the registry this session joined. */ + useCapacity?(capacity: DelegationCapacityGate): void; + /** + * Whether the Thread a queued start waits in already has a Turn; one that does is + * adopted as this start, without sending the task. Undefined when this session cannot tell. + */ + adoptQueued?(queued: DelegationQueuedStart): Promise; executionHostId?(threadId: string): string | undefined; canHandleStart(input: DelegationStartInput): boolean | Promise; ownsThread(threadId: string): boolean | Promise; diff --git a/packages/host-runtime/src/delegation-watch-store.ts b/packages/host-runtime/src/delegation-watch-store.ts index ec86d8cd2..cb9705381 100644 --- a/packages/host-runtime/src/delegation-watch-store.ts +++ b/packages/host-runtime/src/delegation-watch-store.ts @@ -4,6 +4,7 @@ import os from "node:os"; import path from "node:path"; import type { ThreadWatchEntry, ThreadWatchOutcome } from "./delegation-types.js"; +import { withRemoteAppServerSocketInitializationLock as withFileMutationLock } from "./remote-socket-lock.js"; /** A watch as kept on disk. Holds identifiers, deadlines and reasons only, never Thread content. */ export interface PersistedWatch extends ThreadWatchEntry { @@ -16,7 +17,7 @@ export interface PersistedWatch extends ThreadWatchEntry { delivering?: boolean; /** Why an `interrupted` outcome was decided after a restart, for the notification. */ restartNote?: string; - /** Recorded when the watched Thread stops, for client events. */ + /** Recorded when an outcome is observed, for client events. */ harnessId?: string; refs?: string[]; } @@ -109,6 +110,8 @@ function parseWatchFile(text: string): WatchFile { !optionalString(watch.reason) || !Number.isSafeInteger(watch.timeoutMs) || !Number.isFinite(watch.deadline) || + (watch.stallAfterMs !== undefined && + (!Number.isSafeInteger(watch.stallAfterMs) || watch.stallAfterMs <= 0)) || !optionalNumber(watch.deliveryDeadline) || (watch.delivering !== undefined && typeof watch.delivering !== "boolean") || !optionalString(watch.restartNote) || @@ -153,7 +156,9 @@ export class FileDelegationWatchStore implements DelegationWatchStore { const claimer = claimAt >= 0 ? Number(name.slice(claimAt + CLAIM_MARKER.length).split(".")[0]) : NaN; if (!isRecord && !(Number.isSafeInteger(claimer) && !processAlive(claimer))) continue; - const watch = await this.#adoptFile(name).catch((error: unknown) => { + const watch = await withFileMutationLock(`${this.#directory}.adoption`, () => + this.#adoptFile(name), + ).catch((error: unknown) => { this.#diagnose(error); return undefined; }); diff --git a/packages/host-runtime/src/delegation-watch.ts b/packages/host-runtime/src/delegation-watch.ts index b34470abd..2c2ac820c 100644 --- a/packages/host-runtime/src/delegation-watch.ts +++ b/packages/host-runtime/src/delegation-watch.ts @@ -31,6 +31,10 @@ const UNREADABLE_GRACE_MS = 60_000; const DEFAULT_RESTORE_GRACE_MS = 5 * 60_000; interface Watch extends PersistedWatch { + /** Live event clock. Never restored from disk: downtime is not Harness silence. */ + lastActivityAt?: number; + /** This silence episode already alerted. Not stored either: a restart starts a new episode. */ + stallNotified?: boolean; unreadableSince?: number; lastReadError?: string; /** Restored watch: read failures mean the owning session is not back yet. */ @@ -58,6 +62,7 @@ function persisted(watch: Watch): PersistedWatch { restartNote, harnessId, refs, + stallAfterMs, } = watch; return { id, @@ -75,6 +80,7 @@ function persisted(watch: Watch): PersistedWatch { ...(restartNote ? { restartNote } : {}), ...(harnessId ? { harnessId } : {}), ...(refs?.length ? { refs } : {}), + ...(stallAfterMs !== undefined ? { stallAfterMs } : {}), }; } @@ -112,11 +118,20 @@ function duration(milliseconds: number): string { : `${Math.round(milliseconds / 60_000)} min`; } +/** Never rounded: a stalled alert claims at least this much silence. */ +function exactDuration(milliseconds: number): string { + if (milliseconds % 60_000 === 0) return `${milliseconds / 60_000} min`; + if (milliseconds % 1_000 === 0) return `${milliseconds / 1_000} s`; + return `${milliseconds} ms`; +} + function describe(watch: Watch): string { const link = threadLink(watch.threadId); if (watch.outcome === "interrupted" && watch.restartNote) return `${link}: interrupted${watch.turnId ? ` (Turn ${watch.turnId})` : ""}. ${watch.restartNote}`; switch (watch.outcome) { + case "stalled": + return `${link}: stalled${watch.turnId ? ` (Turn ${watch.turnId})` : ""}. It was still running, but no Harness event had been observed for at least ${exactDuration(watch.stallAfterMs ?? 0)}. This may be a long operation; read the Thread for its latest progress before deciding whether to intervene. The watch continues.`; case "timedOut": return `${link} has not reached a terminal state after ${duration(watch.timeoutMs)}; this watch expired. Run 'codexhost thread watch' again to keep waiting.`; case "unreadable": @@ -132,16 +147,19 @@ function describe(watch: Watch): string { } function notification(watches: readonly Watch[]): string { + const event = watches.some((watch) => watch.outcome === "stalled") + ? "Watched Threads stopped, the watch expired, or an enabled silence threshold was reached." + : "Watched Threads stopped or the watch expired."; return [ - "[codexhost thread watch] Watched Threads stopped or the watch expired. This reports execution state only, not that the work is correct or accepted. Inspect each Thread with 'codexhost thread read ' before relying on it.", + `[codexhost thread watch] ${event} This reports execution state only, not that the work is correct or accepted. Inspect each Thread with 'codexhost thread read ' before relying on it.`, ...watches.map((watch) => `- ${describe(watch)}`), ].join("\n"); } /** - * One-shot notifications when a watched Thread stops, built only on the public - * `read` and `send` operations, so it is independent of Harness, Desktop and - * delegation lineage. With a store, watches survive a Host Runtime restart: a + * One-shot stop notifications plus optional silence alerts. State and delivery + * use public `read`/`send`; Host event subscriptions supply activity without + * coupling observation to Harness or delegation lineage. Watches survive a restart: a * send whose outcome was not recorded before the restart is made again, so a * restart may repeat a notification but does not lose one. */ @@ -191,6 +209,14 @@ export class DelegationWatchService { throw new DelegationControlError("INVALID_ARGUMENT", "A Thread cannot watch itself"); if (!Number.isSafeInteger(input.timeoutMs) || input.timeoutMs <= 0) throw new DelegationControlError("INVALID_ARGUMENT", "timeoutMs must be a positive integer"); + if ( + input.stallAfterMs !== undefined && + (!Number.isSafeInteger(input.stallAfterMs) || input.stallAfterMs <= 0) + ) + throw new DelegationControlError( + "INVALID_ARGUMENT", + "stallAfterMs must be a positive integer", + ); const client = originClientName(input.notifyThreadId); if (client !== undefined && (!isValidClientName(client) || !this.#clientEvents)) @@ -208,14 +234,23 @@ export class DelegationWatchService { status: target.status, timeoutMs: input.timeoutMs, }; - if (terminal(target.status)) return { ...result, state: "alreadyTerminal" }; + const stallOption = + input.stallAfterMs !== undefined ? { stallAfterMs: input.stallAfterMs } : {}; + if (terminal(target.status)) return { ...result, ...stallOption, state: "alreadyTerminal" }; const existing = this.#watches.find( (watch) => watch.state === "watching" && watch.threadId === input.threadId && watch.notifyThreadId === input.notifyThreadId, ); - if (existing) return { ...result, state: "watching", timeoutMs: existing.timeoutMs }; + if (existing) { + return { + ...result, + state: "watching", + timeoutMs: existing.timeoutMs, + ...(existing.stallAfterMs !== undefined ? { stallAfterMs: existing.stallAfterMs } : {}), + }; + } if (this.#closed) throw new DelegationControlError("INTERNAL_ERROR", "Host Runtime is closing"); const watch: Watch = { id: randomUUID(), @@ -225,6 +260,9 @@ export class DelegationWatchService { registeredAt: new Date().toISOString(), timeoutMs: input.timeoutMs, deadline: Date.now() + input.timeoutMs, + ...(input.stallAfterMs !== undefined + ? { stallAfterMs: input.stallAfterMs, lastActivityAt: Date.now() } + : {}), }; this.#watches.push(watch); try { @@ -238,14 +276,23 @@ export class DelegationWatchService { ); } this.#schedule(); - return { ...result, state: "watching" }; + return { ...result, ...stallOption, state: "watching" }; } async watches(): Promise { await this.#ready; return { watches: this.#watches.map( - ({ threadId, notifyThreadId, state, outcome, turnId, reason, registeredAt }) => ({ + ({ + threadId, + notifyThreadId, + state, + outcome, + turnId, + reason, + registeredAt, + stallAfterMs, + }) => ({ threadId, notifyThreadId, state, @@ -253,11 +300,27 @@ export class DelegationWatchService { ...(turnId ? { turnId } : {}), ...(reason ? { reason } : {}), registeredAt, + ...(stallAfterMs !== undefined ? { stallAfterMs } : {}), }), ), }; } + /** Any live Harness event, including subagent activity; polling reads never call this. */ + noteActivity(threadIds: readonly string[]): void { + const now = Date.now(); + for (const watch of this.#watches) { + if ( + watch.state !== "watching" || + watch.stallAfterMs === undefined || + !threadIds.includes(watch.threadId) + ) + continue; + watch.lastActivityAt = now; + delete watch.stallNotified; + } + } + /** Stops work in this process. Stored watches stay on disk for the next Runtime. */ close(): void { this.#closed = true; @@ -284,6 +347,7 @@ export class DelegationWatchService { if (watch.state === "watching") { watch.observeGraceUntil = graceUntil; watch.crossedRestart = true; + if (watch.stallAfterMs !== undefined) watch.lastActivityAt = now; } if (watch.state === "pendingDelivery" && now >= (watch.deliveryDeadline ?? 0)) { watch.state = "undeliverable"; @@ -320,6 +384,20 @@ export class DelegationWatchService { const observed = await this.#observe(watch); if (!observed) continue; const { outcome, turnId, restartNote } = observed; + if (outcome === "stalled") { + // Keep the original watch observing; a silence alert is an independent delivery. + const alert: Watch = { + ...persisted(watch), + id: randomUUID(), + state: "pendingDelivery", + outcome, + ...(turnId ? { turnId } : {}), + deliveryDeadline: Date.now() + DELIVERY_WINDOW_MS, + }; + watch.stallNotified = true; + this.#watches.push(alert); + continue; + } if (restartNote) watch.restartNote = restartNote; watch.state = "pendingDelivery"; watch.outcome = outcome; @@ -365,6 +443,17 @@ export class DelegationWatchService { }; } delete watch.unreadableSince; + // Waiting in the capacity queue or being created is not Harness silence. + if (snapshot.status !== "running" && watch.stallAfterMs !== undefined) + watch.lastActivityAt = Date.now(); + if ( + snapshot.status === "running" && + Date.now() < watch.deadline && + watch.stallAfterMs !== undefined && + !watch.stallNotified && + Date.now() - (watch.lastActivityAt ?? Date.now()) >= watch.stallAfterMs + ) + return { outcome: "stalled", ...(turnId ? { turnId } : {}) }; } catch (error) { if (watch.observeGraceUntil !== undefined) { // Until the session that owns the Thread registers again, a failed read says nothing. @@ -443,7 +532,11 @@ export class DelegationWatchService { ...(watch.turnId ? { turnId: watch.turnId } : {}), ...(watch.harnessId ? { harnessId: watch.harnessId } : {}), ...(watch.refs?.length ? { refs: watch.refs } : {}), - ...(watch.restartNote ? { note: watch.restartNote } : {}), + ...(watch.outcome === "stalled" + ? { note: describe(watch) } + : watch.restartNote + ? { note: watch.restartNote } + : {}), })), ); for (const watch of pending) this.#remove(watch); @@ -475,6 +568,8 @@ export class DelegationWatchService { } #persist(watch: Watch, required = false): Promise { + // Silence is measured again after a restart, and older Runtimes set a stalled record aside as unreadable. + if (watch.outcome === "stalled") return Promise.resolve(); const record = persisted(watch); return this.#write((store) => store.save(record), required); } diff --git a/packages/host-runtime/src/desktop-thread-routing.ts b/packages/host-runtime/src/desktop-thread-routing.ts index 33449f622..5a31ce84d 100644 --- a/packages/host-runtime/src/desktop-thread-routing.ts +++ b/packages/host-runtime/src/desktop-thread-routing.ts @@ -157,6 +157,8 @@ export interface DesktopThreadRoutingHost { params: JsonObject, historyFresh: boolean, ): Promise; + listQueuedOfficialHistory(request: JsonRpcRequest, params: JsonObject): Promise; + readOrResumeQueuedOfficialThread(request: JsonRpcRequest, params: JsonObject): Promise; startExternalTurn(request: JsonRpcRequest, thread: ExternalThread): Promise; steerExternalTurn(request: JsonRpcRequest, thread: ExternalThread): Promise; interruptExternalTurn( @@ -536,6 +538,7 @@ export class DesktopThreadRouting { ); return true; } + if (await this.#host.listQueuedOfficialHistory(request, params)) return true; } if (request.method === "turn/start") { const params = routingParams(request); @@ -604,6 +607,11 @@ export class DesktopThreadRouting { return true; } if (location.kind === "official") { + if ( + params.includeTurns === true && + (await this.#host.readOrResumeQueuedOfficialThread(request, params)) + ) + return true; await this.#host.forwardOfficialRequest(request, frame); return true; } @@ -643,6 +651,11 @@ export class DesktopThreadRouting { } return true; } + if ( + request.method === "thread/resume" && + (await this.#host.readOrResumeQueuedOfficialThread(request, params)) + ) + return true; } if (request.method === "thread/unsubscribe") { const params = routingParams(request); diff --git a/packages/host-runtime/src/external-client-events.ts b/packages/host-runtime/src/external-client-events.ts index ec2783b13..9494c3ca4 100644 --- a/packages/host-runtime/src/external-client-events.ts +++ b/packages/host-runtime/src/external-client-events.ts @@ -10,7 +10,7 @@ export const EVENT_RETENTION_DAYS_ENV = "CODEXHOST_CLIENT_EVENT_RETENTION_DAYS"; const DAY_MS = 24 * 60 * 60_000; const SEGMENT = /^(\d{4}-\d{2}-\d{2})\.jsonl$/u; -/** What an external client learns when a watched Thread stops: state and links, never content. */ +/** Watch outcomes for an external client: state and links, never conversation content. */ export interface ClientEvent { /** Increases within a client's log; use it as the cursor. */ id: number; diff --git a/packages/host-runtime/src/external-client-mcp.ts b/packages/host-runtime/src/external-client-mcp.ts index a0a9ec8bc..de185c968 100644 --- a/packages/host-runtime/src/external-client-mcp.ts +++ b/packages/host-runtime/src/external-client-mcp.ts @@ -24,6 +24,12 @@ interface Tool { const string = (description: string) => ({ type: "string", description }); const integer = (description: string) => ({ type: "integer", minimum: 0, description }); const thread = string("Thread id or codex://threads/ link"); +const stallAfterMs = { + type: "integer", + minimum: 1, + description: + "Optional silence threshold in ms; disabled by default. A stalled event keeps the watch active and does not prove failure. Live Harness and subagent events reset the clock.", +}; function text(args: Arguments, name: string, required = false): string | undefined { const value = args[name]; @@ -51,6 +57,13 @@ function pick(args: Arguments, names: readonly string[]): Arguments { ); } +function stallOption(args: Arguments): { stallAfterMs?: number } { + if (args.stallAfterMs === undefined) return {}; + if (!Number.isSafeInteger(args.stallAfterMs) || (args.stallAfterMs as number) <= 0) + throw new DelegationControlError("INVALID_ARGUMENT", "stallAfterMs must be a positive integer"); + return { stallAfterMs: args.stallAfterMs as number }; +} + /** Each tool is one delegation route; parameters and results match the CLI's JSON. */ const TOOLS: Tool[] = [ { @@ -80,7 +93,7 @@ const TOOLS: Tool[] = [ { name: "delegate_start", description: - "Start a task in a new Thread and return at once. Without parentThread the work belongs to this client and cwd is required. watch=true records an event for this client when the Thread stops (read it with events_poll or the SSE stream).", + "Start a task in a new Thread and return at once. Without parentThread the work belongs to this client and cwd is required. watch=true records an event for this client when the Thread stops (read it with events_poll or the SSE stream). When the local limit on running delegations is reached, the Thread is queued (status queued, queue.position) and starts by itself later.", inputSchema: { type: "object", properties: { @@ -99,30 +112,38 @@ const TOOLS: Tool[] = [ }, watch: { type: "boolean", description: "Record an event when the Thread stops" }, watchTimeoutMs: integer("How long to watch; default 29 minutes"), + stallAfterMs, }, required: ["harnessId", "task"], }, - call: (args) => ({ - route: "/v1/delegate/start", - body: { - harnessId: text(args, "harnessId", true), - task: text(args, "task", true), - ...pick(args, ["taskName", "cwd", "requestId", "refs"]), - ...(args.model !== undefined ? { model: { id: text(args, "model") } } : {}), - ...(args.thinking !== undefined ? { thinkingOptionId: text(args, "thinking") } : {}), - ...(args.parentThread !== undefined ? { parentThreadId: text(args, "parentThread") } : {}), - ...(args.watch === true - ? { - watchTimeoutMs: bounded( - args, - "watchTimeoutMs", - DEFAULT_WATCH_TIMEOUT_MS, - Number.MAX_SAFE_INTEGER, - ), - } - : {}), - }, - }), + call: (args) => { + if (args.stallAfterMs !== undefined && args.watch !== true) + throw new DelegationControlError("INVALID_ARGUMENT", "stallAfterMs requires watch=true"); + return { + route: "/v1/delegate/start", + body: { + harnessId: text(args, "harnessId", true), + task: text(args, "task", true), + ...pick(args, ["taskName", "cwd", "requestId", "refs"]), + ...(args.model !== undefined ? { model: { id: text(args, "model") } } : {}), + ...(args.thinking !== undefined ? { thinkingOptionId: text(args, "thinking") } : {}), + ...(args.parentThread !== undefined + ? { parentThreadId: text(args, "parentThread") } + : {}), + ...(args.watch === true + ? { + watchTimeoutMs: bounded( + args, + "watchTimeoutMs", + DEFAULT_WATCH_TIMEOUT_MS, + Number.MAX_SAFE_INTEGER, + ), + ...stallOption(args), + } + : {}), + }, + }; + }, }, { name: "thread_read", @@ -227,10 +248,11 @@ const TOOLS: Tool[] = [ }, { name: "thread_watch", - description: "Record one event for this client when the Thread stops or the watch expires.", + description: + "Record an event for this client when the Thread stops or the watch expires. Optional stallAfterMs also records one stalled event per silence episode without ending the watch.", inputSchema: { type: "object", - properties: { thread, timeoutMs: integer("Default 29 minutes") }, + properties: { thread, timeoutMs: integer("Default 29 minutes"), stallAfterMs }, required: ["thread"], }, call: (args) => ({ @@ -239,6 +261,7 @@ const TOOLS: Tool[] = [ threadId: threadId(args), timeoutMs: bounded(args, "timeoutMs", DEFAULT_WATCH_TIMEOUT_MS, Number.MAX_SAFE_INTEGER) || 1, + ...stallOption(args), }, }), }, diff --git a/packages/host-runtime/src/external-thread-repository.ts b/packages/host-runtime/src/external-thread-repository.ts index 481650ca2..8f51e2a30 100644 --- a/packages/host-runtime/src/external-thread-repository.ts +++ b/packages/host-runtime/src/external-thread-repository.ts @@ -35,6 +35,7 @@ import { materializeExternalSubagent, projectExternalSnapshot, } from "./external-subagent-threads.js"; +import { originClientName } from "./external-client-origin.js"; export interface ExternalThreadStore { initialize(): Promise; @@ -122,10 +123,16 @@ export function createProductionExternalThreadStore( } export class ExternalThreadRepository { + readonly #delegationsByChild = new Map(); + readonly #pendingDelegationsByChild = new Map(); + constructor(private readonly store: ExternalThreadStore) {} - initialize(): Promise { - return this.store.initialize(); + async initialize(): Promise { + await this.store.initialize(); + for (const record of await this.store.listDelegations()) { + this.#delegationsByChild.set(record.childHostThreadId, record); + } } close(): Promise { @@ -168,8 +175,26 @@ export class ExternalThreadRepository { return this.store.getDelegation(delegationId); } - getDelegationByChild(childHostThreadId: HostThreadId): Promise { - return this.store.getDelegationByChild(childHostThreadId); + async getDelegationByChild( + childHostThreadId: HostThreadId, + ): Promise { + const stored = await this.store.getDelegationByChild(childHostThreadId); + if (stored) { + this.#delegationsByChild.set(childHostThreadId, stored); + return stored; + } + return this.#pendingDelegationsByChild.get(childHostThreadId) ?? null; + } + + /** Whether this Thread is a Delegation child of a session Thread rather than a top-level client start. */ + hasSessionDelegationParent(threadId: string): boolean { + const delegation = + this.#delegationsByChild.get(threadId as HostThreadId) ?? + this.#pendingDelegationsByChild.get(threadId as HostThreadId); + if (!delegation) return false; + return ( + delegation.parentHostThreadId !== threadId && !originClientName(delegation.parentHostThreadId) + ); } findDelegationByRequest(requestId: string): Promise { @@ -184,19 +209,42 @@ export class ExternalThreadRepository { return this.store.listDelegations(parentHostThreadId); } - createDelegation(input: CreateDelegationInput): Promise { - return this.store.createDelegation(input); + async createDelegation(input: CreateDelegationInput): Promise { + const timestamp = new Date().toISOString(); + const provisional: StoredDelegationRecordV1 = { + formatVersion: 1, + revision: 1, + ...input, + status: input.status ?? "creating", + createdAt: timestamp, + updatedAt: timestamp, + }; + this.#pendingDelegationsByChild.set(input.childHostThreadId, provisional); + try { + const created = await this.store.createDelegation(input); + this.#delegationsByChild.set(created.childHostThreadId, created); + return created; + } finally { + this.#pendingDelegationsByChild.delete(input.childHostThreadId); + } } - setDelegationStatus( + async setDelegationStatus( delegationId: HostThreadId, status: DelegationStatus, ): Promise { - return this.store.setDelegationStatus(delegationId, status); + const updated = await this.store.setDelegationStatus(delegationId, status); + this.#delegationsByChild.set(updated.childHostThreadId, updated); + return updated; } - removeDelegation(delegationId: HostThreadId): Promise { - return this.store.removeDelegation(delegationId); + async removeDelegation(delegationId: HostThreadId): Promise { + for (const [childId, record] of this.#delegationsByChild) { + if (record.delegationId === delegationId) { + this.#delegationsByChild.delete(childId); + } + } + await this.store.removeDelegation(delegationId); } createProvisional(input: CreateProvisionalThreadInput): Promise { diff --git a/packages/host-runtime/src/harness-delegation-coordinator.ts b/packages/host-runtime/src/harness-delegation-coordinator.ts index e623230e4..d1add67b2 100644 --- a/packages/host-runtime/src/harness-delegation-coordinator.ts +++ b/packages/host-runtime/src/harness-delegation-coordinator.ts @@ -21,12 +21,17 @@ import { harnessIdSchema, hostThreadIdSchema, hostTurnIdSchema, + type HostThreadId, + type HostTurnId, } from "@codexhost/shared-contracts"; import { DELEGATION_THREAD_ID_ENV, DelegationControlError, + type DelegationAdmission, + type DelegationAdmissionRequest, type DelegationConfigurationResult, + type DelegationStartIdentity, type DelegationStartInput, type DelegationStartResult, type DelegationThreadListResult, @@ -153,11 +158,14 @@ export class HarnessDelegationCoordinator { readonly #cancelOfficial: (input: ThreadCancelInput) => Promise; readonly #startOfficial: ( input: DelegationStartInput & { parentThreadId: string; cwd: string; title: string }, + admitted?: DelegationStartIdentity, ) => Promise; readonly #listOfficial: (input: ThreadListInput) => Promise; readonly #officialThreadMetadata: (threadId: string) => Promise; readonly #activeOfficialParents: () => string[]; readonly #externalThreadBusy: (thread: ExternalThread) => boolean; + readonly #admitDelegation: + ((request: DelegationAdmissionRequest) => Promise | undefined) | undefined; constructor(input: { adapters: Map; @@ -180,13 +188,17 @@ export class HarnessDelegationCoordinator { readOfficial(input: ThreadReadInput): Promise; sendOfficial(input: ThreadSendInput): Promise; cancelOfficial(input: ThreadCancelInput): Promise; + /** Native Codex starts apply the capacity limit themselves: their Thread exists while queued. */ startOfficial( input: DelegationStartInput & { parentThreadId: string; cwd: string; title: string }, + admitted?: DelegationStartIdentity, ): Promise; listOfficial(input: ThreadListInput): Promise; officialThreadMetadata(threadId: string): Promise; activeOfficialParents(): string[]; externalThreadBusy(thread: ExternalThread): boolean; + /** The capacity limit for external Harness starts; undefined admits at once. */ + admitDelegation?(request: DelegationAdmissionRequest): Promise | undefined; }) { this.#adapters = input.adapters; this.#environment = input.environment; @@ -204,6 +216,7 @@ export class HarnessDelegationCoordinator { this.#officialThreadMetadata = input.officialThreadMetadata; this.#activeOfficialParents = input.activeOfficialParents; this.#externalThreadBusy = input.externalThreadBusy; + this.#admitDelegation = input.admitDelegation; } async listHarnesses(): Promise { @@ -229,7 +242,10 @@ export class HarnessDelegationCoordinator { }; } - async start(input: DelegationStartInput): Promise { + async start( + input: DelegationStartInput, + admitted?: DelegationStartIdentity, + ): Promise { validateStart(input); // An external client has no workspace of its own for the child to inherit. if (originClientName(input.parentThreadId) && !input.cwd) @@ -242,12 +258,10 @@ export class HarnessDelegationCoordinator { const selectedCwd = input.cwd ?? parent.cwd ?? process.cwd(); const title = delegationTitle(parent.title, input); if (input.harnessId === "codex") { - const result = await this.#startOfficial({ - ...input, - parentThreadId, - cwd: selectedCwd, - title, - }); + const result = await this.#startOfficial( + { ...input, parentThreadId, cwd: selectedCwd, title }, + admitted, + ); return { ...result, parentThreadId, @@ -265,14 +279,17 @@ export class HarnessDelegationCoordinator { } const targetHarnessId = input.harnessId as ExternalHarnessId; const digest = taskDigest(startInput); - const duplicate = input.requestId - ? await this.#repository.findDelegationByRequest(input.requestId) - : await this.#repository.findRecentDelegation({ - parentHostThreadId: hostThreadIdSchema.parse(parentThreadId), - targetHarnessId: harnessIdSchema.parse(targetHarnessId), - taskDigest: digest, - since: new Date(Date.now() - IMPLICIT_DEDUPLICATION_MS), - }); + // A queued start was deduplicated when it was queued. + const duplicate = admitted + ? null + : input.requestId + ? await this.#repository.findDelegationByRequest(input.requestId) + : await this.#repository.findRecentDelegation({ + parentHostThreadId: hostThreadIdSchema.parse(parentThreadId), + targetHarnessId: harnessIdSchema.parse(targetHarnessId), + taskDigest: digest, + since: new Date(Date.now() - IMPLICIT_DEDUPLICATION_MS), + }); if ( duplicate && input.requestId && @@ -302,16 +319,86 @@ export class HarnessDelegationCoordinator { }); this.#validateConfiguration(inspected.inspection, input.model, input.thinkingOptionId); } - const delegationId = hostThreadIdSchema.parse(randomUUID()); - const childThreadId = hostThreadIdSchema.parse(randomUUID()); - const turnId = hostTurnIdSchema.parse(randomUUID()); + const delegationId = hostThreadIdSchema.parse(admitted?.delegationId ?? randomUUID()); + const childThreadId = hostThreadIdSchema.parse(admitted?.threadId ?? randomUUID()); + const turnId = hostTurnIdSchema.parse(admitted?.turnId ?? randomUUID()); + const admission = + admitted || !originClientName(parentThreadId) + ? undefined + : await this.#admitDelegation?.({ + input: startInput, + title, + taskDigest: digest, + identity: { delegationId, threadId: childThreadId, turnId }, + }); + if (admission?.state === "queued") { + const { identity } = admission; + return { + ...this.#result( + identity.delegationId, + identity.threadId, + identity.turnId, + targetHarnessId, + admission.status, + ), + ...(admission.position ? { queue: { position: admission.position } } : {}), + cwd: admission.cwd, + title: admission.title, + parentThreadId: admission.parentThreadId, + }; + } + try { + return await this.#create({ + input, + parent, + parentThreadId, + cwd: startInput.cwd, + title, + digest, + adapter, + targetHarnessId, + delegationId, + childThreadId, + turnId, + }); + } finally { + admission?.release(); + } + } + + async #create(context: { + input: DelegationStartInput; + parent: { harnessId: RoutedHarnessId }; + parentThreadId: string; + cwd: string; + title: string; + digest: string; + adapter: HarnessAdapter; + targetHarnessId: ExternalHarnessId; + delegationId: HostThreadId; + childThreadId: HostThreadId; + turnId: HostTurnId; + }): Promise { + const { + input, + parent, + parentThreadId, + cwd, + title, + digest, + adapter, + targetHarnessId, + delegationId, + childThreadId, + turnId, + } = context; const createRequestId = input.requestId ? `delegation:${input.requestId}` : randomUUID(); let record = await this.#repository.createProvisional( createExternalThreadRecordInput({ hostThreadId: childThreadId, createRequestId, harnessId: harnessIdSchema.parse(targetHarnessId), - cwd: startInput.cwd, + cwd, title, transportModelId: input.model || input.thinkingOptionId diff --git a/packages/host-runtime/src/remote-delegation-read.ts b/packages/host-runtime/src/remote-delegation-read.ts index 1d6dee1b4..d4ddd86bf 100644 --- a/packages/host-runtime/src/remote-delegation-read.ts +++ b/packages/host-runtime/src/remote-delegation-read.ts @@ -11,7 +11,8 @@ const status = z.enum(["creating", "running", "completed", "failed", "interrupte const snapshot = z.object({ threadId: z.string(), harnessId: z.union([z.literal("codex"), harnessIdSchema]), - status, + status: z.union([z.literal("queued"), status]), + queue: z.object({ position: z.number().int().positive() }).optional(), turn: z.object({ turnId: z.string(), status }).nullable(), progress: z.array(z.object({ id: z.string(), turnId: z.string(), text: z.string() })), result: z.object({ diff --git a/packages/host-runtime/src/run-host-runtime.ts b/packages/host-runtime/src/run-host-runtime.ts index 87ce2390b..e2c6579d5 100644 --- a/packages/host-runtime/src/run-host-runtime.ts +++ b/packages/host-runtime/src/run-host-runtime.ts @@ -12,6 +12,10 @@ import { prepareLocalCodex } from "./native-account-host.js"; import { SingleNativeCodexAccount } from "./account/codex-account-control.js"; import { OfficialRuntimeScope } from "./codex-runtime/official-runtime-scope.js"; import { createOwnedUnixBackend } from "./codex-runtime/owned-official-backends.js"; +import { + FileDelegationQueueStore, + defaultDelegationQueueDirectory, +} from "./delegation-capacity-store.js"; import { DelegationControlRegistry } from "./delegation-control-registry.js"; import { FileDelegationWatchStore, @@ -178,6 +182,10 @@ export async function prepareDelegationRuntime(input: { process.stderr.write( `codexhost delegation watch: ${error instanceof Error ? (error.stack ?? error.message) : String(error)}\n`, ); + const diagnoseQueue = (error: unknown) => + process.stderr.write( + `codexhost delegation queue: ${error instanceof Error ? (error.stack ?? error.message) : String(error)}\n`, + ); const events = new ClientEventLog({ directory: path.join(codexhostDataDirectory(input.environment), "client-events"), retentionDays: eventRetentionDays(input.environment), @@ -193,6 +201,13 @@ export async function prepareDelegationRuntime(input: { path.join(codexhostDataDirectory(input.environment), "thread-refs"), ), clientEvents: events, + capacity: { + store: new FileDelegationQueueStore( + defaultDelegationQueueDirectory(input.environment), + diagnoseQueue, + ), + diagnose: diagnoseQueue, + }, }); const token = randomBytes(32).toString("hex"); const server = await startDelegationControlServer({ token, api: registry, watchApi: registry }); @@ -231,7 +246,7 @@ export async function prepareDelegationRuntime(input: { try { return await input.createHost(environment, (value) => registry.register(value), registry); } finally { - registry.close(); + await registry.close(); await externalServer?.close(); await server.close(); } diff --git a/packages/host-runtime/src/shared-thread-bridge.ts b/packages/host-runtime/src/shared-thread-bridge.ts index 9cbc6847c..024d680c1 100644 --- a/packages/host-runtime/src/shared-thread-bridge.ts +++ b/packages/host-runtime/src/shared-thread-bridge.ts @@ -8,6 +8,7 @@ import { LOADED_SESSIONS_METHOD, IDLE_RELEASE_SETTINGS_METHOD, USAGE_STATISTICS_METHOD, + DELEGATION_CAPACITY_SETTINGS_METHOD, } from "@codexhost/shared-contracts"; import { storedSectionPlacementV1Schema, @@ -143,9 +144,12 @@ export class SharedThreadBridge { handlesControl(method: string): boolean { return ( this.options.delegateCreates && - [LOADED_SESSIONS_METHOD, IDLE_RELEASE_SETTINGS_METHOD, USAGE_STATISTICS_METHOD].some( - (value) => value === method, - ) + [ + LOADED_SESSIONS_METHOD, + IDLE_RELEASE_SETTINGS_METHOD, + USAGE_STATISTICS_METHOD, + DELEGATION_CAPACITY_SETTINGS_METHOD, + ].some((value) => value === method) ); } diff --git a/packages/host-runtime/src/thread-activity.ts b/packages/host-runtime/src/thread-activity.ts new file mode 100644 index 000000000..9d525fe50 --- /dev/null +++ b/packages/host-runtime/src/thread-activity.ts @@ -0,0 +1,117 @@ +import type { JsonValue } from "@codexhost/protocol-core"; + +function record(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** + * Live Harness events only: retains ancestry while Threads are open, never event or + * conversation content. + */ +export class ThreadActivity { + readonly #parents = new Map(); + /** Closed Threads kept because an open descendant still reports through them. */ + readonly #closed = new Set(); + readonly #listeners = new Set<(threadIds: readonly string[]) => void>(); + + /** Threads whose ancestry is still retained. */ + get size(): number { + return new Set([...this.#parents.keys(), ...this.#closed]).size; + } + + subscribe(listener: (threadIds: readonly string[]) => void): () => void { + this.#listeners.add(listener); + return () => this.#listeners.delete(listener); + } + + /** Whether an event has shown the Thread to be a subagent of another Thread. */ + isSubagent(threadId: string): boolean { + return this.#parents.has(threadId); + } + + note(threadId: string, parentThreadId?: string): void { + this.#closed.delete(threadId); + const previous = this.#parents.get(threadId); + if (parentThreadId && parentThreadId !== threadId && parentThreadId !== previous) { + this.#parents.set(threadId, parentThreadId); + this.#prune(previous); + } + const active = new Set(); + let current: string | undefined = threadId; + while (current && !active.has(current)) { + active.add(current); + current = this.#parents.get(current); + } + const threadIds = [...active]; + for (const listener of this.#listeners) listener(threadIds); + } + + /** + * Native ancestry is learned only when a Thread starts, so a closed Thread keeps its + * link until every descendant reporting through it has closed too. + */ + close(threadId: string): void { + this.#closed.add(threadId); + this.#prune(threadId); + } + + /** Only for Threads whose links are noted again with every event they report. */ + forget(threadId: string): void { + const parent = this.#parents.get(threadId); + this.#parents.delete(threadId); + this.#closed.delete(threadId); + for (const [child, childParent] of this.#parents) { + if (childParent === threadId) this.#parents.delete(child); + } + this.#prune(parent); + } + + clear(): void { + this.#parents.clear(); + this.#closed.clear(); + } + + observeOfficial(value: JsonValue): void { + // Replies to Host polling are not Harness activity. + if (!record(value) || typeof value.method !== "string" || !record(value.params)) return; + const params = value.params; + const thread = record(params.thread) ? params.thread : undefined; + const threadId = typeof params.threadId === "string" ? params.threadId : thread?.id; + if (typeof threadId !== "string" || !threadId) return; + const source = record(thread?.source) ? thread.source : undefined; + const subagent = record(source?.subAgent) ? source.subAgent : undefined; + const spawn = record(subagent?.thread_spawn) ? subagent.thread_spawn : undefined; + const parent = typeof spawn?.parent_thread_id === "string" ? spawn.parent_thread_id : undefined; + this.note(threadId, parent); + if (value.method === "thread/closed") { + this.close(threadId); + return; + } + const item = record(params.item) ? params.item : undefined; + if ( + item?.type === "collabAgentToolCall" && + item.tool === "spawnAgent" && + Array.isArray(item.receiverThreadIds) + ) { + for (const child of item.receiverThreadIds) { + if (typeof child === "string" && child) this.note(child, threadId); + } + } + } + + /** Drops closed Threads upward until one still has a retained child or is open. */ + #prune(threadId: string | undefined): void { + let current = threadId; + while (current && this.#closed.has(current) && !this.#hasChildren(current)) { + this.#closed.delete(current); + const parent = this.#parents.get(current); + this.#parents.delete(current); + current = parent; + } + } + + #hasChildren(threadId: string): boolean { + for (const parent of this.#parents.values()) if (parent === threadId) return true; + return false; + } +} diff --git a/packages/host-runtime/test/app-server-host.delegation-capacity.test.ts b/packages/host-runtime/test/app-server-host.delegation-capacity.test.ts new file mode 100644 index 000000000..dff9a2140 --- /dev/null +++ b/packages/host-runtime/test/app-server-host.delegation-capacity.test.ts @@ -0,0 +1,208 @@ +import { describe, expect, it, vi } from "vitest"; +import type { JsonObject } from "@codexhost/protocol-core"; + +import { DelegationControlRegistry } from "../src/delegation-control-registry.js"; +import type { DelegationControlRegistration } from "../src/delegation-types.js"; +import { + createFixture, + requestId, + startPiThread, + startPiTurn, + stopFixture, + turnEvent, + writeRequest, +} from "./app-server-host-fixture.js"; + +describe("AppServerHost delegation capacity", () => { + it("reports the setting as unavailable when no delegation registry is attached", async () => { + const fixture = createFixture(); + try { + await fixture.ready; + writeRequest(fixture.desktopInput, { + id: 900, + method: "codexhost/settings/delegation-capacity/set", + params: { enabled: true, maxRunning: 2 }, + }); + expect(await fixture.collector.waitFor((message) => requestId(message, 900))).toMatchObject({ + error: { code: -32601 }, + }); + expect(fixture.official.stdin.read()).toBeNull(); + } finally { + await stopFixture(fixture); + } + }); + + it("queues a top-level client delegation while a manually opened Thread runs, and starts session-parented child delegations immediately", async () => { + const registry = new DelegationControlRegistry({ capacity: { pollIntervalMs: 10 } }); + let registration: DelegationControlRegistration | undefined; + const fixture = createFixture({ + onDelegationApi: (api) => { + registration = api; + return registry.register(api); + }, + }); + try { + await fixture.ready; + writeRequest(fixture.desktopInput, { + id: 900, + method: "codexhost/settings/delegation-capacity/set", + params: { enabled: true, maxRunning: 0 }, + }); + expect(await fixture.collector.waitFor((message) => requestId(message, 900))).toMatchObject({ + error: { code: -32602 }, + }); + writeRequest(fixture.desktopInput, { + id: 901, + method: "codexhost/settings/delegation-capacity/set", + params: { enabled: true, maxRunning: 1 }, + }); + expect(await fixture.collector.waitFor((message) => requestId(message, 901))).toMatchObject({ + result: { enabled: true, maxRunning: 1 }, + }); + expect(fixture.official.stdin.read()).toBeNull(); + + const manual = await startPiThread(fixture); + const manualTurn = await startPiTurn(fixture, manual); + await fixture.collector.waitFor((message) => turnEvent(message, "turn/started", manualTurn)); + + // A child session spawned by an existing session bypasses the top-level queue and does not count as top-level. + const child = await registry.start({ + harnessId: "pi", + task: "child work", + cwd: "/synthetic", + parentThreadId: manual, + }); + expect(child).toMatchObject({ harnessId: "pi", status: "running", parentThreadId: manual }); + expect(fixture.adapter.sessions).toHaveLength(2); + expect(registration?.runningThreadIds?.()).toEqual([manual]); + + const queued = await registry.start({ + harnessId: "pi", + task: "queued work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(queued).toMatchObject({ harnessId: "pi", status: "queued", queue: { position: 1 } }); + expect(fixture.adapter.sessions).toHaveLength(2); + expect(await registry.read({ threadId: queued.threadId, view: "result" })).toMatchObject({ + status: "queued", + queue: { position: 1 }, + }); + + const manualSession = fixture.adapter.sessions[0]; + if (!manualSession) throw new Error("Missing manual Session"); + manualSession.succeedTurn(); + await fixture.collector.waitFor((message) => + turnEvent(message, "turn/completed", manualTurn), + ); + // Even while `child` is still running, ending `manual` frees the top-level slot for `queued`. + await vi.waitFor(() => expect(fixture.adapter.sessions).toHaveLength(3), { timeout: 5_000 }); + await vi.waitFor(async () => + expect(await registry.read({ threadId: queued.threadId, view: "result" })).toMatchObject({ + threadId: queued.threadId, + status: "running", + }), + ); + expect(registration?.runningThreadIds?.()).toEqual([queued.threadId]); + expect(await fixture.mappingStore.getThread(queued.threadId as never)).toMatchObject({ + hostThreadId: queued.threadId, + }); + + // The limit is reached again, yet a follow-up message to an existing Thread still starts. + const followUp = await registry.send({ threadId: manual, message: "follow up" }); + expect(followUp).toMatchObject({ threadId: manual, status: "running" }); + const next = await registry.start({ + harnessId: "pi", + task: "next work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(next).toMatchObject({ status: "queued", queue: { position: 1 } }); + expect(await registry.cancel({ threadId: next.threadId })).toMatchObject({ cancelled: true }); + + manualSession.succeedTurn(); + await fixture.collector.waitFor((message) => + turnEvent(message, "turn/completed", followUp.turnId), + ); + fixture.adapter.sessions[1]?.succeedTurn(); + await fixture.collector.waitFor((message) => + turnEvent(message, "turn/completed", child.turnId), + ); + fixture.adapter.sessions[2]?.succeedTurn(); + await fixture.collector.waitFor((message) => + turnEvent(message, "turn/completed", queued.turnId), + ); + expect(fixture.adapter.sessions).toHaveLength(3); + } finally { + registry.close(); + await stopFixture(fixture); + } + }); + + it("counts running native top-level Threads but not native subagents or Fork children", async () => { + const registry = new DelegationControlRegistry({ capacity: { pollIntervalMs: 10 } }); + let registration: DelegationControlRegistration | undefined; + const fixture = createFixture({ + onDelegationApi: (api) => { + registration = api; + return registry.register(api); + }, + }); + const official = (method: string, params: JsonObject) => + fixture.official.stdout.write(`${JSON.stringify({ method, params })}\n`); + try { + await fixture.ready; + writeRequest(fixture.desktopInput, { + id: 900, + method: "codexhost/settings/delegation-capacity/set", + params: { enabled: true, maxRunning: 1 }, + }); + await fixture.collector.waitFor((message) => requestId(message, 900)); + official("thread/started", { + thread: { + id: "native-sub", + source: { subAgent: { thread_spawn: { parent_thread_id: "native-top", depth: 1 } } }, + }, + }); + official("thread/started", { + thread: { + id: "native-fork", + forkedFromId: "native-top", + }, + }); + official("turn/started", { threadId: "native-sub", turn: { id: "sub-turn" } }); + official("turn/started", { threadId: "native-fork", turn: { id: "fork-turn" } }); + official("turn/started", { threadId: "native-top", turn: { id: "top-turn" } }); + await vi.waitFor(() => expect(registration?.runningThreadIds?.()).toEqual(["native-top"])); + + const queued = await registry.start({ + harnessId: "pi", + task: "queued work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(queued).toMatchObject({ status: "queued", queue: { position: 1 } }); + expect(fixture.adapter.sessions).toHaveLength(0); + + // Only the subagent and Fork child keep running, and neither holds the top-level slot. + official("turn/completed", { + threadId: "native-top", + turn: { id: "top-turn", status: "completed" }, + }); + await vi.waitFor(() => expect(fixture.adapter.sessions).toHaveLength(1), { timeout: 5_000 }); + await vi.waitFor(async () => + expect(await registry.read({ threadId: queued.threadId, view: "result" })).toMatchObject({ + status: "running", + }), + ); + expect(registration?.runningThreadIds?.()).toEqual([queued.threadId]); + fixture.adapter.sessions[0]?.succeedTurn(); + await fixture.collector.waitFor((message) => + turnEvent(message, "turn/completed", queued.turnId), + ); + } finally { + registry.close(); + await stopFixture(fixture); + } + }); +}); diff --git a/packages/host-runtime/test/app-server-host.native-delegation-capacity.test.ts b/packages/host-runtime/test/app-server-host.native-delegation-capacity.test.ts new file mode 100644 index 000000000..98c4cc3dd --- /dev/null +++ b/packages/host-runtime/test/app-server-host.native-delegation-capacity.test.ts @@ -0,0 +1,551 @@ +import { createHash } from "node:crypto"; + +import { describe, expect, it, vi } from "vitest"; +import type { JsonObject } from "@codexhost/protocol-core"; + +import type { PersistedQueuedDelegation } from "../src/delegation-capacity-store.js"; +import { DelegationControlRegistry } from "../src/delegation-control-registry.js"; +import type { DelegationControlRegistration } from "../src/delegation-types.js"; +import { + createFixture, + readJsonLine, + requestId, + stopFixture, + writeRequest, +} from "./app-server-host-fixture.js"; + +type Fixture = ReturnType; + +/** + * Answers the Host's requests to the native backend; `turns` holds each Thread's Turn + * IDs, and a Turn start in a Thread listed in `refuse` is answered with an error. + */ +function serveOfficial(fixture: Fixture, threadIds: string[]) { + const seen: JsonObject[] = []; + const turns = new Map(); + const refuse = new Set(); + let stopped = false; + let nextTurn = 0; + const answer = (request: JsonObject): JsonObject => { + const params = (request.params ?? {}) as JsonObject; + const threadId = typeof params.threadId === "string" ? params.threadId : ""; + if (request.method === "turn/start" && refuse.has(threadId)) { + return { error: { code: -32000, message: "synthetic refusal" } }; + } + switch (request.method) { + case "thread/start": { + const id = threadIds.shift(); + if (!id) throw new Error("No synthetic native Thread left"); + return { thread: { id, cwd: "/synthetic" } }; + } + case "thread/read": + return { + thread: { + id: threadId, + cwd: "/synthetic", + ...(params.includeTurns + ? { turns: (turns.get(threadId) ?? []).map((id) => ({ id, status: "inProgress" })) } + : {}), + }, + }; + case "thread/resume": + return { thread: { id: threadId, cwd: "/synthetic" } }; + case "turn/start": { + const id = `turn-${nextTurn++}`; + turns.set(threadId, [...(turns.get(threadId) ?? []), id]); + return { turn: { id, status: "inProgress" } }; + } + default: + return {}; + } + }; + const reply = (request: JsonObject): JsonObject => { + const answered = answer(request); + return "error" in answered + ? { id: request.id ?? null, error: answered.error ?? null } + : { id: request.id ?? null, result: answered }; + }; + const loop = (async () => { + while (!stopped) { + let request: JsonObject; + try { + request = await readJsonLine(fixture.official.stdin, 50); + } catch { + continue; + } + seen.push(request); + if (request.id !== undefined) { + fixture.official.stdout.write(`${JSON.stringify(reply(request))}\n`); + } + } + })(); + return { + seen, + turns, + refuse, + calls: (method: string, threadId: string) => + seen.filter( + (entry) => + entry.method === method && + (entry.params as JsonObject | undefined)?.threadId === threadId, + ), + stop: async () => { + stopped = true; + await loop; + }, + }; +} + +function official(fixture: Fixture, method: string, params: JsonObject): void { + fixture.official.stdout.write(`${JSON.stringify({ method, params })}\n`); +} + +async function limit(fixture: Fixture, maxRunning: number, id = 900): Promise { + writeRequest(fixture.desktopInput, { + id, + method: "codexhost/settings/delegation-capacity/set", + params: { enabled: true, maxRunning }, + }); + await fixture.collector.waitFor((message) => requestId(message, id)); +} + +function setup(store?: { + adopt(): Promise; + save(entries: readonly PersistedQueuedDelegation[]): Promise; +}) { + const registry = new DelegationControlRegistry({ + capacity: { pollIntervalMs: 10, ...(store ? { store } : {}) }, + }); + let registration: DelegationControlRegistration | undefined; + const fixture = createFixture({ + onDelegationApi: (api) => { + registration = api; + return registry.register(api); + }, + }); + return { registry, fixture, running: () => registration?.runningThreadIds?.() ?? [] }; +} + +describe("AppServerHost native delegation capacity", () => { + it("queues top-level native Codex starts in Threads created up front, while starting session-parented child delegations immediately", async () => { + const { registry, fixture, running } = setup(); + const backend = serveOfficial(fixture, [ + "child-native", + "queued-native", + "second-native", + "third-native", + ]); + try { + await fixture.ready; + await limit(fixture, 1); + official(fixture, "turn/started", { threadId: "native-top", turn: { id: "top-turn" } }); + await vi.waitFor(() => expect(running()).toEqual(["native-top"])); + + // A session-parented native child delegation bypasses the top-level queue and does not count toward running(). + const child = await registry.start({ + harnessId: "codex", + task: "child native work", + cwd: "/synthetic", + parentThreadId: "native-top", + }); + expect(child).toMatchObject({ + threadId: "child-native", + harnessId: "codex", + status: "running", + parentThreadId: "native-top", + }); + official(fixture, "turn/started", { threadId: "child-native", turn: { id: "child-turn" } }); + expect(running()).toEqual(["native-top"]); + + const first = await registry.start({ + harnessId: "codex", + task: "queued native work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(first).toMatchObject({ + threadId: "queued-native", + harnessId: "codex", + deepLink: "codex://threads/queued-native", + status: "queued", + queue: { position: 1 }, + }); + expect( + await fixture.mappingStore.getDelegationByChild("queued-native" as never), + ).toMatchObject({ + parentHostThreadId: "client:claude-desktop", + childHostThreadId: "queued-native", + status: "creating", + }); + expect(backend.calls("thread/name/set", "queued-native")).toHaveLength(1); + const second = await registry.start({ + harnessId: "codex", + task: "second native work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(second).toMatchObject({ threadId: "second-native", queue: { position: 2 } }); + // A retry of a waiting start returns it without creating another Thread. + expect( + await registry.start({ + harnessId: "codex", + task: "second native work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }), + ).toMatchObject({ threadId: "second-native", status: "queued", queue: { position: 2 } }); + expect(backend.seen.filter((entry) => entry.method === "thread/start")).toHaveLength(3); + expect(backend.calls("turn/start", "queued-native")).toHaveLength(0); + expect(backend.calls("turn/start", "second-native")).toHaveLength(0); + + expect(await registry.read({ threadId: "queued-native", view: "result" })).toMatchObject({ + status: "queued", + queue: { position: 1 }, + }); + await expect( + registry.send({ threadId: "queued-native", message: "more" }), + ).rejects.toMatchObject({ code: "THREAD_BUSY" }); + + expect(await registry.cancel({ threadId: "second-native" })).toMatchObject({ + cancelled: true, + }); + // Checked for a Turn first; the empty Thread is left in place. + expect(backend.calls("thread/read", "second-native")).toMatchObject([ + { params: { includeTurns: true } }, + ]); + expect(backend.seen.some((entry) => entry.method === "thread/delete")).toBe(false); + expect(await registry.read({ threadId: "second-native", view: "result" })).toMatchObject({ + status: "interrupted", + }); + writeRequest(fixture.desktopInput, { + id: 910, + method: "thread/resume", + params: { + threadId: "second-native", + initialTurnsPage: { limit: 10, itemsView: "full" }, + }, + }); + expect(await fixture.collector.waitFor((message) => requestId(message, 910))).toMatchObject({ + result: { + thread: { + id: "second-native", + turns: [ + { + status: "interrupted", + items: [ + { type: "userMessage", content: [{ text: "second native work" }] }, + { type: "agentMessage" }, + ], + }, + ], + }, + initialTurnsPage: { + data: [ + { + status: "interrupted", + items: [ + { type: "userMessage", content: [{ text: "second native work" }] }, + { type: "agentMessage" }, + ], + }, + ], + }, + }, + }); + + official(fixture, "turn/completed", { + threadId: "native-top", + turn: { id: "top-turn", status: "completed" }, + }); + await vi.waitFor(() => expect(running()).toEqual(["queued-native"]), { timeout: 5_000 }); + expect(backend.calls("thread/resume", "queued-native")).toHaveLength(1); + expect(backend.calls("turn/start", "queued-native")).toMatchObject([ + { + params: { + threadId: "queued-native", + input: [{ type: "text", text: "queued native work" }], + }, + }, + ]); + // The identity returned while queued now reads the native Thread itself. + await vi.waitFor(async () => + expect(await registry.read({ threadId: "queued-native", view: "result" })).toMatchObject({ + threadId: "queued-native", + harnessId: "codex", + status: "running", + }), + ); + expect(await registry.capacity.claim("queued-native", "codex")).toBeUndefined(); + + // The slot is taken again; the next top-level native start waits behind it. + const third = await registry.start({ + harnessId: "codex", + task: "third native work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(third).toMatchObject({ threadId: "third-native", status: "queued" }); + } finally { + await registry.close(); + await backend.stop(); + await stopFixture(fixture); + } + }); + + it("starts the queued task with a user message sent to the waiting Thread, only once", async () => { + const { registry, fixture, running } = setup(); + const backend = serveOfficial(fixture, ["claimed-native"]); + try { + await fixture.ready; + await limit(fixture, 1); + official(fixture, "turn/started", { threadId: "native-top", turn: { id: "top-turn" } }); + await vi.waitFor(() => expect(running()).toEqual(["native-top"])); + const queued = await registry.start({ + harnessId: "codex", + task: "queued native work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(queued).toMatchObject({ threadId: "claimed-native", status: "queued" }); + + writeRequest(fixture.desktopInput, { + id: 950, + method: "turn/start", + params: { threadId: "claimed-native", input: [{ type: "text", text: "user message" }] }, + }); + expect(await fixture.collector.waitFor((message) => requestId(message, 950))).toMatchObject({ + result: { turn: { id: "turn-0" } }, + }); + expect(backend.calls("turn/start", "claimed-native")).toMatchObject([ + { + params: { + input: [ + { type: "text", text: "queued native work" }, + { type: "text", text: "user message" }, + ], + }, + }, + ]); + expect(await registry.capacity.claim("claimed-native", "codex")).toBeUndefined(); + + official(fixture, "turn/completed", { + threadId: "native-top", + turn: { id: "top-turn", status: "completed" }, + }); + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(backend.calls("turn/start", "claimed-native")).toHaveLength(1); + } finally { + await registry.close(); + await backend.stop(); + await stopFixture(fixture); + } + }); + + it("keeps a queued start in its place when the backend refuses the user's Turn", async () => { + const { registry, fixture, running } = setup(); + const backend = serveOfficial(fixture, ["refused-native"]); + try { + await fixture.ready; + await limit(fixture, 1); + official(fixture, "turn/started", { threadId: "native-top", turn: { id: "top-turn" } }); + await vi.waitFor(() => expect(running()).toEqual(["native-top"])); + await registry.start({ + harnessId: "codex", + task: "queued native work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + backend.refuse.add("refused-native"); + writeRequest(fixture.desktopInput, { + id: 960, + method: "turn/start", + params: { threadId: "refused-native", input: [{ type: "text", text: "user message" }] }, + }); + expect(await fixture.collector.waitFor((message) => requestId(message, 960))).toMatchObject({ + error: { message: "synthetic refusal" }, + }); + await vi.waitFor(async () => + expect(await registry.read({ threadId: "refused-native", view: "result" })).toMatchObject({ + status: "queued", + queue: { position: 1 }, + }), + ); + + backend.refuse.clear(); + official(fixture, "turn/completed", { + threadId: "native-top", + turn: { id: "top-turn", status: "completed" }, + }); + await vi.waitFor(() => expect(running()).toEqual(["refused-native"]), { timeout: 5_000 }); + expect(backend.calls("turn/start", "refused-native").at(-1)).toMatchObject({ + params: { input: [{ type: "text", text: "queued native work" }] }, + }); + } finally { + await registry.close(); + await backend.stop(); + await stopFixture(fixture); + } + }); + + it("keeps the user's message when a cancel and the message cross", async () => { + const gate = { block: false, release: (): void => undefined }; + const { registry, fixture, running } = setup({ + adopt: async () => [], + save: async () => { + if (!gate.block) return; + gate.block = false; + await new Promise((resolve) => (gate.release = resolve)); + }, + }); + const backend = serveOfficial(fixture, ["crossed-native"]); + try { + await fixture.ready; + await limit(fixture, 1); + official(fixture, "turn/started", { threadId: "native-top", turn: { id: "top-turn" } }); + await vi.waitFor(() => expect(running()).toEqual(["native-top"])); + await registry.start({ + harnessId: "codex", + task: "queued native work", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + gate.block = true; + const cancelling = registry.cancel({ threadId: "crossed-native" }); + await vi.waitFor(() => expect(gate.block).toBe(false)); + // The cancel has taken the start out of the queue and is storing that. + writeRequest(fixture.desktopInput, { + id: 970, + method: "turn/start", + params: { threadId: "crossed-native", input: [{ type: "text", text: "user message" }] }, + }); + expect(await fixture.collector.waitFor((message) => requestId(message, 970))).toMatchObject({ + result: { turn: { id: "turn-0" } }, + }); + gate.release(); + expect(await cancelling).toMatchObject({ cancelled: true }); + + expect(backend.calls("turn/start", "crossed-native")).toMatchObject([ + { params: { input: [{ type: "text", text: "user message" }] } }, + ]); + expect(backend.seen.some((entry) => entry.method === "thread/delete")).toBe(false); + expect(backend.turns.get("crossed-native")).toEqual(["turn-0"]); + // The delegation was cancelled; the Thread reads as what it holds, the user's Turn. + expect(await registry.read({ threadId: "crossed-native", view: "result" })).toMatchObject({ + turn: { turnId: "turn-0" }, + }); + expect(await registry.capacity.claim("crossed-native", "codex")).toBeUndefined(); + } finally { + gate.release(); + await registry.close(); + await backend.stop(); + await stopFixture(fixture); + } + }); + + it("adopts a restored native start whose Turn already began, and starts the rest", async () => { + const saved: (readonly PersistedQueuedDelegation[])[] = []; + const { registry, fixture, running } = setup({ + adopt: async () => [ + stored("begun-native", "begun work", true), + stored("waiting-native", "waiting work"), + ], + save: async (entries) => { + saved.push(entries); + }, + }); + const backend = serveOfficial(fixture, []); + backend.turns.set("begun-native", ["begun-turn"]); + try { + await fixture.ready; + // Read before the setting arrives: the Thread is checked, not reported as waiting. + const begun = await registry.read({ threadId: "begun-native", view: "result" }); + expect(begun).toMatchObject({ threadId: "begun-native", turn: { turnId: "begun-turn" } }); + expect(begun).not.toHaveProperty("queue"); + expect(await registry.read({ threadId: "waiting-native", view: "result" })).toMatchObject({ + status: "queued", + queue: { position: 1 }, + }); + await limit(fixture, 4); + await vi.waitFor(() => expect(running()).toEqual(["waiting-native"]), { timeout: 5_000 }); + expect(backend.calls("thread/resume", "begun-native")).toHaveLength(1); + expect(backend.calls("turn/start", "begun-native")).toHaveLength(0); + expect(backend.calls("turn/start", "waiting-native")).toMatchObject([ + { params: { input: [{ type: "text", text: "waiting work" }] } }, + ]); + await vi.waitFor(() => expect(saved.at(-1)).toEqual([])); + } finally { + await registry.close(); + await backend.stop(); + await stopFixture(fixture); + } + }); + + it("cancels a restored native start whose Turn began as a started one, keeping its Thread", async () => { + const { registry, fixture } = setup({ + adopt: async () => [stored("begun-native", "begun work", true)], + save: async () => undefined, + }); + const backend = serveOfficial(fixture, []); + backend.turns.set("begun-native", ["begun-turn"]); + try { + await fixture.ready; + expect(await registry.cancel({ threadId: "begun-native" })).toMatchObject({ + threadId: "begun-native", + turnId: "begun-turn", + cancelled: true, + }); + expect(backend.calls("turn/interrupt", "begun-native")).toHaveLength(1); + expect(backend.seen.some((entry) => entry.method === "thread/delete")).toBe(false); + expect(backend.calls("turn/start", "begun-native")).toHaveLength(0); + } finally { + await registry.close(); + await backend.stop(); + await stopFixture(fixture); + } + }); + + it("sends a user message to a restored native start whose Turn began without the task", async () => { + const { registry, fixture } = setup({ + adopt: async () => [stored("begun-native", "begun work", true)], + save: async () => undefined, + }); + const backend = serveOfficial(fixture, []); + backend.turns.set("begun-native", ["begun-turn"]); + try { + await fixture.ready; + writeRequest(fixture.desktopInput, { + id: 980, + method: "turn/start", + params: { threadId: "begun-native", input: [{ type: "text", text: "user follow-up" }] }, + }); + await fixture.collector.waitFor((message) => requestId(message, 980)); + expect(backend.calls("turn/start", "begun-native")).toMatchObject([ + { params: { input: [{ type: "text", text: "user follow-up" }] } }, + ]); + expect(await registry.capacity.claim("begun-native", "codex")).toBeUndefined(); + expect(await registry.read({ threadId: "begun-native", view: "result" })).toMatchObject({ + status: "running", + }); + } finally { + await registry.close(); + await backend.stop(); + await stopFixture(fixture); + } + }); +}); + +function stored(threadId: string, task: string, starting?: boolean): PersistedQueuedDelegation { + return { + delegationId: `delegation-${threadId}`, + threadId, + turnId: "pending", + harnessId: "codex", + task, + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + title: task, + taskDigest: createHash("sha256").update(task).digest("hex"), + queuedAt: Date.now(), + ...(starting ? { starting } : {}), + }; +} diff --git a/packages/host-runtime/test/app-server-host.projection-2.test.ts b/packages/host-runtime/test/app-server-host.projection-2.test.ts index 4278b14ce..1a790e2a7 100644 --- a/packages/host-runtime/test/app-server-host.projection-2.test.ts +++ b/packages/host-runtime/test/app-server-host.projection-2.test.ts @@ -1512,11 +1512,6 @@ describe("AppServerHost HarnessAdapter projection", () => { fixture.official.stdout.write( `${JSON.stringify({ id: threadStart.id, result: { thread: { id: "native-child" } } })}\n`, ); - await answerOfficialChildName(fixture); - const turnStart = await readJsonLine(fixture.official.stdin); - fixture.official.stdout.write( - `${JSON.stringify({ id: turnStart.id, result: { turn: { id: "native-turn" } } })}\n`, - ); const deletion = await readJsonLine(fixture.official.stdin); expect(deletion).toMatchObject({ method: "thread/delete", diff --git a/packages/host-runtime/test/app-server-host.watch.test.ts b/packages/host-runtime/test/app-server-host.watch.test.ts index c2a355e0d..a50ff0250 100644 --- a/packages/host-runtime/test/app-server-host.watch.test.ts +++ b/packages/host-runtime/test/app-server-host.watch.test.ts @@ -1,14 +1,124 @@ +import { rmSync } from "node:fs"; import { describe, expect, it, vi } from "vitest"; import type { JsonObject } from "@codexhost/protocol-core"; -import type { DelegationControlApi, DelegationThreadSnapshot } from "../src/delegation-types.js"; +import type { + DelegationControlApi, + DelegationControlRegistration, + DelegationThreadSnapshot, +} from "../src/delegation-types.js"; import { DelegationWatchService } from "../src/delegation-watch.js"; +import { ThreadActivity } from "../src/thread-activity.js"; import { bindOfficialThread, createFixture, readJsonLine, + startPiThread, + startPiTurn, stopFixture, } from "./app-server-host-fixture.js"; +describe("watch activity from Host Harness streams", () => { + it("publishes native tool events and external subagent events through the registration", async () => { + let registration: DelegationControlRegistration | undefined; + const fixture = createFixture({ + onDelegationApi: (api) => { + registration = api; + return undefined; + }, + }); + try { + await fixture.ready; + if (!registration?.subscribeActivity) throw new Error("Missing activity subscription"); + const activity = vi.fn(); + const unsubscribe = registration.subscribeActivity(activity); + fixture.official.stdout.write( + `${JSON.stringify({ method: "item/commandExecution/outputDelta", params: { threadId: "native", delta: "output" } })}\n`, + ); + await vi.waitFor(() => expect(activity).toHaveBeenCalledWith(["native"])); + const threadId = await startPiThread(fixture); + await startPiTurn(fixture, threadId); + const session = fixture.adapter.sessions[0]; + if (!session) throw new Error("Missing fake Harness session"); + activity.mockClear(); + session.emitSubagentState("worker", "running"); + await vi.waitFor(() => expect(activity).toHaveBeenCalledWith([threadId])); + unsubscribe(); + activity.mockClear(); + session.emitSubagentTranscriptChanged("worker"); + await new Promise((resolve) => setImmediate(resolve)); + expect(activity).not.toHaveBeenCalled(); + } finally { + await stopFixture(fixture); + } + }); + + it("forgets native subagent ancestry when the official Thread closes", async () => { + let registration: DelegationControlRegistration | undefined; + const fixture = createFixture({ + onDelegationApi: (api) => { + registration = api; + return undefined; + }, + }); + const official = (message: JsonObject) => + fixture.official.stdout.write(`${JSON.stringify(message)}\n`); + const progress = () => + official({ + method: "item/agentMessage/delta", + params: { threadId: "native-child", delta: "progress" }, + }); + try { + await fixture.ready; + if (!registration?.subscribeActivity) throw new Error("Missing activity subscription"); + const activity = vi.fn(); + registration.subscribeActivity(activity); + official({ + method: "thread/started", + params: { + thread: { + id: "native-child", + source: { subAgent: { thread_spawn: { parent_thread_id: "native-parent" } } }, + }, + }, + }); + progress(); + await vi.waitFor(() => + expect(activity).toHaveBeenLastCalledWith(["native-child", "native-parent"]), + ); + official({ method: "thread/closed", params: { threadId: "native-child" } }); + progress(); + await vi.waitFor(() => expect(activity).toHaveBeenLastCalledWith(["native-child"])); + } finally { + await stopFixture(fixture); + } + }); + + it("releases ancestry when an external session ends and when the official backend stops", async () => { + const forget = vi.spyOn(ThreadActivity.prototype, "forget"); + const clear = vi.spyOn(ThreadActivity.prototype, "clear"); + const fixture = createFixture(); + try { + const threadId = await startPiThread(fixture); + await startPiTurn(fixture, threadId); + const session = fixture.adapter.sessions[0]; + if (!session) throw new Error("Missing fake Harness session"); + expect(forget).not.toHaveBeenCalled(); + await session.close(); + await vi.waitFor(() => expect(forget).toHaveBeenCalledWith(threadId)); + + expect(clear).not.toHaveBeenCalled(); + fixture.official.emit("exit", 1, null); + await vi.waitFor(() => expect(clear).toHaveBeenCalled()); + } finally { + forget.mockRestore(); + clear.mockRestore(); + fixture.host.close(); + await fixture.running; + rmSync(fixture.mappingStoreDirectory, { recursive: true, force: true }); + } + }); +}); + async function watchFixture() { let api: DelegationControlApi | undefined; const fixture = createFixture({ diff --git a/packages/host-runtime/test/delegation-capacity-queue.test.ts b/packages/host-runtime/test/delegation-capacity-queue.test.ts new file mode 100644 index 000000000..3eaae9e08 --- /dev/null +++ b/packages/host-runtime/test/delegation-capacity-queue.test.ts @@ -0,0 +1,693 @@ +import { harnessModelRefSchema, harnessThinkingOptionIdSchema } from "@codexhost/shared-contracts"; +import { afterEach, describe, expect, it, vi } from "vitest"; + +import { + DelegationCapacityQueue, + type DelegationCapacityRuntime, +} from "../src/delegation-capacity-queue.js"; +import type { + DelegationQueueStore, + PersistedQueuedDelegation, +} from "../src/delegation-capacity-store.js"; +import type { + DelegationAdmission, + DelegationAdmissionRequest, + DelegationStartIdentity, + DelegationStartInput, +} from "../src/delegation-types.js"; + +let sequence = 0; + +function request( + patch: Partial = {}, + taskDigest?: string, +): DelegationAdmissionRequest { + sequence += 1; + return { + input: { + harnessId: "pi", + task: `task ${sequence}`, + cwd: "/work", + parentThreadId: "parent", + ...patch, + }, + title: `Task ${sequence}`, + taskDigest: taskDigest ?? `digest-${sequence}`, + identity: { + delegationId: `delegation-${sequence}`, + threadId: `child-${sequence}`, + turnId: `turn-${sequence}`, + }, + }; +} + +function record(patch: Partial = {}): PersistedQueuedDelegation { + return { + delegationId: "delegation-stored", + threadId: "child-stored", + turnId: "turn-stored", + harnessId: "pi", + task: "stored task", + cwd: "/work", + parentThreadId: "parent", + title: "Stored", + taskDigest: "digest-stored", + queuedAt: 1, + ...patch, + }; +} + +/** Sessions whose started Threads keep running until a test removes them. */ +function sessions() { + const running = new Set(); + const starts: { input: DelegationStartInput; identity: DelegationStartIdentity }[] = []; + const hold = { next: false }; + const held = new Map(); + const runtime: DelegationCapacityRuntime = { + runningThreadIds: () => running, + canStart: () => true, + start: vi.fn(async (input: DelegationStartInput, identity: DelegationStartIdentity) => { + starts.push({ input, identity }); + if (hold.next) { + hold.next = false; + await new Promise((resolve, reject) => { + held.set(identity.threadId, { resolve, reject }); + }); + } + running.add(identity.threadId); + }), + }; + const started = () => starts.map((start) => start.identity.threadId); + return { running, starts, hold, held, runtime, started }; +} + +class MemoryQueueStore implements DelegationQueueStore { + saved: PersistedQueuedDelegation[] = []; + saves = 0; + #adoptable: PersistedQueuedDelegation[]; + + constructor(adoptable: PersistedQueuedDelegation[] = []) { + this.#adoptable = adoptable; + } + + async adopt() { + const adopted = this.#adoptable; + this.#adoptable = []; + return adopted; + } + + async save(entries: readonly PersistedQueuedDelegation[]) { + this.saves += 1; + this.saved = entries.map((entry) => ({ ...entry })); + } +} + +function queued(admission: DelegationAdmission) { + if (admission.state !== "queued") throw new Error("expected a queued start"); + return admission; +} + +function admitted(admission: DelegationAdmission) { + if (admission.state !== "admitted") throw new Error("expected an admitted start"); + return admission; +} + +const flush = async () => { + for (let index = 0; index < 10; index += 1) await Promise.resolve(); +}; + +describe("DelegationCapacityQueue", () => { + const queues: DelegationCapacityQueue[] = []; + const track = (queue: DelegationCapacityQueue) => { + queues.push(queue); + return queue; + }; + afterEach(() => { + for (const queue of queues.splice(0)) queue.close(); + vi.useRealTimers(); + }); + + it("admits every start at once while the limit is off or not yet delivered", async () => { + const value = sessions(); + for (let index = 0; index < 8; index += 1) value.running.add(`manual-${index}`); + const store = new MemoryQueueStore(); + const queue = track(new DelegationCapacityQueue(value.runtime, { store })); + + const undelivered = admitted(await queue.admit(request())); + undelivered.release(); + undelivered.release(); + expect(queue.configure({ enabled: false, maxRunning: 1 })).toEqual({ + enabled: false, + maxRunning: 1, + }); + const starts = await Promise.all([1, 2, 3].map(() => queue.admit(request()))); + expect(starts.map((start) => start.state)).toEqual(["admitted", "admitted", "admitted"]); + expect(await queue.list({})).toEqual([]); + expect(value.runtime.start).not.toHaveBeenCalled(); + expect(store.saves).toBe(0); + }); + + it("rejects invalid settings", () => { + const queue = track(new DelegationCapacityQueue(sessions().runtime)); + for (const maxRunning of [0, 33, 1.5]) { + expect(() => queue.configure({ enabled: true, maxRunning })).toThrow(); + } + }); + + it("queues starts beyond the limit in arrival order, counting manual Threads and starts in flight", async () => { + const value = sessions(); + value.running.add("manual"); + const store = new MemoryQueueStore(); + const queue = track(new DelegationCapacityQueue(value.runtime, { store })); + queue.configure({ enabled: true, maxRunning: 2 }); + + const first = admitted(await queue.admit(request())); + const secondRequest = request({ taskName: "Second" }); + const second = queued(await queue.admit(secondRequest)); + const thirdRequest = request({ parentThreadId: "other", cwd: "/other" }); + const third = queued(await queue.admit(thirdRequest)); + expect(second).toEqual({ + state: "queued", + identity: secondRequest.identity, + status: "queued", + position: 1, + parentThreadId: "parent", + cwd: "/work", + title: secondRequest.title, + }); + expect(third).toMatchObject({ identity: thirdRequest.identity, position: 2 }); + expect(store.saved).toEqual([ + expect.objectContaining({ ...secondRequest.identity, taskName: "Second", harnessId: "pi" }), + expect.objectContaining(thirdRequest.identity), + ]); + + expect( + await queue.read({ threadId: thirdRequest.identity.threadId, view: "result" }), + ).toMatchObject({ + threadId: thirdRequest.identity.threadId, + harnessId: "pi", + title: thirdRequest.title, + status: "queued", + queue: { position: 2 }, + turn: null, + result: { availability: "pending" }, + }); + expect(await queue.peek(secondRequest.identity.threadId)).toEqual({ + status: "queued", + position: 1, + }); + expect( + (await queue.list({ parentThreadId: "parent" })).map(({ threadId, status, queue }) => ({ + threadId, + status, + queue, + })), + ).toEqual([ + { threadId: secondRequest.identity.threadId, status: "queued", queue: { position: 1 } }, + ]); + expect((await queue.list({ cwd: "/other/." })).map((row) => row.threadId)).toEqual([ + thirdRequest.identity.threadId, + ]); + expect(await queue.list({})).toHaveLength(2); + + // A slot that frees while others wait goes to the head of the queue, never to a newcomer. + first.release(); + await flush(); + expect(value.started()).toEqual([secondRequest.identity.threadId]); + expect(queued(await queue.admit(request())).position).toBe(2); + }); + + it("starts queued work in order as running Threads finish", async () => { + vi.useFakeTimers(); + const value = sessions(); + value.running.add("busy"); + const queue = track(new DelegationCapacityQueue(value.runtime, { pollIntervalMs: 1_000 })); + queue.configure({ enabled: true, maxRunning: 1 }); + const model = harnessModelRefSchema.parse({ id: "model-a" }); + const firstRequest = request({ model, requestId: "first" }); + queued(await queue.admit(firstRequest)); + const secondRequest = request(); + queued(await queue.admit(secondRequest)); + + await vi.advanceTimersByTimeAsync(5_000); + expect(value.runtime.start).not.toHaveBeenCalled(); + + value.running.delete("busy"); + await vi.advanceTimersByTimeAsync(1_000); + expect(value.starts).toEqual([{ input: firstRequest.input, identity: firstRequest.identity }]); + expect( + await queue.read({ threadId: firstRequest.identity.threadId, view: "result" }), + ).toBeUndefined(); + expect(await queue.peek(secondRequest.identity.threadId)).toEqual({ + status: "queued", + position: 1, + }); + + await vi.advanceTimersByTimeAsync(5_000); + expect(value.started()).toEqual([firstRequest.identity.threadId]); + value.running.delete(firstRequest.identity.threadId); + await vi.advanceTimersByTimeAsync(1_000); + expect(value.started()).toEqual([ + firstRequest.identity.threadId, + secondRequest.identity.threadId, + ]); + expect(await queue.list({})).toEqual([]); + }); + + it("starts what fits when the limit is raised, and everything when it is turned off", async () => { + const value = sessions(); + value.running.add("busy"); + const queue = track(new DelegationCapacityQueue(value.runtime)); + queue.configure({ enabled: true, maxRunning: 1 }); + const waiting: string[] = []; + for (let index = 0; index < 4; index += 1) { + const next = request(); + queued(await queue.admit(next)); + waiting.push(next.identity.threadId); + } + queue.configure({ enabled: true, maxRunning: 3 }); + await flush(); + expect(value.started()).toEqual(waiting.slice(0, 2)); + queue.configure({ enabled: false, maxRunning: 3 }); + await flush(); + expect(value.started()).toEqual(waiting); + }); + + it("reports a queued start that fails to begin as failed", async () => { + const value = sessions(); + value.running.add("busy"); + const diagnose = vi.fn(); + const store = new MemoryQueueStore(); + const queue = track(new DelegationCapacityQueue(value.runtime, { store, diagnose })); + queue.configure({ enabled: true, maxRunning: 1 }); + const entry = request(); + queued(await queue.admit(entry)); + value.hold.next = true; + value.running.delete("busy"); + queue.refresh(); + await flush(); + const read = { threadId: entry.identity.threadId, view: "result" as const }; + expect(await queue.read(read)).toMatchObject({ status: "creating" }); + expect(await queue.peek(entry.identity.threadId)).toEqual({ status: "creating" }); + expect(store.saved).toEqual([expect.objectContaining({ starting: true })]); + + const failure = new Error("Harness is unavailable"); + value.held.get(entry.identity.threadId)?.reject(failure); + await flush(); + expect(await queue.read(read)).toBeUndefined(); + expect(await queue.finished(read)).toMatchObject({ + status: "failed", + turn: null, + result: { availability: "unavailable", message: "Harness is unavailable" }, + }); + expect(diagnose).toHaveBeenCalledWith(failure); + expect(store.saved).toEqual([]); + }); + + it("restores the stored queue after a restart and resumes it once the limit arrives", async () => { + const before = sessions(); + before.running.add("busy"); + const store = new MemoryQueueStore(); + const first = track(new DelegationCapacityQueue(before.runtime, { store })); + first.configure({ enabled: true, maxRunning: 1 }); + const thinkingOptionId = harnessThinkingOptionIdSchema.parse("high"); + const head = request({ requestId: "retry" }, "digest-retry"); + const tail = request({ thinkingOptionId }); + queued(await first.admit(head)); + queued(await first.admit(tail)); + first.close(); + + const after = sessions(); + const diagnose = vi.fn(); + const restoredStore = new MemoryQueueStore(store.saved); + const restored = track( + new DelegationCapacityQueue(after.runtime, { store: restoredStore, diagnose }), + ); + expect(await restored.peek(head.identity.threadId)).toEqual({ status: "queued", position: 1 }); + expect(await restored.read({ threadId: tail.identity.threadId, view: "result" })).toMatchObject( + { + status: "queued", + queue: { position: 2 }, + }, + ); + expect(diagnose).toHaveBeenCalledWith("restored 2 queued delegation(s) after a restart"); + // Until the Renderer delivers the limit again, nothing restored starts and nothing overtakes it. + const newcomer = request(); + expect(queued(await restored.admit(newcomer)).position).toBe(3); + // A retried start keeps its Thread across the restart. + expect( + queued(await restored.admit(request({ requestId: "retry" }, "digest-retry"))).identity, + ).toEqual(head.identity); + await flush(); + expect(after.runtime.start).not.toHaveBeenCalled(); + + restored.configure({ enabled: true, maxRunning: 2 }); + await flush(); + expect(after.starts).toEqual([ + { input: head.input, identity: head.identity }, + { input: tail.input, identity: tail.identity }, + ]); + expect(restoredStore.saved.map((entry) => entry.threadId)).toEqual([ + newcomer.identity.threadId, + ]); + }); + + it("reports a start cut off by the restart as interrupted instead of starting it twice", async () => { + const value = sessions(); + const store = new MemoryQueueStore([record({ starting: true })]); + const queue = track(new DelegationCapacityQueue(value.runtime, { store })); + queue.configure({ enabled: false, maxRunning: 4 }); + await flush(); + expect(value.runtime.start).not.toHaveBeenCalled(); + const read = { threadId: "child-stored", view: "result" as const }; + expect(await queue.read(read)).toBeUndefined(); + expect(await queue.finished(read)).toMatchObject({ + status: "interrupted", + title: "Stored", + result: { availability: "unavailable", message: expect.stringContaining("restarted") }, + }); + expect(store.saves).toBe(1); + expect(store.saved).toEqual([]); + }); + + it("starts the restored queue as if the limit were off when no setting arrives", async () => { + vi.useFakeTimers(); + const value = sessions(); + const diagnose = vi.fn(); + const queue = track( + new DelegationCapacityQueue(value.runtime, { + store: new MemoryQueueStore([record()]), + diagnose, + settingsGraceMs: 60_000, + }), + ); + await vi.advanceTimersByTimeAsync(59_000); + expect(value.runtime.start).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1_000); + expect(value.started()).toEqual(["child-stored"]); + expect(diagnose).toHaveBeenCalledWith( + "no delegation capacity setting arrived; starting the restored queue", + ); + expect(await queue.list({})).toEqual([]); + }); + + it("removes a waiting start on cancel and reports it interrupted", async () => { + const value = sessions(); + value.running.add("busy"); + const store = new MemoryQueueStore(); + const queue = track(new DelegationCapacityQueue(value.runtime, { store })); + queue.configure({ enabled: true, maxRunning: 1 }); + const first = request(); + const second = request(); + queued(await queue.admit(first)); + queued(await queue.admit(second)); + + expect(await queue.cancel(first.identity.threadId)).toEqual({ + threadId: first.identity.threadId, + turnId: null, + harnessId: "pi", + cancelled: true, + }); + expect(await queue.peek(second.identity.threadId)).toEqual({ status: "queued", position: 1 }); + expect(store.saved.map((entry) => entry.threadId)).toEqual([second.identity.threadId]); + expect( + await queue.finished({ threadId: first.identity.threadId, view: "result" }), + ).toMatchObject({ + status: "interrupted", + result: { availability: "unavailable", message: "Removed from the queue before it started" }, + }); + expect(await queue.cancel(first.identity.threadId)).toBeUndefined(); + + value.running.delete("busy"); + queue.refresh(); + await flush(); + expect(value.started()).toEqual([second.identity.threadId]); + }); + + it("leaves a start already under way to its session when cancelled", async () => { + const value = sessions(); + value.running.add("busy"); + const queue = track(new DelegationCapacityQueue(value.runtime)); + queue.configure({ enabled: true, maxRunning: 1 }); + const entry = request(); + queued(await queue.admit(entry)); + value.hold.next = true; + value.running.delete("busy"); + queue.refresh(); + await flush(); + let settled = false; + const cancelling = queue.cancel(entry.identity.threadId).then((result) => { + settled = true; + return result; + }); + await flush(); + expect(settled).toBe(false); + value.held.get(entry.identity.threadId)?.resolve(); + await expect(cancelling).resolves.toBeUndefined(); + expect( + await queue.finished({ threadId: entry.identity.threadId, view: "result" }), + ).toBeUndefined(); + }); + + it("returns the waiting entry for a retried start and rejects a conflicting request ID", async () => { + const value = sessions(); + value.running.add("busy"); + const queue = track(new DelegationCapacityQueue(value.runtime)); + queue.configure({ enabled: true, maxRunning: 1 }); + const original = request({ requestId: "same" }, "digest-a"); + queued(await queue.admit(original)); + expect(queued(await queue.admit(request({ requestId: "same" }, "digest-a"))).identity).toEqual( + original.identity, + ); + await expect(queue.admit(request({ requestId: "same" }, "digest-b"))).rejects.toMatchObject({ + code: "INVALID_ARGUMENT", + }); + const implicit = request({}, "digest-c"); + queued(await queue.admit(implicit)); + expect(queued(await queue.admit(request({}, "digest-c"))).identity).toEqual(implicit.identity); + expect(queued(await queue.admit(request({}, "digest-d"))).position).toBe(3); + }); + + it("settles a wait when the start leaves the queue, or reports the timeout", async () => { + const value = sessions(); + value.running.add("busy"); + const queue = track(new DelegationCapacityQueue(value.runtime)); + queue.configure({ enabled: true, maxRunning: 1 }); + const entry = request(); + queued(await queue.admit(entry)); + await expect(queue.settled(entry.identity.threadId, 10)).resolves.toBe(false); + const settling = queue.settled(entry.identity.threadId, 60_000); + value.running.delete("busy"); + queue.refresh(); + await expect(settling).resolves.toBe(true); + await expect(queue.settled("unknown", 10)).resolves.toBe(true); + }); + + it("stops starting work once closed and keeps the stored queue", async () => { + const value = sessions(); + value.running.add("busy"); + const store = new MemoryQueueStore(); + const queue = new DelegationCapacityQueue(value.runtime, { store }); + queue.configure({ enabled: true, maxRunning: 1 }); + const entry = request(); + queued(await queue.admit(entry)); + queue.close(); + value.running.delete("busy"); + queue.refresh(); + await flush(); + expect(value.runtime.start).not.toHaveBeenCalled(); + expect(store.saved.map((saved) => saved.threadId)).toEqual([entry.identity.threadId]); + await expect(queue.admit(request())).rejects.toMatchObject({ code: "INTERNAL_ERROR" }); + }); + + it("prepares a Harness-created Thread only for a start that must wait, keeping arrival order", async () => { + const value = sessions(); + value.running.add("busy"); + const queue = track(new DelegationCapacityQueue(value.runtime)); + queue.configure({ enabled: true, maxRunning: 2 }); + const prepared: string[] = []; + const native = (threadId: string, task: string) => { + const base = request({ harnessId: "codex", task }, task); + return { + ...base, + prepare: vi.fn(async () => { + // A slow backend must not let a later start take this one's place. + await new Promise((resolve) => setTimeout(resolve, threadId === "native-a" ? 20 : 0)); + prepared.push(threadId); + return { ...base.identity, threadId }; + }), + }; + }; + const now = native("unused", "now"); + admitted(await queue.admit(now)); + expect(now.prepare).not.toHaveBeenCalled(); + const [first, second] = await Promise.all([ + queue.admit(native("native-a", "a")), + queue.admit(native("native-b", "b")), + ]); + expect([queued(first).identity.threadId, queued(second).identity.threadId]).toEqual([ + "native-a", + "native-b", + ]); + expect((await queue.list({})).map((entry) => entry.threadId)).toEqual(["native-a", "native-b"]); + // A retry of a waiting start is answered from the queue without preparing again. + const retry = native("native-c", "a"); + expect(queued(await queue.admit(retry)).identity.threadId).toBe("native-a"); + expect(retry.prepare).not.toHaveBeenCalled(); + expect(prepared).toEqual(["native-a", "native-b"]); + }); + + it("holds a claimed start in its place until the Turn start settles", async () => { + const value = sessions(); + value.running.add("busy"); + const adopt = vi.fn(async () => false); + const store = new MemoryQueueStore(); + const queue = track(new DelegationCapacityQueue({ ...value.runtime, adopt }, { store })); + queue.configure({ enabled: true, maxRunning: 1 }); + const claimed = request({ harnessId: "codex" }); + const taken = request({ harnessId: "codex" }); + const behind = request(); + for (const entry of [claimed, taken, behind]) queued(await queue.admit(entry)); + const { threadId } = claimed.identity; + + expect(await queue.claim(threadId, "pi")).toBeUndefined(); + const claim = await queue.claim(threadId, "codex"); + expect(claim).toMatchObject({ + identity: claimed.identity, + input: { ...claimed.input }, + taskDigest: claimed.taskDigest, + }); + expect(await queue.claim(threadId, "codex")).toBeUndefined(); + expect(await queue.peek(threadId)).toEqual({ status: "creating" }); + expect(await queue.peek(taken.identity.threadId)).toEqual({ status: "queued", position: 1 }); + // A crash during the takeover leaves the start to be checked for a Turn. + expect(store.saved.find((saved) => saved.threadId === threadId)?.starting).toBe(true); + + let cancelled = false; + const cancelling = queue.cancel(threadId).then((result) => { + cancelled = true; + return result; + }); + await flush(); + expect(cancelled).toBe(false); + await claim?.settle("rejected"); + expect(store.saved.find((saved) => saved.threadId === threadId)?.starting).toBeUndefined(); + // The rejected takeover kept the start's place, and the waiting cancel goes on from there. + expect(await cancelling).toMatchObject({ cancelled: true }); + expect(adopt).toHaveBeenCalledWith(expect.objectContaining({ identity: claimed.identity })); + + const second = await queue.claim(taken.identity.threadId, "codex"); + await second?.settle("started"); + expect(await queue.peek(taken.identity.threadId)).toBeUndefined(); + // Taken-over work is the session's now, not a start that never happened. + expect(await queue.finished({ threadId: taken.identity.threadId, view: "result" })).toBe( + undefined, + ); + expect(store.saved.map((saved) => saved.threadId)).toEqual([behind.identity.threadId]); + expect(value.runtime.start).not.toHaveBeenCalled(); + }); + + it("returns a takeover with an unknown outcome to its place, to be checked first", async () => { + const value = sessions(); + value.running.add("busy"); + const adopt = vi.fn(async () => true); + const queue = track(new DelegationCapacityQueue({ ...value.runtime, adopt })); + queue.configure({ enabled: true, maxRunning: 1 }); + const entry = request({ harnessId: "codex" }); + queued(await queue.admit(entry)); + await (await queue.claim(entry.identity.threadId, "codex"))?.settle("unknown"); + // Its Turn did begin, so a second message joins that Turn instead of repeating the task. + expect(await queue.claim(entry.identity.threadId, "codex")).toBeUndefined(); + expect(adopt).toHaveBeenCalledOnce(); + expect(await queue.peek(entry.identity.threadId)).toBeUndefined(); + }); + + it("leaves a cancelled start whose Thread has a Turn to its session", async () => { + const value = sessions(); + value.running.add("busy"); + const begun = request({ harnessId: "codex" }); + const empty = request({ harnessId: "codex" }); + const adopt = vi.fn( + async (queuedStart: { identity: DelegationStartIdentity }) => + queuedStart.identity.threadId === begun.identity.threadId, + ); + const queue = track(new DelegationCapacityQueue({ ...value.runtime, adopt })); + queue.configure({ enabled: true, maxRunning: 1 }); + for (const entry of [begun, empty]) queued(await queue.admit(entry)); + + expect(await queue.cancel(begun.identity.threadId)).toBeUndefined(); + expect(await queue.finished({ threadId: begun.identity.threadId, view: "result" })).toBe( + undefined, + ); + expect(await queue.cancel(empty.identity.threadId)).toMatchObject({ cancelled: true }); + expect( + await queue.finished({ threadId: empty.identity.threadId, view: "result" }), + ).toMatchObject({ status: "interrupted" }); + expect(adopt).toHaveBeenCalledTimes(2); + }); + + it("checks a native start cut off by a restart for a Turn before using it", async () => { + const value = sessions(); + const store = new MemoryQueueStore([ + record({ threadId: "native-begun", harnessId: "codex", starting: true }), + record({ threadId: "native-empty", harnessId: "codex", starting: true }), + record({ threadId: "pi-starting", starting: true }), + record({ threadId: "native-waiting", harnessId: "codex" }), + ]); + const adopt = vi.fn( + async (queuedStart: { identity: DelegationStartIdentity }) => + queuedStart.identity.threadId === "native-begun", + ); + const queue = track(new DelegationCapacityQueue({ ...value.runtime, adopt }, { store })); + expect((await queue.list({})).map((entry) => entry.threadId)).toEqual([ + "native-begun", + "native-empty", + "native-waiting", + ]); + expect(await queue.finished({ threadId: "pi-starting", view: "result" })).toMatchObject({ + status: "interrupted", + }); + // Still unknown on disk until checked. + expect(store.saved.filter((saved) => saved.starting).map((saved) => saved.threadId)).toEqual([ + "native-begun", + "native-empty", + ]); + + // Its Turn began before the restart: it is the session's now, not a waiting start. + expect(await queue.read({ threadId: "native-begun", view: "result" })).toBeUndefined(); + const claim = await queue.claim("native-empty", "codex"); + expect(claim?.identity.threadId).toBe("native-empty"); + await claim?.settle("rejected"); + expect(store.saved.some((saved) => saved.starting)).toBe(false); + expect(adopt.mock.calls.map(([queuedStart]) => queuedStart.identity.threadId)).toEqual([ + "native-begun", + "native-empty", + ]); + + queue.configure({ enabled: true, maxRunning: 4 }); + await vi.waitFor(() => expect(value.started()).toEqual(["native-empty", "native-waiting"])); + }); + + it("stores a start that fails after close as one to check, not as failed", async () => { + const value = sessions(); + const store = new MemoryQueueStore(); + const queue = new DelegationCapacityQueue(value.runtime, { store }); + value.running.add("busy"); + queue.configure({ enabled: true, maxRunning: 1 }); + const entry = request({ harnessId: "codex" }); + queued(await queue.admit(entry)); + value.hold.next = true; + value.running.delete("busy"); + queue.refresh(); + await vi.waitFor(() => expect(value.held.has(entry.identity.threadId)).toBe(true)); + await queue.close(); + value.held.get(entry.identity.threadId)?.reject(new Error("backend closed")); + await vi.waitFor(() => expect(store.saves).toBeGreaterThan(2)); + expect(store.saved).toEqual([ + expect.objectContaining({ threadId: entry.identity.threadId, starting: true }), + ]); + expect(await queue.finished({ threadId: entry.identity.threadId, view: "result" })).toBe( + undefined, + ); + }); +}); diff --git a/packages/host-runtime/test/delegation-capacity-registry.test.ts b/packages/host-runtime/test/delegation-capacity-registry.test.ts new file mode 100644 index 000000000..b405bc870 --- /dev/null +++ b/packages/host-runtime/test/delegation-capacity-registry.test.ts @@ -0,0 +1,388 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; + +import { afterEach, describe, expect, it, vi } from "vitest"; + +import { FileDelegationQueueStore } from "../src/delegation-capacity-store.js"; +import { DelegationControlRegistry } from "../src/delegation-control-registry.js"; +import { + DelegationControlError, + type DelegationCapacityGate, + type DelegationControlRegistration, + type DelegationStartIdentity, + type DelegationStartInput, + type DelegationThreadStatus, +} from "../src/delegation-types.js"; + +const PARENT = "parent"; + +/** Rows listed by reference carry no queue place, so the list row type does not declare one. */ +function queueOf(thread: object): unknown { + return "queue" in thread ? thread.queue : undefined; +} + +/** A Host session whose delegated starts pass the capacity gate the way the Harness coordinator does. */ +function session() { + const threads = new Map([ + [PARENT, { status: "completed", turnId: "turn-parent" }], + ]); + let gate: DelegationCapacityGate | undefined; + let created = 0; + const result = ( + identity: DelegationStartIdentity, + status: DelegationThreadStatus, + extra: object = {}, + ) => ({ + delegationId: identity.delegationId, + threadId: identity.threadId, + turnId: identity.turnId, + harnessId: "pi", + deepLink: `codex://threads/${identity.threadId}`, + status, + parentThreadId: PARENT, + cwd: "/work", + next: { read: "read", wait: "wait" }, + ...extra, + }); + const snapshot = (threadId: string) => { + const thread = threads.get(threadId); + if (!thread) throw new DelegationControlError("THREAD_NOT_FOUND", "Thread was not found"); + return { + threadId, + harnessId: "pi", + status: thread.status, + turn: { turnId: thread.turnId, status: thread.status }, + progress: [], + result: + thread.status === "running" + ? { availability: "pending" as const } + : { availability: "available" as const, text: "done" }, + nextCursor: null, + }; + }; + const registration: DelegationControlRegistration = { + listHarnesses: async () => ({ harnesses: ["codex", "pi"] }), + inspect: async () => { + throw new Error("not used"); + }, + canHandleStart: (input) => input.parentThreadId === PARENT, + ownsThread: (threadId) => threads.has(threadId), + useCapacity: (capacity) => { + gate = capacity; + }, + runningThreadIds: () => + [...threads] + .filter(([, thread]) => thread.status === "running") + .map(([threadId]) => threadId), + start: vi.fn(async (input: DelegationStartInput, admitted?: DelegationStartIdentity) => { + created += 1; + const identity = admitted ?? { + delegationId: `delegation-${created}`, + threadId: `child-${created}`, + turnId: `turn-${created}`, + }; + const admission = admitted + ? undefined + : await gate?.admit({ + input: { ...input, parentThreadId: PARENT, cwd: "/work" }, + title: input.taskName ?? input.task, + taskDigest: input.task, + identity, + }); + if (admission?.state === "queued") { + return result(admission.identity, admission.status, { + ...(admission.position ? { queue: { position: admission.position } } : {}), + title: admission.title, + }); + } + try { + threads.set(identity.threadId, { status: "running", turnId: identity.turnId }); + } finally { + admission?.release(); + } + return result(identity, "running"); + }), + send: vi.fn(async ({ threadId }) => ({ + threadId, + turnId: `turn-sent-${threadId}`, + harnessId: "pi", + status: "running" as const, + next: { read: "read", wait: "wait" }, + })), + cancel: vi.fn(async ({ threadId }) => ({ + threadId, + turnId: threads.get(threadId)?.turnId ?? null, + harnessId: "pi", + cancelled: threads.get(threadId)?.status === "running", + })), + read: vi.fn(async ({ threadId }) => snapshot(threadId)), + wait: vi.fn(async ({ threadId }) => ({ ...snapshot(threadId), timedOut: false })), + list: vi.fn(async () => ({ + threads: [...threads].map(([threadId, thread]) => ({ + threadId, + harnessId: "pi", + deepLink: `codex://threads/${threadId}`, + status: thread.status, + })), + nextCursor: null, + })), + }; + const finish = (threadId: string, status: DelegationThreadStatus = "completed") => { + const thread = threads.get(threadId); + if (thread) thread.status = status; + }; + return { registration, threads, finish }; +} + +const start = (registry: DelegationControlRegistry, task: string) => + registry.start({ harnessId: "pi", task, parentThreadId: PARENT }); + +describe("delegation capacity in the registry", () => { + const cleanups: (() => Promise | void)[] = []; + afterEach(async () => { + for (const cleanup of cleanups.splice(0).reverse()) await cleanup(); + vi.useRealTimers(); + }); + const registryWith = ( + options: ConstructorParameters[0] = {}, + ) => { + const registry = new DelegationControlRegistry(options); + const value = session(); + registry.register(value.registration); + cleanups.push(() => registry.close()); + return { registry, ...value }; + }; + + it("starts every delegation at once while the limit is off", async () => { + const { registry, registration } = registryWith(); + for (const task of ["one", "two", "three"]) { + expect(await start(registry, task)).toMatchObject({ status: "running" }); + } + registry.capacity.configure({ enabled: false, maxRunning: 1 }); + const result = await start(registry, "four"); + expect(result).toMatchObject({ status: "running" }); + expect(result).not.toHaveProperty("queue"); + expect(registration.start).toHaveBeenCalledTimes(4); + const listed = await registry.list({ parentThreadId: PARENT, limit: 10, sort: "created-desc" }); + expect(listed.threads.every((thread) => queueOf(thread) === undefined)).toBe(true); + }); + + it("returns a queued Thread at once and shows it in read, list, wait and send", async () => { + const { registry } = registryWith(); + registry.capacity.configure({ enabled: true, maxRunning: 1 }); + const running = await start(registry, "running"); + const first = await start(registry, "first"); + const second = await start(registry, "second"); + expect(running).toMatchObject({ status: "running" }); + expect(first).toMatchObject({ + status: "queued", + queue: { position: 1 }, + deepLink: `codex://threads/${first.threadId}`, + parentThreadId: PARENT, + cwd: "/work", + title: "first", + }); + expect(second).toMatchObject({ status: "queued", queue: { position: 2 } }); + + expect(await registry.read({ threadId: second.threadId, view: "result" })).toMatchObject({ + threadId: second.threadId, + status: "queued", + queue: { position: 2 }, + turn: null, + result: { availability: "pending" }, + }); + const listed = await registry.list({ parentThreadId: PARENT, limit: 1, sort: "created-desc" }); + expect( + listed.threads.map((thread) => ({ + threadId: thread.threadId, + status: thread.status, + queue: queueOf(thread), + })), + ).toEqual([ + { threadId: first.threadId, status: "queued", queue: { position: 1 } }, + { threadId: second.threadId, status: "queued", queue: { position: 2 } }, + { threadId: PARENT, status: "completed", queue: undefined }, + { threadId: running.threadId, status: "running", queue: undefined }, + ]); + expect( + ( + await registry.list({ + parentThreadId: PARENT, + limit: 1, + sort: "created-desc", + cursor: "next-page", + }) + ).threads.some((thread) => thread.status === "queued"), + ).toBe(false); + expect( + await registry.wait({ threadId: first.threadId, view: "result", timeoutMs: 20 }), + ).toMatchObject({ status: "queued", queue: { position: 1 }, timedOut: true }); + await expect( + registry.send({ threadId: first.threadId, message: "more" }), + ).rejects.toMatchObject({ + code: "THREAD_BUSY", + details: { status: "queued", queuePosition: 1 }, + }); + await expect( + registry.wait({ threadId: first.threadId, view: "result", timeoutMs: 0 }), + ).rejects.toMatchObject({ code: "INVALID_ARGUMENT" }); + }); + + it("starts queued Threads under their identity when running Threads finish", async () => { + const { registry, registration, finish } = registryWith({ capacity: { pollIntervalMs: 5 } }); + registry.capacity.configure({ enabled: true, maxRunning: 1 }); + const running = await start(registry, "running"); + const queued = await start(registry, "queued"); + const waiting = registry.wait({ threadId: queued.threadId, view: "result", timeoutMs: 5_000 }); + finish(running.threadId); + expect(await waiting).toMatchObject({ + threadId: queued.threadId, + status: "running", + timedOut: false, + }); + expect(registration.start).toHaveBeenLastCalledWith( + { harnessId: "pi", task: "queued", cwd: "/work", parentThreadId: PARENT }, + { delegationId: queued.delegationId, threadId: queued.threadId, turnId: queued.turnId }, + ); + expect(await registry.read({ threadId: queued.threadId, view: "result" })).toMatchObject({ + status: "running", + turn: { turnId: queued.turnId }, + }); + await expect( + registry.send({ threadId: queued.threadId, message: "more" }), + ).resolves.toMatchObject({ + status: "running", + }); + }); + + it("removes a queued Thread on cancel and then reports it interrupted", async () => { + const { registry, registration, finish } = registryWith({ capacity: { pollIntervalMs: 5 } }); + registry.capacity.configure({ enabled: true, maxRunning: 1 }); + const running = await start(registry, "running"); + const cancelled = await start(registry, "cancelled"); + const kept = await start(registry, "kept"); + + expect(await registry.cancel({ threadId: cancelled.threadId })).toEqual({ + threadId: cancelled.threadId, + turnId: null, + harnessId: "pi", + cancelled: true, + }); + expect(registration.cancel).not.toHaveBeenCalled(); + expect(await registry.read({ threadId: kept.threadId, view: "result" })).toMatchObject({ + queue: { position: 1 }, + }); + expect(await registry.read({ threadId: cancelled.threadId, view: "result" })).toMatchObject({ + status: "interrupted", + result: { availability: "unavailable", message: "Removed from the queue before it started" }, + }); + expect( + await registry.wait({ threadId: cancelled.threadId, view: "result", timeoutMs: 1_000 }), + ).toMatchObject({ status: "interrupted", timedOut: false }); + expect(await registry.cancel({ threadId: cancelled.threadId })).toMatchObject({ + cancelled: false, + }); + await expect( + registry.send({ threadId: cancelled.threadId, message: "more" }), + ).rejects.toMatchObject({ code: "THREAD_NOT_FOUND" }); + const listed = await registry.list({ parentThreadId: PARENT, limit: 10, sort: "created-desc" }); + expect(listed.threads.map((thread) => thread.threadId)).not.toContain(cancelled.threadId); + + finish(running.threadId); + await vi.waitFor(() => + expect(registration.start).toHaveBeenLastCalledWith( + expect.objectContaining({ task: "kept" }), + expect.objectContaining({ threadId: kept.threadId }), + ), + ); + expect(registration.start).not.toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ threadId: cancelled.threadId }), + ); + }); + + it("watches a queued Thread through the queue until it stops after starting", async () => { + vi.useFakeTimers(); + const { registry, registration, finish } = registryWith({ + capacity: { pollIntervalMs: 1_000 }, + }); + registry.capacity.configure({ enabled: true, maxRunning: 1 }); + const running = await start(registry, "running"); + const queued = await start(registry, "queued"); + + expect( + await registry.watch({ + threadId: queued.threadId, + notifyThreadId: PARENT, + timeoutMs: 600_000, + stallAfterMs: 3_000, + }), + ).toMatchObject({ state: "watching", status: "queued" }); + // Time spent waiting for a slot is not Harness silence. + await vi.advanceTimersByTimeAsync(20_000); + expect(registration.send).not.toHaveBeenCalled(); + + finish(running.threadId); + await vi.advanceTimersByTimeAsync(2_000); + expect(await registry.read({ threadId: queued.threadId, view: "result" })).toMatchObject({ + status: "running", + }); + expect(registration.send).not.toHaveBeenCalled(); + finish(queued.threadId); + await vi.advanceTimersByTimeAsync(2_000); + expect(registration.send).toHaveBeenCalledOnce(); + expect(registration.send).toHaveBeenCalledWith({ + threadId: PARENT, + message: expect.stringContaining(`codex://threads/${queued.threadId}: completed`), + }); + }); + + it("notifies a watch on a queued Thread that is removed from the queue", async () => { + vi.useFakeTimers(); + const { registry, registration } = registryWith(); + registry.capacity.configure({ enabled: true, maxRunning: 1 }); + await start(registry, "running"); + const queued = await start(registry, "queued"); + await registry.watch({ threadId: queued.threadId, notifyThreadId: PARENT, timeoutMs: 600_000 }); + await registry.cancel({ threadId: queued.threadId }); + await vi.advanceTimersByTimeAsync(2_000); + expect(registration.send).toHaveBeenCalledWith({ + threadId: PARENT, + message: expect.stringContaining(`codex://threads/${queued.threadId}: interrupted`), + }); + }); + + it("keeps the queue across a Host Runtime restart", async () => { + const directory = await mkdtemp(path.join(os.tmpdir(), "codexhost-capacity-")); + cleanups.push(() => rm(directory, { recursive: true, force: true })); + const before = registryWith({ capacity: { store: new FileDelegationQueueStore(directory) } }); + before.registry.capacity.configure({ enabled: true, maxRunning: 1 }); + await start(before.registry, "running"); + const queued = await start(before.registry, "queued"); + before.registry.close(); + + const after = registryWith({ + capacity: { store: new FileDelegationQueueStore(directory), pollIntervalMs: 5 }, + }); + expect(await after.registry.read({ threadId: queued.threadId, view: "result" })).toMatchObject({ + status: "queued", + queue: { position: 1 }, + }); + expect( + (await after.registry.list({ parentThreadId: PARENT, limit: 10, sort: "created-desc" })) + .threads[0], + ).toMatchObject({ threadId: queued.threadId, status: "queued" }); + expect(after.registration.start).not.toHaveBeenCalled(); + after.registry.capacity.configure({ enabled: true, maxRunning: 1 }); + await vi.waitFor(() => + expect(after.registration.start).toHaveBeenCalledWith( + expect.objectContaining({ task: "queued", parentThreadId: PARENT }), + { delegationId: queued.delegationId, threadId: queued.threadId, turnId: queued.turnId }, + ), + ); + expect(await after.registry.read({ threadId: queued.threadId, view: "result" })).toMatchObject({ + status: "running", + }); + }); +}); diff --git a/packages/host-runtime/test/delegation-capacity-store.test.ts b/packages/host-runtime/test/delegation-capacity-store.test.ts new file mode 100644 index 000000000..7f3a07aa8 --- /dev/null +++ b/packages/host-runtime/test/delegation-capacity-store.test.ts @@ -0,0 +1,195 @@ +import { spawnSync } from "node:child_process"; +import { mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import { + FileDelegationQueueStore, + defaultDelegationQueueDirectory, + type PersistedQueuedDelegation, +} from "../src/delegation-capacity-store.js"; + +const bootSeconds = () => Math.round(Date.now() / 1_000 - os.uptime()); + +function entry(threadId: string, patch: Partial = {}) { + return { + delegationId: `delegation-${threadId}`, + threadId, + turnId: `turn-${threadId}`, + harnessId: "pi", + task: "Review the change", + cwd: "/work", + parentThreadId: "parent", + title: "Review", + taskDigest: "digest", + queuedAt: 1_000, + ...patch, + } satisfies PersistedQueuedDelegation; +} + +/** The pid of a process that has already exited. */ +function exitedPid(): number { + const child = spawnSync(process.execPath, ["-e", ""]); + if (child.pid === undefined) throw new Error("could not start a process"); + return child.pid; +} + +async function writeForeign( + directory: string, + name: string, + owner: { pid: number; boot: number; instance: string }, + queue: unknown[], +): Promise { + await writeFile( + path.join(directory, name), + `${JSON.stringify({ formatVersion: 1, owner, queue })}\n`, + ); +} + +describe("FileDelegationQueueStore", () => { + let directory: string; + + beforeEach(async () => { + directory = path.join(await mkdtemp(path.join(os.tmpdir(), "codexhost-queue-")), "queue"); + }); + afterEach(async () => { + await rm(path.dirname(directory), { recursive: true, force: true }); + }); + + it("lives in the codexhost data directory", () => { + expect(defaultDelegationQueueDirectory({ CODEXHOST_DATA_DIR: "/data" })).toBe( + path.join(path.resolve("/data"), "delegation-queue"), + ); + }); + + it("keeps this Runtime's whole queue in one private file and removes it when empty", async () => { + const store = new FileDelegationQueueStore(directory); + await store.save([entry("a"), entry("b", { starting: true, requestId: "retry" })]); + const [name, ...others] = await readdir(directory); + expect(others).toEqual([]); + expect(name).toMatch(new RegExp(`^${process.pid}-[0-9a-f]+\\.json$`)); + const file = JSON.parse(await readFile(path.join(directory, name ?? ""), "utf8")); + expect(file).toMatchObject({ + formatVersion: 1, + owner: { pid: process.pid }, + queue: [entry("a"), entry("b", { starting: true, requestId: "retry" })], + }); + if (process.platform !== "win32") { + expect((await stat(path.join(directory, name ?? ""))).mode & 0o777).toBe(0o600); + expect((await stat(directory)).mode & 0o777).toBe(0o700); + } + await store.save([]); + expect(await readdir(directory)).toEqual([]); + }); + + it("returns nothing when no Runtime has stored a queue", async () => { + await expect(new FileDelegationQueueStore(directory).adopt()).resolves.toEqual([]); + }); + + it("leaves the queue of a running Runtime alone", async () => { + await mkdir(directory, { recursive: true }); + await writeForeign( + directory, + "live.json", + { pid: process.ppid, boot: bootSeconds(), instance: "other" }, + [entry("live")], + ); + const own = new FileDelegationQueueStore(directory); + await own.save([entry("mine")]); + await expect(own.adopt()).resolves.toEqual([]); + expect((await readdir(directory)).sort()).toHaveLength(2); + }); + + it("adopts the queues of exited Runtimes in arrival order and takes ownership", async () => { + await mkdir(directory, { recursive: true }); + const exited = { pid: exitedPid(), boot: bootSeconds(), instance: "exited" }; + await writeForeign(directory, "one.json", exited, [ + entry("late", { queuedAt: 3_000 }), + entry("early", { queuedAt: 1_000 }), + ]); + await writeForeign(directory, "two.json", { ...exited, instance: "second" }, [ + entry("middle", { queuedAt: 2_000, starting: true }), + entry("early", { queuedAt: 1_000 }), + ]); + + const store = new FileDelegationQueueStore(directory); + const adopted = await store.adopt(); + expect(adopted.map((queued) => queued.threadId)).toEqual(["early", "middle", "late"]); + expect(adopted[1]).toMatchObject({ starting: true }); + const names = await readdir(directory); + expect(names).toHaveLength(1); + const file = JSON.parse(await readFile(path.join(directory, names[0] ?? ""), "utf8")); + expect(file.owner.pid).toBe(process.pid); + expect(file.queue.map((queued: PersistedQueuedDelegation) => queued.threadId)).toEqual([ + "early", + "middle", + "late", + ]); + }); + + it("treats an earlier process with this pid or an earlier boot as exited", async () => { + await mkdir(directory, { recursive: true }); + await writeForeign( + directory, + "same-pid.json", + { pid: process.pid, boot: bootSeconds(), instance: "earlier" }, + [entry("same-pid")], + ); + await writeForeign( + directory, + "rebooted.json", + { pid: process.ppid, boot: 1, instance: "rebooted" }, + [entry("rebooted", { queuedAt: 2_000 })], + ); + const adopted = await new FileDelegationQueueStore(directory).adopt(); + expect(adopted.map((queued) => queued.threadId)).toEqual(["same-pid", "rebooted"]); + }); + + it("adopts a claim left by an adopter that exited mid-way", async () => { + await mkdir(directory, { recursive: true }); + await writeForeign( + directory, + `half.json.claim-${exitedPid()}.abcd`, + { pid: exitedPid(), boot: bootSeconds(), instance: "half" }, + [entry("half")], + ); + const store = new FileDelegationQueueStore(directory); + await expect(store.adopt()).resolves.toEqual([entry("half")]); + expect(await readdir(directory)).toEqual([ + expect.stringMatching(new RegExp(`^${process.pid}-[0-9a-f]+\\.json$`)), + ]); + }); + + it("lets only one of two concurrent adopters take a queue", async () => { + await mkdir(directory, { recursive: true }); + await writeForeign( + directory, + "race.json", + { pid: exitedPid(), boot: bootSeconds(), instance: "race" }, + [entry("race")], + ); + const [left, right] = await Promise.all([ + new FileDelegationQueueStore(directory).adopt(), + new FileDelegationQueueStore(directory).adopt(), + ]); + expect([...left, ...right]).toEqual([entry("race")]); + }); + + it("sets aside an unreadable queue and reports it", async () => { + await mkdir(directory, { recursive: true }); + await writeFile(path.join(directory, "bad.json"), "{ not json"); + const exited = { pid: exitedPid(), boot: bootSeconds(), instance: "exited" }; + await writeForeign(directory, "missing-task.json", exited, [entry("x", { task: "" })]); + await writeForeign(directory, "bad-model.json", exited, [ + entry("y", { model: { id: "" } as never }), + ]); + const diagnose = vi.fn(); + await expect(new FileDelegationQueueStore(directory, diagnose).adopt()).resolves.toEqual([]); + expect(diagnose).toHaveBeenCalledTimes(3); + const names = await readdir(directory); + expect(names.filter((name) => name.endsWith(".invalid"))).toHaveLength(3); + expect(names.some((name) => name.endsWith(".json"))).toBe(false); + }); +}); diff --git a/packages/host-runtime/test/delegation-cli-output.test.ts b/packages/host-runtime/test/delegation-cli-output.test.ts index 63c61e8d2..e7991ad5e 100644 --- a/packages/host-runtime/test/delegation-cli-output.test.ts +++ b/packages/host-runtime/test/delegation-cli-output.test.ts @@ -152,4 +152,71 @@ describe("compact delegation output", () => { nextCursor: null, }); }); + + it("keeps the queue position of a queued Thread", () => { + const queue = { position: 2 }; + expect( + compactDelegationOutput("delegate start", { + delegationId: "private-delegation", + threadId: "child", + turnId: "private-turn", + harnessId: "pi", + deepLink: "codex://threads/child", + status: "queued", + queue, + cwd: "/project", + next: {}, + }), + ).toEqual({ + thread: "codex://threads/child", + harnessId: "pi", + status: "queued", + queuePosition: 2, + cwd: "/project", + }); + expect( + compactDelegationOutput("thread read", { + threadId: "child", + harnessId: "pi", + status: "queued", + queue, + turn: null, + progress: [], + result: { availability: "pending" }, + nextCursor: "bookmark", + }), + ).toEqual({ + thread: "codex://threads/child", + harnessId: "pi", + status: "queued", + queuePosition: 2, + result: { availability: "pending" }, + }); + expect( + compactDelegationOutput("thread list", { + threads: [ + { + threadId: "child", + deepLink: "codex://threads/child", + harnessId: "pi", + status: "queued", + queue, + title: "Task", + }, + ], + nextCursor: null, + }), + ).toEqual({ + threads: [ + { + thread: "codex://threads/child", + harnessId: "pi", + status: "queued", + queuePosition: 2, + title: "Task", + }, + ], + nextCursor: null, + }); + }); }); diff --git a/packages/host-runtime/test/delegation-cli.test.ts b/packages/host-runtime/test/delegation-cli.test.ts index 27253957a..c569a186c 100644 --- a/packages/host-runtime/test/delegation-cli.test.ts +++ b/packages/host-runtime/test/delegation-cli.test.ts @@ -42,6 +42,22 @@ describe("delegation CLI", () => { expect(fetchImpl).not.toHaveBeenCalled(); }); + it("lists stalled alerts in the watch and event help", async () => { + const help = async (...command: string[]) => { + const output = new PassThrough(); + await expect(runDelegationCli({ arguments: [...command, "--help"], output })).resolves.toBe( + 0, + ); + return outputText(output); + }; + expect(await help("thread", "watch")).toContain( + "unreadable, notFound, or stalled (only with --stall-after-ms; the watch continues)", + ); + expect(await help("events")).toContain( + "(outcome stalled, explained in note; the watch continues)", + ); + }); + it("discovers Harnesses without reading a Model catalog", async () => { const fetchImpl = successfulFetch({ harnesses: ["codex", "pi"] }); const output = new PassThrough(); @@ -750,7 +766,15 @@ describe("delegation CLI", () => { const base = ["delegate", "start", "--harness", "pi", "--task", "work"]; await runDelegationCli({ arguments: base, environment, output: new PassThrough(), fetchImpl }); await runDelegationCli({ - arguments: [...base, "--watch", "true", "--watch-timeout-ms", "5000"], + arguments: [ + ...base, + "--watch", + "true", + "--watch-timeout-ms", + "5000", + "--stall-after-ms", + "1000", + ], environment, output: new PassThrough(), fetchImpl, @@ -759,7 +783,7 @@ describe("delegation CLI", () => { .mocked(fetchImpl) .mock.calls.map(([, init]) => JSON.parse(String(init?.body)) as Record); expect(bodies[0]).not.toHaveProperty("watchTimeoutMs"); - expect(bodies[1]).toMatchObject({ watchTimeoutMs: 5_000 }); + expect(bodies[1]).toMatchObject({ watchTimeoutMs: 5_000, stallAfterMs: 1_000 }); const diagnosticOutput = new PassThrough(); expect( @@ -773,4 +797,52 @@ describe("delegation CLI", () => { ).toBe(1); expect(fetchImpl).toHaveBeenCalledTimes(2); }); + + it("accepts a positive optional silence threshold on thread watch", async () => { + const fetchImpl = successfulFetch({}); + const environment = { + [DELEGATION_RUNTIME_ENDPOINT_ENV]: "http://127.0.0.1:4321", + [DELEGATION_RUNTIME_TOKEN_ENV]: "token", + [DELEGATION_THREAD_ID_ENV]: "parent", + }; + await expect( + runDelegationCli({ + arguments: ["thread", "watch", "child", "--stall-after-ms", "3000"], + environment, + output: new PassThrough(), + fetchImpl, + }), + ).resolves.toBe(0); + expect(JSON.parse(String(vi.mocked(fetchImpl).mock.calls[0]?.[1]?.body))).toMatchObject({ + stallAfterMs: 3_000, + }); + for (const args of [ + ["thread", "watch", "child", "--stall-after-ms", "0"], + ["thread", "watch", "child", "--stall-after-ms", "1.5"], + ["delegate", "start", "--harness", "pi", "--task", "work", "--stall-after-ms", "3000"], + [ + "delegate", + "start", + "--harness", + "pi", + "--task", + "work", + "--watch", + "false", + "--stall-after-ms", + "3000", + ], + ]) { + await expect( + runDelegationCli({ + arguments: args, + environment, + output: new PassThrough(), + diagnosticOutput: new PassThrough(), + fetchImpl, + }), + ).resolves.toBe(1); + } + expect(fetchImpl).toHaveBeenCalledTimes(1); + }); }); diff --git a/packages/host-runtime/test/delegation-control-registry.test.ts b/packages/host-runtime/test/delegation-control-registry.test.ts index 4636f3ab7..6788db9b3 100644 --- a/packages/host-runtime/test/delegation-control-registry.test.ts +++ b/packages/host-runtime/test/delegation-control-registry.test.ts @@ -72,6 +72,43 @@ function registration(threadId: string): DelegationControlRegistration { } describe("DelegationControlRegistry", () => { + it("forwards registered Harness activity to silence watches and unsubscribes on removal", async () => { + vi.useFakeTimers(); + const registry = new DelegationControlRegistry(); + try { + const target = registration("target"); + const subscriber = registration("subscriber"); + let noteActivity: ((threadIds: readonly string[]) => void) | undefined; + const unsubscribe = vi.fn(); + target.subscribeActivity = (listener) => { + noteActivity = listener; + return unsubscribe; + }; + const remove = registry.register(target); + registry.register(subscriber); + await registry.watch({ + threadId: "target", + notifyThreadId: "subscriber", + timeoutMs: 60_000, + stallAfterMs: 3_000, + }); + await vi.advanceTimersByTimeAsync(2_000); + noteActivity?.(["subagent", "target"]); + await vi.advanceTimersByTimeAsync(2_000); + expect(subscriber.send).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(2_000); + expect(subscriber.send).toHaveBeenCalledOnce(); + expect(subscriber.send).toHaveBeenCalledWith({ + threadId: "subscriber", + message: expect.stringContaining(": stalled"), + }); + remove(); + expect(unsubscribe).toHaveBeenCalledOnce(); + } finally { + registry.close(); + vi.useRealTimers(); + } + }); it("uses only the calling Thread's execution identity for same-Host reads", async () => { const caller = registration("caller"); caller.executionHostId = (id) => (id === "caller" ? "ssh:mac" : undefined); diff --git a/packages/host-runtime/test/delegation-control-server.test.ts b/packages/host-runtime/test/delegation-control-server.test.ts index df9637cd0..708225c21 100644 --- a/packages/host-runtime/test/delegation-control-server.test.ts +++ b/packages/host-runtime/test/delegation-control-server.test.ts @@ -227,7 +227,7 @@ describe("delegation control server", () => { try { const response = await fetch( `${server.endpoint}/v1/delegate/start`, - authorized({ harnessId: "pi", task: "work", watchTimeoutMs: 5_000 }), + authorized({ harnessId: "pi", task: "work", watchTimeoutMs: 5_000, stallAfterMs: 1_000 }), ); await expect(response.json()).resolves.toMatchObject({ threadId: "child", @@ -239,6 +239,7 @@ describe("delegation control server", () => { threadId: "child", notifyThreadId: "parent", timeoutMs: 5_000, + stallAfterMs: 1_000, }); watch.mockRejectedValueOnce(new Error("synthetic watch failure")); @@ -273,6 +274,17 @@ describe("delegation control server", () => { authorized({ threadId: "child", notifyThreadId: "parent", timeoutMs: 5_000 }), ); expect(rejected.status).toBe(400); + + const starts = start.mock.calls.length; + const withoutWatch = await fetch( + `${server.endpoint}/v1/delegate/start`, + authorized({ harnessId: "pi", task: "work", stallAfterMs: 1_000 }), + ); + expect(withoutWatch.status).toBe(400); + await expect(withoutWatch.json()).resolves.toMatchObject({ + error: { code: "INVALID_ARGUMENT", message: "stallAfterMs requires a watch" }, + }); + expect(start).toHaveBeenCalledTimes(starts); } finally { await server.close(); await unsupported.close(); diff --git a/packages/host-runtime/test/delegation-watch-restart.test.ts b/packages/host-runtime/test/delegation-watch-restart.test.ts index 2b49585ae..013a4007b 100644 --- a/packages/host-runtime/test/delegation-watch-restart.test.ts +++ b/packages/host-runtime/test/delegation-watch-restart.test.ts @@ -83,6 +83,73 @@ describe("DelegationWatchService across a Host Runtime restart", () => { vi.useRealTimers(); }); + it("persists the silence threshold but not its suppression, then starts a fresh silence clock after restart", async () => { + const store = memoryStore(); + const before = runtime({ child: { status: "running" }, parent: { status: "completed" } }); + const first = service(before, store); + await first.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 60 * 60_000, + stallAfterMs: 3_000, + }); + await vi.advanceTimersByTimeAsync(3_000); + expect(before.sent).toHaveLength(1); + await vi.advanceTimersByTimeAsync(10_000); + expect(before.sent).toHaveLength(1); + const records = [...store.records.values()]; + expect(records).toMatchObject([{ state: "watching", stallAfterMs: 3_000 }]); + expect(records[0]).not.toHaveProperty("stallNotified"); + expect(records[0]).not.toHaveProperty("lastActivityAt"); + first.close(); + await vi.advanceTimersByTimeAsync(60_000); + const after = runtime({ child: { status: "running" }, parent: { status: "completed" } }); + const second = service(after, store); + await second.watches(); + await vi.advanceTimersByTimeAsync(2_999); + expect(after.sent).toHaveLength(0); + await vi.advanceTimersByTimeAsync(1); + expect(after.sent).toHaveLength(1); + expect(after.sent[0]?.message).toContain(": stalled"); + await vi.advanceTimersByTimeAsync(10_000); + expect(after.sent).toHaveLength(1); + second.close(); + }); + + it("never stores a pending silence alert and measures silence again after a restart", async () => { + const store = memoryStore(); + const before = runtime({ child: { status: "running" }, parent: { status: "running" } }); + before.send.mockRejectedValue(new DelegationControlError("THREAD_BUSY", "busy")); + const first = service(before, store); + await first.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 60_000, + stallAfterMs: 3_000, + }); + await vi.advanceTimersByTimeAsync(3_000); + expect((await first.watches()).watches).toMatchObject([ + { state: "watching" }, + { state: "pendingDelivery", outcome: "stalled" }, + ]); + expect([...store.records.values()]).toMatchObject([{ state: "watching", stallAfterMs: 3_000 }]); + first.close(); + const after = runtime({ child: { status: "running" }, parent: { status: "completed" } }); + const second = service(after, store); + await second.watches(); + await vi.advanceTimersByTimeAsync(2_000); + expect(after.sent).toHaveLength(0); + await vi.advanceTimersByTimeAsync(POLL_MS); + expect(after.sent).toHaveLength(1); + expect(after.sent[0]?.message).toContain(": stalled"); + expect([...store.records.values()]).toMatchObject([{ state: "watching", stallAfterMs: 3_000 }]); + after.update("child", { status: "completed" }); + await vi.advanceTimersByTimeAsync(POLL_MS); + expect(after.sent).toHaveLength(2); + expect(after.sent[1]?.message).toContain(": completed"); + expect(store.records.size).toBe(0); + }); + it("stores a watch while it is pending and removes it once delivered", async () => { const store = memoryStore(); const fake = runtime({ child: { status: "running" }, parent: { status: "completed" } }); diff --git a/packages/host-runtime/test/delegation-watch-store-race.test.ts b/packages/host-runtime/test/delegation-watch-store-race.test.ts index a03c5b830..a61ac9a7a 100644 --- a/packages/host-runtime/test/delegation-watch-store-race.test.ts +++ b/packages/host-runtime/test/delegation-watch-store-race.test.ts @@ -43,7 +43,7 @@ describe("FileDelegationWatchStore adoption race", () => { }); it.each([undefined, "EEXIST", "EIO"])( - "preserves another adopter's record when returning a claim encounters %s", + "preserves an active owner's record when returning a claim encounters %s", async (linkError) => { await mkdir(directory, { recursive: true }); const dead = spawnSync(process.execPath, ["-e", ""]).pid; @@ -51,7 +51,7 @@ describe("FileDelegationWatchStore adoption race", () => { id: "race", threadId: "child", notifyThreadId: "parent", - state: "watching", + state: "watching" as const, registeredAt: "2026-11-03T00:00:00.000Z", timeoutMs: 60_000, deadline: 1_000, @@ -61,13 +61,11 @@ describe("FileDelegationWatchStore adoption race", () => { `${JSON.stringify({ formatVersion: 1, owner: { pid: dead, boot: bootSeconds() }, watch: record })}\n`, ); const first = new FileDelegationWatchStore(directory); - let firstAdopted: unknown[] = []; - // The late adopter has read the exited owner's record. Before its claim - // rename runs, another adopter takes the record and saves it as its own. + // A live Runtime writes the record after the adopter read its previous owner. hooks.beforeRename = async (_from, to) => { if (!to.includes(".claim-")) return; hooks.beforeRename = undefined; - firstAdopted = await first.adopt(); + await first.save(record); }; hooks.beforeLink = async (from, to) => { if (linkError === "EEXIST") await writeFile(to, await readFile(from)); @@ -76,7 +74,6 @@ describe("FileDelegationWatchStore adoption race", () => { const diagnose = vi.fn(); const late = await new FileDelegationWatchStore(directory, diagnose).adopt(); - expect(firstAdopted).toEqual([record]); expect(late).toEqual([]); const files = await readdir(directory); expect(files).toHaveLength(1); diff --git a/packages/host-runtime/test/delegation-watch-store.test.ts b/packages/host-runtime/test/delegation-watch-store.test.ts index d6b6db9c3..f9e83b838 100644 --- a/packages/host-runtime/test/delegation-watch-store.test.ts +++ b/packages/host-runtime/test/delegation-watch-store.test.ts @@ -74,6 +74,37 @@ describe("FileDelegationWatchStore", () => { await expect(new FileDelegationWatchStore(directory).adopt()).resolves.toEqual([]); }); + it("adopts the silence threshold but not a stalled outcome, which is never stored", async () => { + await mkdir(directory, { recursive: true }); + const record = watch("silent", { stallAfterMs: 3_000 }); + const owner = { pid: exitedPid(), boot: bootSeconds() }; + await writeForeign(directory, "silent.json", owner, record); + await writeForeign( + directory, + "stalled.json", + owner, + watch("stalled", { stallAfterMs: 3_000, state: "pendingDelivery", outcome: "stalled" }), + ); + const diagnose = vi.fn(); + await expect(new FileDelegationWatchStore(directory, diagnose).adopt()).resolves.toEqual([ + record, + ]); + expect(diagnose).toHaveBeenCalledOnce(); + }); + + it.each([0, -1, 1.5])("rejects a stored invalid silence threshold %s", async (stallAfterMs) => { + await mkdir(directory, { recursive: true }); + await writeForeign( + directory, + "bad-stall.json", + { pid: exitedPid(), boot: bootSeconds() }, + watch("bad-stall", { stallAfterMs }), + ); + const diagnose = vi.fn(); + await expect(new FileDelegationWatchStore(directory, diagnose).adopt()).resolves.toEqual([]); + expect(diagnose).toHaveBeenCalledOnce(); + }); + it("leaves records of a running Runtime alone", async () => { await new FileDelegationWatchStore(directory).save(watch("mine")); await expect(new FileDelegationWatchStore(directory).adopt()).resolves.toEqual([]); diff --git a/packages/host-runtime/test/delegation-watch.test.ts b/packages/host-runtime/test/delegation-watch.test.ts index 01e9c4aab..edacaa1b5 100644 --- a/packages/host-runtime/test/delegation-watch.test.ts +++ b/packages/host-runtime/test/delegation-watch.test.ts @@ -7,6 +7,7 @@ import { type ThreadSendInput, } from "../src/delegation-types.js"; import { DelegationWatchService } from "../src/delegation-watch.js"; +import { ThreadActivity } from "../src/thread-activity.js"; const POLL_MS = 1_000; @@ -127,13 +128,16 @@ describe("DelegationWatchService", () => { expect(fake.sent[0]?.message).toContain("has not reached a terminal state after 29 min"); }); - it("states a timeout shorter than a minute in seconds", async () => { + it.each([ + [15_000, "15 s"], + [700, "1 s"], + ])("states a %i ms timeout in whole seconds", async (timeoutMs, text) => { const fake = runtime({ child: { status: "running" }, parent: { status: "completed" } }); const service = new DelegationWatchService(fake, { pollIntervalMs: POLL_MS }); - await service.watch({ threadId: "child", notifyThreadId: "parent", timeoutMs: 15_000 }); + await service.watch({ threadId: "child", notifyThreadId: "parent", timeoutMs }); - await vi.advanceTimersByTimeAsync(15_000 + POLL_MS); - expect(fake.sent[0]?.message).toContain("has not reached a terminal state after 15 s"); + await vi.advanceTimersByTimeAsync(timeoutMs + POLL_MS); + expect(fake.sent[0]?.message).toContain(`has not reached a terminal state after ${text};`); }); it("waits out a replacement Turn and notifies once when the Thread stops", async () => { @@ -382,4 +386,218 @@ describe("DelegationWatchService", () => { service.watch({ threadId: "child", notifyThreadId: "parent", timeoutMs: 60_000 }), ).rejects.toMatchObject({ code: "INTERNAL_ERROR" }); }); + + it("notifies once per silence episode, rearms on activity, and still reports the final stop", async () => { + const fake = runtime({ child: { status: "running" }, parent: { status: "completed" } }); + const service = new DelegationWatchService(fake, { pollIntervalMs: POLL_MS }); + await service.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 60_000, + stallAfterMs: 3_000, + }); + expect(vi.getTimerCount()).toBe(1); + await vi.advanceTimersByTimeAsync(2_999); + expect(fake.sent).toHaveLength(0); + await vi.advanceTimersByTimeAsync(1); + expect(fake.sent).toHaveLength(1); + expect(fake.sent[0]?.message).toContain( + "codex://threads/child: stalled (Turn turn-child). It was still running, but no Harness event had been observed for at least 3 s.", + ); + await vi.advanceTimersByTimeAsync(10_000); + expect(fake.sent).toHaveLength(1); + service.noteActivity(["unrelated"]); + await vi.advanceTimersByTimeAsync(POLL_MS); + expect(fake.sent).toHaveLength(1); + service.noteActivity(["child"]); + await vi.advanceTimersByTimeAsync(2_999); + expect(fake.sent).toHaveLength(1); + await vi.advanceTimersByTimeAsync(1); + expect(fake.sent).toHaveLength(2); + fake.update("child", { status: "completed" }); + await vi.advanceTimersByTimeAsync(POLL_MS); + expect(fake.sent).toHaveLength(3); + expect(fake.sent[2]?.message).toContain("completed (Turn turn-child)"); + expect((await service.watches()).watches).toEqual([]); + expect(vi.getTimerCount()).toBe(0); + }); + + it.each([ + [90_000, "90 s"], + [600_000, "10 min"], + [1_500, "1500 ms"], + ])( + "states a %i ms silence threshold exactly and leaves progress to thread read", + async (stallAfterMs, text) => { + const fake = runtime({ child: { status: "running" }, parent: { status: "completed" } }); + const service = new DelegationWatchService( + { + read: async (input) => ({ + ...(await fake.read(input)), + progress: [ + { id: "progress-1", turnId: "turn-child", text: "Compiling the release build" }, + ], + }), + send: fake.send, + }, + { pollIntervalMs: 60_000 }, + ); + await service.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 60 * 60_000, + stallAfterMs, + }); + await vi.advanceTimersByTimeAsync(10 * 60_000); + expect(fake.sent).toHaveLength(1); + const message = fake.sent[0]?.message; + expect(message).toContain(`no Harness event had been observed for at least ${text}.`); + expect(message).toContain("read the Thread for its latest progress"); + expect(message).not.toContain("Compiling the release build"); + service.close(); + }, + ); + + it("counts Harness subagent events as parent activity without the parent sending progress", async () => { + const fake = runtime({ parent: { status: "running" }, subscriber: { status: "completed" } }); + const service = new DelegationWatchService(fake, { pollIntervalMs: POLL_MS }); + const activity = new ThreadActivity(); + activity.subscribe((threadIds) => service.noteActivity(threadIds)); + await service.watch({ + threadId: "parent", + notifyThreadId: "subscriber", + timeoutMs: 60_000, + stallAfterMs: 3_000, + }); + activity.note("subagent", "parent"); + for (let index = 0; index < 5; index += 1) { + await vi.advanceTimersByTimeAsync(2_000); + activity.observeOfficial({ + method: "item/commandExecution/outputDelta", + params: { threadId: "subagent", delta: "tool output" }, + }); + } + expect(fake.sent).toHaveLength(0); + await vi.advanceTimersByTimeAsync(3_000); + expect(fake.sent).toHaveLength(1); + service.close(); + }); + + it("keeps counting a grandchild's activity after the intermediate native Thread closes", async () => { + const fake = runtime({ A: { status: "running" }, subscriber: { status: "completed" } }); + const service = new DelegationWatchService(fake, { pollIntervalMs: POLL_MS }); + const activity = new ThreadActivity(); + activity.subscribe((threadIds) => service.noteActivity(threadIds)); + await service.watch({ + threadId: "A", + notifyThreadId: "subscriber", + timeoutMs: 60_000, + stallAfterMs: 3_000, + }); + for (const [id, parent] of [ + ["B", "A"], + ["C", "B"], + ] as const) { + activity.observeOfficial({ + method: "thread/started", + params: { + thread: { id, source: { subAgent: { thread_spawn: { parent_thread_id: parent } } } }, + }, + }); + } + activity.observeOfficial({ method: "thread/closed", params: { threadId: "B" } }); + for (let index = 0; index < 5; index += 1) { + await vi.advanceTimersByTimeAsync(2_000); + activity.observeOfficial({ + method: "item/commandExecution/outputDelta", + params: { threadId: "C", delta: "tool output" }, + }); + } + expect(fake.sent).toHaveLength(0); + activity.observeOfficial({ method: "thread/closed", params: { threadId: "C" } }); + expect(activity.size).toBe(0); + await vi.advanceTimersByTimeAsync(3_000); + expect(fake.sent[0]?.message).toContain(": stalled"); + service.close(); + }); + + it("keeps silence detection disabled when the threshold is omitted", async () => { + const fake = runtime({ child: { status: "running" }, parent: { status: "completed" } }); + const service = new DelegationWatchService(fake, { pollIntervalMs: POLL_MS }); + const result = await service.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 60_000, + }); + expect(result).not.toHaveProperty("stallAfterMs"); + await vi.advanceTimersByTimeAsync(59_000); + expect(fake.sent).toHaveLength(0); + await vi.advanceTimersByTimeAsync(POLL_MS); + expect(fake.sent[0]?.message).toContain("watch expired"); + expect(fake.sent[0]?.message).not.toContain(": stalled"); + }); + + it("merges pending silence alerts and terminal notifications for the same subscriber", async () => { + const fake = runtime({ + a: { status: "running" }, + b: { status: "running" }, + parent: { status: "running" }, + }); + const service = new DelegationWatchService(fake, { pollIntervalMs: POLL_MS }); + await service.watch({ + threadId: "a", + notifyThreadId: "parent", + timeoutMs: 60_000, + stallAfterMs: 2_000, + }); + await service.watch({ threadId: "b", notifyThreadId: "parent", timeoutMs: 60_000 }); + fake.sendFailures.push(busy()); + await vi.advanceTimersByTimeAsync(2_000); + expect((await service.watches()).watches).toMatchObject([ + { threadId: "a", state: "watching" }, + { threadId: "b", state: "watching" }, + { threadId: "a", state: "pendingDelivery", outcome: "stalled" }, + ]); + fake.update("b", { status: "completed" }); + await vi.advanceTimersByTimeAsync(POLL_MS); + expect(fake.sent).toHaveLength(1); + expect(fake.sent[0]?.message).toContain("codex://threads/a: stalled"); + expect(fake.sent[0]?.message).toContain("codex://threads/b: completed"); + expect((await service.watches()).watches).toMatchObject([{ threadId: "a", state: "watching" }]); + service.close(); + }); + + it("preserves the original silence threshold on duplicate registration", async () => { + const fake = runtime({ child: { status: "running" }, parent: { status: "completed" } }); + const service = new DelegationWatchService(fake, { pollIntervalMs: POLL_MS }); + await service.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 60_000, + stallAfterMs: 3_000, + }); + await expect( + service.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 5_000, + stallAfterMs: 1_000, + }), + ).resolves.toMatchObject({ timeoutMs: 60_000, stallAfterMs: 3_000 }); + service.close(); + }); + + it.each([0, -1, 1.5, Number.NaN])("rejects invalid stallAfterMs %s", async (stallAfterMs) => { + const service = new DelegationWatchService( + runtime({ child: { status: "running" }, parent: { status: "completed" } }), + ); + await expect( + service.watch({ + threadId: "child", + notifyThreadId: "parent", + timeoutMs: 60_000, + stallAfterMs, + }), + ).rejects.toMatchObject({ code: "INVALID_ARGUMENT" }); + }); }); diff --git a/packages/host-runtime/test/external-client-events.test.ts b/packages/host-runtime/test/external-client-events.test.ts index 886180181..45625ae14 100644 --- a/packages/host-runtime/test/external-client-events.test.ts +++ b/packages/host-runtime/test/external-client-events.test.ts @@ -64,6 +64,85 @@ describe("client event log", () => { expect(need(next).id).toBeGreaterThan(need(last).id); }); + it("records stalled alerts in the client log and keeps observing until the final stop", async () => { + vi.useFakeTimers(); + let status: DelegationThreadStatus = "running"; + const send = vi.fn(); + // Fake timers cannot wait for real file writes, so polling appends to memory and + // the real log stores the recorded events once real timers are back. + const appended: NewClientEvent[] = []; + const append = vi.fn(async (_client: string, events: readonly NewClientEvent[]) => { + appended.push(...events); + }); + const service = new DelegationWatchService( + { + read: vi.fn( + async ({ threadId }: { threadId: string }): Promise => ({ + threadId, + harnessId: "codex", + status, + turn: { turnId: "turn-1", status }, + progress: [], + result: { availability: "pending" }, + nextCursor: null, + }), + ), + send, + }, + { pollIntervalMs: 1_000, clientEvents: { append } }, + ); + try { + await service.watch({ + threadId: "child", + notifyThreadId: "client:bot", + timeoutMs: 60_000, + stallAfterMs: 3_000, + }); + await vi.advanceTimersByTimeAsync(2_999); + expect(appended).toEqual([]); + await vi.advanceTimersByTimeAsync(1); + expect(appended).toEqual([ + { + type: "watch", + threadId: "child", + outcome: "stalled", + turnId: "turn-1", + harnessId: "codex", + deepLink: "codex://threads/child", + note: expect.stringContaining( + "still running, but no Harness event had been observed for at least 3 s", + ), + }, + ]); + expect(appended[0]?.note).toContain("The watch continues"); + await vi.advanceTimersByTimeAsync(10_000); + expect(appended).toHaveLength(1); + service.noteActivity(["child"]); + await vi.advanceTimersByTimeAsync(2_999); + expect(appended).toHaveLength(1); + await vi.advanceTimersByTimeAsync(1); + expect(appended).toHaveLength(2); + expect((await service.watches()).watches).toEqual([ + expect.objectContaining({ state: "watching", stallAfterMs: 3_000 }), + ]); + status = "completed"; + await vi.advanceTimersByTimeAsync(1_000); + expect(appended.map((event) => event.outcome)).toEqual(["stalled", "stalled", "completed"]); + expect((await service.watches()).watches).toEqual([]); + expect(append.mock.calls.map(([client]) => client)).toEqual(["bot", "bot", "bot"]); + expect(send).not.toHaveBeenCalled(); + } finally { + service.close(); + vi.useRealTimers(); + } + // The stalled outcome and its explanation survive the real log and reopening it. + await new ClientEventLog({ directory, retentionDays: 7 }).append("bot", appended); + const reopened = new ClientEventLog({ directory, retentionDays: 7 }); + expect((await reopened.read("bot", 0)).events).toEqual( + appended.map((event) => ({ ...event, id: expect.any(Number), time: expect.any(String) })), + ); + }); + it("removes whole days past retention and reports an expired cursor", async () => { let now = Date.parse("2026-11-01T12:00:00Z"); const log = new ClientEventLog({ directory, retentionDays: 7, now: () => now }); diff --git a/packages/host-runtime/test/external-client-mcp.test.ts b/packages/host-runtime/test/external-client-mcp.test.ts index 7272baa9b..2a68205bc 100644 --- a/packages/host-runtime/test/external-client-mcp.test.ts +++ b/packages/host-runtime/test/external-client-mcp.test.ts @@ -6,7 +6,11 @@ import { Client } from "@modelcontextprotocol/sdk/client/index.js"; import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { DelegationControlError, type DelegationControlApi } from "../src/delegation-types.js"; +import { + DelegationControlError, + type DelegationControlApi, + type ThreadWatchInput, +} from "../src/delegation-types.js"; import { ClientEventLog } from "../src/external-client-events.js"; import { externalClientMcp } from "../src/external-client-mcp.js"; import { externalClientRouter } from "../src/external-client-routes.js"; @@ -20,6 +24,11 @@ describe("external client MCP endpoint", () => { let directory: string; let server: ExternalClientServer | undefined; let token: string; + const watch = vi.fn(async (input: ThreadWatchInput) => ({ + ...input, + state: "watching" as const, + status: "running" as const, + })); const api = { listHarnesses: vi.fn(async () => ({ harnesses: ["codex", "pi"] })), start: vi.fn(async (input: Record) => ({ @@ -60,7 +69,11 @@ describe("external client MCP endpoint", () => { await events.append("grok-bot", [ { type: "watch", threadId: "t0", outcome: "completed", deepLink: "codex://threads/t0" }, ]); - const route = externalClientRouter({ api: api as unknown as DelegationControlApi, events }); + const route = externalClientRouter({ + api: api as unknown as DelegationControlApi, + watchApi: { watch, watches: vi.fn(async () => ({ watches: [] })) }, + events, + }); server = await startExternalClientServer({ port: 0, clients, @@ -90,6 +103,11 @@ describe("external client MCP endpoint", () => { "thread_watches", "events_poll", ]); + for (const name of ["thread_watch", "delegate_start"]) { + expect( + tools.find((tool) => tool.name === name)?.inputSchema.properties?.stallAfterMs, + ).toMatchObject({ type: "integer", minimum: 1 }); + } await client.close(); }); @@ -147,6 +165,55 @@ describe("external client MCP endpoint", () => { await client.close(); }); + it("passes optional silence thresholds to watches and rejects them without watch=true", async () => { + const client = await connect(); + await expect( + client.callTool({ + name: "thread_watch", + arguments: { thread: "codex://threads/t1", stallAfterMs: 3_000 }, + }), + ).resolves.toMatchObject({ + isError: false, + structuredContent: { stallAfterMs: 3_000, notifyThreadId: "client:grok-bot" }, + }); + expect(watch).toHaveBeenLastCalledWith({ + threadId: "t1", + notifyThreadId: "client:grok-bot", + timeoutMs: 1_740_000, + stallAfterMs: 3_000, + }); + await expect( + client.callTool({ + name: "delegate_start", + arguments: { harnessId: "pi", task: "work", cwd: "/w", watch: true, stallAfterMs: 5_000 }, + }), + ).resolves.toMatchObject({ + isError: false, + structuredContent: { watch: { stallAfterMs: 5_000 } }, + }); + expect(api.start).toHaveBeenLastCalledWith({ + harnessId: "pi", + task: "work", + cwd: "/w", + parentThreadId: "client:grok-bot", + }); + const starts = api.start.mock.calls.length; + for (const args of [ + { stallAfterMs: 3_000 }, + { watch: false, stallAfterMs: 3_000 }, + { watch: true, stallAfterMs: 0 }, + ]) { + await expect( + client.callTool({ + name: "delegate_start", + arguments: { harnessId: "pi", task: "work", cwd: "/w", ...args }, + }), + ).resolves.toMatchObject({ isError: true }); + } + expect(api.start).toHaveBeenCalledTimes(starts); + await client.close(); + }); + it("rejects a connection without a valid client token", async () => { await expect(connect("Bearer chc_nope")).rejects.toThrow(); }); diff --git a/packages/host-runtime/test/harness-delegation-coordinator.test.ts b/packages/host-runtime/test/harness-delegation-coordinator.test.ts index a23a0b256..145348701 100644 --- a/packages/host-runtime/test/harness-delegation-coordinator.test.ts +++ b/packages/host-runtime/test/harness-delegation-coordinator.test.ts @@ -18,6 +18,12 @@ import { describe, expect, it, vi } from "vitest"; import { runDelegationCli } from "../src/delegation-cli.js"; import { startDelegationControlServer } from "../src/delegation-control-server.js"; +import type { + DelegationAdmission, + DelegationAdmissionRequest, + DelegationStartIdentity, + DelegationStartInput, +} from "../src/delegation-types.js"; import { HarnessDelegationCoordinator } from "../src/harness-delegation-coordinator.js"; import { ExternalThreadRepository } from "../src/external-thread-repository.js"; import { ExternalThreadRuntime, type ExternalThread } from "../src/external-thread-runtime.js"; @@ -29,6 +35,12 @@ async function fixture( ) => Promise = async () => undefined, environment: NodeJS.ProcessEnv = {}, externalThreadBusy: (thread: ExternalThread) => boolean = (thread) => thread.running, + options: Partial< + Pick< + ConstructorParameters[0], + "admitDelegation" | "startOfficial" + > + > = {}, ) { const directory = await mkdtemp(path.join(os.tmpdir(), "codexhost-delegation-coordinator-")); const store = new MappingStore({ directory }); @@ -75,7 +87,7 @@ async function fixture( readOfficial: vi.fn(), sendOfficial: vi.fn(), cancelOfficial: vi.fn(), - startOfficial: vi.fn(), + startOfficial: options.startOfficial ?? vi.fn(), listOfficial: vi.fn(async () => ({ threads: [], nextCursor: null })), officialThreadMetadata: async (threadId) => { const metadata = await officialThreadCwd(threadId); @@ -83,6 +95,7 @@ async function fixture( }, activeOfficialParents: () => [], externalThreadBusy, + ...(options.admitDelegation ? { admitDelegation: options.admitDelegation } : {}), }); return { adapter, @@ -861,4 +874,172 @@ describe("HarnessDelegationCoordinator", () => { await value.close(); } }); + + it("returns a queued external start for a top-level client origin and starts session-parented child delegations immediately", async () => { + const admitDelegation = vi.fn(async (request: DelegationAdmissionRequest) => ({ + state: "queued" as const, + identity: request.identity, + status: "queued" as const, + position: 2, + parentThreadId: request.input.parentThreadId, + cwd: request.input.cwd, + title: request.title, + })); + const value = await fixture( + undefined, + async () => ({ cwd: "/parent-workspace", title: "Main project" }), + {}, + undefined, + { admitDelegation }, + ); + try { + const result = await value.coordinator.start({ + harnessId: "pi", + task: "Full task", + taskName: "Review", + cwd: "/parent-workspace", + parentThreadId: "client:claude-desktop", + requestId: "queued-request", + }); + const request = admitDelegation.mock.calls[0]?.[0]; + expect(request).toMatchObject({ + input: { + harnessId: "pi", + task: "Full task", + taskName: "Review", + parentThreadId: "client:claude-desktop", + cwd: path.resolve("/parent-workspace"), + requestId: "queued-request", + }, + title: "Review · claude-desktop", + taskDigest: expect.any(String), + }); + expect(result).toEqual({ + ...request?.identity, + harnessId: "pi", + deepLink: `codex://threads/${request?.identity.threadId}`, + status: "queued", + queue: { position: 2 }, + cwd: path.resolve("/parent-workspace"), + title: "Review · claude-desktop", + parentThreadId: "client:claude-desktop", + next: expect.any(Object), + }); + expect(value.adapter.sessions).toHaveLength(0); + expect(await value.repository.list()).toEqual([]); + + // A session-parented child delegation bypasses the top-level queue. + const child = await value.coordinator.start({ + harnessId: "pi", + task: "Child task", + parentThreadId: "parent-thread", + }); + expect(child).toMatchObject({ status: "running", parentThreadId: "parent-thread" }); + expect(admitDelegation).toHaveBeenCalledOnce(); + expect(value.adapter.sessions).toHaveLength(1); + } finally { + await value.close(); + } + }); + + it("starts an admitted external delegation at once and then frees its slot", async () => { + const release = vi.fn(); + const admitDelegation = vi.fn< + (request: DelegationAdmissionRequest) => Promise + >(async () => ({ state: "admitted", release })); + const value = await fixture(undefined, async () => undefined, {}, undefined, { + admitDelegation, + }); + try { + const result = await value.coordinator.start({ + harnessId: "pi", + task: "Now", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }); + expect(result).toMatchObject({ + ...admitDelegation.mock.calls[0]?.[0].identity, + status: "running", + }); + expect(result).not.toHaveProperty("queue"); + expect(release).toHaveBeenCalledOnce(); + expect(value.adapter.sessions).toHaveLength(1); + } finally { + await value.close(); + } + }); + + it("starts a queued delegation under its identity without asking the limit or deduplicating", async () => { + const admitDelegation = vi.fn(async () => ({ state: "admitted" as const, release: vi.fn() })); + const value = await fixture(undefined, async () => undefined, {}, undefined, { + admitDelegation, + }); + try { + const input = { + harnessId: "pi" as const, + task: "Same task", + cwd: "/synthetic", + parentThreadId: "client:claude-desktop", + }; + const earlier = await value.coordinator.start(input); + admitDelegation.mockClear(); + const identity = { + delegationId: "queued-delegation", + threadId: "queued-thread", + turnId: "queued-turn", + }; + // An identical earlier start would satisfy implicit deduplication; the queue already decided. + const result = await value.coordinator.start(input, identity); + expect(result).toMatchObject({ ...identity, status: "running" }); + expect(result.threadId).not.toBe(earlier.threadId); + expect(admitDelegation).not.toHaveBeenCalled(); + expect( + await value.repository.getDelegationByChild(hostThreadIdSchema.parse("queued-thread")), + ).toMatchObject({ delegationId: "queued-delegation", status: "running" }); + } finally { + await value.close(); + } + }); + + it("leaves the limit of native Codex starts to the Host and passes a queued identity on", async () => { + const admitDelegation = vi.fn(); + const startOfficial = vi.fn( + async ( + input: DelegationStartInput & { parentThreadId: string; cwd: string }, + admitted?: DelegationStartIdentity, + ) => ({ + delegationId: admitted?.delegationId ?? "native-delegation", + threadId: "native-thread", + turnId: "native-turn", + harnessId: "codex", + deepLink: "codex://threads/native-thread", + status: "running" as const, + cwd: input.cwd, + next: { read: "read", wait: "wait" }, + }), + ); + const value = await fixture(undefined, async () => undefined, {}, undefined, { + admitDelegation, + startOfficial, + }); + try { + await expect( + value.coordinator.start({ + harnessId: "codex", + task: "native", + cwd: "/synthetic", + parentThreadId: "parent-thread", + }), + ).resolves.toMatchObject({ threadId: "native-thread", status: "running" }); + const identity = { delegationId: "queued", threadId: "native-thread", turnId: "pending" }; + await value.coordinator.start( + { harnessId: "codex", task: "native", cwd: "/synthetic", parentThreadId: "parent-thread" }, + identity, + ); + expect(admitDelegation).not.toHaveBeenCalled(); + expect(startOfficial.mock.calls.map((call) => call[1])).toEqual([undefined, identity]); + } finally { + await value.close(); + } + }); }); diff --git a/packages/host-runtime/test/remote-delegation-read.test.ts b/packages/host-runtime/test/remote-delegation-read.test.ts new file mode 100644 index 000000000..49ffb688a --- /dev/null +++ b/packages/host-runtime/test/remote-delegation-read.test.ts @@ -0,0 +1,55 @@ +import { expect, it, vi } from "vitest"; + +const reply = vi.hoisted(() => ({ value: {} as unknown })); + +vi.mock("@codexhost/desktop-control", () => ({ + requestDesktopRemoteConnections: vi.fn(async () => reply.value), +})); + +const { readRemoteDelegationThread } = await import("../src/remote-delegation-read.js"); + +it("reads a Thread still waiting in a remote Host's delegation queue", async () => { + reply.value = { + result: { + threadId: "remote-queued", + harnessId: "codex", + status: "queued", + queue: { position: 2 }, + turn: null, + progress: [], + result: { availability: "pending" }, + nextCursor: null, + }, + }; + await expect( + readRemoteDelegationThread( + {}, + { threadId: "remote-queued", hostId: "remote-ssh:mac", view: "result" }, + ), + ).resolves.toMatchObject({ + threadId: "remote-queued", + status: "queued", + queue: { position: 2 }, + hostId: "remote-ssh:mac", + }); +}); + +it("still rejects a snapshot with an unknown status", async () => { + reply.value = { + result: { + threadId: "remote-odd", + harnessId: "codex", + status: "paused", + turn: null, + progress: [], + result: { availability: "pending" }, + nextCursor: null, + }, + }; + await expect( + readRemoteDelegationThread( + {}, + { threadId: "remote-odd", hostId: "remote-ssh:mac", view: "result" }, + ), + ).rejects.toMatchObject({ code: "INTERNAL_ERROR" }); +}); diff --git a/packages/host-runtime/test/thread-activity.test.ts b/packages/host-runtime/test/thread-activity.test.ts new file mode 100644 index 000000000..7c56bc7dd --- /dev/null +++ b/packages/host-runtime/test/thread-activity.test.ts @@ -0,0 +1,169 @@ +import { describe, expect, it, vi } from "vitest"; + +import { ThreadActivity } from "../src/thread-activity.js"; + +describe("ThreadActivity", () => { + it("counts any native event or interaction with a Thread identity, excluding polling replies", () => { + const activity = new ThreadActivity(); + const listener = vi.fn(); + const unsubscribe = activity.subscribe(listener); + activity.observeOfficial({ id: 1, result: { thread: { id: "thread" } } }); + activity.observeOfficial({ method: "account/updated", params: {} }); + expect(listener).not.toHaveBeenCalled(); + for (const method of [ + "turn/started", + "item/reasoning/textDelta", + "item/commandExecution/outputDelta", + "item/fileChange/outputDelta", + "item/commandExecution/requestApproval", + "future/event", + ]) { + activity.observeOfficial({ method, params: { threadId: "thread" } }); + } + expect(listener).toHaveBeenCalledTimes(6); + expect(listener).toHaveBeenLastCalledWith(["thread"]); + unsubscribe(); + activity.note("thread"); + expect(listener).toHaveBeenCalledTimes(6); + }); + + it("uses native subagent ancestry across nested children", () => { + const activity = new ThreadActivity(); + const listener = vi.fn(); + activity.subscribe(listener); + activity.observeOfficial({ + method: "thread/started", + params: { + thread: { + id: "child", + source: { subAgent: { thread_spawn: { parent_thread_id: "parent" } } }, + }, + }, + }); + activity.observeOfficial({ + method: "item/completed", + params: { + threadId: "child", + item: { + type: "collabAgentToolCall", + tool: "spawnAgent", + receiverThreadIds: ["grandchild"], + }, + }, + }); + activity.observeOfficial({ + method: "item/agentMessage/delta", + params: { threadId: "grandchild", delta: "progress" }, + }); + expect(listener).toHaveBeenLastCalledWith(["grandchild", "child", "parent"]); + expect(["parent", "child", "grandchild"].map((id) => activity.isSubagent(id))).toEqual([ + false, + true, + true, + ]); + }); + + function nativeChain(activity: ThreadActivity): void { + for (const [id, parent] of [ + ["B", "A"], + ["C", "B"], + ] as const) { + activity.observeOfficial({ + method: "thread/started", + params: { + thread: { id, source: { subAgent: { thread_spawn: { parent_thread_id: parent } } } }, + }, + }); + } + } + + function closeNative(activity: ThreadActivity, threadId: string): void { + activity.observeOfficial({ method: "thread/closed", params: { threadId } }); + } + + it("keeps the whole chain through a closed intermediate Thread until its subtree closes", () => { + const activity = new ThreadActivity(); + const listener = vi.fn(); + activity.subscribe(listener); + nativeChain(activity); + closeNative(activity, "B"); + activity.observeOfficial({ + method: "item/agentMessage/delta", + params: { threadId: "C", delta: "progress" }, + }); + expect(listener).toHaveBeenLastCalledWith(["C", "B", "A"]); + expect(activity.size).toBe(2); + + closeNative(activity, "C"); + expect(activity.size).toBe(0); + activity.note("C"); + expect(listener).toHaveBeenLastCalledWith(["C"]); + activity.note("B"); + expect(listener).toHaveBeenLastCalledWith(["B"]); + }); + + it("keeps the chain when parents close before their running child", () => { + const activity = new ThreadActivity(); + const listener = vi.fn(); + activity.subscribe(listener); + nativeChain(activity); + closeNative(activity, "A"); + closeNative(activity, "B"); + activity.note("C"); + expect(listener).toHaveBeenLastCalledWith(["C", "B", "A"]); + + closeNative(activity, "C"); + expect(activity.size).toBe(0); + activity.note("C"); + expect(listener).toHaveBeenLastCalledWith(["C"]); + }); + + it("releases a closed parent once its only child moves to another parent", () => { + const activity = new ThreadActivity(); + nativeChain(activity); + closeNative(activity, "B"); + activity.note("C", "D"); + expect(activity.size).toBe(1); + }); + + it("forgets a released Thread with its children's links, and all ancestry when cleared", () => { + const activity = new ThreadActivity(); + const listener = vi.fn(); + activity.subscribe(listener); + activity.note("external", "root"); + activity.note("subagent", "external"); + activity.forget("external"); + activity.note("subagent"); + expect(listener).toHaveBeenLastCalledWith(["subagent"]); + activity.note("external"); + expect(listener).toHaveBeenLastCalledWith(["external"]); + + activity.note("subagent", "external"); + activity.note("other", "root"); + activity.clear(); + expect(activity.size).toBe(0); + activity.note("subagent"); + expect(listener).toHaveBeenLastCalledWith(["subagent"]); + activity.note("other"); + expect(listener).toHaveBeenLastCalledWith(["other"]); + }); + + it("does not turn sendInput recipients into children or loop on malformed ancestry", () => { + const activity = new ThreadActivity(); + const listener = vi.fn(); + activity.subscribe(listener); + activity.observeOfficial({ + method: "item/started", + params: { + threadId: "parent", + item: { type: "collabAgentToolCall", tool: "sendInput", receiverThreadIds: ["other"] }, + }, + }); + activity.note("other"); + expect(listener).toHaveBeenLastCalledWith(["other"]); + expect(activity.isSubagent("other")).toBe(false); + activity.note("a", "b"); + activity.note("b", "a"); + expect(listener).toHaveBeenLastCalledWith(["b", "a"]); + }); +}); diff --git a/packages/renderer-extension/src/renderer-delegation-capacity-preference.ts b/packages/renderer-extension/src/renderer-delegation-capacity-preference.ts new file mode 100644 index 000000000..f53d6d3a5 --- /dev/null +++ b/packages/renderer-extension/src/renderer-delegation-capacity-preference.ts @@ -0,0 +1,111 @@ +import { + DEFAULT_DELEGATION_CAPACITY_SETTINGS, + delegationCapacitySettingsSchema, + type DelegationCapacitySettings, +} from "@codexhost/shared-contracts"; +import type { RendererModelClient } from "./renderer-model-client.js"; +import { RendererMethodUnavailableError } from "./renderer-request-sender.js"; + +export const DELEGATION_CAPACITY_STORAGE_KEY = "codexhost.delegation-capacity.v1"; +export const DELEGATION_CAPACITY_CHANGE_EVENT = "codexhost:delegation-capacity-changed"; +export const DELEGATION_CAPACITY_STATUS_EVENT = "codexhost:delegation-capacity-status"; +export type DelegationCapacitySyncStatus = "pending" | "applied" | "unavailable" | "failed"; + +export function readDelegationCapacityPreference(owner: Window): DelegationCapacitySettings { + try { + const raw = owner.localStorage.getItem(DELEGATION_CAPACITY_STORAGE_KEY); + const parsed = delegationCapacitySettingsSchema.safeParse(raw ? JSON.parse(raw) : null); + if (parsed.success) return parsed.data; + } catch { + /* An unavailable preference store must not start queueing delegations. */ + } + return { ...DEFAULT_DELEGATION_CAPACITY_SETTINGS }; +} + +export function writeDelegationCapacityPreference( + owner: Window, + value: DelegationCapacitySettings, +): boolean { + const parsed = delegationCapacitySettingsSchema.safeParse(value); + if (!parsed.success) return false; + try { + owner.localStorage.setItem(DELEGATION_CAPACITY_STORAGE_KEY, JSON.stringify(parsed.data)); + } catch { + return false; + } + owner.dispatchEvent(new Event(DELEGATION_CAPACITY_CHANGE_EVENT)); + return true; +} + +/** One connection synchronizer per Renderer installation; never uses the active remote route. */ +export function installDelegationCapacityPreferenceSync(owner: Window): { + connect(client: RendererModelClient | null): void; + dispose(): void; +} { + let client: RendererModelClient | null = null; + let disposed = false; + let generation = 0; + let work = { pending: false, dirty: false }; + const publish = (value: DelegationCapacitySyncStatus): void => { + owner.dispatchEvent(new CustomEvent(DELEGATION_CAPACITY_STATUS_EVENT, { detail: value })); + }; + const sync = async (): Promise => { + const currentWork = work; + const version = generation; + currentWork.dirty = true; + if (currentWork.pending || disposed) return; + currentWork.pending = true; + try { + while (currentWork.dirty && !disposed && version === generation) { + currentWork.dirty = false; + const target = client; + if (!target) { + publish("pending"); + continue; + } + if (!target.setDelegationCapacitySettings) { + publish("unavailable"); + continue; + } + // Read at send time, never replay a window's cached preference on reconnect. + const settings = readDelegationCapacityPreference(owner); + publish("pending"); + try { + await target.setDelegationCapacitySettings(settings); + if (!disposed && version === generation && !currentWork.dirty) publish("applied"); + } catch (error) { + if (!disposed && version === generation) { + publish(error instanceof RendererMethodUnavailableError ? "unavailable" : "failed"); + } + } + } + } finally { + currentWork.pending = false; + } + }; + const changed = (): void => { + void sync(); + }; + const storage = (event: StorageEvent): void => { + if (event.key === DELEGATION_CAPACITY_STORAGE_KEY || event.key === null) changed(); + }; + owner.addEventListener(DELEGATION_CAPACITY_CHANGE_EVENT, changed); + owner.addEventListener("storage", storage); + return { + connect(next) { + if (disposed) return; + if (next === client) return; + client = next; + generation += 1; + work = { pending: false, dirty: false }; + void sync(); + }, + dispose() { + disposed = true; + generation += 1; + client = null; + owner.removeEventListener(DELEGATION_CAPACITY_CHANGE_EVENT, changed); + owner.removeEventListener("storage", storage); + }, + }; +} diff --git a/packages/renderer-extension/src/renderer-model-client.ts b/packages/renderer-extension/src/renderer-model-client.ts index 5dd4c5368..01420c34f 100644 --- a/packages/renderer-extension/src/renderer-model-client.ts +++ b/packages/renderer-extension/src/renderer-model-client.ts @@ -45,6 +45,9 @@ import { type LoadedSession, idleReleaseSettingsSchema, type IdleReleaseSettings, + DELEGATION_CAPACITY_SETTINGS_METHOD, + delegationCapacitySettingsSchema, + type DelegationCapacitySettings, harnessAccountInspectParamsSchema, harnessAccountInspectResultSchema, harnessAccountSourceListResultSchema, @@ -229,6 +232,9 @@ export interface RendererModelClient extends Partial; setHarnessLaunchSettings?(input: HarnessLaunchSettingsSet): Promise; setIdleReleaseSettings?(settings: IdleReleaseSettings): Promise; + setDelegationCapacitySettings?( + settings: DelegationCapacitySettings, + ): Promise; listLoadedSessions?(): Promise; currentHostId?(composer?: Element): string | null; knownHostIds?(): readonly string[]; @@ -489,6 +495,14 @@ export function createRendererModelClient( await manager.sendRequest(IDLE_RELEASE_SETTINGS_METHOD, params), ); }, + async setDelegationCapacitySettings( + settings: DelegationCapacitySettings, + ): Promise { + const params = delegationCapacitySettingsSchema.parse(settings); + return delegationCapacitySettingsSchema.parse( + await manager.sendRequest(DELEGATION_CAPACITY_SETTINGS_METHOD, params), + ); + }, ...createRendererSessionImportClient(async (method, params) => manager.sendRequest(method, params), ), diff --git a/packages/renderer-extension/src/settings/appearance-page.ts b/packages/renderer-extension/src/settings/appearance-page.ts index 860c6d8df..1827ba5cf 100644 --- a/packages/renderer-extension/src/settings/appearance-page.ts +++ b/packages/renderer-extension/src/settings/appearance-page.ts @@ -5,6 +5,7 @@ import { } from "../renderer-transcript-dom.js"; import type { RendererSettingsPageDefinition, RendererSettingsPageMountContext } from "./core.js"; import type { RendererSettingsMessages } from "./localization.js"; +import { mountDelegationCapacityControls } from "./delegation-capacity-controls.js"; import { mountIdleReleaseControls } from "./idle-release-controls.js"; import type { LoadedSessionsClient } from "./loaded-sessions-table.js"; import { @@ -59,9 +60,11 @@ export function createAppearanceSettingsPage( messages, getLoadedSessionsClient, ); + const disposeDelegationCapacity = mountDelegationCapacityControls(context, messages); return () => { ownerWindow.removeEventListener(REASONING_SOFT_WRAP_CHANGE_EVENT, sync); disposeIdleRelease(); + disposeDelegationCapacity(); }; }, }); diff --git a/packages/renderer-extension/src/settings/delegation-capacity-controls.ts b/packages/renderer-extension/src/settings/delegation-capacity-controls.ts new file mode 100644 index 000000000..dc852c7e1 --- /dev/null +++ b/packages/renderer-extension/src/settings/delegation-capacity-controls.ts @@ -0,0 +1,144 @@ +import { + DELEGATION_CAPACITY_MAX_RUNNING_MAX, + DELEGATION_CAPACITY_MAX_RUNNING_MIN, + delegationCapacitySettingsSchema, + type DelegationCapacitySettings, +} from "@codexhost/shared-contracts"; +import { + DELEGATION_CAPACITY_CHANGE_EVENT, + DELEGATION_CAPACITY_STATUS_EVENT, + DELEGATION_CAPACITY_STORAGE_KEY, + readDelegationCapacityPreference, + writeDelegationCapacityPreference, + type DelegationCapacitySyncStatus, +} from "../renderer-delegation-capacity-preference.js"; +import type { RendererSettingsMessages } from "./localization.js"; +import type { RendererSettingsPageMountContext } from "./core.js"; +import { + createNumberField, + createPreferenceGroup, + createPreferenceItem, + createPreferenceSwitch, + preferenceId, +} from "./preference-ui.js"; + +export function mountDelegationCapacityControls( + context: RendererSettingsPageMountContext, + messages: RendererSettingsMessages, +): () => void { + const { content } = context; + const document = content.ownerDocument; + const owner = document.defaultView; + if (!owner) return () => undefined; + const { group, header, card } = createPreferenceGroup( + document, + messages.delegationCapacitySection, + ); + + const status = document.createElement("div"); + status.className = + "group/status ml-auto inline-flex items-center gap-1.5 text-xs leading-[18px] text-settings-muted data-[state=failed]:text-settings-danger data-[state=unavailable]:text-settings-warning"; + const statusDot = document.createElement("span"); + statusDot.setAttribute("aria-hidden", "true"); + statusDot.className = + "size-1.5 shrink-0 rounded-full bg-settings-subtle group-data-[state=failed]/status:bg-settings-danger group-data-[state=unavailable]/status:bg-settings-warning"; + const statusText = document.createElement("span"); + statusText.setAttribute("role", "status"); + status.append(statusDot, statusText); + header.append(status); + + const enabledId = preferenceId("delegation-capacity-enabled"); + const toggle = createPreferenceItem(document, { + title: messages.delegationCapacityTitle, + description: messages.delegationCapacityDescription, + controlId: enabledId, + help: { label: messages.delegationCapacityHelpLabel, lines: messages.delegationCapacityHelp }, + }); + const enabled = createPreferenceSwitch(document, enabledId, toggle.description.id); + toggle.item.append(enabled); + + const limitId = preferenceId("delegation-capacity-max-running"); + const limit = createPreferenceItem(document, { + title: messages.delegationCapacityLimit, + description: messages.delegationCapacityLimitDescription, + controlId: limitId, + }); + const { field, control: maxRunning } = createNumberField(document, { + id: limitId, + describedBy: limit.description.id, + unit: messages.delegationCapacityUnit, + min: DELEGATION_CAPACITY_MAX_RUNNING_MIN, + max: DELEGATION_CAPACITY_MAX_RUNNING_MAX, + }); + limit.item.append(field); + limit.item.id = preferenceId("delegation-capacity-limit"); + enabled.setAttribute("aria-controls", limit.item.id); + card.append(toggle.item, limit.item); + content.append(group); + + const statusMessages: Record, string> = { + pending: messages.delegationCapacityPending, + unavailable: messages.delegationCapacityUnavailable, + failed: messages.delegationCapacityFailed, + }; + const showStatus = (value: DelegationCapacitySyncStatus): void => { + status.dataset.state = value; + status.hidden = value === "applied"; + statusText.textContent = value === "applied" ? "" : statusMessages[value]; + }; + const setInvalid = (invalid: boolean): void => { + field.dataset.invalid = String(invalid); + limit.description.dataset.invalid = String(invalid); + limit.description.textContent = invalid + ? messages.delegationCapacityInvalid + : messages.delegationCapacityLimitDescription; + maxRunning.setAttribute("aria-invalid", String(invalid)); + maxRunning.setCustomValidity(invalid ? messages.delegationCapacityInvalid : ""); + }; + const sync = (): void => { + const settings = readDelegationCapacityPreference(owner); + enabled.checked = settings.enabled; + limit.item.hidden = !settings.enabled; + if (!settings.enabled && maxRunning.getAttribute("aria-invalid") === "true") setInvalid(false); + if (maxRunning.getAttribute("aria-invalid") !== "true") { + maxRunning.value = String(settings.maxRunning); + } + }; + const save = (settings: DelegationCapacitySettings): void => { + if (writeDelegationCapacityPreference(owner, settings)) return; + sync(); + showStatus("failed"); + }; + + enabled.addEventListener("change", () => { + save({ ...readDelegationCapacityPreference(owner), enabled: enabled.checked }); + }); + maxRunning.addEventListener("change", () => { + const parsed = delegationCapacitySettingsSchema.safeParse({ + enabled: readDelegationCapacityPreference(owner).enabled, + maxRunning: maxRunning.valueAsNumber, + }); + setInvalid(!parsed.success); + if (parsed.success) save(parsed.data); + }); + maxRunning.addEventListener("input", () => { + if (maxRunning.getAttribute("aria-invalid") === "true") setInvalid(false); + }); + const storage = (event: StorageEvent): void => { + if (event.key === DELEGATION_CAPACITY_STORAGE_KEY || event.key === null) sync(); + }; + const onStatus = (event: Event): void => + showStatus((event as CustomEvent).detail); + owner.addEventListener(DELEGATION_CAPACITY_CHANGE_EVENT, sync); + owner.addEventListener(DELEGATION_CAPACITY_STATUS_EVENT, onStatus); + owner.addEventListener("storage", storage); + sync(); + showStatus("pending"); + // Ask the installed connection synchronizer to report whether the saved setting was applied. + owner.dispatchEvent(new Event(DELEGATION_CAPACITY_CHANGE_EVENT)); + return () => { + owner.removeEventListener(DELEGATION_CAPACITY_CHANGE_EVENT, sync); + owner.removeEventListener(DELEGATION_CAPACITY_STATUS_EVENT, onStatus); + owner.removeEventListener("storage", storage); + }; +} diff --git a/packages/renderer-extension/src/settings/localization.ts b/packages/renderer-extension/src/settings/localization.ts index da5e409d4..3776ce2e5 100644 --- a/packages/renderer-extension/src/settings/localization.ts +++ b/packages/renderer-extension/src/settings/localization.ts @@ -72,6 +72,18 @@ export interface RendererSettingsMessages { readonly idleReleasePending: string; readonly idleReleaseUnavailable: string; readonly idleReleaseFailed: string; + readonly delegationCapacitySection: string; + readonly delegationCapacityTitle: string; + readonly delegationCapacityDescription: string; + readonly delegationCapacityHelpLabel: string; + readonly delegationCapacityHelp: readonly string[]; + readonly delegationCapacityLimit: string; + readonly delegationCapacityLimitDescription: string; + readonly delegationCapacityUnit: string; + readonly delegationCapacityInvalid: string; + readonly delegationCapacityPending: string; + readonly delegationCapacityUnavailable: string; + readonly delegationCapacityFailed: string; readonly reasoningSoftWrapTitle: string; readonly reasoningSoftWrapDescription: string; readonly pageUnavailable: string; @@ -343,6 +355,22 @@ const ENGLISH_MESSAGES: RendererSettingsMessages = Object.freeze({ idleReleasePending: "Syncing…", idleReleaseUnavailable: "Not supported by this Host", idleReleaseFailed: "Sync failed, try again", + delegationCapacitySection: "Delegation", + delegationCapacityTitle: "Limit running delegations", + delegationCapacityDescription: + "When the limit is reached, new delegations wait in a queue and start in order.", + delegationCapacityHelpLabel: "About the delegation limit", + delegationCapacityHelp: Object.freeze([ + "Counts top-level sessions running on this computer, including ones you opened yourself. Subagents are not counted.", + "Only new delegations wait. Your messages and follow-up messages to existing sessions are never held back. Queued delegations are kept across Host restarts and can be cancelled.", + ]), + delegationCapacityLimit: "Running limit", + delegationCapacityLimitDescription: "1–32 sessions.", + delegationCapacityUnit: "sessions", + delegationCapacityInvalid: "Enter a whole number from 1 to 32.", + delegationCapacityPending: "Syncing…", + delegationCapacityUnavailable: "Not supported by this Host", + delegationCapacityFailed: "Sync failed, try again", reasoningSoftWrapTitle: "Wrap thinking text", reasoningSoftWrapDescription: "Wrap long lines in thinking blocks. Shell output is unaffected.", pageUnavailable: "Page unavailable", @@ -657,6 +685,21 @@ const CHINESE_MESSAGES: RendererSettingsMessages = Object.freeze({ idleReleasePending: "同步中…", idleReleaseUnavailable: "当前 Host 不支持", idleReleaseFailed: "同步失败，请重试", + delegationCapacitySection: "委派", + delegationCapacityTitle: "限制同时运行的委派", + delegationCapacityDescription: "达到上限后，新的委派排队等待，有空位时按先后自动开始。", + delegationCapacityHelpLabel: "同时运行的委派上限说明", + delegationCapacityHelp: Object.freeze([ + "计入本机正在运行的顶层会话，包括你手动打开的会话；子代理不计入。", + "只有新的委派会排队；你发送的消息和对已有会话的后续消息不受限制。排队中的委派在 Host 重启后保留，也可以取消。", + ]), + delegationCapacityLimit: "同时运行上限", + delegationCapacityLimitDescription: "1～32 个会话。", + delegationCapacityUnit: "个", + delegationCapacityInvalid: "请输入 1～32 之间的整数。", + delegationCapacityPending: "同步中…", + delegationCapacityUnavailable: "当前 Host 不支持", + delegationCapacityFailed: "同步失败，请重试", reasoningSoftWrapTitle: "换行显示思考文本", reasoningSoftWrapDescription: "思考块中的长行自动换行，不影响 Shell 输出。", pageUnavailable: "页面不可用", diff --git a/packages/renderer-extension/src/versioned-renderer-adapter.ts b/packages/renderer-extension/src/versioned-renderer-adapter.ts index 280354c1d..34dc679a3 100644 --- a/packages/renderer-extension/src/versioned-renderer-adapter.ts +++ b/packages/renderer-extension/src/versioned-renderer-adapter.ts @@ -10,6 +10,7 @@ import { type RendererHostRouting, } from "@codexhost/desktop-control/renderer-bindings"; import { createRendererHostClients } from "./renderer-host-clients.js"; +import { installDelegationCapacityPreferenceSync } from "./renderer-delegation-capacity-preference.js"; import { installIdleReleasePreferenceSync } from "./renderer-idle-release-preference.js"; import { installRendererHarnessPreferenceSync } from "./renderer-harness-preferences.js"; import { @@ -889,6 +890,7 @@ export function installCurrentRendererAdapter(): { const usageSubscription = createThreadUsageSubscriptionRelay(); const idleReleaseSync = installIdleReleasePreferenceSync(window); const preferencesSync = installRendererHarnessPreferenceSync(window); + const delegationCapacitySync = installDelegationCapacityPreferenceSync(window); const clients = createRendererHostClients(() => window.__codexhostHostRoutingV1, window); const stopGroupSync = startAgentGroupSync( getSharedAgentGroupPreferenceStore(), @@ -918,6 +920,7 @@ export function installCurrentRendererAdapter(): { stopGroupSync.connect(); preferencesSync.connect("local", localClient); if (route?.hostId && route.hostId !== "local") preferencesSync.connect(route.hostId, client); + delegationCapacitySync.connect(localClient); // Adapter readiness describes native connections, not a unique global // Composer route. Never use this aggregate readiness to choose a request Host. const connected = @@ -1160,6 +1163,7 @@ export function installCurrentRendererAdapter(): { () => usageSubscription.dispose(), () => idleReleaseSync.dispose(), () => preferencesSync.dispose(), + () => delegationCapacitySync.dispose(), ]; selectedPolicies.clear(); for (const cleanup of cleanups) { diff --git a/packages/renderer-extension/test/renderer-delegation-capacity-preference.test.ts b/packages/renderer-extension/test/renderer-delegation-capacity-preference.test.ts new file mode 100644 index 000000000..dc9584698 --- /dev/null +++ b/packages/renderer-extension/test/renderer-delegation-capacity-preference.test.ts @@ -0,0 +1,137 @@ +import { describe, expect, it, vi } from "vitest"; +import { DELEGATION_CAPACITY_SETTINGS_METHOD } from "@codexhost/shared-contracts"; +import { createRendererModelClient } from "../src/renderer-model-client.js"; +import { + DELEGATION_CAPACITY_STATUS_EVENT, + DELEGATION_CAPACITY_STORAGE_KEY, + installDelegationCapacityPreferenceSync, + readDelegationCapacityPreference, + writeDelegationCapacityPreference, +} from "../src/renderer-delegation-capacity-preference.js"; + +function fixture() { + const values = new Map(); + const storage = { + getItem: vi.fn((key: string) => values.get(key) ?? null), + setItem: vi.fn((key: string, value: string) => { + values.set(key, value); + }), + }; + const owner = Object.assign(new EventTarget(), { localStorage: storage }) as unknown as Window; + const send = vi.fn(async (_method: string, params: unknown) => params); + const client = createRendererModelClient([{ sendRequest: send }]); + if (!client) throw new Error("Missing Renderer client"); + const statuses: string[] = []; + owner.addEventListener(DELEGATION_CAPACITY_STATUS_EVENT, (event) => { + statuses.push((event as CustomEvent).detail); + }); + return { owner, values, storage, send, client, statuses }; +} +const flush = async () => { + for (let i = 0; i < 15; i += 1) await Promise.resolve(); +}; + +describe("delegation capacity preference", () => { + it("defaults off, keeps the saved limit while disabled, and rejects invalid limits", () => { + const f = fixture(); + expect(readDelegationCapacityPreference(f.owner)).toEqual({ enabled: false, maxRunning: 4 }); + expect(writeDelegationCapacityPreference(f.owner, { enabled: true, maxRunning: 2 })).toBe(true); + expect(writeDelegationCapacityPreference(f.owner, { enabled: false, maxRunning: 2 })).toBe( + true, + ); + expect(readDelegationCapacityPreference(f.owner)).toEqual({ enabled: false, maxRunning: 2 }); + for (const maxRunning of [0, 33, 1.5, NaN, Infinity]) { + expect(writeDelegationCapacityPreference(f.owner, { enabled: true, maxRunning })).toBe(false); + } + expect(readDelegationCapacityPreference(f.owner).enabled).toBe(false); + }); + + it("fails safely for corrupt and unavailable storage", () => { + const f = fixture(); + for (const raw of ["broken", "null", '{"enabled":true,"maxRunning":0}']) { + f.values.set(DELEGATION_CAPACITY_STORAGE_KEY, raw); + expect(readDelegationCapacityPreference(f.owner).enabled).toBe(false); + } + f.storage.getItem.mockImplementation(() => { + throw new Error("SecurityError"); + }); + f.storage.setItem.mockImplementation(() => { + throw new Error("QuotaExceededError"); + }); + expect(readDelegationCapacityPreference(f.owner)).toEqual({ enabled: false, maxRunning: 4 }); + expect(writeDelegationCapacityPreference(f.owner, { enabled: true, maxRunning: 4 })).toBe( + false, + ); + }); + + it("sends complete validated settings on connect and changes, then removes listeners", async () => { + const f = fixture(); + const sync = installDelegationCapacityPreferenceSync(f.owner); + sync.connect(f.client); + await flush(); + expect(f.send).toHaveBeenLastCalledWith(DELEGATION_CAPACITY_SETTINGS_METHOD, { + enabled: false, + maxRunning: 4, + }); + writeDelegationCapacityPreference(f.owner, { enabled: true, maxRunning: 32 }); + await flush(); + expect(f.send).toHaveBeenLastCalledWith(DELEGATION_CAPACITY_SETTINGS_METHOD, { + enabled: true, + maxRunning: 32, + }); + expect(f.statuses.at(-1)).toBe("applied"); + sync.connect(f.client); + expect(f.send).toHaveBeenCalledTimes(2); + sync.dispose(); + writeDelegationCapacityPreference(f.owner, { enabled: false, maxRunning: 32 }); + await flush(); + expect(f.send).toHaveBeenCalledTimes(2); + }); + + it("rereads shared storage for another window's change and on reconnect", async () => { + const f = fixture(); + const sync = installDelegationCapacityPreferenceSync(f.owner); + sync.connect(f.client); + await flush(); + f.values.set(DELEGATION_CAPACITY_STORAGE_KEY, JSON.stringify({ enabled: true, maxRunning: 3 })); + f.owner.dispatchEvent( + Object.assign(new Event("storage"), { key: DELEGATION_CAPACITY_STORAGE_KEY }), + ); + await flush(); + expect(f.send).toHaveBeenLastCalledWith(DELEGATION_CAPACITY_SETTINGS_METHOD, { + enabled: true, + maxRunning: 3, + }); + const secondSend = vi.fn(async (_method: string, params: unknown) => params); + sync.connect(createRendererModelClient([{ sendRequest: secondSend }])); + await flush(); + expect(secondSend).toHaveBeenLastCalledWith(DELEGATION_CAPACITY_SETTINGS_METHOD, { + enabled: true, + maxRunning: 3, + }); + sync.dispose(); + }); + + it.each([ + [{ code: -32601 }, "unavailable"], + [new Error("Host is restarting"), "failed"], + ])("reports %o as %s without marking it applied", async (rejection, expected) => { + const f = fixture(); + f.send.mockRejectedValue(rejection); + const sync = installDelegationCapacityPreferenceSync(f.owner); + sync.connect(f.client); + await flush(); + expect(f.statuses.at(-1)).toBe(expected); + expect(f.statuses).not.toContain("applied"); + sync.dispose(); + }); + + it("does not send invalid settings through the public Renderer client", async () => { + const f = fixture(); + if (!f.client.setDelegationCapacitySettings) throw new Error("Missing settings method"); + await expect( + f.client.setDelegationCapacitySettings({ enabled: true, maxRunning: 0 }), + ).rejects.toThrow(); + expect(f.send).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/renderer-extension/test/renderer-external-queue.test.ts b/packages/renderer-extension/test/renderer-external-queue.test.ts index 555f4d0b3..f036dc31a 100644 --- a/packages/renderer-extension/test/renderer-external-queue.test.ts +++ b/packages/renderer-extension/test/renderer-external-queue.test.ts @@ -1,5 +1,7 @@ import { + DELEGATION_CAPACITY_SETTINGS_METHOD, IDLE_RELEASE_SETTINGS_METHOD, + delegationCapacitySettingsSchema, idleReleaseSettingsSchema, } from "@codexhost/shared-contracts"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -77,6 +79,9 @@ function fixture() { async sendRequest(method: string, params: unknown) { // Settings sync shares the connection but is not a queue operation. if (method === IDLE_RELEASE_SETTINGS_METHOD) return idleReleaseSettingsSchema.parse(params); + if (method === DELEGATION_CAPACITY_SETTINGS_METHOD) { + return delegationCapacitySettingsSchema.parse(params); + } if ( method.startsWith("thread/queue/") && params !== null && diff --git a/packages/renderer-extension/test/renderer-host-routing.test.ts b/packages/renderer-extension/test/renderer-host-routing.test.ts index 30cce484b..c0823b3ec 100644 --- a/packages/renderer-extension/test/renderer-host-routing.test.ts +++ b/packages/renderer-extension/test/renderer-host-routing.test.ts @@ -27,6 +27,7 @@ function manager(hostId: string) { if (method === "codexhost/harness/inspect") return ready; if (method === "codexhost/thread/inspect") return { owner: "codex", locked: true }; if (method === "codexhost/settings/idle-release/set") return params; + if (method === "codexhost/settings/delegation-capacity/set") return params; return { threadId: "thread-test", usage: null }; }); const requestClient = { @@ -134,6 +135,10 @@ it.each(["local", remoteId])( "codexhost/settings/idle-release/set", expect.anything(), ); + expect(fixture.remote.nativeSend).not.toHaveBeenCalledWith( + "codexhost/settings/delegation-capacity/set", + expect.anything(), + ); } finally { adapter.dispose(); } diff --git a/packages/renderer-extension/test/renderer-model-client.test.ts b/packages/renderer-extension/test/renderer-model-client.test.ts index 39df89c2b..805820e19 100644 --- a/packages/renderer-extension/test/renderer-model-client.test.ts +++ b/packages/renderer-extension/test/renderer-model-client.test.ts @@ -454,6 +454,7 @@ describe("Renderer fixed Model request client", () => { "selectThreadModel", "selectThreadPermissionMode", "selectThreadThinking", + "setDelegationCapacitySettings", "setHarnessDisplaySettings", "setHarnessLaunchSettings", "setIdleReleaseSettings", diff --git a/packages/renderer-extension/test/versioned-renderer-adapter.test.ts b/packages/renderer-extension/test/versioned-renderer-adapter.test.ts index dc71cb0cc..127abd9e9 100644 --- a/packages/renderer-extension/test/versioned-renderer-adapter.test.ts +++ b/packages/renderer-extension/test/versioned-renderer-adapter.test.ts @@ -482,6 +482,10 @@ describe("current Codex Renderer Agent adapter", () => { "codexhost/harness/preferences/get", { scope: "external" }, ); + expect(requestTarget.sendRequest).toHaveBeenCalledWith( + "codexhost/settings/delegation-capacity/set", + { enabled: false, maxRunning: 4 }, + ); const localClient = adapter.modelControl?.clientForHost?.("local"); expect(localClient).toBeTruthy(); // The local target remains discoverable while the remote route is active. @@ -507,7 +511,7 @@ describe("current Codex Renderer Agent adapter", () => { fakeWindow.__codexhostDraftPrewarmPolicyV1 = policy; expect(adapter.modelControl?.clientForHost?.("local")).toBe(localClient); } - expect(requestTarget.sendRequest).toHaveBeenCalledTimes(2); + expect(requestTarget.sendRequest).toHaveBeenCalledTimes(3); expect(requestTarget.sendRequest).toHaveBeenCalledWith( "codexhost/harness/display-settings/get", {}, @@ -552,7 +556,10 @@ describe("current Codex Renderer Agent adapter", () => { ).toBe(false); } expect( - requestTarget.sendRequest.mock.calls.some(([method]) => method.endsWith("/set")), + requestTarget.sendRequest.mock.calls.some( + ([method]) => + method.endsWith("/set") && method !== "codexhost/settings/delegation-capacity/set", + ), ).toBe(false); } finally { adapter.dispose(); diff --git a/packages/shared-contracts/src/delegation-capacity.ts b/packages/shared-contracts/src/delegation-capacity.ts new file mode 100644 index 000000000..3695d7c11 --- /dev/null +++ b/packages/shared-contracts/src/delegation-capacity.ts @@ -0,0 +1,22 @@ +import { z } from "zod"; + +export const DELEGATION_CAPACITY_SETTINGS_METHOD = "codexhost/settings/delegation-capacity/set"; +export const DELEGATION_CAPACITY_MAX_RUNNING_MIN = 1; +export const DELEGATION_CAPACITY_MAX_RUNNING_MAX = 32; + +export const delegationCapacitySettingsSchema = z.strictObject({ + enabled: z.boolean(), + maxRunning: z + .number() + .int() + .min(DELEGATION_CAPACITY_MAX_RUNNING_MIN) + .max(DELEGATION_CAPACITY_MAX_RUNNING_MAX), +}); + +export type DelegationCapacitySettings = z.infer; + +export const DEFAULT_DELEGATION_CAPACITY_SETTINGS: Readonly = + Object.freeze({ + enabled: false, + maxRunning: 4, + }); diff --git a/packages/shared-contracts/src/index.ts b/packages/shared-contracts/src/index.ts index f0578d899..bf72db5e9 100644 --- a/packages/shared-contracts/src/index.ts +++ b/packages/shared-contracts/src/index.ts @@ -35,6 +35,14 @@ export { idleReleaseSettingsSchema, type IdleReleaseSettings, } from "./idle-release.js"; +export { + DELEGATION_CAPACITY_SETTINGS_METHOD, + DELEGATION_CAPACITY_MAX_RUNNING_MAX, + DELEGATION_CAPACITY_MAX_RUNNING_MIN, + DEFAULT_DELEGATION_CAPACITY_SETTINGS, + delegationCapacitySettingsSchema, + type DelegationCapacitySettings, +} from "./delegation-capacity.js"; export { LOADED_SESSIONS_METHOD, loadedSessionsSchema, diff --git a/tests/e2e/renderer-delegation-capacity.spec.ts b/tests/e2e/renderer-delegation-capacity.spec.ts new file mode 100644 index 000000000..1a1007745 --- /dev/null +++ b/tests/e2e/renderer-delegation-capacity.spec.ts @@ -0,0 +1,127 @@ +import { expect, test, type Page } from "@playwright/test"; +import { build } from "esbuild"; +import path from "node:path"; + +import { tailwindEsbuildPlugin } from "../../packages/renderer-extension/scripts/tailwind-esbuild-plugin.mjs"; + +const browserExecutable = process.env.CODEXHOST_PLAYWRIGHT_EXECUTABLE_PATH; +if (browserExecutable) test.use({ launchOptions: { executablePath: browserExecutable } }); +const { outputFiles } = await build({ + stdin: { + contents: ` + import { createAppearanceSettingsPage } from "./packages/renderer-extension/src/settings/appearance-page.ts"; + import { createRendererSettingsPageRegistry } from "./packages/renderer-extension/src/settings/core.ts"; + import { rendererSettingsMessages } from "./packages/renderer-extension/src/settings/localization.ts"; + import { mountRendererSettingsShell } from "./packages/renderer-extension/src/settings/shell.ts"; + import { createRendererModelClient } from "./packages/renderer-extension/src/renderer-model-client.ts"; + import { installIdleReleasePreferenceSync } from "./packages/renderer-extension/src/renderer-idle-release-preference.ts"; + import { installDelegationCapacityPreferenceSync } from "./packages/renderer-extension/src/renderer-delegation-capacity-preference.ts"; + const METHOD = "codexhost/settings/delegation-capacity/set"; + globalThis.setupDelegationCapacity = (unsupported = false) => { + const calls = []; + const client = createRendererModelClient([{sendRequest: async (method, params) => { + if (method === METHOD) calls.push(params); + if (unsupported && method === METHOD) throw {code:-32601}; + return params; + }}]); + const idle = installIdleReleasePreferenceSync(window); + const capacity = installDelegationCapacityPreferenceSync(window); + idle.connect(client); + capacity.connect(client); + const messages = rendererSettingsMessages("zh-CN"); + const registry = createRendererSettingsPageRegistry([createAppearanceSettingsPage(messages)]); + const shell = mountRendererSettingsShell(registry, document, messages); + shell.openSettings(undefined, "appearance"); + globalThis.capacityFixture = {calls, dispose: () => {shell.dispose(); capacity.dispose(); idle.dispose();}}; + }; + `, + resolveDir: path.resolve(import.meta.dirname, "../.."), + sourcefile: "delegation-capacity-e2e-entry.ts", + loader: "ts", + }, + bundle: true, + format: "iife", + platform: "browser", + target: "es2024", + loader: { ".css": "text", ".png": "dataurl", ".svg": "dataurl" }, + plugins: [tailwindEsbuildPlugin()], + write: false, +}); +const bundle = outputFiles[0]?.text ?? ""; +if (!bundle) throw new Error("Missing settings fixture bundle"); +async function setup(page: Page, unsupported = false) { + await page.route("http://localhost/delegation-capacity-test", (route) => + route.fulfill({ + contentType: "text/html", + body: "", + }), + ); + await page.goto("http://localhost/delegation-capacity-test"); + await page.addScriptTag({ content: bundle }); + await page.evaluate( + (value) => Reflect.get(globalThis, "setupDelegationCapacity")(value), + unsupported, + ); +} +function lastSent(page: Page): Promise { + return page.evaluate(() => Reflect.get(globalThis, "capacityFixture").calls.at(-1) ?? null); +} + +test("the delegation limit is off by default and edits apply to the Host at once", async ({ + page, +}) => { + await setup(page); + const region = page.getByRole("region", { name: "委派", exact: true }); + const enabled = region.getByRole("switch", { name: "限制同时运行的委派" }); + const limit = region.getByRole("spinbutton", { name: "同时运行上限" }); + await expect(enabled).not.toBeChecked(); + // The limit has no effect while the switch is off, so it is not shown. + await expect(limit).toBeHidden(); + await expect.poll(() => lastSent(page)).toEqual({ enabled: false, maxRunning: 4 }); + + await region.getByRole("button", { name: "同时运行的委派上限说明" }).hover(); + const tooltip = page.getByRole("tooltip"); + await expect(tooltip).toContainText("子代理不计入"); + await expect(tooltip).toContainText("排队中的委派在 Host 重启后保留"); + + await enabled.click(); + await expect(enabled).toBeChecked(); + await expect(limit).toBeVisible(); + await expect(limit).toHaveValue("4"); + await expect(limit).toHaveAttribute("min", "1"); + await expect(limit).toHaveAttribute("max", "32"); + await expect.poll(() => lastSent(page)).toEqual({ enabled: true, maxRunning: 4 }); + + await limit.fill("0"); + await limit.press("Tab"); + await expect(limit).toHaveAttribute("aria-invalid", "true"); + await expect(region.getByText("请输入 1～32 之间的整数。")).toBeVisible(); + await expect.poll(() => lastSent(page)).toEqual({ enabled: true, maxRunning: 4 }); + await limit.fill("2"); + await limit.press("Tab"); + await expect(limit).toHaveAttribute("aria-invalid", "false"); + await expect.poll(() => lastSent(page)).toEqual({ enabled: true, maxRunning: 2 }); + // Successful sync is silent. + await expect(region.getByText("同步中…")).toBeHidden(); + expect( + await page.evaluate(() => + JSON.parse(localStorage.getItem("codexhost.delegation-capacity.v1") ?? "null"), + ), + ).toEqual({ enabled: true, maxRunning: 2 }); + + await page.reload(); + await page.addScriptTag({ content: bundle }); + await page.evaluate(() => Reflect.get(globalThis, "setupDelegationCapacity")()); + await expect(enabled).toBeChecked(); + await expect(limit).toHaveValue("2"); + await expect.poll(() => lastSent(page)).toEqual({ enabled: true, maxRunning: 2 }); +}); + +test("a Host without the delegation limit is reported rather than shown as applied", async ({ + page, +}) => { + await setup(page, true); + await expect( + page.getByRole("region", { name: "委派", exact: true }).getByRole("status"), + ).toHaveText("当前 Host 不支持"); +}); diff --git a/tests/e2e/renderer-idle-release.spec.ts b/tests/e2e/renderer-idle-release.spec.ts index 466f93106..e532f0441 100644 --- a/tests/e2e/renderer-idle-release.spec.ts +++ b/tests/e2e/renderer-idle-release.spec.ts @@ -120,7 +120,7 @@ test("General groups appearance and idle release controls without dialogs or lon ) .toBe(6); // Successful sync is silent; the scope badge is no longer shown. - await expect(page.getByText("同步中…")).toBeHidden(); + await expect(page.getByRole("region", { name: "资源管理" }).getByText("同步中…")).toBeHidden(); await expect(page.getByText("仅本地 Host", { exact: true })).toHaveCount(0); expect( await page.evaluate(() =>