Skip to content

Make experiment launches recoverable and network delivery explicit - #467

Merged
ctfbruce merged 10 commits into
mainfrom
fix/reliable-experiments
Nov 5, 2026
Merged

ctfbruce merged 10 commits into
mainfrom
fix/reliable-experiments

Conversation

@ctfbruce

@ctfbruce ctfbruce commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Coordinated experiments could lose queued TCP messages during close, and canceled uploads or queued runs could leave incomplete cancellation evidence. This change makes those paths bounded and recoverable while improving how experiments are authored and inspected.

  • Drain native TCP writes before releasing socket accounting, propagate resets and close failures, and preserve message deadlines.
  • Record cancellation outcomes independently of an expired caller, finalize queued cancellations, and bound persistent admission waits.
  • Show generated Go previews without compilation and preserve explicit manual functions alongside the original block diagram. Restoring blocks is an explicit operation.
  • Add execution-linked activity events and composable byte payloads with text, hex and base64 encoding, exact byte counts, binary HTTP bodies and TCP delimiters.
  • Constrain OpenAI proposals with a strict graph schema, validate and generate every participant, allow one bounded correction, and return safe, actionable failure reasons through the additive API 1.32 error field. Other compatible providers retain their existing document format.

Validation: focused package tests and race checks passed. Two-executor live checks covered 512 message frames, bounded canceled-peer failure, and 14 byte/network cases per executor with independent receiver verification. The generated DNS comparison also ran successfully: both executors exchanged their selected addresses, all four TCP connections succeeded, and both reported the correctly ordered timing difference. Provider-backed TCP, DNS and coordinated drafts all compiled; observed generation times were 2.8 seconds, 3.9 seconds and 40.2 seconds respectively. The coordinated draft used 17 blocks per participant and required one correction. All 17 checks on the final pull-request head passed, including normal tests, race, kernel, package/install, compatibility and the required aggregate. The complete console candidate also passed 423 unit tests and 49 browser tests. A real SDK reset regression found during integration was fixed without weakening its assertion. Locally, the API package needed 252.961 seconds and exceeded the four-minute advisory run; the isolated 779-test run and the final GitHub normal-test check passed.

Limits: model output still requires review, and these measurements do not establish a latency guarantee. Manual Go cannot be converted back into blocks; destination and capability hints still come from the retained graph. Run policies accept explicit IP addresses and DNS names, not CIDRs; a resolver-selected address outside the executor's allowed resolved set remains blocked. TCP acknowledgment establishes transport delivery, not peer application processing. Event timestamps use each guest's clock and do not establish cross-machine ordering.

Closes #466.

@ctfbruce
ctfbruce marked this pull request as ready for review November 5, 2026 09:51
@ctfbruce
ctfbruce merged commit 721e35b into main Nov 5, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recover admitted experiments after upload timeout and queued cancellation

1 participant