Skip to content

Add scoped renewable guest TCP tunnels for SSH adapters - #79

Merged
jerryfane merged 3 commits into
mainfrom
forge/raw-ssh-transport
Oct 31, 2026
Merged

jerryfane merged 3 commits into
mainfrom
forge/raw-ssh-transport

Conversation

@jerryfane

@jerryfane jerryfane commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Closes #78.

Assignment: Jarvis376089; Forge #4 integration/acceptance belongs to deimos-forge. Contract accepted in 376122/376124. Implementer: sandboxd seat (in-session), not a review job.

Change

  • Controller-authenticated, bounded, memory-only tunnel leases scoped to one live E2B sandbox and exposed guest port. Explicit renewal preserves tokens and connections; expired/revoked leases cannot revive.
  • HTTP/1.1 CONNECT with exclusive E2b-Tunnel-Access-Token, sandboxID:port authority, no host resolution or credential forwarding, shared guest-port stream cap.
  • Active and dialing streams are cancelled on revoke/expiry, sandbox teardown, worker forgetting or daemon shutdown. Disconnect does not kill guest work.
  • New framed guest-agent operation and remote-worker upgrade preserve directional EOF with bounded buffers/backpressure. Unsupported old agents/workers fail explicitly. Dev driver preserves socket half-close. Apple driver remains unsupported for raw tunnels.
  • Existing compatibility documentation covers API, lifecycle and guest/worker version requirements.

Verification

  • Installed Go1.26.10, GOMAXPROCS=2, go test -p 2 ./...: all packages passed.
  • test-check prove --test "env GOTOOLCHAIN=go1.26.10 GOMAXPROCS=2 go test -p 2 ./internal/control ./internal/guestagent -v -run '^{name}$'": all four new tests PROVEN, red on old code and green on new. Receipt: /root/.local/state/test-check/receipts/20261031T093520Z-5e306e7ed76c9f1bc7ec38d940c26402/receipt.json.
  • Live sandboxd-dev on isolated loopback43478: CONNECT200; 65536 binary bytes sent, client write-EOF, exact 65552-byte reverse response; renew200 same token; revoke204 closes active stream; reconnect403; guest remains200; guest cleanup204; dedicated daemon exits0. No shared services or Forge image changed.
  • Evidence: /root/fleet-tools/state/sandboxd-78-tunnel/verification.md.

Deployment and acceptance boundary

No shared daemon, listener, credentials, firewall, Phobos or Mac changes. Forge must rebuild its own image with matching guest agent and pin reviewed source before product acceptance. Actual Firecracker and HerdrUp/Citadel SSH acceptance are not yet verified. Memory-only lease rollback is fail-closed: restart invalidates leases, Forge remints. Old binaries do not offer the new CONNECT API.

Final expanded proof: all FIVE tests proven, including remote worker directional
EOF. Command added ./internal/worker to the proof package list.
Receipt: /root/.local/state/test-check/receipts/20261031T093823Z-ef1e2d8746c17126b6a4e53069902b8a/receipt.json.
Race check: go test -p 2 -race ./internal/control ./internal/guestagent ./internal/worker -run TestTunnel -count=1 -v passed all five tests.

Independent review correction: cancellation after directional EOF

Review job local-review-review-router-18e3a7ab4a909cce-1 requested changes
at 446ac8326b18a521a1c7cd9795168313c0150350: after a framed EOF,
the bridge stopped reading that transport. A later transport close left an
idle reverse copy and its downstream connection alive. The reviewer reproduced
this both directly and through the actual worker HTTP upgrade; the earlier
passing tests did not exercise this sequence.

The bridge now reports directional EOF, continues observing its framed source
for closure or illegal post-EOF bytes, and closes both connections and joins
both copy goroutines on cancellation or completion. No extra reader goroutine,
unbounded buffer, timeout workaround or protocol change.

Added direct bidirectional bridge coverage, an actual worker upgrade/TCP guest
regression, and a guest-agent accepted-socket disconnect case. They keep the
opposite peer open and idle, so cleanup cannot pass because the guest closes.
The direct regression also verifies reverse traffic still works after EOF.
Existing operator documentation now states this lifecycle guarantee explicitly.

At correction head 1920aa086904bca8cda1eae836bc477b4caee94d, independent
review job local-review-review-router-18e3a8d9c2494785-1 approved and marked
the prior P2 answered. Reviewer executed build, focused vet, four affected
package suites, all tunnel regressions 20 times under race detection, and a
memory-only production-Bridge smoke (transport close, illegal post-EOF data,
both directional EOFs, reverse payload preservation): all passed. Initial
PATH-based attempts failed because Go was absent from PATH; absolute-path
commands succeeded. These are reviewer executions, not local reruns.

The raw verdict declares evidence=executed, but Gitmoot normalized it to
static_only; this discrepancy is escalated in message377713, not overridden.
CI did not run on1920aa0 because merging PR81 into main introduced one service
field conflict. Integrated reviewed main at c9c7e0709b538bac9c185a8269c8ce4c7cc92608,
retaining both tunnel leases and volume transfer slots. Final integrated-head
CI and independent review remain required.

Source-host build/test/go-run remains prohibited by the disk hold. The original
review supplies failing-before runtime evidence and the correction review
supplies passing-after evidence. New permanent tests are green, but their
own isolated red-on-old proof/test-check run is still unverified.
Remote reviewer: if feasible, use a memory-only Go overlay replacing ONLY
internal/tunnel/stream.go with its 446ac832 version to establish red/green
for TestBridgeTransportCloseAfterDirectionalEOF,
TestTunnelRemoteCloseAfterDirectionalEOF and
TestTunnelDirectionalEOF/host-disconnect without editing repository files.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Review status at head 446ac83: level3_required. Implementer attribution saved as session-implement-sandboxd-18e393c543132728 (sandboxd seat). g7-review independent review-only dispatch was refused before creating a job: Janitor disk-pressure admission_hold on /root/.gitmoot, cleanup_needed. Escalated to Jarvis; no bypass or merge. Actual Forge Firecracker/HerdrUp acceptance remains pending reviewed pin. All five new tests proven red on old/green on new; race run passed.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Canonical independent review requested at exact 446ac83: job local-review-review-router-18e3a7ab4a909cce-1. Remote execution backend, read-only, no automatic fix target; sandboxd owns findings. Reviewer g7-review is independent of implementation but uses the same model family. Current observed state: queued, not yet evidence of remote execution or approval. No local build or merge; existing CI remains passing.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e3a7ab4a909cce-1

Decision: changes_requested
Summary: Reviewed the complete diff at head 446ac83 against base b3247af. Builds, vet, existing tunnel tests, race checks, and correctly tagged dev-driver tests passed. Two independent runtime probes reproduced a cancellation defect after directional EOF, including through the actual worker HTTP upgrade. The frozen prior-verdicts file contains no PR #79 verdicts as of 2026-10-31T15:53:55Z, so there are no prior findings to answer. No repository files were edited.
Severity: P2

Findings

  • Transport closure after directional EOF leaves downstream tunnel resources alive (P2)
    • detail: After a client sends the framed directional EOF, the client-to-guest copy completes successfully and Bridge waits for the reverse copy. Nothing continues reading the client transport, so subsequently closing that transport cannot interrupt the reverse copy while the guest remains open and idle. This affects disconnect and lease revocation across the remote-worker and guest-agent framed transports: the controller closes its connection, but downstream guest connections and bridge goroutines remain until the guest independently sends data or closes, or separate teardown intervenes. A direct production-Bridge probe reproduced this, and a second probe using the actual worker Client.DialTunnel and server upgrade confirmed that closing the client connection did not close the worker's guest connection; closing the guest released it. Preserve observation of transport cancellation after directional EOF and add regression coverage for this sequence.
    • locator: 446ac83: internal/tunnel/stream.go:161-179, especially 175-179; callers internal/worker/server.go:560-565 and internal/guestagent/server_linux.go:150-155. Executed evidence: TestReviewTransportCloseAfterDirectionalEOF and TestReviewRemoteCloseAfterDirectionalEOF, detailed in tests_run.

Tests Run

  • {"command":"/usr/local/go/bin/go version && /usr/local/go/bin/go build ./...","outcome":"PASS; Go 1.26.4 linux/amd64."}
  • {"command":"/usr/local/go/bin/go vet ./...","outcome":"PASS."}
  • {"command":"/usr/local/go/bin/go test -count=1 -v -timeout=120s ./internal/control ./internal/guestagent ./internal/worker -run Tunnel","outcome":"PASS; exercised CONNECT binary transfer and buffering, credential isolation, stream limits, both directional EOF orders, lease renewal/expiry/revocation, sandbox destruction, shutdown, and remote-worker transport."}
  • {"command":"/usr/local/go/bin/go test -count=1 -timeout=180s ./internal/control ./internal/envd ./internal/guestagent ./internal/tunnel ./internal/worker ./internal/vm ./internal/vm/devvm","outcome":"Control, envd, guestagent, worker, and vm suites passed; tunnel has no committed tests. Overall command failed because devvm requires the sandboxd_devdriver build tag. The correctly tagged check below passed."}
  • {"command":"/usr/local/go/bin/go test -race -count=3 -timeout=120s ./internal/control ./internal/guestagent ./internal/worker -run Tunnel","outcome":"PASS; all three repetitions passed without reported races."}
  • {"command":"/usr/local/go/bin/go build -tags sandboxd_devdriver ./... && /usr/local/go/bin/go vet -tags sandboxd_devdriver ./...","outcome":"PASS; tagged build and vet."}
  • {"command":"/usr/local/go/bin/go test -race -tags sandboxd_devdriver -count=1 -timeout=120s ./internal/vm/devvm","outcome":"PASS."}
  • {"command":"/usr/local/go/bin/go test -overlay=/proc/1606/fd/10 -count=1 -v -timeout=10s ./internal/tunnel -run ^TestReviewTransportCloseAfterDirectionalEOF$","outcome":"FAIL, reproducing the finding. After the real TCP guest observed directional EOF, closing the framed peer transport left production Bridge blocked for the 500 ms observation window; closing the guest released it. Probe source and overlay existed only in memory and were released afterward."}
  • {"command":"/usr/local/go/bin/go test -overlay=/proc/1606/fd/10 -count=1 -v -timeout=10s ./internal/worker -run ^TestReviewRemoteCloseAfterDirectionalEOF$","outcome":"FAIL, independently reproducing the finding through the actual worker client/server upgrade. Output: 'worker retained guest connection after client transport close; only guest close released it'. Probe source and overlay existed only in memory and were released afterward."}

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Addressing P2 from review local-review-review-router-18e3a7ab4a909cce-1 at new head1920aa086904bca8cda1eae836bc477b4caee94d. Framed EOF now reports directional completion while its existing copy goroutine continues observing raw transport closure; cancellation closes both endpoints and joins both copies. Added real worker-upgrade/TCP and guest-agent socket regressions plus both-direction framed bridge coverage preserving reverse traffic. Documentation updated. Local execution prohibited by source-host disk hold; no local builds/tests/go-run, only standalone gofmt. Same reviewer remote re-review local-review-review-router-18e3a8d9c2494785-1 queued, baseline446ac832; independent runtime smoke and new-tests red/green requested in PR description. Verification pending, not claiming the correction passes yet.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e3a8d9c2494785-1

Decision: approved
Summary: Reviewed exact head 1920aa0 against prior head 446ac83, limited to the five listed changed files and necessary dependencies. The prior P2 is resolved; no new defects found in this delta. Build, focused vet, affected-package suites, repeated race-enabled regressions, and an independent production-Bridge probe passed. No repository files changed.

Findings

  • Resolved: transport closure after directional EOF leaves downstream tunnel resources alive (P2)
    • continues_uid: Add scoped renewable guest TCP tunnels for SSH adapters #79-f1
    • detail: Bridge now continues reading the framed transport after directional EOF, closes both connections on transport termination or post-EOF bytes, and waits for both copy goroutines to exit. Executed TestBridgeTransportCloseAfterDirectionalEOF in both directions, TestTunnelRemoteCloseAfterDirectionalEOF through the worker upgrade, and TestTunnelDirectionalEOF/host-disconnect through the guest-agent socket wrapper, each 20 times with the race detector. All passed while the opposite peer remained open and idle. Existing directional-EOF tests also passed. An independent production-Bridge probe confirmed reverse payload preservation and downstream closure after transport cancellation, invalid post-EOF data, and completion of both directional EOFs.
    • file: internal/tunnel/stream.go
    • line: 175
    • locator: 1920aa0:internal/tunnel/stream.go:165-195
    • state: answered

Tests Run

  • Initial PATH-based attempts of go build ./..., go vet ./internal/tunnel ./internal/worker ./internal/guestagent, and the focused go test command failed with exit 127 because go was absent from PATH. Located /usr/local/go/bin/go and executed the checks successfully using its absolute path.
  • /usr/local/go/bin/go build ./... — PASS.
  • /usr/local/go/bin/go vet ./internal/tunnel ./internal/worker ./internal/guestagent — PASS, no diagnostics.
  • /usr/local/go/bin/go test -race -count=20 -timeout=180s ./internal/tunnel ./internal/worker ./internal/guestagent -run 'TestBridgeTransportCloseAfterDirectionalEOF|TestTunnelRemoteDirectionalEOF|TestTunnelRemoteCloseAfterDirectionalEOF|TestTunnelDirectionalEOF' -v — PASS in all three packages; all selected tests and subtests passed across 20 repetitions, with no race reports.
  • /usr/local/go/bin/go test -count=1 -timeout=180s ./internal/tunnel ./internal/worker ./internal/guestagent ./internal/control — PASS in all four packages.
  • /usr/local/go/bin/go run -race -overlay /proc/1602/fd/10 /home/user/workspace/review_probe.go — PASS, exit 0. An in-memory overlay supplied the probe without creating or editing repository files. Direct production Bridge scenarios transport-close, data-after-eof, and both-directional-eofs each preserved reverse payload, terminated the bridge, and closed the downstream connection.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Correction to evidence-metadata diagnosis: Gitmoot established issue gitmoot/gitmoot#2465 from the actual clamp event. The earlier1920aa0 review was downgraded because its tests_run honestly recorded initial PATH exit127 failures, despite subsequent successful absolute-path build/vet/tests/race/runtime smoke. This is NOT a string-versus-object schema issue. Raw verdict and failed attempts are preserved; no evidence rewrite or gate bypass. Final integrated headc9c7e0709b538bac9c185a8269c8ce4c7cc92608 has its own canonical review pending; old verdict is not treated as exact-head approval.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e3aa764d1296db-1

Decision: approved
Summary: Reviewed exact head c9c7e07 against prior head 1920aa0, limited to the listed changes, necessary dependencies, and the prior finding. No new correctness defects found. The prior tunnel-resource finding remains resolved. Build, vet, race-enabled package tests, focused regressions, and an independent HTTP smoke probe passed. No repository files were edited or live-service actions performed.

Findings

  • Resolved: transport closure after directional EOF releases downstream resources (P2)
    • continues_uid: Add scoped renewable guest TCP tunnels for SSH adapters #79-f1
    • detail: Checked at this exact head. Bridge continues observing framed transport after directional EOF, closes both connections on transport termination or illegal post-EOF bytes, and waits for both copy goroutines. Executed TestBridgeTransportCloseAfterDirectionalEOF, TestTunnelRemoteCloseAfterDirectionalEOF, TestTunnelRemoteDirectionalEOF, and TestTunnelDirectionalEOF with the race detector and -count=20. All passed, including both bridge directions and the guest-agent host-disconnect case.
    • file: internal/tunnel/stream.go
    • line: 175
    • locator: c9c7e07:internal/tunnel/stream.go:165-195
    • state: answered

Tests Run

  • Initial attempts using go build ./... && go vet ./... and go test -race ./internal/vm ./internal/control ./internal/tunnel ./internal/worker ./internal/guestagent -count=1 exited 127 because Go was absent from PATH. Located the installed toolchain and successfully executed the checks below.
  • /usr/local/go/bin/go version && /usr/local/go/bin/go build ./... && /usr/local/go/bin/go vet ./... — PASS; Go 1.26.4 linux/amd64.
  • /usr/local/go/bin/go test -race ./internal/vm ./internal/control ./internal/tunnel ./internal/worker ./internal/guestagent -count=1 — PASS for all five packages.
  • /usr/local/go/bin/go test -race ./internal/vm -run '^(TestFirecrackerColdBackupRoundTrip|TestFirecrackerRestoreFailureCleanup|TestFirecrackerRestoreReservesQuotaAndName|TestVolumeBackupEnvelopeBounds|TestFirecrackerVolumeRecoveryRequiresClaim|TestFirecrackerVolumeCrashLeftovers|TestFirecrackerVolumeStartupRepairs)$' -count=1 -v — PASS. Exercised native backup/restore, attachment and deletion exclusion, truncation/digest/trailing-data/cancellation cleanup, name and quota reservations, manifest bounds, and claim-gated recovery. The startup-repair test did not exercise its root-only foreign-owner check because the process was non-root; Firecracker host operations use the existing test fixture.
  • /usr/local/go/bin/go test -race ./internal/tunnel ./internal/worker ./internal/guestagent -run '^(TestBridgeTransportCloseAfterDirectionalEOF|TestTunnelRemoteCloseAfterDirectionalEOF|TestTunnelRemoteDirectionalEOF|TestTunnelDirectionalEOF)$' -count=20 — PASS for all three packages.
  • /usr/local/go/bin/go test -overlay=/proc/1601/fd/10 ./internal/control -run ^TestReviewHTTPBackupSmoke$ -race -count=1 -v — PASS. Independent in-memory overlay probe exercised the production controller over a local HTTP server with a fixture storage backend: authenticated 64 MiB backup with verified digest, chunked restore returning 201, authentication rejection, malformed/media/query rejection, missing-volume 404, transfer saturation 429, unsupported-driver 501, and backup-reader closure. No source files were written; overlay descriptors were closed afterward.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Final exact-head independent review APPROVED at c9c7e07: job local-review-review-router-18e3aa764d1296db-1, g7-review, normalized evidence=executed. Prior P2 #79-f1 explicitly answered; no new defects. Reviewer executed Go1.26.4 build/vet, all five affected packages with race detection, focused storage/claim regressions, and tunnel lifecycle regressions20 times under race detection. Independent in-memory production-controller HTTP smoke passed64MiB integrity/restore/auth/input/capacity/cleanup scenarios. Failed initial PATH attempts and successful absolute-path retries are retained honestly. Job event omp_provider_verified reports actual openai-codex; reviewer independent from implementation, same family disclosed. No live-service actions or source edits by reviewer. Root-only foreign-owner repair branch skipped under non-root execution. Both exact-head CI jobs pass and L3 classification rerun; merging with exact-head guard under existing assignment. New permanent tests isolated old-code/test-check proof remains unverified; independent runtime probes failed at old446ac832 and passed correction1920aa0, and final-head regressions pass. Historical evidence-clamp issue2465 is not rewritten or used as current approval. No shared deployment, Forge guest acceptance or credential-reset/release claim.

@jerryfane
jerryfane merged commit 5fd1d29 into main Oct 31, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authenticated raw guest TCP transport for Forge SSH, with half-close and revocation

1 participant