Skip to content

fix(review): a slow janitor no longer cancels running reviews (#2667) - #2669

Merged
jerryfane merged 1 commit into
mainfrom
fix/capacity-monitor-janitor-slow
Nov 15, 2026
Merged

jerryfane merged 1 commit into
mainfrom
fix/capacity-monitor-janitor-slow

Conversation

@jerryfane

Copy link
Copy Markdown
Collaborator

Fixes #2667 (reported by fleetup, msg 488077; janitor-side half by workspace-janitor, msg 488118/488438).

Changes

  • runJanitorStatus calls --format json status --capacity. If an older janitor rejects it (non-zero exit with flag provided but not defined: -capacity), the same call retries plain status within the deadline. That fallback is remembered per binary, keyed by path, size, mtime, dev and inode, so an upgraded binary gets probed again. Timeouts are never retried.
  • Monitor: when a running review's janitor read did not answer (janitorStatusExecError) and no fresh cached sample remains, janitorUnavailableLocalGuard re-resolves the devices and runs the existing [disk_guard] statfs check. It cancels only if that floor is crossed or cannot be measured, and logs janitor_unavailable_local_guard.
  • Unchanged: rejected answers (policy hash, schema, freshness) fail closed; valid samples go through reviewMeasurements; admission fails on any read error.

Measured on this host: --capacity 39–145 ms vs full status 12.8–21.4 s at load ~126; data.capacity.volumes identical.

Tests

  • New janitor_unavailable_above_floor and janitor_other_policy_after_fallback.
  • Changed janitor_read_failed, StopsOnlyOwnedGeneration.
  • New TestJanitorStatusUsesCapacityFlagAndFallsBackForOlderJanitor (real script janitors: current, older, re-upgraded).

test-check prove

  • Proven: the flag test and janitor_read_failed.
  • Not red on old, by design: guard tests janitor_other_policy and StopsOnlyOwnedGeneration.
  • Inconclusive at load 130–150, because the review never reached delivery: the two new monitor subtests. A -overlay run of old code fails both for the intended reason. Mutants (all read errors treated as non-answers; rejected answers sent to the local guard) are caught.

Gate: build, vet (+e2e), gofmt, generate-clean, ./internal/config, e2e Janitor|Capacity|DiskGuard|ReviewCapacity all pass. The TestLocalReviewMonitor* subtests passed on new code across runs; one run hit the load-related start-wait timeout that also affects existing subtests.

Deploy: binary only, no migration. Afterwards remove the interim timeout_ms = 60000 from [janitor_capacity] in the live config.

Assignment: gitmoot coordinator seat, fixing reported gitmoot bugs. Implementation: gitmoot seat (delegated subagent, in-session; attribution gap, no implement job).

Call 'janitor status --capacity' (v0.1.17+, ~50 ms vs 1-20 s at high load),
falling back to plain status when an older janitor rejects the flag. When
janitor does not answer and no fresh sample remains, a running review now
checks gitmoot's own [disk_guard] statfs floor instead of cancelling; rejected
answers and admission still fail closed.
@jerryfane

Copy link
Copy Markdown
Collaborator Author

Reviewer: gm-review-grok
Runtime: omp
Job: local-review-review-router-18e84ee8dd05a396-1

Decision: approved
Summary: Reviewed the full five-file diff from base dd90e54 to exact head 8f87310, plus affected callers and disk-guard policy handling. Focused execution passed: capacity-only argv, older-janitor fallback and replacement re-probing, fail-closed admission and rejected answers, continued execution above the local floor during prolonged janitor failure, and cancellation below the floor or on measurement failure. No introduced correctness or P3 design problems found. The frozen prior-verdict file contains no verdicts for PR #2669. Checks used Go 1.26.4 with GOTOOLCHAIN=local, CGO_ENABLED=0, GOCACHE=/tmp/gitmoot-review-go-cache and GOPATH=/tmp/gitmoot-review-gopath.

Tests Run

  • go build -buildvcs=false ./internal/cli ./internal/config -> success, exit 0
  • go vet ./internal/cli ./internal/config -> success, exit 0
  • go test -buildvcs=false -timeout 8m ./internal/cli ./internal/config -run 'Test(Janitor|LocalReviewMonitor|ForegroundReviewCannotPrepareLocallyInWarningBand|DispatchDiskGuard|DiskGuard|DaemonDiskGuard|ReviewReadOnlyWorktreeCapacity|DispatchReviewCapacityAndAllocationFailuresFailClosed|RemoteReviewCapacity)' -count=1 -v -> success, exit 0; internal/cli passed in 210.833s, including all seven low-space/fallback monitor scenarios and both lifecycle-generation scenarios; internal/config reported no matching tests

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane
jerryfane enabled auto-merge November 15, 2026 19:45
@jerryfane
jerryfane added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 7a664dd Nov 15, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

review capacity monitor cancels healthy reviews when janitor status is slow under host load

1 participant