Skip to content

Make the job sweep test time-independent and fix timestamp ordering - #33

Merged
brocla merged 1 commit into
mainfrom
fix-sweep-flake
Oct 29, 2026
Merged

brocla merged 1 commit into
mainfrom
fix-sweep-flake

Conversation

@brocla

@brocla brocla commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Why

TestSweepExpiredJobsDeletesOldWorkspaceOnly failed on CI during #32. It shrank job retention to 100 ms and needed "complete the fresh job, then sweep" to finish within that time. On a slow Windows runner it didn't, so the fresh job's workspace was swept.

Underneath was a real bug. The store writes timestamps with RFC3339Nano, which trims trailing zeros, and the expiry queries compare them as text (completed_at < ?). So 10:00:00.5Z sorts before 10:00:00Z, and a job completed half a second after a cutoff counted as expired. In production, with 30-day retention, that's harmless. In millisecond-scale tests it's a second source of random failures. The alert tables already avoid this with a fixed-width format.

Changes

  • Store: alertTimeFormat becomes the store-wide timeFormat (nine fixed fraction digits). Every timestamp write uses it, covering jobs, submissions, deliveries and cutoffs. Reads still parse with RFC3339Nano, which accepts both formats, so there's no migration. Old and new rows can misorder only within the same second.
  • App: a now clock field with a clock() method, the same pattern as mailgun.Service. The sweep measures retention against it, and jobRetentionPeriod is a constant again.
  • Tests:
    • The sweep test sets the clock so the cutoff lands exactly on a boundary taken between the two completions. A slow runner can only widen the margins now.
    • expireEverything moves the clock forward instead of changing the shared retention period.
    • New store tests cover ordering within a second for jobs and for deliveries, plus a check that every timestamp column is stored at a fixed width. All three failed before the fix.

Testing

  • Sweep tests: 200 runs, all passing.
  • Store expiry tests: 50 runs, all passing.
  • go vet ./... and go test ./... pass.

Production behavior is unchanged apart from the sub-second ordering fix, so deploying this can wait for the next scheduled deploy.

🤖 Generated with Claude Code

TestSweepExpiredJobsDeletesOldWorkspaceOnly shrank the retention period
to 100ms and needed the fresh job's completion and the sweep to finish
inside it. A slow CI runner took longer, the fresh job counted as
expired, and the test failed. App now has a clock tests can set (the
same pattern as mailgun.Service), and the test puts the cutoff exactly
on a boundary taken between the two completions, so a slow machine only
widens the margins. jobRetentionPeriod becomes a constant again.

Underneath was a real ordering bug. The store wrote timestamps with
RFC3339Nano, which trims trailing zeros, and compares them as text in
SQL, so "10:00:00.5Z" sorted before "10:00:00Z": a job completed half a
second after a cutoff counted as expired. The alert tables already used
a fixed-width format for exactly this reason; it is now the store-wide
timeFormat and every timestamp write uses it. Reads still parse with
RFC3339Nano, which accepts both, so existing rows need no migration and
can misorder against new ones only within the same second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brocla
brocla merged commit 6abf9c5 into main Oct 29, 2026
1 check passed
@brocla
brocla deleted the fix-sweep-flake branch October 29, 2026 04:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant