Repository navigation
fix: stop starting work on a host whose clock is measurably wrong - #895
Merged
Merged
Conversation
HugoXK
enabled auto-merge (squash)
November 16, 2026 04:22
Owner
Author
|
Written-by: Fledge Platform Lead PASS requested at
|
HugoXK-bot
approved these changes
Nov 16, 2026
HugoXK-bot
left a comment
Collaborator
There was a problem hiding this comment.
Verdict: PASS at 0f7ea68
Written-by: HugoXK-bot
Author: Fledge Platform Verdict
src/repos/app.ts:140-149turns the stored reading into oneclockState()that readiness,enqueueand the JWT all read, and:160-165stops the JWT signing from the old fail-openskewMscopy, so a host 28 minutes fast can no longer keep talking to GitHub.src/app.ts:1041-1048answers 503 only infail;src/jobs/service.ts:1154-1163refuses before any job row and countsfledge_admission_rejected_total{reason="clock_skew"};src/lib/failure.ts:270names the measured seconds; andsrc/repos/app.ts:48is the same 120 asdeploy/prometheus/fledge-alerts.yml:69.- The tests pin the properties, not the words:
test/clockSkewFail.test.ts:105(503),:117(no job row),:120(no JWT),:149-160(120 inside / 121 outside),:178-190(15-minute window). Non-blocking:docs/deploy.md:1071says the next measurement "at most five minutes after a fail" clears it, butmeasureSkewis only called fromappJwt(src/repos/app.ts:154) and/readyzdoes not re-measure, so an idle instance staysfailuntil an App call.
Ran:
npm run checkin a fresh tree of0f7ea6829142(git-init'd;package-lock.jsonsha256ef264428…matches the one CI used): lint, typecheck, 2322/2322 tests, build — exit 0npx tsx --test test/clockSkew.test.ts test/clockSkewFail.test.ts— 12/12 at the tip- red half: the same head test over base
410f16b2sources (src/app.ts,src/jobs/service.ts,src/lib/failure.ts,src/repos/app.ts; blobs identical to the PR diff base side) — 0/5, all five fail node scripts/risk-gate.mjs --base 410f16b2 --head judge—risk-tier: T3; basis: src/jobs/service.ts, 10 files+418/−46, matching the PRGITHUB_REPOSITORY=HugoXK/fledge GITHUB_TOKEN=… node scripts/pr-lint.mjs --pr 895—pr-lint: ok (#895, T3, 10 files)
Not run: no deployed instance, no live GitHub App installation, no real GitHub outage — the 15-minute age-out and the 503 were driven by a fake Date header and a frozen Date.now, not by api.github.com.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Written-by: Fledge Platform Dev-2
Bot: Fledge Platform Dev-2
Fixes #758
What changed
The skew was measured and surfaced but never used: the App JWT corrected for it, and everything that decides whether work may start did not. A host 1684 s fast keeps starting jobs and signs JWTs 28 minutes into the future, and the old code failed open — a failed refresh reused the last correction for another five minutes.
src/repos/app.ts:clockState()returnsnot-configured | ok | fail | unknown;CLOCK_SKEW_FAIL_SECONDS = 120andCLOCK_SKEW_REUSE_MS = 15 min.appJwt()signs only inokand refuses withCLOCK_SKEW_EXCEEDED/CLOCK_SKEW_UNVERIFIED; the fail-openskewMscopy is gone./readyzanswers 503 onfail, carryingclock.state,clock.thresholdSecondsandclock.measuredAt.Jobs.enqueue()refusesCLOCK_SKEWbefore any job row is written, counted asfledge_admission_rejected_total{reason="clock_skew"}.src/lib/failure.ts; docs indocs/deploy.md,docs/api.md,docs/architecture.md; one changelog fragment.Eval
test/clockSkewFail.test.tson410f16b2with the source untouched: 5 fail / 0 pass (clockStateis not a function).Date1680 s behind:measured 1680,/readyz200{"ok":true,…,"clock":{"skewSeconds":1680,"source":"github"}}(nostate),enqueueok with 1 job row — both removed by this tree.test/clockSkew.test.tsneeded one honest update: the unparseable-Datecase now asserts the last real reading stands for the reuse window (90 s,state: 'ok', gauge present, JWT signing from that correction).Ran:
npm run check— exit 0, 2322/2322 (lint at the 11-warning baseline, typecheck, test, build)node scripts/risk-gate.mjs --base origin/main—risk-tier: T3; basis: src/jobs/service.tsNot run: no live GitHub App, no production host, no Docker image.
Risk and rollback
T3 (
src/repos,src/jobs,src/lib). A host that cannot verify its clock now refuses to start work and answers/readyz503 instead of running 28 minutes ahead — that is the point. Rollback isgit revert; no schema or config key changes, and the metric simply stops being written.Docs
docs/deploy.md(the two constants and the readiness semantics),docs/api.md(/readyz503 shape),docs/architecture.md(sink versus admission). Changelogchangelog.d/758-clock-skew-admission.fixed.md.10 files,
+187/−46plus the new test file.maintainer_can_modify: true