Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/758-clock-skew.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The measured clock offset is now used, not only reported. Above the same 120 seconds the alert rule fires on, `/readyz` answers 503 with `clock.state: "fail"`, `Jobs.enqueue` refuses with `CLOCK_SKEW` before any job row is written (counted as `fledge_admission_rejected_total{reason="clock_skew"}`), and the App JWT refuses with `CLOCK_SKEW_EXCEEDED` instead of signing from a stale correction. A reading is trusted for fifteen minutes, after which the clock is `unknown` and signing fails with `CLOCK_SKEW_UNVERIFIED`, so a GitHub outage can no longer keep a wrong correction alive. Stored timestamps are unchanged and a wrong host clock still needs the host fix.
36 changes: 29 additions & 7 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -732,7 +732,7 @@ Public, and answerable without a token, because a readiness probe that needs a
credential fails when the credential does. The body is:

```json
{ "ok": true, "db": true, "instances": { "ready": 2, "failed": 0, "total": 2 }, "clock": { "skewSeconds": 1680, "source": "github" } }
{ "ok": false, "db": true, "instances": { "ready": 2, "failed": 0, "total": 2 }, "clock": { "skewSeconds": 1680, "measuredAt": "2026-11-15T21:14:02.000Z", "thresholdSeconds": 120, "state": "fail", "source": "github" } }
```

`instances` counts the per-repository instances the database knows about.
Expand All @@ -750,14 +750,36 @@ API, and an outage cannot make readiness flap. An operator watches an absent
`fledge_clock_skew_seconds` series as "not measured", not as "aligned" (see
[`docs/deploy.md`](deploy.md)).

`clock.state` is the decision the measurement supports, and `ok` in the body
follows it:

| state | meaning | response |
|---|---|---|
| `not-configured` | this deployment has no GitHub App, so nothing is measured | `200` |
| `ok` | measured at most 15 minutes ago and within `thresholdSeconds` | `200` |
| `unknown` | never measured, or the last reading is older than 15 minutes | `200` |
| `fail` | the last measurement is outside `thresholdSeconds` | **`503`, `ok:false`** |

`clock.thresholdSeconds` is `CLOCK_SKEW_FAIL_SECONDS` (`src/repos/app.ts`), the
same `120` the `FledgeClockSkewHigh` alert fires on, and `clock.measuredAt` is
when the reported reading was taken. `unknown` and `not-configured` stay `200`
on purpose: a GitHub outage, or a deployment without an App, is not a clock
verdict. In `fail` the platform also refuses work — `Jobs.enqueue` answers
`CLOCK_SKEW` before any job row is written, counted as
`fledge_admission_rejected_total{reason="clock_skew"}` — and the App JWT refuses
to sign (`CLOCK_SKEW_EXCEEDED`), where `unknown` refuses with
`CLOCK_SKEW_UNVERIFIED`.

The measurement itself lives on the GitHub App client
(`src/repos/app.ts`): `measureSkew()` reads the `Date` header of one
unauthenticated `GET /rate_limit`, records it, and leaves `null` behind when the
request fails, the header is missing, or its value is present but unparseable
(a rewriting proxy) — the last case keeps the JWT's last real correction rather
than handing `NaN` to `iat`. `clockSkewSeconds()` returns that stored
reading. The App JWT still refreshes the same figure at most every five minutes
and corrects `iat` with it; the JWT path is unchanged by this field.
unauthenticated `GET /rate_limit`, records it, and leaves the last successful
reading untouched when the request fails, the header is missing, or its value is
present but unparseable (a rewriting proxy) — an unparseable header used to hand
`NaN` to `iat`, and it never replaces a real reading now. `clockSkewSeconds()`
returns that stored reading and `clockState()` turns it into the state above.
The App JWT refreshes the same figure at most every five minutes and signs only
from a reading inside the 15-minute window, so a failed refresh can no longer
keep a stale correction signing.

## Conventions

Expand Down
21 changes: 15 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -865,18 +865,27 @@ DSH token never reaches the browser.
### GitHub App onboarding and webhooks

The optional GitHub App removes the PAT from the common path. Fledge generates a
short-lived App JWT (with a clock-skew correction from GitHub's `Date` header,
the same measurement reported by `/readyz` as `clock.skewSeconds` and by
`/metrics` as `fledge_clock_skew_seconds`; a failed measurement reports
`null` and no series at all rather than a zero that would read as aligned),
exchanges it for an installation token, lists the installation repositories and
stores only `repos.installation_id`; `repoGithubToken()` prefers a user OAuth token authorized with `repo` scope for human attribution and otherwise uses a fresh installation
short-lived App JWT, exchanges it for an installation token, lists the
installation repositories and stores only `repos.installation_id`; `repoGithubToken()` prefers a user OAuth token authorized with `repo` scope for human attribution and otherwise uses a fresh installation
token when that column is set and falls back to the encrypted PAT otherwise. The
installation callback binds the installation to the logged-in user (verified
through the user's OAuth token or an individual account match). GitHub webhooks
are verified with `X-Hub-Signature-256` before parsing; verified events are
recorded in `events` for the follow-up loop. PAT onboarding stays available.

The JWT corrects `iat`/`exp` for the process clock's offset from GitHub's `Date`
header — the same measurement `/readyz` reports as `clock.skewSeconds` +
`clock.state` and `/metrics` as `fledge_clock_skew_seconds`. The offset is used,
not only surfaced (#758): a reading is `ok` for 15 minutes and inside 120 seconds,
`fail` above 120 seconds, and `unknown` when nothing was measured inside the
window. While it is `fail`, `/readyz` answers 503 and `Jobs.enqueue` refuses with
`CLOCK_SKEW` before any job row is written, so no run stores a time from a clock
nobody can trust; the JWT signs only in `ok` and otherwise raises
`CLOCK_SKEW_EXCEEDED` (`fail`) or `CLOCK_SKEW_UNVERIFIED` (`unknown`) instead of
signing from a stale or zero correction. A failed measurement reports `null` and
no series rather than a zero that would read as aligned, and it never replaces
the last reading that really was measured.

### Inbound email replies

When `INBOUND_EMAIL_SECRET` is set, the daily digest gets a
Expand Down
22 changes: 22 additions & 0 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -1053,6 +1053,28 @@ merged, so a tag on `main` is evidence of a released version, not of a rollout.
minutes. Fix the host clock (`timedatectl set-ntp true`, or start the container
with a host that keeps time); Fledge reports the drift and never rewrites a
stored timestamp to hide it.

The measured offset is used, not only shown. `clock.state` says what the last
measurement means, and `clock.thresholdSeconds` carries the number it is
compared against — `CLOCK_SKEW_FAIL_SECONDS` in `src/repos/app.ts`, which must
stay equal to the `120` in `deploy/prometheus/fledge-alerts.yml`:

| `clock.state` | meaning | `/readyz` |
|---|---|---|
| `not-configured` | no GitHub App on this deployment, so nothing is measured | `200` |
| `ok` | measured at most 15 minutes ago and within 120 s | `200` |
| `unknown` | never measured, or the last reading is older than 15 minutes | `200` |
| `fail` | the last measurement is outside 120 s | **`503`, `ok:false`** |

In `fail` the platform refuses to start work: `Jobs.enqueue` answers
`CLOCK_SKEW` (counted as `fledge_admission_rejected_total{reason="clock_skew"}`)
before any job row is written, and the console banner names the measured
seconds. App JWTs are signed only in `ok`; `fail` raises `CLOCK_SKEW_EXCEEDED`
and `unknown` raises `CLOCK_SKEW_UNVERIFIED` instead of signing with a stale or
zero correction. A GitHub outage therefore stops App calls after at most 15
minutes rather than serving a signature nobody re-measured. Fix the clock, then
confirm `/readyz` is `200` again — the next measurement (at most five minutes
after a `fail`) clears it.
- The Admin panel shows the audit trail and recent jobs.
- Quota state: `GET /v1/models` with a user token, or the dashboard.
- Disk: SQLite WAL files and logs.
Expand Down
19 changes: 15 additions & 4 deletions src/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1031,10 +1031,21 @@ export async function buildApp(options: BuildOptions): Promise<BuiltApp> {
const instances = store.listRepoInstances()
const ready = instances.filter((entry) => entry.status === 'ready').length
const failed = instances.filter((entry) => entry.status === 'failed').length
// #758: the skew the App JWT already corrects for, reported rather than
// re-measured. `null` means no measurement has succeeded — never 0, which
// would read as a clock that is exactly right.
return { ok: true, db: true, instances: { ready, failed, total: instances.length }, clock: { skewSeconds: githubApp.clockSkewSeconds(), source: 'github' } }
// #758 slice 2: readiness fails on a measured clock outside the threshold, so
// a monitor that reads only the status code stops seeing a "healthy" instance
// whose every stored time is 28 minutes ahead. `unknown` and
// `not-configured` stay 200: a GitHub outage, or a deployment with no App, is
// not a clock verdict. The object keeps `source` and adds `state` and
// `thresholdSeconds`, so a reader can tell "no number" from "a number that is
// too far".
const clock = githubApp.clockState()
const body = {
ok: clock.state !== 'fail',
db: true,
instances: { ready, failed, total: instances.length },
clock: { ...clock, source: 'github' },
}
return clock.state === 'fail' ? reply.code(503).send(body) : body
})

app.get('/lang/:locale', async (request, reply) => {
Expand Down
17 changes: 17 additions & 0 deletions src/jobs/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1144,6 +1144,23 @@ export class Jobs {
// implying this one repository is disabled, and it is checked in the admission
// path itself: a due repository reaches `enqueue` without a route.
if (this.platformStop() !== null) return { ok: false, error: '平台已紧急停止,暂不接受新任务。', code: 'PLATFORM_PAUSED' }
// #758 slice 2: a clock the last measurement puts outside the threshold must
// not start work that stores or schedules time. It sits after the platform
// stop (which outranks everything) and before the work-type gates, and it is
// here rather than on the routes because every trigger — manual, scheduled,
// follow-up, review, discovery — funnels through `enqueue`. Nothing is
// written: the refusal is before any job row, so no row carries a time from a
// clock nobody can trust.
const clock = this.githubApp.clockState()
if (clock.state === 'fail') {
incCounter('fledge_admission_rejected_total', { reason: 'clock_skew' }, 1, 'Queue admission rejections')
return {
ok: false,
error: `平台时钟与 GitHub 相差 ${String(clock.skewSeconds)} 秒,超过 ${String(clock.thresholdSeconds)} 秒上限,任务没有开始。`,
code: 'CLOCK_SKEW',
detail: String(clock.skewSeconds),
}
}
// #390: the work type is resolved before the gates, and an unknown one is
// refused. A default here would run work under a contract nobody declared.
const workType = resolveWorkType(options.workType ?? workTypeForKind(kind))
Expand Down
6 changes: 6 additions & 0 deletions src/lib/failure.ts
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,8 @@ export const ENQUEUE_REFUSAL_CODES = [
'RUNNER_ERROR',
'RULES_UNAVAILABLE',
'RULES_INVALID',
// #758 slice 2: the platform clock is outside the threshold, so no job starts.
'CLOCK_SKEW',
] as const

export type EnqueueRefusalCode = (typeof ENQUEUE_REFUSAL_CODES)[number]
Expand Down Expand Up @@ -262,6 +264,10 @@ export const REFUSAL_SENTENCES: Record<EnqueueRefusalCode, [string, string]> = {
// the file that crossed the limit and the byte count the maintainer can act on.
RULES_UNAVAILABLE: ['无法读取这个仓库的规则或技能,任务没有开始。', "This repository's rules or skills could not be read, so the job did not start."],
RULES_INVALID: ['这个仓库的规则或技能不合法:{detail}。', 'The repository rules or skills are invalid: {detail}.'],
// #758 slice 2: the refusal carries the measured seconds, because "the clock is
// wrong" without the number sends the reader to the wrong machine. The
// threshold rides on `clockState()` and in docs/deploy.md, not in this sentence.
CLOCK_SKEW: ['平台时钟与 GitHub 相差 {detail} 秒,超过 120 秒上限,任务没有开始。', 'The platform clock is {detail} seconds from GitHub time, above the 120-second threshold, so the job did not start.'],
}

const BRANCH_EMPTY: [string, string] = ['目标分支不存在或不可访问。', 'The target branch does not exist, or this account cannot read it.']
Expand Down
90 changes: 69 additions & 21 deletions src/repos/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,38 @@ function base64url(value: Buffer | string): string {
const CLOCK_SKEW_LOG_SECONDS = 60
const SKEW_REFRESH_MS = 5 * 60_000

// #758 slice 2: the number that turns a measurement into a decision. It is the
// same 120 the alert rule fires on (`deploy/prometheus/fledge-alerts.yml`) and
// `docs/deploy.md` names both, so a failing readiness probe and a firing alert
// cannot disagree about what "too far" means.
export const CLOCK_SKEW_FAIL_SECONDS = 120

// How long one successful measurement stays usable. Inside the window a GitHub
// outage is tolerated; past it the clock is unverified, which is not the same as
// good, and nothing may sign from a number nobody has re-measured.
export const CLOCK_SKEW_REUSE_MS = 15 * 60_000

export type ClockStateName = 'not-configured' | 'ok' | 'fail' | 'unknown'

export interface ClockState {
/** The last successful measurement: seconds the process is ahead of GitHub. */
skewSeconds: number | null
/** When that measurement was taken, ISO-8601, or null when none succeeded. */
measuredAt: string | null
thresholdSeconds: number
state: ClockStateName
}

export class GithubApp {
private key: string | null = null
private skewMs = 0
private skewAt = 0
// The last successful measurement, kept apart from `skewMs`: the JWT needs a
// number, while a report has to be able to say "unknown". A 0 here would read
// as a clock that is exactly right, which is the one lie this slice exists to
// prevent.
// The last successful measurement. It is kept when a refresh fails — the
// reading is still what was measured, it just stops being trusted once
// `CLOCK_SKEW_REUSE_MS` passes — and it is the only number the JWT may sign
// with. `skewMs` used to be a private fail-open copy of it and is gone: that
// copy is how a host 28 minutes fast kept signing.
private measuredSkewSeconds: number | null = null
private measuredAt: number | null = null
private readonly tokens = new Map<string, { token: string; expiresAt: number }>()

constructor(private readonly config: Config) {}
Expand All @@ -71,37 +94,34 @@ export class GithubApp {
async measureSkew(): Promise<number | null> {
if (!this.configured) {
this.measuredSkewSeconds = null
this.measuredAt = null
return null
}
try {
const response = await fetch(`${this.config.github.apiBase}/rate_limit`, { headers: { 'user-agent': 'fledge/0.1' }, signal: AbortSignal.timeout(10_000) })
const date = response.headers.get('date')
this.skewAt = Date.now()
if (date === null) {
this.measuredSkewSeconds = null
return null
}
const at = Date.now()
this.skewAt = at
if (date === null) return null
const remote = Date.parse(date)
if (!Number.isFinite(remote)) {
// #758 finding: a `Date` header that is present but unparseable (a
// rewriting proxy is the realistic case) is a failed measurement, not a
// reading. `skewMs` keeps the last real correction for the JWT, and the
// report says "unknown" — `NaN` here used to travel to /readyz, to the
// metric, and into the JWT's `iat` without failing anything.
this.measuredSkewSeconds = null
// reading — `NaN` here used to travel to /readyz, to the metric, and into
// the JWT's `iat` without failing anything. The previous reading stays;
// it ages out of `ok` on its own rather than being replaced by a guess.
return null
}
this.skewMs = Date.now() - remote
const seconds = Math.round(this.skewMs / 1000)
const seconds = Math.round((at - remote) / 1000)
this.measuredSkewSeconds = seconds
this.measuredAt = at
if (Math.abs(seconds) > CLOCK_SKEW_LOG_SECONDS) log.warn('clock.skew', { skewSeconds: seconds, source: 'github' })
return seconds
} catch {
// The JWT keeps its last correction here; slice 2 of #758 owns that hole.
// The report does not keep it, because a measurement that failed is not a
// reading, and a stale number would hide the next drift.
// A failed refresh is not a reading: the last successful one stays as the
// report's answer and ages out of `ok` after `CLOCK_SKEW_REUSE_MS`. The JWT
// has no correction of its own to fall back on any more.
this.skewAt = Date.now()
this.measuredSkewSeconds = null
return null
}
}
Expand All @@ -111,10 +131,38 @@ export class GithubApp {
return this.measuredSkewSeconds
}

// #758 slice 2: one answer to "may anything act on this clock", read by
// /readyz, by `appJwt` and by `Jobs.enqueue`. `not-configured` is a deployment
// with no App, which is not a clock verdict; `unknown` is "nobody measured, or
// the last reading is stale"; `fail` is a measurement above the threshold, and
// it stays `fail` once it ages out — an outage does not repair a clock that was
// measured 28 minutes wrong.
clockState(): ClockState {
const thresholdSeconds = CLOCK_SKEW_FAIL_SECONDS
if (!this.configured) return { skewSeconds: null, measuredAt: null, thresholdSeconds, state: 'not-configured' }
const seconds = this.measuredSkewSeconds
const at = this.measuredAt
if (seconds === null || at === null) return { skewSeconds: null, measuredAt: null, thresholdSeconds, state: 'unknown' }
const measuredAt = new Date(at).toISOString()
if (Math.abs(seconds) > thresholdSeconds) return { skewSeconds: seconds, measuredAt, thresholdSeconds, state: 'fail' }
if (Date.now() - at > CLOCK_SKEW_REUSE_MS) return { skewSeconds: seconds, measuredAt, thresholdSeconds, state: 'unknown' }
return { skewSeconds: seconds, measuredAt, thresholdSeconds, state: 'ok' }
}

private async appJwt(): Promise<string> {
if (!this.configured) throw new Error('GitHub App 未配置')
if (Date.now() - this.skewAt > SKEW_REFRESH_MS) await this.measureSkew()
const now = Math.floor((Date.now() - this.skewMs) / 1000)
// #758 slice 2: sign only with a measurement that is inside both the reuse
// window and the threshold. Everything else refuses with a named error
// instead of signing from a stale or zero correction — that fail-open is what
// let a host 28 minutes fast keep talking to GitHub as if its clock were
// right, and a 401 from GitHub was the only symptom.
const clock = this.clockState()
if (clock.state === 'fail') {
throw new Error(`CLOCK_SKEW_EXCEEDED: the process clock is ${String(clock.skewSeconds)}s from GitHub time, above the ${String(clock.thresholdSeconds)}s threshold`)
}
if (clock.state !== 'ok') throw new Error('CLOCK_SKEW_UNVERIFIED: no clock measurement inside the reuse window')
const now = Math.floor(Date.now() / 1000) - (clock.skewSeconds ?? 0)
const header = base64url(JSON.stringify({ alg: 'RS256', typ: 'JWT' }))
const payload = base64url(JSON.stringify({ iat: now - 60, exp: now + 540, iss: this.config.githubApp.id }))
const signer = createSign('RSA-SHA256')
Expand Down
Loading
Loading