Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/694-mask-log-identities.security.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Account identifiers no longer reach the log sink, and a development outbox file no longer carries the recipient in its name. A log line masks an email, a GitHub login, a user id, a token and an IP address in every string field — including fields the sink does not name and values nested inside an object or an array — while the stored job row and the job page keep the real value an operator needs; `npm run mail:outbox` expires every `.eml` older than `MAIL_OUTBOX_RETENTION_DAYS` (7 days by default) and then prints the newest one.
10 changes: 9 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -1295,7 +1295,15 @@ same pull request and work item; a review that already recorded a
posted. `summary` is the job's progress line, written on every running heartbeat,
not a post marker, so it does not block a retry; it and `error` pass through
`src/lib/secrets.ts` before the row is written and again when the page renders
them (`docs/decisions/0015`). If a review posted its comments
them (`docs/decisions/0015`).

Identifier masking is the other half and lives on the sink, not in `redact()`:
`src/lib/log.ts` masks an account identifier in every line it writes — by shape in
every string value, by field name for a login the shape cannot see, and through
nested objects and arrays — while the row, the inbox and the job page keep the
real value the operator acts on (#694). `maskEmail` and `maskIp` are in
`src/lib/redact.ts` and are imported back by `src/views/pages.ts`, so a page and a
log line cannot drift onto two masks. If a review posted its comments
and then lost the terminal callback, nothing stored tells that apart from a
review that never posted; that residual case is manual. `github`, `quota`,
`credential` and `review` failures stay manual.
Expand Down
11 changes: 8 additions & 3 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Key settings:
| `MAIL_TRANSPORT` | `smtp` in production, `console` for a local demo |
| `SMTP_HOST/PORT/SECURE/USER/PASS` | For yeah.net: `smtp.yeah.net`, 465, secure |
| `MAIL_FROM` / `ADMIN_EMAIL` | `fledge@hugoxk.online` / `admin@example.com` |
| `MAIL_OUTBOX_RETENTION_DAYS` | Window for `npm run mail:outbox` to expire `$DATA_DIR/mail-outbox/*.eml`, in days; `7` by default |
| `FEEDBACK_EMAIL` | Optional feedback/error-report destination; defaults to `ADMIN_EMAIL` |
| `FLEDGE_LEAD_REPO_ID` | Repo connected to Fledge whose FlyBot Lead triages feedback |
| `FLEDGE_LEAD_BRIDGE_URL/TOKEN/SESSION_ID` | Direct bridge target when the Fledge Lead is external |
Expand Down Expand Up @@ -1117,9 +1118,13 @@ your nginx config), skip this file and let the distribution's nginx rotation own
those paths — never rotate the same file from two stanzas.

Development mail (`MAIL_TRANSPORT=console`) is written to
`$DATA_DIR/mail-outbox/*.eml` and is never pruned: it is a dev aid
(`npm run mail:outbox`), an instance with SMTP does not write it, and the
directory may be deleted at any time.
`$DATA_DIR/mail-outbox/*.eml`. It is a dev aid (`npm run mail:outbox`), and an
instance with SMTP does not write it. Each run of `npm run mail:outbox` first
deletes every `.eml` older than `MAIL_OUTBOX_RETENTION_DAYS` (7 by default) and
then prints the newest one, so the directory is never a store of account
identifiers: the file name is a stamp plus an opaque id, never the recipient,
and the address lives only inside the file. The directory may be deleted at any
time.

### Workspace snapshot retention

Expand Down
22 changes: 21 additions & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -912,8 +912,28 @@ and settings do not offer a disabled provider as an allowlist entry.

`redact()` removes GitHub tokens, `sk-…` keys, `flg_…` tokens, bearer values and
Upstream `api_key:` hashes from log messages. Request bodies are never logged.

The log sink (`src/lib/log.ts`) additionally masks **account identifiers**, which
are not secrets and which `redact()` deliberately leaves alone: an email becomes
`a***@example.com`, a GitHub login or a `usr_` id keeps its first character, and
an address loses its last IPv4 octet. Every string field value goes through the
shape pass — an email or a `usr_` id is masked wherever it appears, including in
a field whose name the sink does not know — and nested objects and arrays are
walked rather than copied, so `{ user: { email } }` cannot carry the address
through. Names still add what shape cannot see: a login has no shape, so the key
table (`email`, `to`, `login`, `user`, `userId`, `ip`, `admin`, …) masks those
values by name, and a `repo`/`repository` field keeps the repository name while
masking the `owner/` login in front of it. The mask lives in the sink, never in
`redact()`: the stored job row, the inbox and the job page still carry the real
identifier an operator has to act on, and that difference is asserted in
`test/logMask.test.ts`. A post-hoc rewrite of `fledge.log` is not a policy; the
identifier must not be written in the first place.

The development mail outbox is written with mode 0600 and contains approval links;
treat it as a secret.
treat it as a secret. A file is named `<stamp>-<opaque id>.eml`, never the
recipient, so the name alone keeps no account alive, and `npm run mail:outbox`
expires every `.eml` older than `MAIL_OUTBOX_RETENTION_DAYS` (7 by default)
before it prints the newest one.

The evolution export uses a read-only database handle and passes every exported
string through the same `redact()` before serialisation. It reads only
Expand Down
38 changes: 35 additions & 3 deletions scripts/outbox.mjs
Original file line number Diff line number Diff line change
@@ -1,12 +1,44 @@
#!/usr/bin/env node
// Print the newest development email and any approval links it contains.
import { readdirSync, readFileSync } from 'node:fs'
// Print the newest development email and any approval links it contains, and
// expire the ones past the retention window.
//
// #694: the file name carries a stamp and an opaque id, never the recipient, so
// the window below can delete every `.eml` in the directory without keeping one
// account identifier alive forever. Deletion is by mtime, takes no exception,
// and happens before the newest file is read.
//
// DATA_DIR=./data npm run mail:outbox
// MAIL_OUTBOX_RETENTION_DAYS=14 npm run mail:outbox # 7 by default
import { readdirSync, readFileSync, statSync, unlinkSync } from 'node:fs'
import { join } from 'node:path'

const DAY_MS = 86_400_000

function retentionDays() {
const raw = String(process.env.MAIL_OUTBOX_RETENTION_DAYS ?? '7').trim()
const days = Number(raw)
if (raw === '' || !Number.isFinite(days) || days < 0) {
console.error(`MAIL_OUTBOX_RETENTION_DAYS must be a non-negative number of days, got "${raw}".`)
process.exit(1)
}
return days
}

const dir = join(process.env.DATA_DIR ?? './data', 'mail-outbox')
const cutoff = Date.now() - retentionDays() * DAY_MS
let files = []
try {
files = readdirSync(dir).filter((name) => name.endsWith('.eml')).sort()
for (const name of readdirSync(dir)) {
if (!name.endsWith('.eml')) continue
const path = join(dir, name)
if (statSync(path).mtimeMs < cutoff) {
unlinkSync(path)
console.log(`expired ${name}`)
continue
}
files.push(name)
}
files.sort()
} catch {
console.error(`No outbox at ${dir}. Start with MAIL_TRANSPORT=console and log in once.`)
process.exit(1)
Expand Down
7 changes: 6 additions & 1 deletion src/email/mailer.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { randomUUID } from 'node:crypto'
import { mkdirSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'
import nodemailer, { type Transporter } from 'nodemailer'
Expand Down Expand Up @@ -26,7 +27,11 @@ export class ConsoleMailer implements Mailer {
async send(message: MailMessage): Promise<void> {
mkdirSync(this.outbox, { recursive: true })
const stamp = new Date().toISOString().replace(/[:.]/g, '-')
const file = join(this.outbox, `${stamp}-${message.to.replace(/[^a-z0-9@._-]/gi, '_')}.eml`)
// #694: the name is not the recipient. An addressed name means "keep the
// newest file" keeps one identifier forever, so no retention window could
// ever delete the whole directory. The stamp keeps the lexical order
// `scripts/outbox.mjs` reads, and the address stays inside the file.
const file = join(this.outbox, `${stamp}-${randomUUID().slice(0, 8)}.eml`)
const body = [
`Date: ${new Date().toUTCString()}`,
`From: Fledge <${this.from}>`,
Expand Down
62 changes: 57 additions & 5 deletions src/lib/log.ts
Original file line number Diff line number Diff line change
@@ -1,15 +1,67 @@
import { redact } from './secrets.js'
import { maskEmail, maskIp, redact } from './secrets.js'

export type Level = 'debug' | 'info' | 'warn' | 'error'

export type LogFields = Record<string, unknown>

// #694: a log line outlives the account it names — it is rotated, shipped and
// read long after a purge — so the sink is the one surface that never carries an
// account identifier. The row, the inbox and the job page keep the real value
// (masking lives here, not in `redact()`, which cleans what is stored and shown).
// Fields are decided by name because a login and a `usr_` id have no shape; text
// is decided by shape because a call site can glue one into a sentence.
const EMAIL_KEY = /^(?:admin|admin_?email|e-?mail|feedback_?email|from|recipient|reply_?to|to)$/i
const LOGIN_KEY = /^(?:account|login|user|user_?name)$/i
const ID_KEY = /^(?:account_?id|user_?id)$/i
const IP_KEY = /^(?:client_?ip|ip|ip_?address|remote_?addr|remote_?ip)$/i
// A repository field is `owner/name`: the owner is a GitHub login and is masked,
// the repository name is not an account and is kept so a line still says which
// repo it is about. Left out of the table, `owner/name` passed through whole.
const REPO_KEY = /^(?:repos?|repository|repo_?name)$/i

const EMAIL_SHAPE = /[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/g
const USER_ID_SHAPE = /\busr_[A-Za-z0-9]{3,}\b/g
const IPV4_SHAPE = /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g

function maskName(value: string): string {
return value === '' ? '' : `${value.slice(0, 1)}***`
}

function maskText(text: string): string {
return text.replace(EMAIL_SHAPE, maskEmail).replace(USER_ID_SHAPE, 'usr_***').replace(IPV4_SHAPE, maskIp)
}

function maskField(key: string, value: string): string {
if (EMAIL_KEY.test(key)) return value.includes('@') ? maskEmail(value) : maskName(value)
if (LOGIN_KEY.test(key) || ID_KEY.test(key)) return maskName(value)
if (IP_KEY.test(key)) return maskIp(value)
if (REPO_KEY.test(key)) return value.replace(/^([^/]+)\//, (_match, owner: string) => `${maskName(owner)}/`)
return value
}

// The key table is an optimisation, not the contract: a field the table does not
// name is still a string on its way to a sink that promises no account
// identifier, so the shape pass runs over every value (#694 review). An email
// glued into an unlisted field, a nested `{ user: { email } }` or an array of
// logins therefore cannot slip through just because a call site invented a key.
function maskValue(key: string, value: unknown, seen: WeakSet<object>): unknown {
if (typeof value === 'string') return redact(maskText(maskField(key, value)))
if (value === null || typeof value !== 'object') return value
if (seen.has(value)) return '[circular]'
seen.add(value)
if (Array.isArray(value)) return value.map((item) => maskValue(key, item, seen))
const nested: Record<string, unknown> = {}
for (const [nestedKey, nestedValue] of Object.entries(value)) {
nested[nestedKey] = maskValue(nestedKey, nestedValue, seen)
}
return nested
}

function write(level: Level, message: string, fields?: LogFields): void {
const line: Record<string, unknown> = { ts: new Date().toISOString(), level, msg: redact(message) }
const line: Record<string, unknown> = { ts: new Date().toISOString(), level, msg: redact(maskText(message)) }
if (fields !== undefined) {
for (const [key, value] of Object.entries(fields)) {
line[key] = typeof value === 'string' ? redact(value) : value
}
const seen = new WeakSet<object>()
for (const [key, value] of Object.entries(fields)) line[key] = maskValue(key, value, seen)
}
const output = JSON.stringify(line)
if (level === 'error') process.stderr.write(`${output}\n`)
Expand Down
26 changes: 26 additions & 0 deletions src/lib/redact.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,29 @@
// #694: an account identifier is not a secret, so `redact()` leaves it alone — a
// job row, the inbox and the job page keep the real address because that is where
// an operator acts on it. These two helpers exist for the surfaces that must not
// carry it: the log sink and the session/list pages. One function per shape, so a
// page and a log line cannot drift onto different masks.
export function maskEmail(email: string): string {
const at = email.lastIndexOf('@')
if (at <= 0) return '***'
const local = email.slice(0, at)
const domain = email.slice(at + 1)
return `${local.slice(0, 1)}***@${domain}`
}

// #356: a session row is recognisable without handing out the address. The last
// IPv4 octet and everything past the second IPv6 group are dropped, so a user can
// tell their own sessions apart but the page never carries a full address.
export function maskIp(ip: string | null): string {
if (ip === null || ip === '') return '—'
if (ip.includes(':')) {
const groups = ip.split(':').filter((group) => group !== '')
return `${groups.slice(0, 2).join(':')}::`
}
const parts = ip.split('.')
return parts.length === 4 ? `${parts[0]}.${parts[1]}.${parts[2]}.0` : ip
}

// Configured secrets (an SMTP password, a model key) have no shape a pattern can
// recognise, so the config registers its literal values here. The set is
// replaced per call, not appended: tests call loadConfig() repeatedly, and
Expand Down
5 changes: 4 additions & 1 deletion src/lib/secrets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@ const VERSION = 'v1'

// The recogniser lives in one module so a job log, a callback and a URL cannot
// drift onto different rules. This re-export keeps the existing import path.
export { redact, registerRedactionSecrets } from './redact.js'
// `maskEmail` / `maskIp` (#694) are shapes an account is rendered by, not
// secrets, so they live beside `redact()` and are re-exported here too: the log
// sink and the pages import from one place.
export { maskEmail, maskIp, redact, registerRedactionSecrets } from './redact.js'

export function decodeKey(keyBase64: string): Buffer {
const key = Buffer.from(keyBase64, 'base64')
Expand Down
23 changes: 1 addition & 22 deletions src/views/pages.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import { DECISION_KINDS, DEFAULT_DECISION, decisionKindLabel } from '../lib/task
import { ITERATION_INTERVALS, REVIEW_INTERVALS, intervalLabel, normalizeInterval } from '../lib/intervals.js'
import { escapeHtml, FLEDGE_GITHUB_URL, layout, money, quotaMoney, starIcon, type PageNotification, type Viewer } from './layout.js'
import { renderState, stateCell, type StateAction } from './states.js'
import { redact } from '../lib/secrets.js'
import { maskEmail, maskIp, redact } from '../lib/secrets.js'
import { REVIEW_INSTRUCTION_LABEL, reviewInstructionFromGoal } from '../lib/followUpInstruction.js'
import { isTemporalStatus, summariseDelta, type TemporalDelta } from '../lib/temporal.js'
import { firstRunOutcome, setupChecklist, type SetupState, type SetupStepKey } from './checklist.js'
Expand Down Expand Up @@ -258,27 +258,6 @@ function readinessBadge(t: Translate, state: 'ready' | 'blocked' | 'unknown'): s
return `<span class="badge ${cls}">${escapeHtml(readinessLabel(t, state))}</span>`
}

function maskEmail(email: string): string {
const at = email.lastIndexOf('@')
if (at <= 0) return '***'
const local = email.slice(0, at)
const domain = email.slice(at + 1)
return `${local.slice(0, 1)}***@${domain}`
}

// #356: a session row is recognisable without handing out the address. The last
// IPv4 octet and everything past the second IPv6 group are dropped, so a user can
// tell their own sessions apart but the page never carries a full address.
function maskIp(ip: string | null): string {
if (ip === null || ip === '') return '—'
if (ip.includes(':')) {
const groups = ip.split(':').filter((group) => group !== '')
return `${groups.slice(0, 2).join(':')}::`
}
const parts = ip.split('.')
return parts.length === 4 ? `${parts[0]}.${parts[1]}.${parts[2]}.0` : ip
}

function agreementHtml(text: string): string {
return text
.split(/\n\n/)
Expand Down
Loading
Loading