Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,18 @@ jobs:
echo "healthz ok, version ${version}"
docker rm -f fledge-smoke >/dev/null 2>&1 || true
cleaned=1
# #109: the FlyBot boundary is proven from inside a real container, with the
# product's own run arguments, in the one job that has a Docker daemon. The
# node test job has none, so `npm test` keeps the argument assertions and
# this step carries the container-level proof. A run without a daemon exits
# 3 instead of passing, as the check itself does everywhere else.
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: 22
cache: npm
- run: npm ci
- name: Prove the FlyBot container boundary from inside a container
run: npx tsx scripts/isolation-check.ts --image fledge:ci

audit:
name: audit
Expand Down
1 change: 1 addition & 0 deletions changelog.d/109-flybot-isolation.security.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A per-repository FlyBot container now starts with a read-only root, a single writable tmpfs at `/tmp`, every Linux capability dropped, `no-new-privileges`, a non-root user and the existing CPU, memory and PID ceilings, and the host Docker socket is never mounted into it. A new isolation check starts a throwaway container with exactly those arguments and asserts each fact from inside it — the root refuses a write, the effective capability set is empty, a seccomp filter is loaded, the socket is absent, only the repository's own data directory is a host bind, and a container on another tenant's network is unreachable — and exits 3 where no Docker daemon exists instead of reporting a pass. The builder also refuses a data directory that is not an absolute existing directory, so the one host path a tenant container sees cannot be a relative path the daemon resolves elsewhere or a path Docker would create as root. Egress is unchanged in this slice.
30 changes: 30 additions & 0 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,36 @@ long runs; lowering it makes that failure more likely, so change it together
with the agent's own time budget. `JOB_STALL_MINUTES` (default 30) is the
separate no-heartbeat stall sweep; a heartbeat moves only that one.

### Per-repository FlyBot containers

Each hosted repository gets its own container, started from
`flybotContainerArgs()` in `src/flybot/isolation.ts`. The flags are the boundary,
so they are not configurable per repository: `--read-only` with one `/tmp` tmpfs
(`rw,noexec,nosuid,size=64m`), a single bind of that repository's data directory
at `/data`, `--cap-drop ALL`, `--security-opt no-new-privileges`, a non-root
`--user`, and the CPU, memory and PID ceilings. `NARB_DISABLE_NATIVE_CACHE=1` is
not optional: the DSH native-addon loader copies a binary into the temp directory
and executes it, so a `noexec` `/tmp` otherwise stops the web profile from
starting.

Those flags reach production through the normal deploy and take effect the next
time an instance is provisioned or restarted. Before shipping, prove the boundary
on a host that runs the containers:

```sh
npm run isolation:check # node:22-bookworm-slim probe
npm run isolation:check -- --image <flybot image>
```

The check starts its own throwaway container with exactly those arguments, then
asserts from inside it that the root refuses a write, the effective capability set
is empty, a seccomp filter is loaded, the Docker socket is absent, only the
repository data directory is a host bind, and a container on another tenant's
network is unreachable. It exits 3 where no Docker daemon exists rather than
reporting a pass. Egress is not restricted yet: a fresh data volume installs its
profile from `registry.npmmirror.com` on first start, so deny-by-default egress
with an allowlist is still open work and not something a deploy changes today.

## 5. Run

### Docker
Expand Down
40 changes: 40 additions & 0 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,46 @@ operator FlyBot is not used for user jobs. A container is limited by CPU,
memory, PID and filesystem mounts; stopping or deleting a repository affects
only that container.

The container's run arguments are a security boundary, so they are built in one
place (`flybotContainerArgs` in `src/flybot/isolation.ts`) and nothing else adds
a mount or a network:

- **Read-only root.** `--read-only` plus one tmpfs at `/tmp`
(`rw,noexec,nosuid,size=64m`). Only the repository's own data directory is a
host bind; the container cannot rewrite the image the next job starts from.
- **No capabilities.** `--cap-drop ALL` with `--security-opt no-new-privileges`
and a non-root `--user`, so a process inside cannot gain one back through a
setuid binary.
- **No host Docker socket.** The arguments never carry `/var/run/docker.sock`;
the socket is root on the host, so mounting it into a tenant container would
make every other tenant reachable.
- **One checked mount source.** The data directory must be an absolute path to an
existing directory before it becomes a bind: the builder throws otherwise,
because a relative path is resolved by the daemon against its own working
directory and a missing one is created by Docker as root — either way the mount
would come from outside the tenant's tree.
- **Seccomp stays on.** The runtime's default profile is what refuses the rare
syscalls an agent runtime never needs; the isolation check asserts that a
filter really is loaded (`Seccomp: 2`) rather than replacing the profile with a
weaker hand-written one.
- **PID, CPU and memory ceilings** stay as they were, which is what keeps one
tenant from starving the host.

`npm run isolation:check` starts a throwaway container with exactly those
arguments and asserts each fact from inside it: the root refuses a write, the
effective capability set is empty, a seccomp filter is loaded, the Docker socket
is absent, exactly one host bind is mounted, and a container on another tenant's
network cannot be reached. It exits 3 where no Docker daemon exists, because a
silent pass would hide the boundary it exists to test. CI runs it in the `docker`
job; a developer can run it on any host with a daemon.

Egress is not restricted yet: the container still reaches the outside network,
because a fresh data volume installs its profile from `registry.npmmirror.com` on
first start. Deny-by-default egress with an allowlist is the remaining close
condition, and it needs a mechanism decision (an egress proxy on an internal
network, or a warm-store seal before the long-lived container starts) before it
can land.

### API tokens

- A personal `flg_` token is a gateway capability with two properties: a `scope`
Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
"mock:snapshot": "tsx scripts/mock-snapshot.ts",
"mock:check": "tsx scripts/mock-snapshot.ts --check",
"evolution:consent": "tsx scripts/evolution-consent.ts",
"isolation:check": "tsx scripts/isolation-check.ts",
"lint:update": "node scripts/lint-warnings.mjs --update"
},
"dependencies": {
Expand Down
165 changes: 165 additions & 0 deletions scripts/isolation-check.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
// #109: the isolation boundary is proven from inside a real container that is
// started with exactly the flags the product starts, never by reading flags.
// This check needs a Docker daemon; where there is none it exits 3 with a loud
// message, because a silent pass would hide the very boundary it exists to test.
import { execFile } from 'node:child_process'
import { mkdtempSync, rmSync } from 'node:fs'
import { createServer } from 'node:net'
import { join } from 'node:path'
import { promisify } from 'node:util'
import { flybotContainerArgs } from '../src/flybot/isolation.js'

const execFileAsync = promisify(execFile)

const NO_DAEMON = 3
const BOUNDARY_BROKEN = 1

interface Options {
docker: string
image: string
}

function parseOptions(argv: string[]): Options {
const options: Options = { docker: 'docker', image: 'node:22-bookworm-slim' }
for (let index = 0; index < argv.length; index += 1) {
const flag = argv[index]
const value = argv[index + 1]
if (flag === '--docker' && value !== undefined) { options.docker = value; index += 1 }
else if (flag === '--image' && value !== undefined) { options.image = value; index += 1 }
else throw new Error(`unknown argument: ${flag ?? ''}`)
}
return options
}

function docker(options: Options, argv: string[], timeout = 120_000): Promise<string> {
return execFileAsync(options.docker, argv, { timeout, maxBuffer: 8 * 1024 * 1024 }).then((result) => result.stdout.trim())
}

async function freePort(): Promise<number> {
return new Promise((resolve, reject) => {
const server = createServer()
server.on('error', reject)
server.listen(0, '127.0.0.1', () => {
const address = server.address()
const port = typeof address === 'object' && address !== null ? address.port : 0
server.close(() => resolve(port))
})
})
}

// Runs inside the probe container. It reports what the kernel says about the
// process, not what the flags intended, and it tries one thing it must not be
// able to do: reach a sibling container on another tenant's network.
const PROBE = `
const fs = require('node:fs')
const net = require('node:net')
const fact = {}
try { fs.writeFileSync('/__isolation_probe', 'x'); fact.rootWritable = true } catch (error) { fact.rootWritable = false; fact.rootError = error.code }
const status = fs.readFileSync('/proc/self/status', 'utf8')
fact.caps = (/^CapEff:\\s*([0-9a-f]+)/m.exec(status) || [])[1] || null
fact.seccomp = (/^Seccomp:\\s*(\\d+)/m.exec(status) || [])[1] || null
fact.dockerSocket = fs.existsSync('/var/run/docker.sock')
fact.dataMounts = fs.readFileSync('/proc/mounts', 'utf8').split('\\n')
.filter((line) => line.includes(' /data ')).map((line) => line.split(' ')[1])
const sibling = process.argv[1]
const reach = (host, port, timeout) => new Promise((resolve) => {
const socket = net.connect({ host, port })
const done = (value) => { socket.destroy(); resolve(value) }
socket.setTimeout(timeout, () => done(false))
socket.once('connect', () => done(true))
socket.once('error', () => done(false))
})
reach(sibling, Number(process.argv[2]), 1500).then((ok) => { fact.siblingReachable = ok; process.stdout.write(JSON.stringify(fact)) })
`

interface Fact {
rootWritable: boolean
rootError?: string
caps: string | null
seccomp: string | null
dockerSocket: boolean
dataMounts: string[]
siblingReachable: boolean
}

async function main(): Promise<void> {
const options = parseOptions(process.argv.slice(2))
try {
await docker(options, ['info', '--format', '{{.ServerVersion}}'], 30_000)
} catch {
console.error(`isolation-check: no Docker daemon reachable through ${options.docker} — this check cannot pass without one`)
process.exit(NO_DAEMON)
}

const tag = `${process.pid}-${Math.random().toString(16).slice(2, 8)}`
const network = `fledge-isolation-${tag}`
const siblingNetwork = `fledge-isolation-sibling-${tag}`
const probe = `fledge-isolation-probe-${tag}`
const sibling = `fledge-isolation-sibling-${tag}`
const dataDir = mkdtempSync(join(process.cwd(), '.isolation-check-'))
const failures: string[] = []
try {
await docker(options, ['network', 'create', network])
await docker(options, ['network', 'create', siblingNetwork])
// The sibling stands in for another tenant: its own network, its own
// container, one open port. Nothing a job container does may reach it.
await docker(options, [
'run', '-d', '--name', sibling, '--network', siblingNetwork, '--entrypoint', 'node', options.image,
'-e', 'require("node:http").createServer((q, s) => s.end("sibling")).listen(8080)',
])
const siblingIp = await docker(options, ['inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}', sibling])
const dshPort = await freePort()
const args = flybotContainerArgs({
container: probe,
network,
dataDir,
uid: typeof process.getuid === 'function' ? process.getuid() : 1000,
gid: typeof process.getgid === 'function' ? process.getgid() : 1000,
dshPort,
memory: '512m',
cpus: '1',
image: options.image,
modelKey: 'isolation-probe',
command: `node -e '${PROBE.replace(/'/g, "'\\''")}' ${siblingIp} 8080`,
})
// The product starts the container detached and waits for the web port; the
// check wants the probe's stdout, so it runs the same arguments attached.
const output = await docker(options, ['run', '--rm', ...args.slice(2)], 180_000)
.catch((error: { stdout?: string; stderr?: string }) => error.stdout || error.stderr || '')

let fact: Fact | null = null
try { fact = JSON.parse(output) as Fact } catch { fact = null }
if (fact === null) {
console.error(`isolation-check: probe produced no facts\n${output}`)
process.exit(BOUNDARY_BROKEN)
}

const report = (ok: boolean, line: string): void => {
console.log(`isolation-check: ${ok ? 'ok' : 'FAILED'} — ${line}`)
if (!ok) failures.push(line)
}
report(fact.rootWritable === false, `root filesystem is read-only (write failed with ${fact.rootError ?? 'an error'})`)
report(fact.caps === '0000000000000000', `every capability is dropped (CapEff ${fact.caps ?? 'unreadable'})`)
report(fact.seccomp === '2', `a seccomp filter is loaded (Seccomp ${fact.seccomp ?? 'unreadable'})`)
report(fact.dockerSocket === false, 'the host Docker socket is not visible inside the container')
report(fact.dataMounts.length === 1, `exactly one host bind is mounted, the repository's own data directory (${fact.dataMounts.join(', ') || 'none'})`)
report(fact.siblingReachable === false, 'another tenant on another Docker network is unreachable')

if (failures.length > 0) {
console.error(`isolation-check: ${failures.length} condition(s) failed`)
process.exit(BOUNDARY_BROKEN)
}
console.log('isolation-check: boundary holds')
} finally {
await docker(options, ['rm', '-f', probe], 60_000).catch(() => '')
await docker(options, ['rm', '-f', sibling], 60_000).catch(() => '')
await docker(options, ['network', 'rm', network], 60_000).catch(() => '')
await docker(options, ['network', 'rm', siblingNetwork], 60_000).catch(() => '')
rmSync(dataDir, { recursive: true, force: true })
}
}

main().catch((error: unknown) => {
console.error(`isolation-check: ${error instanceof Error ? error.message : String(error)}`)
process.exit(BOUNDARY_BROKEN)
})
34 changes: 13 additions & 21 deletions src/flybot/instances.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { log } from '../lib/log.js'
import { decryptSecret, encryptSecret, randomToken, sha256Hex } from '../lib/secrets.js'
import { nowIso } from '../lib/time.js'
import { PLAYBOOK_VERSION, TEAM_ROLES, playbookFingerprint, playbookFor } from './playbook.js'
import { flybotContainerArgs } from './isolation.js'
import { clearWorkspaceCookie, dshCookie } from './workspace.js'
import { readZipText } from '../lib/zip.js'
import { parseChiefTranscript, type ChiefMessage } from '../lib/chiefTranscript.js'
Expand Down Expand Up @@ -302,27 +303,18 @@ server.listen(3081, '0.0.0.0')
const gid = typeof process.getgid === 'function' ? process.getgid() : 1000
writeFileSync(join(dataDir, 'dsh-web.log'), '')
clearStaleDshLock(dataDir)
await this.docker([
'run', '-d',
'--name', container,
'--network', network,
'-p', `127.0.0.1:${dshPort}:3081`,
'-v', `${dataDir}:/data`,
'--user', `${uid}:${gid}`,
'--memory', this.config.flybot.memory,
'--cpus', this.config.flybot.cpus,
'--pids-limit', '512',
'--security-opt', 'no-new-privileges',
'--restart', 'no',
'--entrypoint', '/bin/sh',
'-e', 'HOME=/data',
'-e', 'DSH_HOME=/data',
'-e', 'NPM_CONFIG_REGISTRY=https://registry.npmmirror.com',
'-e', 'npm_config_registry=https://registry.npmmirror.com',
'-e', `FLEDGE_MODEL_KEY=${modelKey}`,
this.config.flybot.instanceImage,
'-c', 'node /opt/flybot/docker/entrypoint.mjs >> /data/dsh-web.log 2>&1 & exec node /data/tcp-proxy.mjs',
], 180_000)
await this.docker(flybotContainerArgs({
container,
network,
dataDir,
uid,
gid,
dshPort,
memory: this.config.flybot.memory,
cpus: this.config.flybot.cpus,
image: this.config.flybot.instanceImage,
modelKey,
}), 180_000)

await this.waitForDsh(dshPort, row)
if (this.store.getRepo(repo.id) === undefined) throw new Error('仓库已删除,停止初始化')
Expand Down
Loading
Loading