Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion packages/vx/docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -1541,7 +1541,9 @@ vx watch build -- --sourcemap # forwarded args carry through every cycle
Any other (an input of a task that already finished, a new lock) is
found as the loop arms: a file git lists as untracked or modified,
written since the run started by none of its commands, is a change,
and the first cycle runs on it. A deletion made while it runs is
and the first cycle runs on it. One written while a command of its
own project ran may be that command's: the keys are taken again, and
it is a change when one moved. A deletion made while it runs is
still unheard: it has no date to tell it from one made before.
2. **Watch loop.** After the initial run finishes (or once it waits only
on a server's readiness, X-139), the directory of
Expand Down
7 changes: 5 additions & 2 deletions packages/vx/docs/modules/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -498,8 +498,11 @@ too: they format one run's result.
(`InitialRun.from`, on the mtime clock) outside every window of its
commands (`InitialRun.windows`) is an event, marked `unseen` so the
judgement takes it for a change; a re-lock among them is the first
cycle's (`InitialRun.lock`). A start with no edit pays `-m`'s stats
alone (`tests/watch-initial-edit-done.test.ts`).
cycle's (`InitialRun.lock`). One inside a window may be the command's
own or an edit to another task of that project, already keyed: a
keys-only plan (`hearIfKeysMoved`, no store asked) is compared with
the run's keys, and the files are events when one moved. A start with
no edit pays `-m`'s stats alone (`tests/watch-initial-edit-done.test.ts`).
5. **Watch loop** (`runWatchLoop`):
- For each project a cycle can run (`watchedProjects`: the scope
plus its transitive dependencies through `buildPackageGraph` with
Expand Down
2 changes: 2 additions & 0 deletions packages/vx/docs/modules/util.md
Original file line number Diff line number Diff line change
Expand Up @@ -1096,6 +1096,8 @@ run against.
`vx watch` calls `restartTimings()` as each cycle starts, so a cycle's
table holds its own stages alone, timed from its start rather than
across the idle wait before it (`tests/watch-timing.test.ts`).
`untimed(fn)` runs `fn` with no table printed and none left behind:
the keys-only plan `vx watch` takes as it arms is no cycle.
`prepareRun` calls `beginRun()` first: a process's first run keeps
its stages from this module's load, and a later one (an embedder's
second `run()`) restarts the table the same way.
Expand Down
93 changes: 74 additions & 19 deletions packages/vx/src/cli/watch-loop.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { namedIgnoredReach, repoFacts } from '../cache/index.js'
import {
type CommandWindows,
forwardedSignal,
planCore,
runCore as runOrchestrator,
type HeldPersistent,
type CoreRunOptions,
Expand All @@ -32,11 +33,11 @@ import {
import { CLOSED, type WatchHandle, WatcherPool } from './watch-fs.js'
import { ChangeJudge } from './watch-judge.js'
import { type WatchCycle, watchCycle } from './watch-cycle.js'
import { maskedLine, printable, restartTimings } from '../util/index.js'
import { maskedLine, printable, restartTimings, untimed } from '../util/index.js'
import * as ui from '../ui/index.js'
import { sayNotice } from '../ui/index.js'
import { isCacheHit } from '../graph/index.js'
import { resolveOutputView } from '../report/index.js'
import { type Logger, resolveOutputView } from '../report/index.js'

/** One line for a watcher or re-read the OS refused; the loop goes on without it. */
function sayCannot(what: string, err: unknown): void {
Expand Down Expand Up @@ -64,6 +65,18 @@ function isDirectory(abs: string): boolean {
}
}

/** A plan asked only for its keys asks no store. */
const KEYS_ONLY = { localRead: false, localWrite: false, remoteRead: false, remoteWrite: false }

/** A plan asked only for its keys says nothing. */
const SILENT: Logger = {
status() {},
warn() {},
taskStdout() {},
taskStderr() {},
taskComplete() {},
}

/** Wait this long after the last filesystem event before re-running. */
const DEBOUNCE_MS = 150
/**
Expand Down Expand Up @@ -102,6 +115,8 @@ interface WatchLoopArgs {
tasks: () => number | undefined
/** A task the initial run left unsaved because its inputs moved under it (`TaskOutcome.cache.unkeyed`). */
drifted: string | undefined
/** Each task's key as the initial run took it; none when the loop armed while it ran. */
keys: ReadonlyMap<string, string> | undefined
/** Aborted by the SIGINT/SIGTERM handlers `watchCmd` installed; the loop drains its cycle and resolves. */
stop: AbortSignal
workspaceRoot: string
Expand Down Expand Up @@ -407,9 +422,14 @@ class WatchLoop {
// An edit made during the initial run predates the arm, so no event finds
// it, and watch idled over stale output. One to a task the run was running
// its re-check found; any other is heard here.
const heard = this.heardSinceRun()
const { heard, inWindow } = this.heardSinceRun()
if (args.drifted !== undefined && !heard)
void this.cycle(`${args.drifted} inputs, edited during the first run`)
// Heard, the cycle keys every task itself.
else if (!heard && inWindow.length > 0 && args.keys !== undefined && !this.running) {
this.running = true
this.inFlight = this.hearIfKeysMoved(inWindow, args.keys)
}

return await new Promise<void>((resolve) => {
const cleanup = async (): Promise<void> => {
Expand Down Expand Up @@ -1165,10 +1185,10 @@ class WatchLoop {
* `atArm` asked git already, so a start with no edit costs only stats. An
* edit before its task keyed it re-runs to a hit.
*/
private heardSinceRun(): boolean {
private heardSinceRun(): { heard: boolean; inWindow: string[] } {
const { initialRun } = this.args
const workspaceRoot = this.workspaceRoot
let heard = false
const inWindow: string[] = []
for (const abs of this.atArm?.dirty ?? []) {
let t: number
try {
Expand All @@ -1182,23 +1202,58 @@ class WatchLoop {
if (
initialRun.windows.writerDuring('', abs, t, t + racyWindowMs(t, FILE_HASH_RACY_MS)) !==
undefined
)
) {
inWindow.push(abs)
continue
const rel = path.relative(workspaceRoot, abs)
if (this.workspaceWide) this.wideEvent(rel)
else {
if (!rel.includes(path.sep)) this.rootFileEvent(rel)
else if (rel.startsWith(`.vx${path.sep}`) && !rel.slice(4).includes(path.sep))
this.dotVxEvent(rel.slice(4))
for (const proj of this.watched)
if (abs.startsWith(proj.dir + path.sep))
this.projectEvent(proj, path.relative(proj.dir, abs))
}
if (!this.changes.pending.has(abs)) continue
this.changes.unseen([abs])
heard = true
if (this.hear(abs)) heard = true
}
return { heard, inWindow }
}

/** `abs` as an event no watcher delivered: true when it is pending, a change at its judgement. */
private hear(abs: string): boolean {
const rel = path.relative(this.workspaceRoot, abs)
if (this.workspaceWide) this.wideEvent(rel)
else {
if (!rel.includes(path.sep)) this.rootFileEvent(rel)
else if (rel.startsWith(`.vx${path.sep}`) && !rel.slice(4).includes(path.sep))
this.dotVxEvent(rel.slice(4))
for (const proj of this.watched)
if (abs.startsWith(proj.dir + path.sep))
this.projectEvent(proj, path.relative(proj.dir, abs))
}
if (!this.changes.pending.has(abs)) return false
this.changes.unseen([abs])
return true
}

/**
* Files written inside a command's window of the initial run: the
* command's own, or an edit to another task of its project that had
* already keyed (a hit, or done). The write cannot say which, the keys can:
* taken again, one that moved is a task built from bytes the tree no
* longer holds, and the files are heard. Dropped unasked, the edit waited
* for the next save. Holds `running`, so an event meanwhile queues.
*/
private async hearIfKeysMoved(
inWindow: readonly string[],
keys: ReadonlyMap<string, string>,
): Promise<void> {
try {
const plan = await untimed(() => planCore({ ...this.opts, log: SILENT, cache: KEYS_ONLY }))
const moved = plan.tasks.some((t) => {
const was = keys.get(t.node.id)
return was !== undefined && was !== t.hash
})
if (moved) for (const abs of inWindow) this.hear(abs)
} catch {
// The cycle an edit starts says what a plan could not.
} finally {
this.running = false
const next = pendingAfterCycle(this.changes.pending, this.stop.aborted)
if (next !== undefined) this.trigger(next[1], next[0])
}
return heard
}
}

Expand Down
12 changes: 10 additions & 2 deletions packages/vx/src/cli/watch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,8 @@ export async function watchCmd(args: readonly string[]): Promise<number> {
// (see config-eval.ts). The edit is found instead: one to the inputs of a
// task the run was running leaves that task unsaved, and the loop starts on
// it (`drifted`); any other is a file the arm finds written since the run
// started by none of its commands (`InitialRun`). The exception: once the run
// started by none of its commands (`InitialRun`), or inside one's window
// with a key moved since (`hearIfKeysMoved`). The exception: once the run
// waits on a server's readiness alone, every config is loaded, and the loop
// arms with the run as its cycle in flight. Unarmed, a server that never got
// ready held the initial run for good, and the fix to it was never heard.
Expand Down Expand Up @@ -223,9 +224,14 @@ export async function watchCmd(args: readonly string[]): Promise<number> {
let refusedToStart = false
/** A task whose inputs the user edited while the initial run ran it: no watcher heard the edit. */
let drifted: string | undefined
/** Each task's key as the initial run took it. */
let keys: ReadonlyMap<string, string> | undefined
initial.held = runOrchestrator(initial.cycle.opts).then(
async (run) => {
drifted = run.outcomes.find((o) => o.cache?.unkeyed === true)?.node.id
keys = new Map(
run.outcomes.flatMap((o) => (o.hash === undefined ? [] : [[o.node.id, o.hash]])),
)
if (run.refused !== undefined) {
const hint = run.refused.startsWith('no projects declare')
? await spacedHint(parsed, cwd)
Expand Down Expand Up @@ -268,13 +274,14 @@ export async function watchCmd(args: readonly string[]): Promise<number> {
sayNotice('error', 'vx watch', 'no projects in scope')
return 1
}
await armLoop(undefined, held, drifted)
await armLoop(undefined, held, drifted, keys)
return stopped()

async function armLoop(
initial?: InitialCycle,
held?: HeldPersistent,
drifted?: string,
keys?: ReadonlyMap<string, string>,
): Promise<void> {
// `--affected`'s diff is the tree at start; judged again, it held every
// later edit out of the scope it picked. A cycle is an edit, and the
Expand All @@ -295,6 +302,7 @@ export async function watchCmd(args: readonly string[]): Promise<number> {
initialRun,
tasks: openingTasks,
drifted,
keys,
stop: stop.signal,
workspaceRoot,
projects: watched,
Expand Down
1 change: 1 addition & 0 deletions packages/vx/src/util/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ export {
mark,
timingLines,
restartTimings,
untimed,
span,
stageTimes,
type StageTime,
Expand Down
18 changes: 17 additions & 1 deletion packages/vx/src/util/timing.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,25 @@ export function span(label: string): () => void {
}
}

let muted = false

/**
* Run `fn` with no table printed and none left behind: a plan `vx watch`
* asks only for its keys is no cycle, and its table read as one.
*/
export async function untimed<T>(fn: () => Promise<T>): Promise<T> {
muted = true
try {
return await fn()
} finally {
muted = false
restartTimings()
}
}

/** The stage table's lines (once, at the end of a run); none unless enabled. */
export function timingLines(): string[] | undefined {
if (!enabled || marks.length === 0) return undefined
if (!enabled || muted || marks.length === 0) return undefined
const width = Math.max(...marks.map(([l]) => l.length))
let prev = 0
const lines = ['[vx timing] stage'.padEnd(width + 14) + ' own cumulative']
Expand Down
59 changes: 56 additions & 3 deletions packages/vx/tests/watch-initial-edit-done.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,10 @@ const output = (lib: string) => async (): Promise<string> =>
* lib#build's row: its command's window has closed. A write inside it may
* be the command's own, and watch cannot tell, so the edit comes after.
*/
async function finished(w: Watch, lib: string): Promise<void> {
async function finished(w: Watch, lib: string, row = 'lib#build'): Promise<void> {
await until(
async () => w.out().includes('lib#build') && (await output(lib)()) === 'old\n',
'lib#build to finish',
async () => w.out().includes(row) && (await output(lib)()) === 'old\n',
`${row} to finish`,
)
}

Expand All @@ -85,6 +85,59 @@ it.each([
40_000,
)

/** `lib#lint` is done at once; `lib#hold`, behind it, holds the run until `go` exists, then writes a file. */
async function oneProject(): Promise<{ lib: string; go: string; root: string }> {
const root = await makeWorkspace({ prefix: 'vx-watch-initial-edit-done-' })
const outside = await mkdtemp(path.join(os.tmpdir(), 'vx-watch-initial-edit-done-out-'))
made.push(root, outside)
const go = path.join(outside, 'go')
const lib = await addProject(
root,
'lib',
`export default { tasks: {
lint: {
exec: { command: 'mkdir -p dist && cat src/a.txt > dist/out.txt' },
inputs: { files: ['src/**'] }, outputs: { files: ['dist/**'] }, cache: true,
},
hold: {
exec: { command: 'while [ ! -f ${go} ]; do sleep 0.02; done; echo held > held.txt' },
dependsOn: ['lint'],
},
} }`,
)
await mkdir(path.join(lib, 'src'), { recursive: true })
await writeFile(path.join(lib, 'src', 'a.txt'), 'old\n')
await writeFile(path.join(root, '.gitignore'), 'dist\n')
return { lib, go, root }
}

// A write while a command of the same project runs may be that command's,
// so it was dropped: `lint` was done, `hold` was running, and the edit to
// lint's input waited for the next save. The keys tell: one moved.
it('an edit to a finished task, made while another task of its project runs, starts a cycle', async () => {
const { lib, go, root } = await oneProject()
watch = startWatch(root, ['--all'], {}, 'hold')
const w = watch
const out = output(lib)
await finished(w, lib, 'lib#lint')
await writeFile(path.join(lib, 'src', 'a.txt'), 'new\n')
await writeFile(go, '')
await until(async () => (await out()) === 'new\n', 'the output built from the edit')
}, 40_000)

// CONTROL: the command's own write, in its window, moves no key.
it('a file the initial run wrote in its own project starts no cycle', async () => {
const { lib, go, root } = await oneProject()
watch = startWatch(root, ['--all'], {}, 'hold')
const w = watch
await finished(w, lib, 'lib#lint')
await writeFile(go, '')
await until(() => w.out().includes('watching'), 'the watching marker')
expect(await readFile(path.join(lib, 'held.txt'), 'utf8')).toBe('held\n')
await Bun.sleep(1_000)
expect(w.cycles()).toBe(0)
}, 40_000)

it('a `vx lock` during the initial run starts a cycle on the new lock', async () => {
const { lib, go, root } = await fixture(false)
watch = startWatch(root, ['--all'])
Expand Down
Loading