Epic: converge the start phase into the plan engine #14081

"This issue was filed by an AI agent on a human's behalf. The human submitter may not have independently verified the report."

Context

Two lifecycle engines coexist today (#14074, section C). The plan-based reconciler (reconcile.go → executor.go, single entry point create.go) is pure, deterministic and golden-tested — but create-only: containers leave the plan in created state, and OpStartContainer is only ever emitted for exotic states (paused/dead). Everything that makes an application actually start lives in the second, imperative engine (InDependencyOrder): waitDependencies (health/completion polling), secret/config injection, pre_start/post_start hooks. up chains the two with two different daemon snapshots, no canonical project object across the phases, two event-emission systems, and latent bugs at the seam (startMx not held on the path actually exercised; the start-phase ContainerList skips the config-hash filter; dependency-wait timeouts swallowed by ctx.Done() → nil).

This epic tracks converging the start phase into the plan engine, one reviewable PR at a time.

Target architecture

Guiding split: decision vs execution — not "plan vs imperative". The Plan is the pivot execution format of Compose; the reconciler is just one producer of plans.

• One snapshot, one project, one plan with two phases. ReconcileOptions gains a scope (Create, Start, or both); PlanNode gains a Phase field. up -d builds a single Create+Start plan; start/scale/watch-rebuild use scope Start (never recreating); compose create keeps scope Create unchanged — the existing golden tests do not change.
• New operations (explicit numbering, 40+):
  • OpWaitCondition — one node per (awaited service, condition ∈ healthy / completed_successfully / running_or_healthy), deduplicated across dependents (a waitNodes map, like networkNodes). required: false is absorbed locally: Skipped event, node succeeds — same pattern as the existing BestEffort. condition: service_started needs no node: a plain DAG edge expresses it. Health is re-observed at execution time (the node runs today's waitDependency polling); ObservedState deliberately does not grow a Health field — it would be stale by construction.
  • OpRunPreStart — per service. Emitted at plan time only when no replica was running at observation (today's rule in startService), targeting the lowest-numbered replica (lowestNumberedContainer).
  • OpRunPostStart — per container, after its start.
• OpStartContainer enriched: secret/config injection folds into execStartContainer (they always run as a pair right before start — a separate node would be noise), and the target resolves either from an observed container or from the CreateNodeID of a create in the same plan (the reconciliationContext mechanism already used by OpRenameContainer). Side effect by construction: every ContainerStart in the codebase now goes through the one call site that holds startMx.
• Replica chains: inject→start→post_start of replica n+1 depends on the end of replica n's chain — today's sequential start order, preserved and now visible in golden plans. serviceNodes[svc] points at the end of the chain, so a service_started dependent waits for the whole service, matching InDependencyOrder semantics.
• Events: on converged paths the executor is the only emitter. Exact parity of observable sequences (Waiting→Healthy|Exited|Skipped, Starting→Started with Started emitted after post_start) via a start:<svc>:<n> group on the existing groupTracker.
• Interactive up: prepare the plan once; execute the Create phase; set up attach/printer/monitor (upSession unchanged); execute the Start phase under context.WithoutCancel with the printer as listener. The phase boundary replaces today's create/start seam without reintroducing a second snapshot.
• --wait stays post-plan: it is a final verification with a global timeout and synthetic conditions (getDependencyCondition), running on the shared waitDependency primitive.
• Shared primitives, no decisions inside: waitDependency, injectSecrets/injectConfigs, runHook, createMobyContainer — consumed by the executor and by what stays imperative (restart, run, --wait). startService/startServiceContainer/waitDependencies are deleted at the end of the series.
• Executor constraint (to document in code): never add errgroup.SetLimit to the plan executor while it schedules one blocking goroutine per node — instant deadlock. A concurrency cap requires moving to a ready-queue scheduler first.

Non-goals (explicit)

• stop / down: reverse-order teardown converges later as a plan-builder producing a Plan directly (no reconciler — down without a compose file has no desired state to diff; --rmi, anonymous volumes and duplicate-named networks don't fit a diff model). Separate epic.
• restart: ContainerRestart is atomic on the daemon side; decomposing it into Stop+Start changes semantics. Stays imperative on the shared primitives.
• kill / pause / unpause: unordered by design; a plan would introduce an ordering that doesn't exist.
• run one-off container: by definition outside desired state (number=-1, AutoRemove, unique slug). Its dependencies converge for free (they go through Create/Start); its own creation stays imperative on the shared primitives.
• Parallel replica starts (relaxing today's sequential order): becomes a trivial edge change after this series; not mixed into it.

PR breakdown

Lot 0 — foundations (no behavior change except deliberate bug fixes; independent, can start immediately)

• test: unit-lock the imperative start path — characterization tests only (start idempotence, "no container to start", pre_start gating ×3, inject→start→post_start order, required:false skip, getDependencyCondition, restart restart:true). The imperative engine is currently locked only by e2e. → test: unit-lock the imperative start path #14104 (merged)
• fix: dependency wait timeout silently ignored (Epic: make the codebase agent-legible — fix misleading self-description, ambiguous contracts, and legacy leftovers #14074 C) — waitDependency returns the error on DeadlineExceeded; user cancellation stays silent; audit of the 4 callers. → fix: dependency wait timeout surfaced only by luck #14105 (merged)
• fix|chore: startMx on the real start path — maintainer decision requested below. → fix: hold startMx on the start path actually exercised #14106 (merged)
• refactor: NewGraph stops mutating the project — pruning of unresolvable optional deps becomes an explicit step; precondition for a canonical project object. → refactor: NewGraph stops mutating the project #14124 (+ introduce Project.WithoutUnresolvedOptionalDependencies compose-spec/compose-go#922 for the upstream Project.WithoutUnresolvedOptionalDependencies)

Lot 1 — vocabulary (the plan learns to start; inert code, no consumer)

• feat: reconciler plans the start phase — new ops, Phase field, planStartPhase (deduplicated waits reproducing shouldWaitForDependency, started-edges, plan-time pre_start gating, replica chains, exited/created → start chain under scope Start). Golden tests only; enabled by an option nobody passes yet. → plan engine: reconciler plans the start phase (epic #14081, lot 1) #14200 (merged)
• feat: pre_start hook runners become reconciliation-plan resources — runner creation moves into the create phase (deterministic naming, DAG ordering, purge of stale runners), landing independently of the start-phase plan and setting up hook execution/removal to become start-phase operations next. → pre_start hook runners become reconciliation-plan resources #14221 (merged)
• feat: executor runs start-phase operations — execWaitCondition delegates to waitDependency (no polling rewrite), enriched execStartContainer, listener plumbing for hook logs, start:* event groups with word-for-word event parity. → executor: run start-phase operations (epic #14081, lot 1) #14274 (merged)
• refactor: split create() into preparePlan + execute — pure extraction, gives up access to plan/snapshot/canonical project. → refactor: split create() into preparePlan + execute #14289 (merged)

Lot 2 — migration, consumer by consumer (increasing risk)

• fix(executor): a wait node holds no --parallel slot while it polls — prerequisite of the detached-up switchover, where wait nodes are first scheduled by a production path. → fix(executor): a wait node holds no --parallel slot while it polls #14303
• feat: detached up runs on a single plan — the semantic switchover: the Start phase now emits starts for exited/created containers (the role of isNotRunning today); the second snapshot disappears. Best e2e coverage in the repo backs this path. → feat: detached up runs on a single plan #14290
• feat: scale and watch rebuild use the unified plan — reproduce current behavior (StartOptions.Services is dead today; wiring it is a separate decision). → feat: scale and watch rebuild use the unified plan #14291
• feat: compose start builds a start-only plan — including the label-reconstructed project path (projectFromName); run's dependency startup migrates for free. → feat: compose start builds a start-only plan #14296
• feat: interactive up on the plan engine — the riskiest step, kept surgical: Create phase → attach/printer/monitor → Start phase under WithoutCancel. No opportunistic refactoring. → feat(up): interactive up runs on a single plan #14305
  • test(e2e): lock the behavior of the foreground up — characterization scenarios, green on the current code, so the switchover is reviewed against a fixed specification. → test(e2e): lock the behavior of the foreground up #14300
  • refactor(executor): let a caller drive the create and start phases itself — begin/planRun, inert: a caller can act between the two phases. → refactor(executor): let a caller drive the create and start phases itself #14302
  • fix(monitor): subscribe to events since before the initial listing — Since taken before the snapshot, so a container exiting as soon as it starts is not missed; replay is idempotent. → fix(monitor): subscribe to events since before the initial listing #14304

Lot 3 — demolition

• chore: remove the imperative start path — delete startService/startServiceContainer, the InDependencyOrder start path; move waitDependency helpers to a dedicated file (remaining clients: restart, run, --wait). Separate from the interactive-up switchover so its revert stays trivial.

Critical path: reconciler → executor → detached up → interactive up. Up to and including the detached-up switchover, abandoning the effort still leaves the repo strictly better off (bugs fixed, start path unit-locked, vocabulary tested but inert).

Design decisions where maintainer input is requested before Lot 1

1. Waits as plan nodes (OpWaitCondition, deduplicated, golden-testable) vs conditional edges — this epic proposes nodes; edges cannot emit the Waiting→Healthy events users see and would evaluate conditions once per dependent.
2. startMx (fix: hold startMx on the start path actually exercised #14106): the global mutex serializing ContainerStart (engine port-range race) is currently only held on a dead code path. Take it on the real path, or drop it entirely? The engine-side fix is Avoid selecting duplicate host ports for mappings to 0.0.0.0 and specific addresses moby/moby#50054 (Engine 28.3.0, explicitly fixes [BUG] docker engine and desktop expose random ports differently #12846, the issue startMx was working around via run ContainerStart sequentially #12851) — and the moby networking maintainer assessed the original problem was not a start race, so the mutex may never have protected anything. This also gates future replica parallelism.
3. Phase boundary mechanism: single bi-phase plan executed in two steps (proposed) vs two separate plans over the same snapshot.
4. scale start scope: today scale db=3 also restarts any stopped container of the project (StartOptions.Services is never read). Reproduce first; changing it would be a separate PR.

Verification

Every PR keeps make test and the e2e suites green. The ~48 observable behaviors inventoried from the imperative engine (silent start idempotence, one-offs untouched, leaf/root ordering, event sequences, integer-second timeouts, --wait honoring service_completed_successfully, …) serve as the non-regression checklist for lot 2; #14104 locks the unit-testable part. Event parity is checked against the e2e checks.go vocabulary, which greps actual output.

Supersedes the first exploration in #14082 and #14083 (closed): this design keeps their wait-as-node and soft-fail ideas, but replaces the startWithPlan/listener-based split with explicit plan phases, folds injection into the start operation, and re-sequences the work so every step is independently mergeable on current main.