Repository navigation
feat: let a preloaded deployment skip its universe load, and deploy_env load the universe - #183
mandykwok-scale wants to merge 7 commits into
Conversation
a0ec06a to
0d4ec73
Compare
…ts universe
A provider can hand over an env whose universe it loaded before anyone asked: a warm
pool, say. The load that follows re-ingested the whole universe anyway, on two counts.
The resume check reads per-service progress off the instance record, and a provider's
record is registered fresh, so it finds none. And resume never skips a complete load,
because it can't tell a caller's explicit load from a retry.
`DeployedEnv.preloaded_universe` lets the provider say so instead: `{"id", "version"}` of
the universe the env was delivered with. `MultiEnv.load_environment_universe_artifact`
skips the data load when it names the universe and version being loaded and the
changelog shows the data untouched since. That's the same gate resume uses, so data
touched since voids the guarantee. Container mode, which has no changelog to read, loads
as usual.
A skipped load still stamps the universe on the instance and still stages the metadata
files, since later steps read paths that are fresh per load. It consults no snapshot and
bakes none, because nothing was ingested.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he data Review found the hole. A load reinstalls each service's changelog triggers, which empties the changelog. So a deployment delivered with universe A that then loaded B still named A, passed the changelog gate on the next request for A, and skipped a load the data needed. The guarantee is now dropped the moment a load is about to replace the data: from the deployment in memory and from the stored record, via `EnvInstanceStore.clear_preloaded_universe`, so a reattach can't revive it. A skipped load keeps it, since nothing was replaced. `preloaded_universe` also joins the store's round-trip test, which exists to catch a new field that doesn't survive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skip is now opt-in. `MultiEnv.load_environment_universe_artifact` takes `skip_if_preloaded=False`, and `load_artifact` carries the same flag, so every existing load loads exactly as before, preloaded mark or not. That keeps agent-env's rule that an explicit load is never silently a no-op; only a caller that asks may skip. A pooled DAG keeps `load_artifact` and sets the flag: a hit skips, a miss loads. `LoadEnvironmentUniverseArtifactResult.delivered_preloaded` and the recorded `loaded_environment_universes` entry say when a skip happened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0d4ec73 to
fd51ba1
Compare
kaihackney-scale
left a comment
There was a problem hiding this comment.
Looks good overall. The opt-in flag keeps existing flows unchanged, and Greptile's open P1 ("Wrong universe stays loaded") is addressed by b8ab33d2 (_void_preloaded_universe, persisted to the record), with test_loading_another_universe_voids_the_guarantee and test_the_void_reaches_the_stored_record covering it, so that thread can be resolved. One remaining gap, the single-service load path, is inline.
Two cross-PR notes:
- The title says "…and deploy_env load the universe", but no
deploy_envchange is in the diff. The title becomes the squash subject, so worth fixing. - Nothing sets
skip_if_preloaded=Trueyet: scaleapi#166525'sdeploy_composedoesn't pass it, so after both merge a pool hit still reloads. Flagged on that PR too.
…mises it deploy_env with an artifact set is a contract for a provider like the warm pool's: the caller gets an env holding that universe whether the pool had one ready or the env was built cold. A provider opts in with `delivers_universe`. For one that does, deploy_env loads its universe once the deploy returns, skipped when the env arrived with that universe untouched (`skip_if_preloaded`), and records the load in `loaded_environment_universes` as load_artifact does. Its task needs no load_artifact step. Every built-in provider leaves the flag off, so nothing else changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment only; the syntax tree is identical. The flag means: a provider that sets it promises the deploy step's universe is loaded once deploy_env finishes, hit or miss, so deploy_env loads it unless the provider delivered it and the task needs no load_artifact. No built-in provider sets it. deploy_env's _load_delivered_universe docstring keeps that explanation in code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review. Reloading one service through load_environment_artifact reinstalls its changelog triggers, which empties that service's part of the changelog, but only the universe path voided `preloaded_universe`. A pooled env delivered with universe A whose gmail was edited and then reloaded from another artifact would pass the empty-changelog check, and a later load of A would wrongly skip. load_environment_artifact now voids the guarantee first, as the universe path does. Also moves `preloaded_universe` above the "set by the registry and the deploy step, never by a provider" group, since a provider is what sets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! The design changed: a provider that sets |
From review: nothing sets it. A pooled deploy's universe is loaded by deploy_env itself, and its task carries no load_artifact step, so the option on load_artifact had no caller. The skip itself stays on MultiEnv.load_environment_universe_artifact, which deploy_env uses, as does recording `delivered_preloaded` in loaded_environment_universes. A hand-written pooled task that keeps load_artifact now reloads the universe: slower, still correct. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Closing: replaced by loading the universe in the pooled provider itself (scaleapi/scaleapi#166525, d1f3511e359). On a miss the provider builds cold, then loads the universe through an env opened on the record with |
Why
deploy_envwith anartifact_iddoesn’t load that universe: a cold build comes back empty, so a task needs a separateload_artifact. With a warm pool, that means a hit loads the same ~6 minutes of data again, and a miss depends onload_artifactbeing there. This PR lets a provider promise a loaded env (delivers_universe):deploy_envthen hands back an env holding the universe whether the pool had one ready or the env was built cold.What
EnvironmentProvider.delivers_universe(off by default). For a provider that sets it, such as the warm pool'spooledprovider,deploy_envloads its universe after deploying and records the load asload_artifactdoes. Its task needs noload_artifact.DeployedEnv.preloaded_universe: a provider sets it on an env it hands over already holding a universe. The load above is skipped when that names the universe and version asked for and the env's data hasn't changed since (empty changelog, VM mode only).preloaded_universe, so it can't cause a wrong skip later.No built-in provider sets the flag, so existing flows are unchanged.
Testing
Unit tests cover the deploy-time load for a provider that sets the flag (and none for one that doesn't, or with no universe), the skip and each case that must load instead, and clearing
preloaded_universe. Only those test files were run, not the full suite.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new actionable defect was found.Summary
Providers can opt into loading a universe during
deploy_env. A matching, untouched preloaded universe skips the data load while still recording the result and staging metadata.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Deploy environment] --> B{Provider promises a loaded universe?} B -->|No| C[Return deployment] B -->|Yes| D{Requested universe already loaded and untouched?} D -->|Yes| E[Skip data load] D -->|No| F[Clear preloaded guarantee] F --> G[Restore snapshot or load services] E --> H[Record universe and stage metadata] G --> H H --> CReviews (9) · Last reviewed commit: "refactor(load_artifact): drop the unused..." · Reviewed by Greptile