Release 0.23.0 - #68
Open
phy0x1a79ed wants to merge 500 commits into
Open
phy0x1a79ed wants to merge 500 commits into
phy0x1a79ed wants to merge 500 commits into
Conversation
…date them The wire format the Groovy orchestrator emits existed nowhere but a gitignored 56MB agent home that no fixture ever resets. Metadata from both runs is now in tests/metasmith/fixtures/real_runs, and the home drops runs/ and task_cache/ whenever the cache-key or lineage-payload contract it was written under differs from this tree's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ohVkjgAjzYg4o7C9YtTK6
…corded it CollectResults reconstructed provenance from trace edges keyed on compile-time slot ids, in a third path-hash identity space, and guessed when a slot named more than one producer. It could not resolve a parent that was another step's output at all, which is why eight docker trace tests were skipped. Execution records the fact instead. `.command.metadata` already carries PROV -- the index behind each staged input file -- so promote reads each output's direct parents off it, stores them in the shard so a cache hit answers the same way, and both trace emitters put them on ProducedFile.parents. CollectResults becomes a topological merge of those rows. Three things had to be true first. A step's produce instance and the downstream step's require instance both carried the transform archetype's id, so `produces` and `consumes` were in different identity spaces; cache_decisions now stamps the slot id on both, which also makes a downstream key move when its producer does (CACHE_KEY_VERSION 3 -> 4). The stub lane wrote no .command.metadata, so the one lane the trace tests run in recorded nothing. And an uncacheable step promoted nothing and emitted no event, leaving a hole every consumer downstream fell into. The runtime doubles now call what they emulate: one payload builder for the index, FILES and PROV; LinPayload.mint_file_id as the only file identity; the codegen's spelling for publish paths; and the by-channel, not the plan's instance list, as the count of tasks a step runs. Verified: TestTraceLinearChain, TestTraceFanOutMerge and TestTraceMultiStepDiamond all pass under real Nextflow (14/14), with new per-sample assertions that fail when the old conflation is reintroduced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ohVkjgAjzYg4o7C9YtTK6
…ef50/interproscan) Ports the kofamscan + diamond-uniref50 + interproscan trio from the solved nano-bandicoot workflow into a shipped template. Metabid deliberately absent: its 744 GB GTDB reference is unreachable for a local run. Solves in 11 steps.
The driver now lives in templates/<name>/ next to the compiled spec. build_templates resolves authors from either the package root (the other ten) or the template folder.
…ng that is true stays `msm gui` printed werkzeug's red production-deployment warning and three more banner lines on top of metasmith's own `serving at [...]`, on every start. Neither Flask's banner nor werkzeug's is configurable, so `serve` drops to the layer under both: `make_server` + `serve_forever`, which is also what lets the address line report the port that was actually bound rather than the one that was asked for. Request logging is untouched -- it goes through the same werkzeug logger either way. Silencing it unconditionally would be papering over the advice, because there is one case where it applies: `--host 0.0.0.0`. So the bind address is checked, and a host outside the loopback range prints one warning saying the thing that is actually true of this API -- it stages and runs work on remote hosts and reads local files, with no authentication -- rather than a generic one about throughput. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…way it runs tools Staging a workflow and launching it were the only things the GUI could do with an agent; nothing prepared the host first. Tool images fetch lazily inside the first task that needs one, which fails on a compute node with no route to a registry -- `metasmith workflow materialise` existed for that, but only from a terminal. Conda environments were never created at all: the library ships one recipe per tool and a `--runtime mamba` user was expected to build them by hand. `SetupEnvironment` is both halves behind one verb, dispatching on the agent rather than on the person asking: an image store to fill for a container runtime, envs to build for mamba or native. The container half is the proven one reused as-is. The conda half is its sibling in `agents/conda.py` -- free functions taking a shell, skipping an env that already answers `mamba run -n <name>`, which is the wrapper every task is launched under, so a second press does nothing and `force` removes before rebuilding. What it cannot do it reports rather than guesses. An env whose recipe the library does not ship is named, because a package spec inferred from a container tag makes a plausible env that is not the one the transform was written against. A step whose env resource carries no `conda:` at all is named with the container it does have -- the same fact `_check_env_portability` raises on at launch, produced early and as a list. A host with neither mamba nor conda is answered once instead of failing every create. In the GUI it is a split button beside `stage and run`, with `force setup` behind its chevron -- the gesture AgentView's force-redeploy already teaches, now a shared `SplitButton`. The endpoint stages with `on_exist=update` first, because the manifest that says which images and envs are needed is written by staging, and writes no run record: this is preparation, not a run. The page had one job id and assumed it was the solve, so it now tracks which kind is running or a setup would paint the solve's stage bar. Verified end to end against a local mamba agent: an env built from the library's recipe, skipped on the second run, removed and rebuilt under force, with a container-only step reported beside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…es a reload showDeployBar now follows jobId alone, not "still running or failed" -- it is the record of the last attempt, not a spinner that vanishes when the wait is over. The last stage now actually turns green on success instead of getting stuck at "running". On mount, AgentView asks GET /jobs?agent=<name> for the most recent deploy job and re-attaches to it, so revisiting or reloading the page shows what a tab left open the whole time would have shown. Co-Authored-By: Opal <noreply@anthropic.com>
…swallowing it NonBlockingReader only flushed its buffer on `\n`, so a pull's `\r`-redrawn progress bar sat invisible in the pty buffer (up to the 1MB truncation cap) until the command finally printed a real newline. It now flushes on `\r` too, via a universal-newline split that keeps each boundary on the segment it ends -- lone `\r` marks a redraw, `\n`/`\r\n` commits a line. LiveShell's tee carries that distinction forward as a trailing `\r` on the message text, since every listener downstream is `Callable[[str], None]`. Job.emit collapses a still-mid-redraw last line in place rather than letting every tick of a pull pile up as its own log entry (and evict real setup output once the 5000-line cap is hit); JobLog.svelte applies the identical rule client-side so backlog replay and a live stream never disagree on what they show. Two unit tests were asserting exact call signatures from before `on_phase` was added to Agent.Deploy/op_agent.deploy and had gone stale; updated both. Co-Authored-By: Opal <noreply@anthropic.com>
…tead of swallowing it" This reverts commit aa1509a.
…ct dir ftp.uniprot.org 404s on every path now; served from the EBI mirror instead, same as fabfos/build_references/transforms/acquire/uniref.py already does. Dropped uniref50's memory ask to 14GB. InterProScan's mkdir created the output slot's container path while tar extracted into a literal `ipr_data` that was never created, so every run failed at the extract step after a successful multi-GB download. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A given input with zero out-degree (no step consumes it, and it isn't itself a target) added clutter with no information in it. Prune those nodes -- and the synthetic "given" root too, if nothing is left under it -- after the full graph is built, so a real "requires this" edge always wins over the prune. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…used A run's trace/log resolve through the agent's `logs.latest` symlink until `run_number` is assigned, and that symlink still points at the PREVIOUS run's directory until its launcher script relinks it -- so a restart briefly painted the old run's failure onto the new one's bar. Ignore `traceFailed` (and the "N tasks failed" banner) while `rec.state` is `staging`/`launching`; it self-corrects once `running` lands anyway. Separately, the workflow tab's agent picker now starts on the agent last used anywhere (`metasmith.lastAgent` in localStorage) instead of blank, and seeds that agent's params on load rather than only on the select's own `onchange`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Typed into the launch panel and previously held only in page state, so a refresh silently dropped them. Persisted to localStorage per workflow name (`metasmith.overrides.<name>`), loaded on mount and whenever the workflow switches, saved on every edit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…effect that settles The in-flow `.res-head` pixel-chased onto the diagram's first node was always the redundant copy: a plan tall enough to scroll past that point had already carried the sticky bar up to answer the same question. It goes, and with it `HEAD_H` and the `topCy` measurement it needed. The one remaining header, the guide lines and the rows now read a single `--res-cols` template, so they cannot drift apart. The row is right-justified rather than centred, so the fixed-width columns anchor the right edge and only the diagram pans. `seedFromAgent` reads and writes `runParams`/`seededParams`, writing a fresh object each time, so the `$effect` that called it tracked its own writes and re-fired forever. `untrack` keys the effect to `chosenAgent`/`agentChoice`, which is the actual condition for re-seeding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CitDf2CY1HNbefFwufrkrh
…s, params-executor precedence, docs and tests
Ports uncommitted work from the engine/dev worktree, which was on the
same commit but ahead in working-tree state:
- downloadKofamDB.py: extracts the profiles archive instead of moving it
raw into the slot, fixing kofamscan's "not a directory" guard.
- downloadInterProScanDB.py: runs setup.py's indexing in a subshell so
its exit-trap marker doesn't try to write to a directory the task's
uid can't write to.
- kofamscan.py: updated guard message to describe both producers.
- workflow_ops.py: appends an explicit `executor {}` override from
`params.executor` after the free -b/nproc auto-detect, since the
preset's own `executor { cpus = params.executor.cpus }` block is
evaluated before -params-file is merged and silently never applies.
- dev/metasmith.sh, dev/fabfos.sh, docs, RELEASE_PROTOCOL.md, tests:
carried-over fixes and doc updates from the dev-scope session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ftp.uniprot.org has gone down whole-host before (every path 404s, not just this one) with no warning. Pinning a single replacement mirror just resets the same fragility clock -- fall through EBI, uniprot.org, then Expasy instead of failing the step the moment one host is down. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both /log and /trace resolved through logs.latest whenever run_number was still null (staging/staged/launching) -- a symlink shared across every run of a task_key, so a brand-new run briefly showed the PREVIOUS run's log lines and per-step trace rows, painting steps as already done/failed before any real work happened. api.py: short-circuit both endpoints with an empty/pending result the instant run_number is None, instead of ever resolving through the shared symlink. RunView.svelte: key the log-panel display and the progress bar's failure check off `rec.run_number == null` rather than enumerating state names, so this can't reopen if another pre-run state gets added later. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…en resources column
Resource overrides were keyed by step.order and persisted to
localStorage per workflow name. Regenerating a workflow (e.g. pointing
a given at a different source) can drop or add upstream steps and
renumber everything after them, so a stale numeric key silently
reattached to whichever step now sat at that position instead of the
one it was meant for -- the step that actually needed the override got
none. Key by transform name instead (a stable identifier the backend
already matched via its string-keyed `.*__{tr}` branch), and drop any
override naming no step in the current plan before sending it.
Also: "setup environment" hidden behind `{#if false}` pending real
testing -- untested feature, do not ship to release. Resources column
widened ~5% so memory-column chips don't squish, and its header
centered over the grid it names rather than sitting at its left edge.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Staging re-sent every registered library in full on every click -- 3 of 54 transforms, all 85 env resources -- because PrepTransfer queues self.location and nothing narrows it. engine/gui asked for a prune. Two mechanisms were already here for it and neither reached the staging path. PruneTypes was broken in two ways that cancelled out. It kept a namespace only when a *manifest entry* named one of its types, and every entry of a transform library is `transforms::transform`, so it deleted all but that one; the whitelist built from the transforms' own requires/produces could not rescue a namespace, being consulted only inside the branch for namespaces that had already survived. And _persist never unlinked while Load re-reads the whole types/ directory, so the deletion never reached disk and the over-deletion was invisible. Fixing either alone breaks the library. responseSurface needs four namespaces and shipped all 28: 188 KB of type YAML for 28 KB of types. So survival is a union now, the whitelist matches on properties rather than on a node hash a cloned Dependency does not share, index `parents:` entries count as used, and what the directory holds is what survives -- including files left by an older build with a wider --types, which are namespaces nobody declared. The views were the other half. A mask meant "the subset that matters" to the solver and nothing else, because __getattr__ forwarded PruneTypes and PrepTransfer straight to _original. Both are real on the view now. PrepTransfer materializes an image and queues one transfer of it, so the ssh executor still opens one rsync per library rather than one per file, and WorkflowTask.SaveAs derives the mask from the plan: the steps' transforms, plan.given -- already narrowed to the endpoints the plan consumes -- and the ancestor closure of both, which crosses libraries because a parent entry names the library it lives in. The image is the tree MINUS the unused entries, not a selection of the used ones, and that is load-bearing twice. The manifest is never narrowed because the library key is a hash of it and that key names the staged directory, prefixes every packed DataInstance and appears in every step's transform reference. And `build` keeps `_`-prefixed files out of the manifest, so _extract_noncoding_chunks.py and its two siblings -- which promotech.py and bakta_noncoding.py copy out by __file__ -- are runtime dependencies a selection would drop with nothing to say so. On the annotation trio: 370 files / 824 KiB -> 41 / 104. Same task key, same plan.Pack(), and a workflow.nf identical byte for byte modulo the bundle path. Two things this turned up on the way. DataTypeLibrary.Save was a truncating open(..., "w"), which writes THROUGH a hardlink -- and an image links what it images, so pruning the copy rewrote the original. It goes through write_yaml_atomic now, like the index, and the image copies its own metadata rather than linking it. And narrowing the types exposed Spec._get_endpoint resolving a target by finding the first library holding its *namespace* and then asserting the type was in it. That only ever worked because every library carried every type; seven of eleven templates failed until it resolved by the type instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Killing a run left tools running on the agent. Every kill targeted a single
pid, nothing escalated past SIGTERM, an ssh disconnect half-killed a run, and
nothing could say what survived.
A run now has one identity and two groups. start.sh backgrounds the driver
under `set -m` and exports METASMITH_RUN=<key>.<timestamp>, recording both as
RUN.pgid and RUN.token beside PID.lock; the driver puts nextflow in a second
group so a cancel stops the workload and leaves the driver to snapshot logs
and promote the cache. Docker tool containers carry the token as the msm.run
label, apptainer puts it back after --cleanenv, and it survives a setsid, so
the environ scan catches anything that leaves the group.
The relay job protocol now records who asked (.owner) and for which run
(.run), before the .start rename. Its .pid holds a process group -- the
launcher runs each job under `set -m` -- and every stop path signals the
group, TERM then KILL. A job whose requester died is reclaimed by the watcher,
which is what repairs the ownership inversion the bounce creates. The watcher
setsids out of the login shell's group, so a disconnect no longer stops it,
and a shutdown that cannot kill a job keeps that job's records.
CancelWorkflow is a ladder that returns {stopped, survived, rung}; the GUI
writes `cancelled` only when nothing survived, `cancelling` otherwise.
`metasmith workflow ps|reap` are the recovery surface.
Tests: a new `lifecycle` axis (dev/metasmith.sh -tl, 13 tests) covering group
kills, relay reclaim, kill-run scoping, watcher survival across SIGHUP and the
environ-scan backstop; one fast file pinning the generated scripts; two docker
e2e tests for cancel and reap end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e the files are
Leaf identity was a blake3 digest of the file's bytes, taken on whichever
machine registered the row. For an input living on the agent's host the
client cannot read it, so every registration minted a fresh random id and
every re-submission of the same inputs re-keyed the whole cache.
An id is now multihash("stat" || abspath || mtime_ns), and StageWorkflow
re-derives every one of them before compiling -- on the agent, where the
files are, a local stat rather than anything crossing the wire. The plan
goes back to task.yml because CollectResults joins the trace against it,
and the task key is recorded there too: it names the staged directory, and
WorkflowPlan._update_hash folds the given ids in.
The trade is deliberate. Two hosts holding identical bytes at different
paths no longer agree, and an in-place edit that restores mtime is
invisible. The tests that pinned cross-location collapse now pin the same
property against one shared library, which is what a re-run actually is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lishes only its targets Each of kofamscan, diamond_uniref50 and interproscan now produces two files, both carried through the chunk/merge pair the way predictf already does it. UniRef50's hit table gains a header and loses `stitle`, which moves to sseqid -> description; the other two never carried a description at all and gain one from data they were already parsing and discarding -- interproscan's `signature_desc` attribute, and the definition column of the ko_list that transform already requires. The consumers of the uniref50 table follow: the GPR lanes and fabfos_evidence read the header and join the description table, and each now refuses a header it does not recognise rather than emptying the lane. `_MB, _SB, _CB` in metagenomics_from_paired_reads move with the targets inserted above them -- they are positions in TARGETS, and stale they pinned checkm/gtdbtk to the wrong nodes. `publish_intermediates` defaults to False. The filter already existed in nextflow_codegen and nothing outside one test ever set it, so a results folder carried every per-step output. CollectResults still registers an unpublished file, since that manifest entry is what lets a target name it as an ancestor. The run view's file tree moves into the results card; the side panel becomes the selected result -- its full ancestry above the preview, read off the parent graph the collected library re-expands on load. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWL4GMBLduxY85TUM5t3V5
# Conflicts: # src/metasmith/models/workflow/task.py
# Conflicts: # src/metasmith/agents/runner.py
`_write_input` already skips rewriting an input whose content matches, because a leaf id is the file's path and mtime and several tests here model a re-run by calling a builder twice against one tmp_path. `_build_transform_lib` never got that treatment: it rewrote every transform .py, transforms.yml and index.yml unconditionally, and reached for `shutil.copy` -- not copy2, so mtime is not preserved -- for the types file. This extends the same rule to those five writes. It fixes no currently-failing test, and it is not a cache fix. Measured: bumping the mtime of all nine files of a built transform library leaves the hit set identical to baseline, as does the samples library's own _metadata and the shared types.yml. Transform-library mtimes do not reach a cache key. The change is here because a fixture that rewrites identical bytes to model "nothing changed" is saying something it does not mean, in a suite where mtime is load-bearing elsewhere -- not because anything currently depends on it. What does reach a cache key is a leaf inside a GIVEN data library, and there one file is enough: bump one and the hit set empties, including steps that never read it, since the re-keyed leaf moves its consumer's key and so on down the chain. That is `stat_multihash_key(abs_path, st_mtime_ns)` with no content fallback, and it is a property of the identity scheme rather than of anything in this file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWL4GMBLduxY85TUM5t3V5
… from it The engine declared `metasmith_libraries` as a conda run dependency and nothing had ever published that package, so `-bd`, `-bc`, `-bs` and `-uc` all died on an unresolvable spec -- the Dockerfile's `mamba env create` and `conda mambabuild` read the same base.yml. Removing that one line is what unblocks them. It goes back to being vendored, which a52d497 deleted, but not to what a52d497 deleted. That reversal rested on two things, and the change that replaced vendoring is what dissolves both. `clone_stdlib` now copies the library into the project and compiles it there, so nothing shipped needs compiled metadata -- which removes the compile-before-vendor ordering hazard entirely rather than guarding it. `--no-metadata` makes that explicit on the shared verb: fabfos still ships a compiled bundle and is still refused a hollow one, metasmith ships content and compiles its own copy. The other reason was that `templates/` was never in the `--src` list, so a released GUI offered none; it is in it now, and `_assert_library_bundle` counts them. The bundle lands at `src/metasmith/vendor/` specifically because `_build_hash.py` walks that tree. Library content therefore moves the engine's build hash, the wheel, the image tag and the SIF together -- measured, not assumed: editing a transform took the hash d7898e5 -> 99cdd1f and restoring it took it back. A library and the engine that reads it can no longer disagree about which is which, which is the property the monorepo exists to provide and the reason the library stops being separately versioned and published at all. There were two resolvers, not one. `agents/conda.py` finds the `envs/` recipes a `--runtime mamba` deployment creates its tool envs from, and it had its own copy of the find_spec lookup; fixing only `gui/stdlib.py` would have left mamba resolving nothing. Both call `standard_library_root()` now: vendored bundle first, an importable `metasmith_libraries` second, which is what a source checkout has and an installed package never needs. Also deletes `src/metasmith/std/`. It was generated by a research script, shipped in the wheel, the conda package, the image and the SIF, and read by nothing in `src/` -- its only consumer reads the `research/` copy. Shipping it beside the vendored library would have put two transform libraries in one wheel. Verified with `metasmith_libraries` not importable at all: the bundle resolves, clones, compiles 13 `_metadata` directories, and yields 20 data types, 11 transform libraries and all 11 templates, stamped with the engine's own version. That case -- a wheel, a `pip install`, a mamba or native runtime -- previously got no library by any mechanism, because conda's solve was the only thing that could ever have delivered one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lished The staging guard proves the bundle was in the tree; it cannot prove it reached the artifact. This is the same three-place shape the solver engine already has, for the same reason and with the same failure mode: an image whose bundle is absent or empty installs, imports and runs, and the only symptom is a blank type panel and no templates -- which nobody sees until a user opens the page. Runs at -bs and -ud, beside _assert_engine_in_image, and asks the metasmith inside the image to resolve its own library and count what it found. MSM_SKIP_LIBRARY_CHECK is the shared override, the same way the two solver checks share MSM_SKIP_SOLVER_CHECK -- they are one claim checked in more than one place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and docs that no longer vendor
The executor is sized by appending a literal `executor { }` block to the config
file rather than by `params`: the preset's block is evaluated at parse time,
before the `-params-file` merge, so every caller was heard at runtime and
ignored at submit.
downloadKofamDB unpacks profiles.tar.gz with --strip-components=1 and fails on
an empty profile set instead of moving the archive into the slot;
downloadInterProScanDB runs setup.py in a subshell so the exit-code trap still
fires in a writable cwd; kofamscan's staging error names both producers.
dev/metasmith.sh --idev/--ibase reach envs/metasmith. AGENTS.md drops
--vendor-library, the consolidation followup re-points the unrecoverable-pin
entry at data/fabfos/runs/aska/gpr, and RELEASE_PROTOCOL.md carries a measured
budget for the full sweep.
Tests follow the sandbox short-circuit and the Deploy on_phase signature; the
flow fan-out budget is split from the batching assertion and xfailed; the
setup-environment tests shut off the installed-library recipe root so a staged
envs/tools cannot change the answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scope's own commit (8b6a796) was already contained in release verbatim, so the merge resolves entirely to release's side: the merged tree is byte-identical to release. The one conflict, in workflow_ops.py, was release adding the local executor's free -b/nproc auto-detect ahead of the params-driven executor block both sides already carried.
The last untested code in the MAG lane has run. On 84,992 contigs of a 2019 groundwater assembly and ten million read pairs, the three binners give 52, 143 and 87 bins, the refinement scores all seven combinations plus the merged set, and 34 bins survive at 50% completion and 10% contamination. The best is 95.79% complete at 1.09% contamination. That exercises the three things the twelve-contig test never reached: the fan-out wrote 34 bin products, the contig-to-bin table has 6,612 rows across 34 distinct bins, and the statistics table carries one row per bin with the binner each came from. MetaWRAP confirms the memory fix in its own words -- "There is 40 RAM and 12 threads available ... use 1 threads for pplacer" -- where before it would have said zero. The host database fetch needed --http1.1. Over HTTP/2 the portal drops the stream part way into a 10 GiB chunk with INTERNAL_ERROR, every time, and six array tasks each burnt 45 minutes on it. On 1.1 the transfer is still cut but resumes, so the retry loop accumulates rather than restarting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes, each paid for by a run that failed without it. `_stable_id` + `RegisterItem` pin the given instances' leaf ids. `AddItem` stats the file to mint one and returns a fresh uuid4 when it cannot, which is every input here -- the reads and the reference databases live on fir while this driver runs on a workstation. The plan key hashes those ids, so two identical submissions minutes apart keyed differently and `-resume` never once fired. Measured `daqL9cFU` then `WwhBaN3k` before, one key after. `ref::kofamscan_profiles` now points at the unpacked `profiles/` rather than the `profiles.tgz` beside it. The transform asserts the staged path is a directory, so every chunk failed instantly: 135 chunks, four tries, 540 FAILED rows reading as a dead tool. After the fix, 1,493 chunks and zero failures. comebin's allocation is 48 cores with a 3d wall, and `--comebin-device` selects the lane. Measured on ten marine samples: 25.0 s an epoch on a 20 GB MIG slice against 110.6 at 96 cores and 145 at 64, for a total of 2 h 20 m against 6 h 28 m to 10 h 07 m. Under the old 8 h wall the CPU lane reached 191, 195 and 28 epochs of 200 across three attempts and never finished. The comments carry the numbers and the one defect this does not fix: the Leiden sweep forks a pool from a loky-threaded parent and hung on 2 of 10 samples at 96 cores, each burning the full wall. Also fixes verify.sbatch resolving its helper through `$(dirname "$0")`, which SLURM breaks by copying the batch script to a per-job spool dir. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QZhY5iQVRrEHjbnsSWFzch
Brings `_stable_id` + `RegisterItem`, the kofamscan profiles path, and comebin's device flags into the campaign branch. The two branches were separate lineages, one commit against 27, which is why the engine lane's merge survey missed this side. `verify.sbatch` resolved to ours. Both sides fix the same `$(dirname "$0")` defect that Slurm's spool copy creates; ours keeps the `VERIFY_PY` override in front of it, so it is a superset rather than a competing fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZvtnGpGTeQ6JA54Meycr5
An untracked file in the tree blocks the aggregation merge, and this one is worth keeping rather than ignoring. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The one commit release holds that feat/dev lacks. Cutting 0.23 without it regresses the image back to building from a source distribution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the caching store the rest of this aggregation builds on. An entry says what made it, when it arrived and what it was called; admission is the single write door; the pool projects into a data instance library. A direct run binds items from a library and refuses a filesystem path, which is the rule givens are about to adopt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three branches grew the direct-run model in parallel. The binding model is direct-run's: a transform is located by the path of its .py, inputs name items in a data instance library, and a filesystem path is refused. The viromics lane brought the pieces that made collecting transforms runnable -- a slot-channel map, a provenance union so SourceOf answers, and caller-stated cpus/memory/ attempt -- and those are re-fitted onto the binding API rather than merged textually. Two things from the viromics side are deliberately not taken. `host` was declared and immediately discarded, so it is noise on the signature until relay bounce exists. The agent stub is dropped: a transform runs on an agent's runtime, and inventing one from the ambient environment hides a missing deploy behind a run that does the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ispatch KBase collapsed ExecWithEnv's container and virtual-env arms into one call and added a scan that refuses the old names, because a body that still splits its command per runtime is an unmigrated body. Nineteen transforms in the viromics and metagenomics lanes were written against the old shape while that landed. Every one of them passed the same command to both arms, so the collapse is mechanical and loses nothing; gtdbtk's container-only binds carry over, because the entry point drops mounts where there is no mount namespace to put them in. Three content disagreements resolved by intent rather than by text. The lineage stamp keeps KBase's comment, which says why an entry needs a KEY at all, and the viromics provenance union and slot-channel map are additive on top. The dereplicated MAG reference takes KBase's note: round 6 gave it a producer, so the viromics "built once outside the DAG" reading is out of date. The metabuli database keeps the viromics version, which pins a release, flattens a nesting that moves between releases, and asserts the extracted database rather than the command's exit code -- a campaign lost a day to the version that did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ransform PruneTypes kept a type only when its property set equalled a declared type's, which dropped every ancestor -- so a supertype target resolved to nothing unless some unrelated transform happened to declare that ancestor by hand. Subset is the right test, because a type is satisfied by any type holding at least its properties. The two SRA transforms fuse into one. Both were modified here only by the dispatch migration, so the deletion is taken as it stands and the replacement is migrated the same way. Its memory declaration drops from 64 GB to 8 GB, which is the honest number for a download bounded by disk and network -- worth knowing on a box whose executor ceiling refuses a step that asks for more than it has. The aggregator reached the same per-binner gtdbtk declarations on both lanes. One comment, keeping the failure this branch observed and the precedent the other names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gain Cherry-picked from the bake lane, where it blocked a reference build. The cache pass rewrites each instance's id into its cache slot id after the plan the agent reloads was serialised, so a step whose input came from another step in the same plan looks its inputs up in the wrong id space and raises. The produces path had always fallen back to the step's own dependency map for the mirror-image mismatch; the requires path now does the same, one line away. The conflict was an add-add at one seam: this branch grew a cache-promote helper where the fix adds its resolver. Both are kept. The five tests that came with it pass against the merged tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`spades.py` was a hand-rolled metaSPAdes call. It now runs the DOE JGI Metagenome Workflow's assembly step, so a pipeline built on it is citable to a published protocol rather than defended: bbcms at mincount=2 and highcountfraction=0.6, then `--only-assembler -k 33,55,77,99,127`, then a 200 bp floor over the contigs product. Each step carries the protocol's own sentence verbatim in the comment above it. One spades, not two. A sibling transform with its own product types was written first and thrown away: it avoided disturbing anything, but a third `assembly` subtype buys nothing the library mask does not already have to do, and every extra matched property set is another chance to re-route something. Three invariants survive unchanged, and the comments explaining them with them. `-m` is a hard setrlimit at 95%, not a hint. `OMP_NUM_THREADS` is re-exported inside each command, now bbcms's too, because every runtime here pins it to 1 and SPAdes silently honours that over `-t`. All three products ship, with only the contigs length-filtered, because the graph's path names are scaffold names that a filtered contig set would no longer index against. TWO THINGS THIS BREAKS, both recorded in REFERENCES.md. The template fingerprint reports no difference. The transform's name and its product types are unchanged, so the tool sees nothing -- it catches a re-route, not a behaviour change inside a transform. Identical output from it is not evidence that the library is unchanged. Every shard `spades.py` wrote is retired, since a transform's identity hashes its whole source file. And every template naming `spades_assembly` now assembles under this protocol, `ecspr_survey_from_pooled_reads` included. That template was written against the old behaviour; reconciling it is deliberately out of scope. REFERENCES.md is new: one entry per tool with its version, the parameters it actually runs, its citation, and its deviations stated rather than hidden. The JGI paper is a workflow description, not a benchmark -- it names both metaSPAdes and MEGAHIT and compares neither -- so it supports "this is the DOE JGI reference workflow" and nothing stronger. `cache_probe.py` is the harness for a measurement not yet taken: whether a member key survives a change in plan shape, not merely a re-submission of the same plan. Two arms over one sample differing only at the binner. It waits on a pinned engine, because a number about a tree nobody ships is not worth the cluster time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZvtnGpGTeQ6JA54Meycr5
The assembly follows the DOE JGI Metagenome Workflow rather than a hand-rolled call: bbcms error correction first, then --only-assembler with fixed kmers, then a 200 bp contig floor over the contigs product alone. REFERENCES.md states each tool's version, the parameters it actually runs, and its deviations. Written against the two-arm dispatch, which this branch had already collapsed. All three calls pass the same command to both arms, so the same mechanical collapse applies and nothing about the commands changes. Landed before the campaign pin on purpose. A transform's identity hashes its whole source file, so this retires every shard spades wrote. Once is the cost of the change; landing it after the pin would charge it a second time, mid-campaign, where partial reuse reads as a broken cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The viromics lane gave every input the union of every input's index, so `SourceOf` could answer where a library recorded nothing. It also overwrote ancestry that WAS recorded: two metadata files and two read files, each read parented to its own metadata, came out with every read descending from both -- `AmbiguousProvenance` instead of an answer, and the direct-run lane's test for exactly that case failed on the merged tree. `given_index` already carries the one hop of parents the data library knows about, which is a statement about these files rather than an inference from their being in one run. So it wins whenever there is any, and the union is the fallback for a library that records no parents at all. The choice is made once for the whole entry, never per item. A root item has a one-key index by definition, so a per-item fallback hands the union to exactly the items whose ancestry matters most, and two roots each claiming to parent every descendant is the ambiguity the recorded ancestry existed to prevent. That version was written and it failed the same test. Both provenance models now hold: 16 of 16 direct-run tests pass under Docker, which is the check `research/viromics/implementation_handoff.md` records as impossible to run at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cherry-picked from the viromics lane. `RenderLauncher` gains `background=False` and an explicit workdir, so a caller that is itself already a durable host for the run -- a Slurm batch job's own script -- runs the driver in the foreground and lets the job's wall keep it alive. `METASMITH_DRIVER_SLURM=1` selects it. Verified rather than taken on report: the default branch is textually the code it replaced, and both renders come out at 24 lines with the right markers on each side. The foreground variant writes RUN.pgid before the run, not after, which matters because there is no backgrounding step to hand back a `$!` and the recovery path reads that file. The argument for it does not rest on a mechanism. Why one 29-hour JVM died is unknown -- the cgroup's oom_kill counter is zero and the memory that looked alarming was reclaimable page cache. What is true without a cause is that a long-lived driver on a login node has no supervision and no record, and a driver in a Slurm job has both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ridge AMBER is the scorer the CAMI challenge uses. Making it a transform rather than a script beside the run means the published ground truth is a declared typed input, the score is a declared product, and provenance, caching and the DAG cover it like anything else. A sample that cannot be scored now fails at solve time rather than after three days. THE BRIDGE, WHICH IS THE WHOLE REASON THIS IS NOT ONE TRANSFORM CAMI's published `binning_gs.tsv` keys on the CAMI-provided gold-standard assembly's own contig ids -- `S0C878739`. Our assemblies emit `k141_*` or `NODE_*`. The two share no id space, so pointing a scorer at the published gold standard joins on nothing and returns a number that is not wrong so much as meaningless. Verified against the real file on fir, not inferred. `cami_contig_truth.py` builds the gold standard AMBER actually needs: for each of OUR contigs, majority-vote the true genome of the reads the BAM mapped there, using CAMISIM's per-read truth table that sits beside the reads. The vote is in `resources/lib/cami_gold_standard.py`. `amber.py` requires the shared `contig_to_bin_table` supertype, so one transform matches all three binners. Its label comes from the BOUND instance's type rather than the requirement's, which is what keeps three binners writing to three directories instead of colliding in one. THREE THINGS REVIEW CAUGHT, ALL IN THE WIRING RATHER THAN THE TRANSFORMS The driver never gave `resources/lib`, only `resources/env`, so `lib::cami_gold_standard.py` had no producer and the chain was unsatisfiable. The planner does not report that. It explores the whole library and then blames every unrelated target, dead-ending at `ncbi::genome_name` and `sequences::background_genome` -- a driver that reads as comprehensively broken when one resource library is missing. The fix is one `Load`. Amber was targeted once, pinned to the assembly, with a comment claiming it would fan out per binner. It does not. A target is a slot the planner satisfies once, so that scored ONE binner: a solve that succeeds and silently answers a third of the question. Now pinned per binner's own table, exactly as `checkm_stats` already was in this same file, and the solve shows three. `amber_bin_metrics` was targeted alongside `amber_results`. Redundant -- amber emits both products in one step -- and a target costs a slot whether or not it buys anything. Bisected rather than guessed: transforms, types and the new given alone leave the plan byte-identical at 22 steps and key q4QR3rmL. The targets were the whole difference. With the resource library given and amber pinned per binner: 26 steps, key hTsuobCM, three amber steps beside three checkm. The two Pratama recovery transforms are designed and implemented but have never run against real data, because our own dereplicated MAG and vOTU sets do not exist yet. Their command shapes were checked against a real skani binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZvtnGpGTeQ6JA54Meycr5
…pe change
Added `--assembler` so the harness can drive the JGI-protocol spades route as
well as megahit. That was for calibration, but it answered the cache question
for free on the way past.
Two runs, different assemblers, different plan keys: `aNmwq814` (megahit) then
`9SsVv8oh` (spades). Both steps above the assembler hit the first run's shards.
seqkit_reads 1m 07s paid -> 36ms
bbduk 1m 57s paid -> 45.6s
So a member key does survive a change in plan SHAPE, not merely a
re-submission of the same plan, which is the property the campaign's reuse
measurement rests on. It cost no cluster time of its own.
Note the bbduk hit is 45.6s rather than milliseconds. A hit still copies the
shard's outputs into a task work dir, and bbduk's are gigabytes of cleaned
reads -- and metasmith is forcing publish-by-copy here because it reads one
Lustre directory bind-mounted twice as two mounts. Reuse is near-free in
compute and not free in I/O. Report it that way.
This is on the pre-pin tree. It re-runs against the pinned engine, where it
also serves as a check that the aggregation did not change plan shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ZvtnGpGTeQ6JA54Meycr5
Four transforms, their types and an environment, reviewed on the viromics lane before landing. The bridge exists because CAMI's published bin assignments key on CAMISIM's own gold-standard contig ids while our assemblies emit k141_* or NODE_*: the two share no id space, so a scorer pointed straight at the published standard joins on nothing and returns a number that looks like an answer. The truth transform majority-votes our contigs' reads against CAMISIM's read mapping instead. Lands after the campaign pin at 34401df on purpose. It adds transforms rather than changing existing ones, so it retires no shard the pin did not already. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All four came from driving the aggregated engine at campaign scale, and none is reachable from the test suite. Recorded now rather than at the end of the plan, because each carries a measurement that would not survive being remembered. The first is the one that reaches the caching architecture: a shard's manifest attests that a file exists, never that it is whole, so a product with unflushed holes was promoted and would have been served as a hit. The others are a missing version lock between a library and the agent image, a planner that reports an unsatisfiable requirement against every target except the one that caused it, and a publish path that copies gigabytes because it reads one bind-mounted directory as two filesystems. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool minted an import's id from the type and the name the caller spelled, so importing one path twice was one entry and two physically separate files handed one name collapsed into one. Neither is something the pool can know: a declaration is not a statement about bytes, and reading the bytes is the cost an import exists to avoid -- a reference is 17 GB and a folder is six hundred thousand files. So the question changes from "what is this data" to "which act put it here", which the pool observes rather than infers. `mint_import_id` digests the declaration with a nonce, keeping the multihash shape and a real preimage. Two acts are two entries whatever they are called, and re-importing a path is how a caller says this declaration is a new thing. What that costs, recorded beside the call: an assigned identity cannot be recomputed. The pool is authoritative state rather than a cache of a calculation, and every shard keyed on an import dies with the pool that holds it. Two things follow from the id no longer carrying the declaration. The manifest records the name, because nothing else does and a reference by name has to match on something. It also records the arrival time in nanoseconds, because the store's `created_at` is whole seconds and two imports of one path inside one second are ordinary -- the projection and the path index both need to say which came second, and both now let the newest claim win rather than the first. The displaced entry stays in the store and stays reachable by id, so shards keyed on it are still valid. The cache epoch does not move. An import's key was already outside it and still is: a product is re-derivable and a bump may strand it, but an import cannot be re-derived at all, so an id that moved with the epoch would strand what may be the user's only copy on every bump. fabfos's pinned references keep their derived identity, deliberately. A DVC pin is an md5 DVC computed over the real bytes, so the name there IS a content address and every host that checks out the pin agrees on it -- a stronger claim than assignment, not a weaker one. The engine's old function is renamed `pinned_import_id` with its payload frozen, which is why `test_refs_pin.py`'s literals did not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A campaign's pool lives in an agent home on a cluster, and the client that builds the plan has no mount there. It never needed one: a given is a name, a type and an assigned identity, and all three are records the pool keeps. Nothing has to be derived from bytes the client cannot open, which is what the identity change bought. `Agent.ReadPool` asks the agent's own metasmith what it holds and reads the answer. One ssh command, the same transport as every other remote read here, running the `msm` wrapper the deploy already writes -- so the pool is read by the engine that owns it and this end parses JSON rather than a database file. The agent's setup commands go first, because a cluster reaches its container runtime through a module load and a non-login ssh command inherits none of it. The wrapper prints its bind list and the runtime prints its own noise, so the answer is taken as the first well-formed JSON value rather than as the whole of stdout. `ResolvePoolRefs` turns names into identities. A name the pool has never seen is refused by naming the import call, because there is nothing this end could mint for it. A name two imports share is refused by listing both ids: re-importing under a name is exactly how a caller says the second is a different thing, so the pool holding both is expected and choosing between them is not this end's call. A home on this host skips the shell entirely and reads in process. `cache list` gains a `--name` filter and the rows carry the name, which is the only field that tells two imports of one path apart now that the key no longer carries the declaration. `metasmith agent pool` is the command that exercises all of it. The tests report paths that exist on no filesystem this process can reach. A read that starts passing because a path happened to resolve is the regression they are watching for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From the campaign: the overlay carried a 229-sample run against an image two API-breaking commits behind, and every user of it still hand-rolls the rsync and the tar that populate it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Agent.GivenLibrary` turns pool references into a data instance library whose items carry the pool's identities. The pool already projected into one and the planner already took that projection unchanged, so this is a reference-resolution step in front of something that fits rather than a new layer. Nothing reads the data: the paths are the agent's, the types come from the libraries the caller attaches, and the ancestry is the edges the pool recorded. The measurement this exists for, now a test: two builds from one reference plan to one key, and touching the file does not move it. The identity was assigned at import and is a record from then on, so the only thing that moves the key is the plan's shape. Before this, a given the client could not stat took a fresh uuid4 per call, and a live campaign measured `daqL9cFU` then `WwhBaN3k` from back-to-back dry runs of one driver -- which renamed the run directory and discarded `-resume`. The opposite direction is a test too: re-importing under a new name does move the key, because that is a caller saying this is a different thing. Cache rows now carry an entry's ancestry as identities, and its lineage payload where there is one, so a reader holding the rows can rebuild the parent edges without the store. An edge to an entry not among the references is dropped rather than written, because `Unpack` walks a parent by path and would walk into nothing. An import into a pool under a swept path now says so, once, at the act that first gives the pool something to lose. An assigned identity cannot be rebuilt, so a scratch filesystem deleting on age since creation has a delete scheduled against the meaning of every shard keyed on an import there. The campaign's own agent home is such a path. This warns and does not refuse: where an agent home lives is the operator's call, and the point is that the call is made knowingly rather than discovered sixty days later with nothing to diagnose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A given's identity has to be a record. Registering a path mints one from the filesystem this process can see: it moves when the file is touched, and it is invented outright when the path belongs to another host, which is every input of a run driven from a workstation. The plan key hashes the givens' ids and the run directory is named after the key, so an invented identity discarded -resume on every submission. `_mint_leaf_id` now marks what it invented, and `CollectSolverInputs` refuses what carries the mark, naming `metasmith data import` and the `GivenLibrary` call that replaces it. `Pack` does not carry the mark, so the question survives exactly one process: a library saved and loaded again is a record. The test is a mint rather than a pool, and that is the limit of this refusal. A shipped template, the env and container libraries and a DVC-pinned reference all carry recorded leaf ids, and none of them is data a pool should hold, so "imported or nothing" would refuse three categories that are right. What no longer passes is an identity nobody wrote down. An invalidate and a fork clear the mark after moving an id. Both act on a record the library already held rather than inventing one, and an invalidate has already refused every path it could not stat. An invalidate now skips an imported item and says to import it again, which is what moves an assigned identity, and a test pins that path. A deferred given is exempt. There is nothing at the path yet, so there is nothing to import, and `StageWorkflow.RefuseIfDeferred` is what catches one that never got a source. Direct-run's refusal names the import too, so the rule is stated one way. The suite's fixtures import what they build. `metasmith.testing.pool_backed` imports a hand-built library into a pool beside it and takes the identities back, referencing an entry the pool already holds under the same name, which is what lets a fixture be built twice and plan to one key. Three files about leaf identity itself keep their leaves by saving and loading instead. unit, flow, solver, cache, bootstrap, audit and gui: 2025 passed, 6 skipped, 5 xfailed, 0 failed. Also records the campaign's report that a driver placed on a compute node by METASMITH_DRIVER_SLURM finds no relay there and submits nothing, without failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`AddItem` did two jobs at once: it declared what a file was, and it minted an identity for that file from a filesystem the driver often could not see. The two are now separate. A driver says what it has and what to call it, the pool assigns the identity once, and every run after that cites the name. `Agent.PoolGivens()` is the collector a driver writes instead. `Add` declares something already on the agent's host and returns the name, so a name is what `parents` takes and an input-building function keeps its shape, the same variables threading the same edges. `Value` is the counterpart of `AddValue` for a small document the driver authors, minus its defect: that one wrote the file and then read back the mtime it had just created, so identical content was handed a new identity every call. `Build` imports whatever the pool lacks and cites all of it, and `ensure=False` cites only -- so a run that is not doing setup fails by name rather than importing as a side effect. Three transports underneath, each the write counterpart of the read path. `ImportToPool` runs the agent's own `data import`. `EnsurePoolEntries` reads the pool once and imports only the names it does not hold, which is what keeps a plan built today keying like the one built last week. `WriteImportable` puts an authored document on the agent, by base64 rather than a quoted heredoc, because the content is arbitrary and the command crosses a shell, a login profile and possibly a container. The CAMI campaign's driver loses `_stable_id` entirely. That workaround supplied an identity by hand because nothing else would hold one; the pool holds one now. `import` is its own subcommand, and `run --import-inputs` is the same act inline. fabfos's three pipelines move with it. `stage_ref` names a reference by its type, so one pipeline has one pool entry for a 17 GB database however many times it runs, and a stub gets a name that says it is a stub -- planning against a stub and then against the real thing must not read as one reference that moved. Copy staging and reference staging now reach the same entry rather than forking the campaign in two. fabfos: 64 passed. metasmith unit and the two pool axes: 686 passed, 3 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three groups, and which group a driver is in is a decision rather than a detail. **On the pool.** The CAMI campaign driver, fabfos's three shipped pipelines and four cluster drivers declare their givens and let the pool assign the identities. Every one of them registers a path this host cannot stat, which is the case the whole change exists for: `AddItem` answered that with a fresh uuid4 per call, so two identical submissions planned to different keys and -resume found nothing. **Written down instead.** Template authoring, the DAG renderers, the figure scripts and the KBase parity probe build synthetic placeholders that nothing will ever cite again. `record_library` writes such a library down and reads it back, which is what makes its ids a record rather than this process's invention. Its docstring says plainly that it is wrong for a campaign's real inputs, because it will not stop the ids moving the next time the declaration is rebuilt. **Left on the pin they already had.** The aspire r1 campaign and nine fabfos benchmark drivers each wrote their identities into a file of their own. Moving them to the pool would move every id and strand the shards those completed benchmarks earned, so they stay. The shared pin now covers every given rather than only the ones the client cannot see -- leaving the stattable ones alone left half of each task key still moving -- and its docstring names it as the pre-pool workaround and points new work at `Agent.PoolGivens`. Tests for what this touches: the fabfos driver axes, 13 passed; templates, the two pool axes and the remote pool read, 54 passed. The wider tiers were green on the previous commit and nothing here reaches them. Also records the wart this leaves: a driver that overrides identity through the pipelines' `on_inputs` hook still causes the import to happen, so the pool collects entries no plan ever cites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the Data tab could not do. **A pool on another host was refused rather than read.** The tab checked the agent's home type and told you to run the commands there. It no longer looks: every store route hands to the agent, which runs its own metasmith over one ssh command and answers in JSON. Listing, importing, tagging and forgetting all work against a cluster home now, and a home on this machine takes the same path in process. An unreachable host is reported as a failed read naming the pool and the host, which is a different complaint from "we will not look". **A plan's inputs could not cite the pool.** An input row gains a third mode beside file and value: `pool`, carrying whose pool and the name the import recorded. `sync` resolves it and registers the entry with the identity the import assigned, so the plan key does not move between one solve and the next and this machine never has to stat the file. The row does not carry a type. The import is what said what the data is, and a row disagreeing with it is a row about a different thing, so the pool's declaration wins and a row that states one is corrected. A pool row binds no sheet column either: a sheet expands a row over paths, and this one names an identity. The resolver is passed in rather than reached for. `ops/inputs` knows nothing about which agents a project has, so the caller supplies the reader, and a row with no reader is refused rather than quietly registered as a plain path -- which would mint a second identity for something the pool has already identified. One read per agent for the length of a sync: fifty samples out of one pool is one question. gui, the given refusal and the remote pool read: 417 passed. The frontend bundle builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four changes, two of them to paragraphs that were actively wrong. The caution about a staged given's id folding the task key described a mechanism 0.22.1 removed, and it cost a campaign a wrong diagnosis before anyone noticed. It is replaced by the rule as it stands, including where the rule stops: the test is whether this process invented the identity, not whether a pool holds it, because a shipped template's placeholders, the env and container libraries and a DVC-pinned reference all carry recorded leaf ids and none of them is data a pool should hold. The import paragraph named `structural_import_id`, which no longer exists. It now carries the reversal and the argument for it -- a derived id answered "what is this data" with what the caller spelled, and the question the pool can actually observe is "which act put this here" -- plus the two things the manifest has to record once the id stops carrying the declaration. The pool's lifetime is written where an operator meets it, beside the agent home. Reuse does not cross a campaign, shards die with the pool, and a two-batch comparison has to finish inside the retention window. The failure is silent and arrives sixty days after the placement that caused it. The declined epoch bump is recorded beside the constant, with the campaign's argument rather than an abstract one: every affected key moves on its own arithmetic, so a bump protects against nothing and would delete the measurement two pinned batches exist to produce. Say so before a future bump, not after. Two register entries change rather than close. Mtime sensitivity is closed for givens and still open for a transform's own outputs. The stale directory hit survives in a new form -- an import records a declaration and never re-reads the data, which is what makes citing a 17 GB reference free -- so the entry now asks for a change detector rather than for a re-read. The user-facing data page gains the rule it was missing: registering here does not make a given, and the import call is what does. flow and cache: 350 passed, 3 skipped, 4 xfailed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
phy0x1a79ed
added a commit
to phy0x1a79ed/Metasmith
that referenced
this pull request
Sep 12, 2026
The publish section claimed there is no standing PR to reuse, because each one closes on merge. That holds only for a PR a maintainer actually merged. hallamlab#68 was opened for 0.22.1, never merged, and was still open when 0.23.0 came to be published -- and GitHub refuses a second pull request for the same head and base, so the new commits simply moved the open one. The instruction now says to look before opening, and what to do with what is already there: retitle it, rewrite the body for the version now shipping, and say which earlier version it still carries. The false sentence is gone rather than annotated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.23.0.
This pull request originally carried 0.22.1, which was never merged upstream. The
fork's
releasebranch has moved on, so this one now delivers both.What changes
A given's identity is assigned by an import into the agent's data pool and recorded
there, rather than derived from the filesystem at the moment the given is registered.
That fixes a reproduction campaign which could not reuse any cached work: every
submission handed its inputs a new identity, and the plan key moved with them.
files are two imports and two identities, whatever they are called, and importing the
same file again is how a caller says a re-declaration is a different thing.
fixes it. The refusal tests whether this process invented the identity, not whether a
pool holds it, so shipped templates, the env and container libraries and DVC-pinned
references are unaffected.
relay, without the data being reachable from the client.
research/and the fabfos pipelines take their givens from the pool.CACHE_KEY_VERSIONstays at 6 deliberately. Every affected key moves on its ownarithmetic, so a bump protects against nothing and would delete a measurement that two
pinned campaign batches exist to produce.
An assigned identity cannot be re-derived, so an agent's pool is authoritative state
rather than a cache of a calculation. Reuse does not cross a campaign, shards die with
the pool that holds their identities, and a comparison between two batches must finish
inside the agent home's retention window. This is written where an operator configures
an agent home.
Also merged in from the engine lane: the direct-run binding model and its lineage-key
fix, a bootstrap input-resolution fix, the solver ratchet, and the docker image
installing the wheel rather than the source distribution.
Artifacts
🤖 Generated with Claude Code