diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..8f6f0cb6 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,234 @@ +# Changelog + +All notable changes to this project are documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +## [0.4.0] — 2026-07-25 + +Run a lightcone project on a JupyterHub deployment with nothing to +configure, and scaffold a report alongside the analysis. + +### Added + +- **`kubernetes` container runtime.** On a Dask Gateway deployment the + worker pod *is* the container, so recipe wrapping is a passthrough and + Containerfile specs resolve to registry refs + (`/lc-:`) instead of local-store tags. Never + auto-detected from `PATH` — selected by site detection or pinned in + `~/.lightcone/config.yaml`. +- **GCP Cloud Build backend** (`lightcone/engine/cloudbuild.py`) for + hosts with no local OCI runtime. The staged build context is uploaded + to a deployment bucket and the image is pushed to the deployment's + Artifact Registry. Auth is the pod's Workload Identity via the GCE + metadata server — no stored credentials, no SDK dependency. + Environment contract: `LIGHTCONE_REGISTRY`, `LIGHTCONE_BUILD_BUCKET`, + and the optional `LIGHTCONE_BUILD_SERVICE_ACCOUNT`. Freshness is a + single registry `HEAD` on the content-addressed ref, so unchanged + files never rebuild and never even upload. +- **Run-scoped Dask Gateway clusters.** A new `cluster_for_run` branch, + activated by `DASK_GATEWAY__ADDRESS`, creates a cluster with the + project's image, scales it adaptively from 1 to `--jobs`, waits for + the first worker (bounded by `LIGHTCONE_GATEWAY_WORKER_TIMEOUT`, + default 600s), asserts that workers advertise the `cpus`+`memory` + resource contract, and culls the cluster on exit. Create-per-run is + what makes image updates seamless: a Gateway cluster's image is fixed + at creation. +- **`jupyterhub` site** in the site registry, detected from the + `DASK_GATEWAY__ADDRESS` env marker; declares the `kubernetes` + container runtime and `$HOME` as the scratch root. +- **Pre-flight image build in `lc run`.** The image pass that `lc build` + performs now also runs at the start of `lc run`, so the first + invocation after editing a Containerfile no longer fails mid-DAG on a + missing image. +- **`lc build --runtime kubernetes`** to select the Cloud Build path + explicitly. +- **`worker_image` manifest field** — the image the executing worker pod + reported, recorded alongside the `container_image` the spec declared. + Optional and additive; `null` on every other backend. +- **MyST report scaffold in `lc init`** — `myst.yml` (MySTRA plugin, + book theme) and a TODO-driven `index.md` using the `{astra}` role and + directive plus an `outputs` embed, referencing the boilerplate + `example_method` / `main_result` ids. `_build/` was added to the + scaffolded `.gitignore`, the project `CLAUDE.md` template gained a + `## Report` section, and `lc init`'s next-steps now suggest + `myst start` (requires the MyST CLI, `npm i -g mystmd`). + +### Changed + +- **Image digests now cover every `COPY` / `ADD` source** referenced + from the Containerfile — files hashed directly, directories walked + recursively (with `.git`, `.venv`, `results`, `.lightcone`, + `node_modules`, and the usual caches excluded). *Behavior change:* + projects whose Containerfile copies source code will rebuild once on + upgrade, and will subsequently rerun downstream outputs when that + copied code changes — which previously went undetected. +- `lc run` **rejects a spec resolving to more than one container image** + on the Gateway backend, where a single worker-pod image serves the + whole run. Other backends wrap per rule and are unaffected. +- `cluster_for_run` now yields an **env overlay dict** rather than a + scheduler address string (the Gateway branch passes a cluster name, + not a dialable address) and takes new `worker_image` and `max_workers` + keyword arguments. +- `dask-gateway` is now a **regular dependency**, not an extra, so + `lc run` works out of the box on a hub and scaffolded project images + inherit it through their `lightcone-cli` pin. +- The `snakemake` invocation passes `--shared-fs-usage persistence + input-output sources storage-local-copies source-cache` + unconditionally — omitting `software-deployment` keeps the driver's + `sys.executable` out of spawned job commands, which would not exist + inside a worker image. + +## [0.3.7] — 2026-06-30 + +### Changed + +- Removed the ASTRA `narrative` field from the shipped skills; bumped + the `astra-tools` floor to 0.2.10. +- Refreshed the README badges. + +## [0.3.6] — 2026-05-14 + +### Added + +- PyPI trove classifiers. + +## [0.3.5] — 2026-05-14 + +### Added + +- The **paper-reproduction skill bundle** (`/lc-from-paper` and its + supporting skills). +- Documentation site: landing page, lightcone styling, GitHub Pages + deployment (automatic and manual). + +### Changed + +- `lc init` uses `uv` for venv creation, falling back to `python -m venv`. +- Upgraded init terminal output; broad cleanup of the CLI and user guides. +- Dropped the credentials requirement now that ASTRA is public. + +### Fixed + +- `curl` missing from the python-slim base image. +- CI: dedicated lint workflow, mypy errors, no expensive runs on draft PRs. + +## [0.3.4] — 2026-04-30 + +### Changed + +- Streamlined the Claude Code project hooks. + +## [0.3.3] — 2026-04-30 + +### Fixed + +- Install `lightcone-cli` (not just `astra-tools`) into the project venv. + +## [0.3.2] — 2026-04-30 + +### Fixed + +- Install `astra-tools` into the project venv on `lc init`. + +## [0.3.1] — 2026-04-30 + +### Added + +- `lc run` refuses to execute on a Perlmutter login node. + +### Changed + +- Trimmed and realigned the Claude Code skills, hooks, and reference docs. + +## [0.3.0] — 2026-04-30 + +Version bump; content shipped in 0.2.1. + +## [0.2.1] — 2026-04-30 + +### Added + +- **Snakemake-based execution layer** with content-addressed manifests + on a Dask substrate — the architecture the CLI still rests on. +- Post-materialization result-file validation. +- Intent-based targets with dynamic SLURM discovery. + +### Fixed + +- Gracefully retire Dask workers so `srun` exits silently; drive worker + log level by env var and silence INFO logs unless `--verbose`. +- Thread analysis-level inputs into manifest `input_versions`. +- Prefer `podman-hpc` over `podman` in runtime auto-detection. +- Insert `--` before snakemake targets so `--rerun-triggers` stops + swallowing them. + +## [0.2.0] — 2026-04-21 + +### Changed + +- **Renamed Prism to lightcone-cli**: the executable is now `lc` and the + code lives in the `lightcone.*` namespace. +- Initial contributor documentation. + +### Fixed + +- Crash with dagster 1.13 (removed `get_all_asset_specs`). + +## [0.1.3] — 2026-04-14 + +### Changed + +- Published to PyPI as `lightcone-prism`. +- Simplified the container spec to a single string instead of a build dict. +- Relicensed from Apache 2.0 to BSD 3-Clause. +- Consolidated the guides into `astra-reference` and `prism-reference`. + +## [0.1.2] — 2026-03-28 + +### Added + +- Sub-analysis redesign. +- Eval harness for quantitative build-loop evaluation. +- Podman support and a venv fallback backend. + +### Changed + +- Recipe output is streamed in real time. + +## [0.1.1] — 2026-03-19 + +### Added + +- `prism update` with project sync and a startup update check. +- `--existing-project` init mode. + +### Changed + +- Recommended permissions flipped from an allowlist to a denylist. +- Internals moved into a `.prism/` directory. +- `prism-build` rewritten on the ralph stop hook, without plan mode. + +## [0.1.0] — 2026-03-03 + +Initial release. + +[Unreleased]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.4.0...HEAD +[0.4.0]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.7...v0.4.0 +[0.3.7]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.6...v0.3.7 +[0.3.6]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.5...v0.3.6 +[0.3.5]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.4...v0.3.5 +[0.3.4]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.3...v0.3.4 +[0.3.3]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.2...v0.3.3 +[0.3.2]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.1...v0.3.2 +[0.3.1]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.3.0...v0.3.1 +[0.3.0]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.2.1...v0.3.0 +[0.2.1]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.2.0...v0.2.1 +[0.2.0]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.1.3...v0.2.0 +[0.1.3]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.1.2...v0.1.3 +[0.1.2]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.1.1...v0.1.2 +[0.1.1]: https://github.com/LightconeResearch/lightcone-cli/compare/v0.1.0...v0.1.1 +[0.1.0]: https://github.com/LightconeResearch/lightcone-cli/releases/tag/v0.1.0 diff --git a/README.md b/README.md index 3a923a31..dd770182 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,9 @@ Then tell the agent what you have to start from — a research question - **Multiverse analysis** — define methodological decisions with multiple options; `lc` runs your analysis across all defensible paths automatically - **Provenance integrity** — every output gets a content-addressed manifest; `lc verify` detects tampering or broken chains -- **HPC-ready execution** — Snakemake-backed DAG dispatch with SLURM and container support (Docker, Podman, Apptainer) out of the box +- **HPC-ready execution** — Snakemake-backed DAG dispatch with SLURM and container support (Docker, Podman, podman-hpc) out of the box +- **Cloud-ready execution** — on a lightcone JupyterHub, `lc` builds images through GCP Cloud Build and runs the pipeline in a run-scoped Dask Gateway cluster, with nothing to configure +- **Report from the spec** — `lc init` scaffolds a MyST report that references your analysis by path, so figures and numbers stay single-sourced - **Reproducible publishing** — `lc export wrroc` emits a [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) bundle ready for Zenodo or WorkflowHub → [Full documentation](https://docs.lightconeresearch.org) diff --git a/claude/lightcone/scripts/session-start.sh b/claude/lightcone/scripts/session-start.sh index 121f7c9f..3e9f36db 100755 --- a/claude/lightcone/scripts/session-start.sh +++ b/claude/lightcone/scripts/session-start.sh @@ -52,9 +52,10 @@ Materialization: ok=$ok_count stale=$stale_count missing=$missing_count alias=$a Substrate CLIs (use --help on any): lc init / lc run / lc status / lc verify / lc build / lc export wrroc astra validate / astra paper add / astra universe generate + myst start — preview the MyST report scaffolded by lc init (needs mystmd) Reference skills (invoke when the surface above isn't enough): - /astra — astra.yaml spec: decisions, prior_insights, findings, evidence, sub-analyses, narrative anchors + /astra — astra.yaml spec: decisions, prior_insights, findings, evidence, sub-analyses /lc-cli — lc workflow: spec-code invariant, status interpretation, failure diagnosis" if [ "$validation_ok" -ne 0 ]; then diff --git a/claude/lightcone/skills/README.md b/claude/lightcone/skills/README.md index 5501fdda..acace026 100644 --- a/claude/lightcone/skills/README.md +++ b/claude/lightcone/skills/README.md @@ -1,6 +1,6 @@ # lightcone-cli skills -Each subdirectory is one Claude Code skill: `SKILL.md` plus optional `references/`, `assets/`, and `scripts/`. `lc init` copies these into a project's `.claude/skills/` so they are discoverable to Claude Code sessions. +Each subdirectory is one Claude Code skill: `SKILL.md` plus optional `references/`, `scripts/`, and `templates/`. `lc init` copies these into a project's `.claude/skills/` so they are discoverable to Claude Code sessions. ## Project lifecycle skills diff --git a/claude/lightcone/skills/astra/SKILL.md b/claude/lightcone/skills/astra/SKILL.md index 51430183..7336e550 100644 --- a/claude/lightcone/skills/astra/SKILL.md +++ b/claude/lightcone/skills/astra/SKILL.md @@ -372,15 +372,21 @@ description: | astra init [DIRECTORY] # Scaffold a new analysis astra validate astra.yaml # Validate (run after every change) astra validate astra.yaml --verify-evidence # + verify insight quotes against PDFs +astra validate astra.yaml --skip-evidence # Skip evidence checks even when insights exist +astra spec [TERM] [--full] # Agent-friendly schema reference (concept summary, one entry, or full dump) astra info [--decisions|--inputs|--outputs] # Analysis summary / element details astra universe generate -n NAME [-d "desc"] # Generate universe from defaults astra universe check universes/x.yaml # Check universe constraints -astra viz [--fmt ascii|mermaid] # Visualize decision space -astra schema show analysis|universe|insights # Show JSON schema +astra viz [--format mermaid|ascii] # Visualize decision space +astra schema show analysis|universe|insights # Print a LinkML schema +astra schema export [-o DIR] # Export LinkML schemas to files astra paper add DOI [--version N] [--pdf PATH] # Cache a paper for evidence checks astra paper list # List cached papers astra paper show DOI # Show metadata for a cached paper astra paper path DOI [--version N] # Print the cached PDF's path +astra paper remove DOI # Remove a paper from the cache +astra paper fetch-metadata [DOI|--all] # Fetch title/authors for cached papers +astra paper verify-quote DOI -q "TEXT" # Verify a single quote astra paper verify-quotes DOI # Batch-verify quotes; reads {"quotes":[...]} JSON from stdin ``` diff --git a/claude/lightcone/skills/check-sentence-by-sentence/SKILL.md b/claude/lightcone/skills/check-sentence-by-sentence/SKILL.md index 7e3e8fdc..6ce359dd 100644 --- a/claude/lightcone/skills/check-sentence-by-sentence/SKILL.md +++ b/claude/lightcone/skills/check-sentence-by-sentence/SKILL.md @@ -58,9 +58,10 @@ Pandoc. > "Which paper source should I audit? Please give me a `.tex` path or > `work/reference/document.md`." - If only `work/reference/paper.pdf` exists, ask the user to run the PARSE - phase first so `work/reference/document.md` exists. Do not audit PDFs - directly. + If only `work/reference/paper.pdf` exists, ask the user to run + `/paper-extraction` first (in lc-from-paper projects this happens during + the ORIENT stage) so `work/reference/source/` or + `work/reference/document.md` exists. Do not audit PDFs directly. ## Section enumeration diff --git a/claude/lightcone/skills/figure-comparison/SKILL.md b/claude/lightcone/skills/figure-comparison/SKILL.md index 0025e4c4..0f0d9376 100644 --- a/claude/lightcone/skills/figure-comparison/SKILL.md +++ b/claude/lightcone/skills/figure-comparison/SKILL.md @@ -48,20 +48,20 @@ results. 3. **Locate the paper reference substrate.** The user may have passed a path. Resolve it in this order: - 1. If the argument is a directory containing `metadata.json`, + 1. If the argument is a directory containing `index.json`, `document.md`, `figures/`, or `tables/`, use that directory as the paper reference root. 2. If the argument is an arXiv source directory containing `.tex` files, use it as `source_root`, and use its parent `work/reference/` as the paper reference root when that parent exists. 3. If no argument was supplied, prefer lc-from-paper's layout: - - `work/reference/source/` when arXiv TeX source exists. Use the TeX - files there for labels/captions and the parsed artifacts under - `work/reference/{figures,tables,metadata.json}` for renderable - reference files. + - `work/reference/source/` when arXiv TeX source exists. Use + `work/reference/index.json` for the figure/table inventory, the TeX + files for labels/captions, and the extracted artifacts under + `work/reference/{figures,tables}` for renderable reference files. - `work/reference/document.md` plus - `work/reference/{figures,tables,metadata.json}` when no TeX source - exists. This is the PDF + Docling fallback from lc-from-paper. + `work/reference/{figures,tables,index.json}` when no TeX source + exists. This is the PDF + Docling fallback from paper-extraction. 4. Only after lc-from-paper paths fail, look for a legacy unzipped arXiv dir in cwd: a directory containing both a `*.tex` file and figure files (`*.pdf`, `*.png`, `*.eps`). Common names: `paper_source/`, @@ -70,12 +70,14 @@ results. If no usable reference substrate is found, ask: > "Where is the paper reference directory? In a lc-from-paper project this - > should usually be `work/reference/`, containing `document.md`, - > `metadata.json`, and extracted `figures/` / `tables/`." + > should usually be `work/reference/`, containing `index.json`, extracted + > `figures/` / `tables/`, and either `source/` (arXiv TeX) or + > `document.md` (Docling fallback)." - If only `work/reference/paper.pdf` exists, ask the user to run the PARSE - phase first so Docling or the TeX parser populates `work/reference/`. - Do not compare directly against a whole PDF. + If only `work/reference/paper.pdf` exists, ask the user to run + `/paper-extraction` first (in lc-from-paper projects this happens during + the ORIENT stage) so `work/reference/index.json` and the extracted + artifacts are populated. Do not compare directly against a whole PDF. ## Phase 1 -- Understand the paper's main results @@ -100,15 +102,20 @@ Read, in this order: come from the output ID and the result resolver in Phase 2. 3. **The paper reference substrate**, in this order: - - Read `work/reference/metadata.json` when present. It is the primary - index for paper figures and tables; its paths are relative to - `work/reference/` and usually point into `figures/` or `tables/`. + - Read `work/reference/index.json` when present. It is the canonical + structural index for paper figures and tables, written by + paper-extraction on both paths; each figure/table entry carries `id`, + `label`, `caption`, and a `file` path relative to `work/reference/` + (usually pointing into `figures/` or `tables/`; multi-panel figures + also carry `files`). On the Docling path a `metadata.json` + intermediate may also exist; ignore it -- its content is already + folded into `index.json`. - If `work/reference/source/` exists, grep its TeX files for `\includegraphics`, `\label{fig:...}`, `\caption{...}`, and - `\begin{table}` to recover labels/captions that metadata may have + `\begin{table}` to recover labels/captions that the index may have missed. - If only `work/reference/document.md` exists, use the markdown plus - `metadata.json` as the source of captions, table text, and in-text + `index.json` as the source of captions, table text, and in-text numerical claims. This is the Docling/Pandoc fallback; preserve its line numbers and do not pretend it is TeX. - Grep the abstract, results, and discussion sections of the TeX or @@ -155,23 +162,26 @@ For project-side result paths, resolve every output ID with this order: - Use an explicit `reproduced_file` from `comparison-report.yaml` or an explicit reproduced path/glob from `targets/targets.md`, if present and the file exists. -- Search for flat files at `results//.` with the - first suitable type-specific extension: images (`.png`, `.jpg`, `.jpeg`, +- Look inside the output directory `results///` -- + every `lc run` output is a directory containing the artifact file(s) + plus `.lightcone-manifest.json`. Glob within it for the first suitable + type-specific extension: images (`.png`, `.jpg`, `.jpeg`, `.pdf`, `.eps`), tables (`.csv`, `.parquet`, `.md`, `.txt`), values - (`.json`, `.yaml`, `.yml`, `.txt`, `.md`). + (`.json`, `.yaml`, `.yml`, `.txt`, `.md`) -- always ignoring + `.lightcone-manifest.json` and `.snakemake_timestamp`. - If still unmatched and no scoped ledger exists, fall back to filename-stem - similarity within `results//`. + similarity across the artifact files inside `results//*/`. - If no match is found, use `project_path: null` and render a red `NOT PRODUCED` panel. Do not include unrelated result files; the report is target-driven when target/report files exist, and paper-driven otherwise. -For tables: use `work/reference/metadata.json` and `work/reference/tables/` +For tables: use `work/reference/index.json` and `work/reference/tables/` when present. If TeX source exists, capture the raw LaTeX of the `tabular` block and any `\caption{...}`. If only `work/reference/document.md` exists, capture the Docling/Pandoc markdown table or the extracted table artifact under `work/reference/tables/`. The project side is whatever artifact -carries the same content -- typically a CSV / parquet / markdown file at -`results//.`. If `astra.yaml` declares no matching +carries the same content -- typically a CSV / parquet / markdown file +inside `results///`. If `astra.yaml` declares no matching output, use `project_path: null`. **If the paper contains no tables at all, leave the manifest's `tables` list empty; the helper must omit the entire Tables section from the HTML in that case (no header, no "no tables" @@ -182,8 +192,8 @@ project_value?, project_value_source?, paper_quote}`. Pull `paper_value` from the in-text claim or `astra.yaml`'s `findings.*.paper_value`. Pull `project_value` from `astra.yaml`'s `findings.*.replicated_value` if present, otherwise from -a scoped `comparison-report.yaml` entry or a flat result summary file at -`results//.` that you can read statically. +a scoped `comparison-report.yaml` entry or a result summary file inside +`results///` that you can read statically. **Never compute or re-derive values yourself.** If no project value can be located statically, leave it null and flag in the HTML. @@ -218,9 +228,11 @@ Use a small Python helper rather than embedding base64 inline through your tool calls -- multi-MB image base64 strings would balloon your context. -Use the existing `.lightcone/` directory in the project root. Do not create -directories in this skill. All three files this skill writes -- manifest, -helper, and final HTML -- live there. +Write into the `.lightcone/` directory in the project root. It exists in +any `lc init`-ed project; if it is missing, tell the user the project is +not initialized (`lc init`) instead of silently creating directories. All +three files this skill writes -- manifest, helper, and final HTML -- live +there. 1. **Write the manifest** as JSON to `.lightcone/comparison_manifest.json`. Schema: @@ -238,7 +250,7 @@ helper, and final HTML -- live there. "paper_caption": "...", "paper_path": "targets/main_result.pdf", "project_output_id": "primary_metric_plot", - "project_path": "results/baseline/primary_metric_plot.png" + "project_path": "results/baseline/primary_metric_plot/plot.png" } ], "tables": [ @@ -247,7 +259,7 @@ helper, and final HTML -- live there. "paper_caption": "...", "paper_latex": "\\begin{tabular}{...}\\end{tabular}", "project_output_id": "...", - "project_path": "results/baseline/summary_table.csv" + "project_path": "results/baseline/summary_table/summary.csv" } ], "values": [ @@ -259,7 +271,7 @@ helper, and final HTML -- live there. "paper_quote": "we find $\\mathrm{metric} = 12.5 \\pm 0.4$ ", "project_value": "12.47", "project_uncertainty": "0.41", - "project_value_source": "results/baseline/metric.json" + "project_value_source": "results/baseline/primary_metric/metric.json" } ] } @@ -519,8 +531,9 @@ solid-fill buttons), it is wrong. `scripts/` or `results/`. The only files this skill writes are `.lightcone/comparison_manifest.json`, `.lightcone/build_comparison.py`, and - `.lightcone/comparison.html`. Assume `.lightcone/` already exists; never - write into `results/`. + `.lightcone/comparison.html`. `.lightcone/` exists in any `lc init`-ed + project; if it is missing, stop and tell the user to run `lc init` + rather than creating it. Never write into `results/`. - You MUST NOT fabricate values. If a paper number is not stated in the paper source, `targets/targets.md`, `comparison-report.yaml`, or `astra.yaml`, leave it null. If a project number is not recorded in a diff --git a/claude/lightcone/skills/lc-cli/SKILL.md b/claude/lightcone/skills/lc-cli/SKILL.md index 66e9b643..9a0c875d 100644 --- a/claude/lightcone/skills/lc-cli/SKILL.md +++ b/claude/lightcone/skills/lc-cli/SKILL.md @@ -4,8 +4,8 @@ description: > Reference for `lc` CLI execution: commands (init/run/status/verify/build/export), the Spec-Code Invariant (`astra.yaml` and code never diverge), status interpretation (ok/stale/missing/alias), failure diagnosis, multiverse - runs, scratch overrides for HPC, sub-analysis scaffolding, publishing - via WRROC. Invoke whenever running, debugging, or diagnosing `lc` + runs, scratch overrides for HPC, JupyterHub/Dask Gateway deployments, + sub-analysis scaffolding, publishing via WRROC. Invoke whenever running, debugging, or diagnosing `lc` workflows; whenever interpreting `lc status` / `lc verify` output; or whenever the user asks about the development workflow surrounding `astra.yaml`. @@ -19,25 +19,33 @@ Reference for lightcone-cli execution: CLI commands, development workflow, statu ## CLI Reference ```bash -lc init [DIR] [--permissions yolo|recommended|minimal] [--scratch PATH] # Scaffold a new ASTRA project -lc run [OUTPUTS...] [--universe NAME] [--force] [--verbose] [--rerun-triggers TRIGGERS] # Materialize outputs -lc build [--force] [--runtime docker] # Build container images from specs +lc init [DIR] [--no-git] [--no-venv] [--permissions yolo|recommended|minimal] [--scratch PATH] # Scaffold a new ASTRA project +lc run [OUTPUTS...] [--universe NAME] [--jobs N] [--force] [--verbose] [--rerun-triggers TRIGGERS] # Materialize outputs +lc build [--force] [--runtime docker|podman|podman-hpc|kubernetes] # Build container images from specs lc status [--universe NAME] [--json] # Materialization status (text or JSON) lc verify [--universe NAME] # Recompute hashes and walk the provenance chain lc export wrroc [--output PATH] [--universe NAME] [--zip] [--metadata-only] [--author "NAME "] # Export Workflow Run RO-Crate bundle ``` -`lc run` is quiet by default — pass `--verbose` to see worker output. `--scratch` is only relevant on HPC sites where `$HOME` doesn't honor `flock` (NERSC etc.); it redirects Snakemake state and Dask spill onto the named filesystem. +`lc run` is quiet by default — pass `--verbose` to see worker output. `--jobs` bounds parallel dispatch (default: local CPU count). `--scratch` is only relevant on HPC sites where `$HOME` doesn't honor `flock` (NERSC etc.); it redirects Snakemake state and Dask spill onto the named filesystem. + +`lc init` also scaffolds a template MyST report (`myst.yml` + `index.md`; `_build/` is gitignored) — preview it with `myst start` (requires the MyST CLI: `npm i -g mystmd`). The first `lc` invocation auto-creates `~/.lightcone/config.yaml`: ```yaml container: - runtime: auto # or: docker | podman | podman-hpc | none + runtime: auto # or: docker | podman | podman-hpc | kubernetes | none ``` +On the `kubernetes` runtime there is no local OCI runtime: `lc build` submits each Containerfile to the deployment's GCP Cloud Build service (running under the deployment's service account) and pushes `/lc-:` — same content-addressed identity. Freshness is a registry check, not a local image-store check. + **Always run via `lc`.** Recipes must execute through `lc run` so that container builds, option resolution, resource limits, and result paths are applied. Treat the underlying execution engine as a black box — never invoke schedulers or container runtimes directly, that will bypass reproducibility guarantees. +## JupyterHub Deployments (Dask Gateway) + +A lightcone JupyterHub is detected via `DASK_GATEWAY__ADDRESS` (site `jupyterhub` in the site registry); the container runtime resolves to `kubernetes` — the worker pod *is* the container, and recipes run natively inside it. `lc run` pre-flight-builds images through Cloud Build, then creates a run-scoped Gateway cluster sized by `--jobs`, with workers running the project image; the cluster is torn down when the run finishes. Constraint: every output in one run must resolve to a single container image, or `lc run` fails — consolidate on one Containerfile (or one shared prebuilt image). If a run sits at zero workers (e.g. a slow first-time image pull), the wait for the first worker is bounded by `LIGHTCONE_GATEWAY_WORKER_TIMEOUT` (seconds, default 600). + ## Creating Sub-Analyses Sub-analyses are scaffolded by hand, since each one is just another `astra.yaml` nested in a directory. To add one: diff --git a/claude/lightcone/skills/lc-feedback/SKILL.md b/claude/lightcone/skills/lc-feedback/SKILL.md index 23cf7b10..63e1ee4d 100644 --- a/claude/lightcone/skills/lc-feedback/SKILL.md +++ b/claude/lightcone/skills/lc-feedback/SKILL.md @@ -41,9 +41,9 @@ Default to **lightcone-cli** if ambiguous. Collect versions silently: ```bash -python3 -c "import astra; print(astra.__version__)" 2>/dev/null || echo "n/a" -python3 -c "import lightcone.cli; print(lightcone.cli.__version__)" 2>/dev/null || echo "n/a" -python3 --version 2>&1 +astra --version 2>/dev/null || echo "n/a" +lc --version 2>/dev/null || echo "n/a" +python --version 2>&1 uname -s -r ``` diff --git a/claude/lightcone/skills/lc-from-code/SKILL.md b/claude/lightcone/skills/lc-from-code/SKILL.md index f81ac86f..df87f2aa 100644 --- a/claude/lightcone/skills/lc-from-code/SKILL.md +++ b/claude/lightcone/skills/lc-from-code/SKILL.md @@ -59,7 +59,7 @@ For reference, here are the decision criteria for classifying candidates: When the codebase is large enough that one Explore pass risks missing depth (a multi-project monorepo, a workflow folder plus a notebooks tree plus a `src/` package), spawn Explores in parallel against the named subtrees — one Explore per coherent region. Aggregate their inventories into the final scan output. -Write the scan results to `CLAUDE.md` under `## Project Notes` (fresh migration) or to the path the invocation prompt specifies (scan-only — typically `work/reference/code-index.md`) as a script inventory, then in fresh migration mode draft or add to `astra.yaml` from the scan results following the spec structure documented in `/astra`. In scan-only mode, stop after the inventory file lands; do not touch `astra.yaml`. Use the decision criteria from `/astra` (Decisions section) to filter candidate decisions down to only true analytical choices — most hardcoded values are implementation details, not decisions. Use current hardcoded values as defaults. +Write the scan results to `CLAUDE.md` (fresh migration — append a `## Project Notes` section; the file `lc init` generates has an orientation intro, an `lc` CLI quick reference, and a `## Report` section, but no notes section) or to the path the invocation prompt specifies (scan-only — typically `work/reference/code-index.md`) as a script inventory, then in fresh migration mode draft or add to `astra.yaml` from the scan results following the spec structure documented in `/astra`. In scan-only mode, stop after the inventory file lands; do not touch `astra.yaml`. Use the decision criteria from `/astra` (Decisions section) to filter candidate decisions down to only true analytical choices — most hardcoded values are implementation details, not decisions. Use current hardcoded values as defaults. In augment mode, preserve the existing paper-derived or user-derived `inputs`, `outputs`, `decisions`, `findings`, and `description` unless the code scan shows a real conflict. Attach code evidence to the nearest existing home first. Create new ASTRA structure only when the code reveals a real analysis object that has no suitable home in the current spec. @@ -103,6 +103,8 @@ Parameterize the code from ASTRA decisions so the baseline run reproduces the ex - No container setup but a `requirements.txt`: write a minimal `Containerfile` (`FROM python:3.12-slim`, copy and `pip install -r requirements.txt`, then `COPY . .`) and point `container:` at it. - Nothing to go on: set `container: python:3.12-slim` as a starting point — the user can swap to a real `Containerfile` later. +Prefer a single shared image unless recipes genuinely need different stacks: on a JupyterHub/Dask-Gateway deployment, `lc run` rejects specs that resolve to more than one container image per run. + Whatever approach you use: - **Don't refactor, restructure, or improve the code.** Just add the parameter plumbing. @@ -120,7 +122,7 @@ If it fails, read the error, fix it, and retry. Iterate until `lc status` shows If the scan found existing results elsewhere in the project, compare them against the new outputs in `results/baseline//` to verify the migration preserved behavior. -Then validate the spec and the provenance chain: `astra validate astra.yaml` and `lc verify`. Present summary to user. +Then validate the spec and the provenance chain: `astra validate astra.yaml` and `lc verify`. Present summary to user. If the project was scaffolded by `lc init`, also flag that the MyST report (`index.md`) still references the boilerplate ids (`example_method`, `main_result`) and needs updating to the reshaped spec's element paths. ## Rules diff --git a/claude/lightcone/skills/lc-from-paper/SKILL.md b/claude/lightcone/skills/lc-from-paper/SKILL.md index b48c1c65..afc43344 100644 --- a/claude/lightcone/skills/lc-from-paper/SKILL.md +++ b/claude/lightcone/skills/lc-from-paper/SKILL.md @@ -64,7 +64,7 @@ ORIENT runs as one phase in **seven stages**: These get drafted into **two files** plus the substrate, all in the reproduction workdir: -- **`constitution.md`** — the ralph loop's driving document. Goal, Fidelity intent, Scope, Quality bar, Evidence (paper DOI, arXiv ID, code repo URL), Open dimensions. Starts with YAML frontmatter `status: active` so the ralph launcher accepts it. Authored using the `/ralph` skill's authoring discipline (the constitution-authoring mode of `/ralph` — see its references on voice and sections). +- **`constitution.md`** — the ralph loop's driving document. Goal, Fidelity intent, Scope, Quality bar, Evidence (paper DOI, arXiv ID, code repo URL), Open dimensions. Starts with YAML frontmatter `status: active` (the launcher accepts `open` or `active`; `active` is the convention here). Authored using the `/ralph` skill's authoring discipline (the constitution-authoring mode of `/ralph` — see its references on voice and sections). - **`CLAUDE.md`** — the auto-loading walk-up. Paper identity at the top, Rules (universal across reproductions; leave the template's defaults), Disagreements log (starts empty), Open opportunities (starts empty), Pointers (to `constitution.md`, `work/reference/`, etc.). - **`work/reference/`** — paper substrate from `/paper-extraction` + code substrate from `/lc-from-code` scan-only (when a code repo exists). diff --git a/claude/lightcone/skills/lc-from-paper/references/architect.md b/claude/lightcone/skills/lc-from-paper/references/architect.md index c93e4b3b..8c4db48b 100644 --- a/claude/lightcone/skills/lc-from-paper/references/architect.md +++ b/claude/lightcone/skills/lc-from-paper/references/architect.md @@ -30,7 +30,7 @@ Read `constitution.md`, `CLAUDE.md`, `work/reference/index.json`, `work/referenc 2. **Choose: one analysis or sub-analyses?** If the paper has only one stage end-to-end (no clean intermediate handoffs), write a single analysis. If it has genuinely independent stages (each stage's output flows as the next's input), write sub-analyses. Sub-analysis IDs must be noun phrases: `reconstruction`, `clustering`, `bao_fit`. Avoid reserved names: `inputs`, `outputs`, `decisions`, `findings`, `prior_insights`, `analyses`, `options`, `content`. 3. **Wire inputs and outputs at the sub-analysis level.** For each sub-analysis: - Declare `inputs:` from `code-index.md`'s External-data-dependencies plus any paper-named external datasets. The depth (acquisition path, selection criteria) is SPECIFY's; ARCHITECT names the input and gives it a stable id. - - Declare `outputs:` matching the result loci from `index.json` (figures + tables) plus any intermediate artifacts a downstream sub-analysis consumes. Tag each output's `priority:` from the paper's emphasis (primary / secondary). **The reproduction's targeted scope from `constitution.md`'s Scope takes precedence** — if the user only wants Figure 3 and Table 2, only those land as `outputs:`; the rest are out-of-scope and noted as such. + - Declare `outputs:` matching the result loci from `index.json` (figures + tables) plus any intermediate artifacts a downstream sub-analysis consumes. Note each output's priority (primary / secondary, from the paper's emphasis) in its `description:` — there is no `priority:` slot in the schema; SPECIFY carries the priority formally into `targets/targets.md`. **The reproduction's targeted scope from `constitution.md`'s Scope takes precedence** — if the user only wants Figure 3 and Table 2, only those land as `outputs:`; the rest are out-of-scope and noted as such. 4. **Author the root and per-analysis `description`.** Write a short `description:` (one or two paragraphs) on the root analysis and on each sub-analysis — enough to orient a reader on what the analysis is and what it produces. The root `description:` should give a top-down, end-to-end sketch of how the sub-analyses' outputs flow into one another when sub-analyses exist. Keep it high-level; per-decision and per-finding prose lives on those entries' own `rationale:` / `notes:` fields, authored in SPECIFY. 5. **Validate.** `astra validate astra.yaml` must return clean — even with empty `decisions:` / `prior_insights:` / `findings:` blocks, the structural fields and descriptions must pass schema checks. @@ -38,27 +38,33 @@ Read `constitution.md`, `CLAUDE.md`, `work/reference/index.json`, `work/referenc ```yaml # Stub: structure + descriptions. SPECIFY fills decisions/findings/prior_insights. -id: -title: "" -doi: +# Paper: "" — DOI (cached via `astra paper add `; +# title/doi are not Analysis slots, so they live here and in constitution.md) +version: "0.0.12" +name: description: | +inputs: [] # externals live on the sub-analyses that consume them + +outputs: # root-level re-exports of the replication targets + - id: + from: . + analyses: : description: | inputs: - : - + - id: + type: data + description: "" outputs: - : - type: figure | table | metric | data-product - priority: primary | secondary - description: | - + - id: + type: figure # figure | table | metric | data | report + description: "" decisions: {} # SPECIFY fills prior_insights: {} # SPECIFY records placeholders (Evidence with doi:, no quote: yet), LITERATURE fills the quote: selectors findings: {} # SPECIFY fills diff --git a/claude/lightcone/skills/lc-from-paper/references/compare.md b/claude/lightcone/skills/lc-from-paper/references/compare.md index 2dd64106..eecaee56 100644 --- a/claude/lightcone/skills/lc-from-paper/references/compare.md +++ b/claude/lightcone/skills/lc-from-paper/references/compare.md @@ -20,11 +20,11 @@ COMPARE is what a ralph iteration does when the workdir signals "RUN done (`resu ## Result path convention -For an output with `id: X`, the reproduced result lives at `results//X.`: +For an output with `id: X`, the reproduced result lives in the directory `results//X/` — the artifact(s) the recipe wrote plus the run's `.lightcone-manifest.json`: -- metrics: `.json` containing `{"value": ...}` -- figures: `.png` -- tables: `.csv` +- metrics: a `.json` containing `{"value": ...}` +- figures: a `.png` +- tables: a `.csv` ## Task diff --git a/claude/lightcone/skills/lc-from-paper/references/implement.md b/claude/lightcone/skills/lc-from-paper/references/implement.md index e0e6c1df..0033b219 100644 --- a/claude/lightcone/skills/lc-from-paper/references/implement.md +++ b/claude/lightcone/skills/lc-from-paper/references/implement.md @@ -53,7 +53,7 @@ The iteration merges scripts and recipes after the per-output sub-agents finish. ### Rules for the first pass 1. **One script per output** (or a shared script for tightly-coupled outputs). -2. **Parameterize by decisions.** Each decision is a CLI argument; scripts also receive `--universe `. See lightcone-cli's `CLAUDE.md` for the full convention. +2. **Parameterize by decisions.** Each decision the output declares becomes a CLI argument, wired explicitly in the recipe via `{decisions.}`; paths flow via `{output}` and `{inputs}` / `{inputs.}`. Nothing is auto-injected — the recipe's `command:` must pass everything the script needs, and any other placeholder is rejected at run time. 3. **Add recipes** to each output in `astra.yaml` with `command:` and `inputs:` (dependencies). Recipe inputs use the `.` tree-path form that wires outputs to downstream inputs. 4. **Create `requirements.txt`** with needed packages. Do not install them — the RUN phase manages environments. 5. **Do not execute scripts** — the RUN phase handles execution via `lc run`. diff --git a/claude/lightcone/skills/lc-from-paper/references/review.md b/claude/lightcone/skills/lc-from-paper/references/review.md index 5a080bf7..dcf7e196 100644 --- a/claude/lightcone/skills/lc-from-paper/references/review.md +++ b/claude/lightcone/skills/lc-from-paper/references/review.md @@ -1,6 +1,6 @@ # REVIEW — close-out in the user's main session -The reproduction has converged: the constitution's `status:` is `closed` (after COMPARE returned `pass`, or `partial` with the un-acted opportunities logged, and the next cold-survey iteration found nothing left to do). The ralph loop's tmux session has exited. REVIEW runs back in the user's main session — the second of two interactive bookends, the first being ORIENT. It runs in the user's main session (not as an iteration) because both `/figure-comparison` and `/check-sentence-by-sentence` use `AskUserQuestion`, which isn't available inside detached ralph iterations. +The reproduction has converged: the constitution's `status:` is `closed` (after COMPARE returned `pass`, or `partial` with the un-acted opportunities logged, and the next cold-survey iteration found nothing left to do). The ralph loop's tmux session has exited. REVIEW runs back in the user's main session — the second of two interactive bookends, the first being ORIENT. It runs in the user's main session (not as an iteration) because both `/figure-comparison` and `/check-sentence-by-sentence` are interactive — they prompt the user for anything they can't infer, which doesn't work inside detached ralph iterations. Its job is to render the validation surfaces, walk the user through the accumulated open questions, land the resolutions, and draft the final report — in one interactive arc. The Open opportunities list in CLAUDE.md already carries un-acted-on opportunities from the latest COMPARE (those iterations logged them directly); REVIEW just reads them. @@ -21,7 +21,7 @@ The phase name **REVIEW** is freed by the old pre-implement REVIEW phase folding ## Outputs - `.lightcone/comparison.html` — `/figure-comparison`'s portable side-by-side report (paper artifacts vs reproduced) -- (Optional) `.lightcone/check-sentence-by-sentence.md` — `/check-sentence-by-sentence`'s claim audit (file:line or NOT FOUND per sentence) +- (Optional) `/check-sentence-by-sentence`'s claim audit, rendered in-chat (file:line or NOT FOUND per sentence — the skill writes no files) - `open-questions.md` — same file, but with `## Resolutions` section appended capturing what the user said for each entry - Edits to `astra.yaml` / `implementation-notes.md` / `universes/baseline.yaml` if any open-question resolution warrants a spec change - `REPRODUCTION-SUMMARY.md` — final report; concise (~1–2 pages); the canonical record of what the reproduction landed on @@ -32,19 +32,19 @@ The phase name **REVIEW** is freed by the old pre-implement REVIEW phase folding ### `/figure-comparison` (mandatory) -Invoke the `/figure-comparison` skill from the user's main session. It builds a portable HTML side-by-side comparing paper artifacts (from `targets/`) to reproduced artifacts (from `results//`). The skill uses `AskUserQuestion` for any inputs it can't infer from the workdir; that works because REVIEW runs back in the user's main session — the prompts land here, not in a detached iteration. +Invoke the `/figure-comparison` skill from the user's main session. It builds a portable HTML side-by-side comparing paper artifacts (from `targets/`) to reproduced artifacts (from `results//`). The skill asks the user about any inputs it can't infer from the workdir; that works because REVIEW runs back in the user's main session — the prompts land here, not in a detached iteration. Output lands at `.lightcone/comparison.html`. Show the user the path and offer to open it (`open` on macOS, `xdg-open` on Linux, or just print the path so they click in their terminal). -**Do not spawn `/figure-comparison` under the `Task` tool or inside a ralph iteration.** It has `AskUserQuestion` in its `allowed-tools`; sub-agents and detached iterations have no user-reach, so the prompt fires into nothing. +**Do not spawn `/figure-comparison` under the `Task` tool or inside a ralph iteration.** It prompts the user for inputs it can't infer; sub-agents and detached iterations have no user-reach, so the prompt fires into nothing. ### `/check-sentence-by-sentence` (opt-in) Ask the user via `AskUserQuestion` whether they want the claim audit. It's optional because for many reproductions the figure-comparison already settles "did it match?"; the sentence-by-sentence audit earns its keep when the paper makes many specific quantitative claims and the user wants each one anchored to a code location. -If yes, invoke `/check-sentence-by-sentence`. Same discipline as `/figure-comparison` — it can prompt the user; do not spawn under `Task` or inside a ralph iteration. +If yes, ask the user to run `/check-sentence-by-sentence` themselves — that skill is user-invoked by design and should not be launched from here. Same discipline as `/figure-comparison` once it runs: it can prompt the user, so it belongs in the main session, never under `Task` or inside a ralph iteration. -Output lands at `.lightcone/check-sentence-by-sentence.md` (or wherever the skill writes it). Show the user the path. +The audit renders directly in chat as a markdown report organized by paper section — the skill writes no files. ## Step 2: walk `open-questions.md` with the user @@ -99,8 +99,8 @@ This commit is the durable mark that the reproduction has reached close-out. Fut ## Notes -- **This phase runs in the user's main session.** Do not invoke it from inside a ralph iteration. The whole point of REVIEW is that the user is reachable — every step uses `AskUserQuestion` (directly, or via the sibling skills it invokes), and iterations are detached. -- **`/figure-comparison` and `/check-sentence-by-sentence` use `AskUserQuestion`.** That's why REVIEW runs in the user's main session and they live here, not in any iteration. Invoking either inside an iteration fires prompts into nothing. +- **This phase runs in the user's main session.** Do not invoke it from inside a ralph iteration. The whole point of REVIEW is that the user is reachable — every step prompts the user (via `AskUserQuestion` directly, or via the interactive sibling skills it invokes), and iterations are detached. +- **`/figure-comparison` and `/check-sentence-by-sentence` are interactive.** They prompt the user when they need input — that's why REVIEW runs in the user's main session and they live here, not in any iteration. Invoking either inside an iteration fires prompts into nothing. - **The user owns the verdict-acceptance decision.** REVIEW's purpose is to let the user see what the loop's iterations did and decide whether they accept it. The skill renders surfaces and asks; it does not unilaterally close. - **Don't confuse with the per-phase reviews inside the loop.** ARCHITECT, SPECIFY, LITERATURE, and IMPLEMENT each have their own fresh-context review discipline that happens by iteration boundary. Those are unrelated to this close-out — same word, different jobs. The phase boundary makes them unambiguous: per-phase reviews live inside their host phase's reference; this one is the post-loop close-out in the user's main session. - **Open-question resolutions are durable.** Append to `open-questions.md`'s `## Resolutions` section so the next re-run / future session sees what was decided. Do not delete the original questions. diff --git a/claude/lightcone/skills/lc-from-paper/references/run.md b/claude/lightcone/skills/lc-from-paper/references/run.md index bbc1bba0..97210637 100644 --- a/claude/lightcone/skills/lc-from-paper/references/run.md +++ b/claude/lightcone/skills/lc-from-paper/references/run.md @@ -29,11 +29,11 @@ Check status: lc status --universe baseline ``` -Status states are `ok` (materialized), `pending` (has recipe, not run), `no_recipe` (declared, no recipe — bug). Every output declared in `astra.yaml` must reach `ok`. +Status states are `ok` (materialized, current), `missing` (not yet materialized), `stale` (materialized, but the spec has drifted since — recipe, container, or decisions changed; needs a re-run), `alias` (a re-export of another output; resolves through its source). Every output declared in `astra.yaml` must reach `ok`. If outputs fail: -1. **Read the script's error.** `results///.log` (or wherever the runner emits stderr) usually has the message. +1. **Read the script's error.** `lc run` re-emits each recipe's stdout / stderr on its own terminal output, and on failure prints a summary pointing at the saved snakemake stderr log. There is no per-output log file on disk — the run's terminal output is the record. 2. **Diagnose.** Common failures: missing data dependency (a referenced URL changed; the data archive moved), missing Python package (`requirements.txt` was incomplete), spec / script mismatch (the recipe's `inputs:` does not match what the script reads). 3. **Fix.** Edit the script or `requirements.txt` or the spec, whichever applies. 4. **Re-run.** `lc run --universe baseline` resumes from where things failed; it does not re-execute already-materialized outputs. @@ -52,6 +52,6 @@ If outputs fail: ## Notes -- The runner backend (Docker / local / SLURM) comes from the project's target configuration — `~/.lightcone/config.yaml` and `.lightcone/lightcone.yaml`. RUN does not need to choose; the runner picks based on config. -- For long-running computations, the script's stdout / stderr stream into the result directory's log file. The iteration should use the Monitor tool on the log file to stream events (each stdout line surfaces as a notification), not poll `lc status` repeatedly. For one-shot waits, Bash with `run_in_background` notifies on completion. +- The runner backend (Docker / local / SLURM) comes from the project's target configuration — `~/.lightcone/config.yaml` and `.lightcone/lightcone.yaml`. RUN does not need to choose; the runner picks based on config. One caveat: on a JupyterHub / Dask-Gateway deployment, `lc run` requires every output in a run to resolve to a single container image — specs declaring several distinct containers are rejected there; consolidate on one Containerfile. +- For long-running computations, run `lc run` under Bash with `run_in_background` — recipe stdout / stderr surfaces on the `lc run` process's own output as rules finish (there is no per-output log file to monitor), and the background run notifies on completion instead of the iteration polling `lc status` repeatedly. - **Commit the materialized results' state when RUN settles.** The actual `results/` artifacts are gitignored heavy data, but the run-level outcome (which outputs reached `ok`, any failures logged) is worth a commit so the next iteration can read `git log` to know RUN landed. diff --git a/claude/lightcone/skills/lc-from-paper/references/specify.md b/claude/lightcone/skills/lc-from-paper/references/specify.md index aa715e16..9101a158 100644 --- a/claude/lightcone/skills/lc-from-paper/references/specify.md +++ b/claude/lightcone/skills/lc-from-paper/references/specify.md @@ -79,7 +79,7 @@ Read the paper's section(s) covering this sub-analysis. Author: # quote: omitted at SPECIFY time — LITERATURE fills the TextQuoteSelector in ``` - Evidence with `doi:` and no `quote:` is structurally valid in 0.0.10 (`quote:` is optional on Evidence); the placeholder passes `astra validate` and waits for LITERATURE to fill the quote. `astra validate --verify-evidence` should only be run after LITERATURE has resolved every placeholder. + Evidence with `doi:` and no `quote:` is structurally valid (`quote:` is optional on Evidence); the placeholder passes `astra validate` and waits for LITERATURE to fill the quote. `astra validate --verify-evidence` should only be run after LITERATURE has resolved every placeholder. When the citation's DOI is unresolved (`citations[].doi: null` — flagged in `extraction_warnings`), the placeholder still needs a `doi:` (Evidence requires exactly one of `doi` or `artifact`). In that case, omit the Evidence entry entirely or fall back to an artifact reference if the gap will be resolved internally — and log the unresolved citation to `open-questions.md` so the user can supply the DOI at REVIEW close-out. Don't pre-emptively fetch the cited paper or guess its content; LITERATURE does that with fresh context per paper. diff --git a/claude/lightcone/skills/lc-new/SKILL.md b/claude/lightcone/skills/lc-new/SKILL.md index 4cac53cc..523c808b 100644 --- a/claude/lightcone/skills/lc-new/SKILL.md +++ b/claude/lightcone/skills/lc-new/SKILL.md @@ -1,7 +1,7 @@ --- name: lc-new description: Use this skill whenever the user starts a new ASTRA analysis from a research question — scoping the question, structuring inputs and outputs, identifying decisions through literature, and landing astra.yaml + project CLAUDE.md. Triggers on verbs (`new`, `start`, `scope`) combined with nouns (`analysis`, `project`, `question`, `research`) — e.g. "new analysis", "start project", "scope research question" — even if the user doesn't say "project" explicitly. Don't use this for working inside an existing ASTRA project; this is for fresh scoping only. -allowed-tools: Read, Write(astra.yaml), Write(universes/*), Write(CLAUDE.md), Edit(astra.yaml), Edit(universes/*), Edit(CLAUDE.md), Glob, Grep, Bash(astra:*), Bash(lc:*), WebSearch, WebFetch, AskUserQuestion, Agent +allowed-tools: Read, Write(astra.yaml), Write(universes/*), Write(CLAUDE.md), Edit(astra.yaml), Edit(universes/*), Edit(CLAUDE.md), Edit(index.md), Glob, Grep, Bash(astra:*), Bash(lc:*), WebSearch, WebFetch, AskUserQuestion, Agent --- # /lc-new @@ -70,7 +70,7 @@ Spawn all in a single message (parallel). Show progress as results come in: ○ Wu & He 2018 (reading...) ``` -Write extracted prior insights to astra.yaml immediately. Synthesize them by topic for the user. +Write extracted prior insights to astra.yaml immediately. The extractor's `decision_links:` block is not an ASTRA field — translate it into `insights: [...]` lists on the corresponding decision options when merging (Option.insights in `/astra`); never copy `decision_links` into astra.yaml. Synthesize the insights by topic for the user. ### Decision Identification @@ -121,7 +121,11 @@ Replace the TODO `description:` in `astra.yaml` with a short one-or-two-paragrap ### Populate CLAUDE.md -Read the existing `CLAUDE.md` (created by `lc init`). Fill the `## Project Notes` section per the inline guidance there — context from the conversation that's not in `astra.yaml` and would be lost after `/clear`. The spec is the source of truth for structure, decisions, and evidence. +Read the existing `CLAUDE.md` (created by `lc init` — an orientation intro, an `lc run`/`lc status`/`lc verify` quick reference, and a `## Report` section). Append a `## Project Notes` section capturing the scoping outcome — context from the conversation that's not in `astra.yaml` and would be lost after `/clear` — and update the intro's "has not been scoped yet" framing now that the spec is real. The spec is the source of truth for structure, decisions, and evidence. + +### Populate Report + +`lc init` scaffolds `index.md` with references to the boilerplate `astra.yaml` elements — `decisions.example_method` (an inline `{astra}` mention and a `:::{astra} decisions.example_method\n:::` block) and `outputs.main_result` (an `{astra:value}` mention). Those IDs no longer exist once the boilerplate is replaced, which breaks the MyST build. Swap them for one representative top-level decision ID and one representative top-level output ID from the finalized spec. Only fix the reference IDs — leave the surrounding TODO narrative (Introduction/Methods/Results prose) for the user to fill in. ### Review with User @@ -154,7 +158,7 @@ Then tell the user the spec is ready and they can begin implementation. Recommen You MUST NOT write Python, R, or other implementation code. -You MUST ONLY create/modify: `astra.yaml`, `universes/*.yaml`, `CLAUDE.md` (Finalize only). +You MUST ONLY create/modify: `astra.yaml`, `universes/*.yaml`, `CLAUDE.md` (Finalize only), `index.md` (Finalize only, reference IDs only). You MUST NOT fabricate quotes -- all evidence must pass `astra validate --verify-evidence`. diff --git a/claude/lightcone/skills/ralph/SKILL.md b/claude/lightcone/skills/ralph/SKILL.md index f65013f2..51f2a50a 100644 --- a/claude/lightcone/skills/ralph/SKILL.md +++ b/claude/lightcone/skills/ralph/SKILL.md @@ -49,7 +49,7 @@ For deeper voice / section guidance and the discipline that keeps a constitution --- ``` - That's what the launcher checks; it refuses to start otherwise. + That's what the launcher checks; it refuses to start otherwise. (`status: open` is accepted too; `active` is the convention.) 3. **Refine** — Show the draft, get feedback, revise. Use `AskUserQuestion` for structured choices. Apply the qualitative ambiguity self-check from [`references/crafting.md`](references/crafting.md) — goal, constraints, success — before launching. Reach for the crafting rhythm and stances when the conversation has careful-thinking character; skip when it doesn't. diff --git a/claude/lightcone/skills/ralph/references/constitution.md b/claude/lightcone/skills/ralph/references/constitution.md index 39eb28b5..c93bdb18 100644 --- a/claude/lightcone/skills/ralph/references/constitution.md +++ b/claude/lightcone/skills/ralph/references/constitution.md @@ -39,7 +39,7 @@ Read relevant files, understand existing patterns. This informs the **constituti ### 2. Draft -Create the constitution as a markdown file with `status: active` in YAML frontmatter (that's what the launcher checks). Some workflows expect a specific path so a runner picks it up — e.g. `/lc-from-paper` writes `constitution.md` at the reproduction workdir root. Otherwise put it wherever the work lives. The section block in the SKILL's "What goes in a constitution" is your starting shape; fill what fits, drop what doesn't. +Create the constitution as a markdown file with `status: active` in YAML frontmatter (that's what the launcher checks; it accepts `open` or `active`, with `active` as the convention). Some workflows expect a specific path so a runner picks it up — e.g. `/lc-from-paper` writes `constitution.md` at the reproduction workdir root. Otherwise put it wherever the work lives. The section block in the SKILL's "What goes in a constitution" is your starting shape; fill what fits, drop what doesn't. Use the crafting process from [`crafting.md`](crafting.md): diff --git a/docs/api/cli.md b/docs/api/cli.md index d0634b3d..9cd5d2c7 100644 --- a/docs/api/cli.md +++ b/docs/api/cli.md @@ -1,7 +1,10 @@ # lightcone.cli.commands -The Click surface. Defined in `src/lightcone/cli/commands.py`. Six -public commands: `init`, `run`, `status`, `verify`, `build`, `setup`. +The Click surface. Defined in `src/lightcone/cli/commands.py`. Five +commands plus one group: `init`, `run`, `status`, `verify`, `build`, and +`export` (with `export wrroc` under it). `lc eval` is registered at +import time from `lightcone.eval.cli` and silently skipped when the +optional `eval` extra isn't installed. The user-facing reference is in [CLI Overview](../cli/index.md). This page is a tour of the module internals. @@ -14,12 +17,14 @@ page is a tour of the module internals. @click.pass_context def main(ctx: click.Context) -> None: ctx.ensure_object(dict) - if ctx.invoked_subcommand in ("setup", "init", "eval"): - return - if not _config_path().exists(): - # print friendly error, sys.exit(1) + _ensure_global_config() ``` +`_ensure_global_config()` runs on **every** invocation (no +`invoked_subcommand` gate): if `~/.lightcone/config.yaml` is missing it +is created with `container: {runtime: auto}`. Nothing errors out on a +first run. + `main` is exposed as `lightcone.cli.main` (re-exported from `lightcone.cli.__init__`) and is the entry point declared in `pyproject.toml::project.scripts`: @@ -40,18 +45,21 @@ The constant lives at module top so tests and external tools can read it directly. To add a new tier, edit this dict and update the `click.Choice` on `lc init`. +`recommended` splits scratch paths into an `ask` list (prompt before +writing — `//scratch/**`, `//pscratch/**`) and a `deny` list for things +that are never acceptable (`~/.ssh`, `sudo`, `rm -rf`, `git push`). + ## Helpers ### `_config_path() → Path` -Returns `~/.lightcone/config.yaml`. Used by the `main` group's -auto-init check and by `setup`. +Returns `~/.lightcone/config.yaml`. Used by `_ensure_global_config()`. ### `_project_root(start: Path | None = None) → Path` Walks up from `start` (or `cwd`) looking for `astra.yaml`. Raises `click.ClickException` if none found. Used by `run`, `status`, `verify`, -`build`. +`build`, and `export wrroc`. ### `_target_for(project: Path, output_id: str, universe: str) → str` @@ -60,11 +68,72 @@ the Snakemake target path that materializes it — specifically the manifest file `results///.lightcone-manifest.json`. Raises `click.ClickException` if the id is unknown or ambiguous. -### `_run_filtered(cmd, *, env)` +### `_abort_on_perlmutter_login() → None` + +Pre-flight on `lc run`. Refuses to run when `NERSC_HOST=perlmutter` and +no `SLURM_JOB_ID` is set — that conjunction unambiguously marks a login +node. Bypassed by `DASK_SCHEDULER_ADDRESS` (matching the external- +scheduler branch of `cluster_for_run`) or by setting +`LIGHTCONE_ALLOW_LOGIN_NODE`. + +### `_build_snakemake_cmd(*, snakefile_path, project, n, rerun_triggers, targets, force, has_outputs) → list[str]` + +Builds the `snakemake` argv. Two non-obvious details it encodes: + +- `--rerun-triggers` uses `nargs=+` in snakemake's argparse, so a `--` + separator is emitted before positional targets or the first target + gets eaten as a trigger value. +- `--shared-fs-usage` lists everything *except* `software-deployment`. + With snakemake's default, spawned job commands embed the driver's + `sys.executable` — a path that doesn't exist inside a Dask Gateway + worker image. Without it, workers invoke plain `python` from their own + environment, which is equally correct on every other backend. + +`--force` is used with explicit targets and `--forceall` without (the +`rule all` target has no recipe of its own). + +### `_run_snakemake(cmd, *, env, scratch_root, verbose) → int` + +Spawn `snakemake` and forward the run's narrative: the executor plugin +prints each finished rule's block prefixed with +`lightcone.engine.runner.SENTINEL`, and those lines — prefix stripped — +go to the terminal while everything else (DAG chatter, job stats) is +dropped. `--verbose` forwards the noise too. -Spawn `snakemake`, line-filter its stdout/stderr to suppress executor -banner chatter, and return the exit code. The recipe's own output -streams through untouched, as do unfamiliar diagnostic lines. +stderr is tailed into a bounded ring buffer (400 lines) on a daemon +thread. On a non-zero exit it is written to +`/snakemake-stderr-.log` and the path is printed — +so a crash leaves a real log behind without polluting a successful run. +Verbose mode passes stderr straight through instead. + +### `_ensure_images(project, *, runtime, force=False) → list[str]` + +Build or pull every container image referenced in `astra.yaml`, and +return the distinct resolved images in declaration order. Idempotent — +present images are skipped unless `force`. No-op returning `[]` when +`runtime == "none"`. + +Three paths per spec: + +- **Prebuilt image** (`python:3.12-slim`, `ghcr.io/...`) → `pull_image`, + so `lc run` can pass `--pull=never`. Skipped entirely on the + `kubernetes` runtime: worker pods pull from the registry themselves. +- **Containerfile on `kubernetes`** → `_cloudbuild_image()`. +- **Containerfile anywhere else** → `compute_image_tag` + + `build_image`. + +Used by `lc build` (which exposes `--force`) and as a `lc run` +pre-flight, so the first run after editing a Containerfile doesn't fail +mid-DAG on a missing image. + +### `_cloudbuild_image(project, spec_str, project_name, force) → str` + +Ensure one Containerfile's image through +[`engine.cloudbuild.ensure_image`](cloudbuild.md), rendering its +`on_progress(phase, detail)` callbacks into a live Rich status line. +Raises a `click.ClickException` with hub-admin guidance when +`cloudbuild_available()` is False (the `kubernetes` runtime has no local +OCI CLI and no other build backend to fall back on). ### `_status_label(s: str) → str` @@ -77,16 +146,53 @@ Map a status literal to the Rich-formatted display label: | `missing` | `[red]✗ miss[/red]` | | `alias` | `[dim]→ alias[/dim]` | -## Boilerplate text - -`_BOILERPLATE_ASTRA`, `_GITIGNORE`, and `_PROJECT_CLAUDE_MD` are -multi-line strings written at `lc init` time. Edit them to change what -new projects look like. +## What `lc run` does, in order + +1. `_abort_on_perlmutter_login()`. +2. [`scratch.prepare_run_dirs`](scratch.md) + `ensure_snakemake_symlink` + — resolve scratch and repoint `/.snakemake`. +3. `container.load_runtime()`, then `_ensure_images()`. +4. `snakefile.generate()`. +5. If [`dask_cluster.gateway_branch_active()`](dask_cluster.md): reject + a spec resolving to more than one distinct image (the worker pod is + the container for every rule, so only one image can be honoured), and + pass the single image on as `worker_image`. +6. Warn when `runtime: auto` silently fell back to `none` while the spec + declares containers — the manifest's `container_image` would + misrepresent what executed. +7. `scratch.acquire_run_lock()`, then `cluster_for_run(...)`, whose + yielded env overlay is merged into the child snakemake's environment. + +## Scaffold constants + +`lc init` writes these module-level strings: + +| Constant | Written to | +|----------|------------| +| `_CONTAINERFILE` | `Containerfile` (`python:3.12-slim`, install requirements, `COPY . .`) | +| `_REQUIREMENTS` | `requirements.txt`, plus `_lightcone_requirement()` | +| `_GITIGNORE_APPEND` | appended to the `.gitignore` `astra init` wrote | +| `_MYST_YML`, `_INDEX_MD_BODY` | `myst.yml` + `index.md`, via `_create_report_template()` | +| `_PROJECT_CLAUDE_MD` | the project's root `CLAUDE.md` | + +`astra.yaml`, `universes/baseline.yaml`, the base `.gitignore` and `src/` +come from `astra init`, which `lc init` calls first; the only edit made +to the generated spec is swapping `container: python:3.12-slim` for +`container: Containerfile`. + +### `_lightcone_requirement() → str` + +The requirements lines that pin `lightcone-cli` into the project image. +The image must be able to execute rules on any backend — including *as* +a Dask Gateway worker pod, where the dask worker and the child snakemake +run inside it — and `lightcone-cli` carries that whole stack. The pin +mirrors the version running `lc init`; dev versions fall back to +unpinned. ## Plugin install `_install_claude_plugin(project_dir, plugin_source, permissions)` copies the bundled plugin into `project_dir/.claude/` (`skills`, `agents`, `scripts`, `guides`, `templates`) and writes `.claude/settings.json` -from the chosen permission tier. Existing subdirectories are removed -before copying. +from the chosen permission tier plus the plugin's own `hooks.json`. +Existing subdirectories are removed before copying. diff --git a/docs/api/container.md b/docs/api/container.md index dad10864..67759e61 100644 --- a/docs/api/container.md +++ b/docs/api/container.md @@ -10,14 +10,24 @@ Source: `src/lightcone/engine/container.py`. | Constant | Value | |----------|-------| -| `RUNTIMES` | `("podman", "docker", "podman-hpc")` — detection priority order | -| `DEPENDENCY_FILES` | `("requirements.txt", "requirements-dev.txt", "requirements-test.txt", "pyproject.toml", "setup.py", "setup.cfg", "poetry.lock", "Pipfile.lock")` | - -Detection priority is podman before docker for two reasons: it's -rootless (less surprising on shared machines), and the docker probe -includes `docker info` so a stopped daemon doesn't silently win over a +| `RUNTIMES` | `("podman-hpc", "podman", "docker")` — detection priority order | +| `KUBERNETES` | `"kubernetes"` — deliberately **not** in `RUNTIMES` | +| `REGISTRY_ENV` | `"LIGHTCONE_REGISTRY"` — deployment-injected registry prefix | +| `DEPENDENCY_FILES` | `requirements{,-dev,-test}.txt`, `pyproject.toml`, `setup.py`, `setup.cfg`, `poetry.lock`, `Pipfile.lock`, `uv.lock`, `conda-lock.yml`, `environment.y{a,}ml` | + +Detection order: `podman-hpc` first, because anyone who installed the HPC +wrapper did so on purpose and plain podman would build images compute +nodes can't read; then podman (rootless, no daemon); docker last, gated +behind a `docker info` probe so a down daemon doesn't silently win over a healthy podman. +`kubernetes` is separate from `RUNTIMES` because it names no binary and +is never found on PATH. It is selected only by site detection (a Dask +Gateway deployment — see [api/site_registry](site_registry.md)) or by an +explicit pin in `~/.lightcone/config.yaml`, and it takes a different code +path throughout: no local build, no recipe wrap, registry refs instead of +local-store tags. + ## Runtime detection ### `detect_runtime() → str | None` @@ -26,6 +36,12 @@ Returns the first usable runtime in `RUNTIMES`. "Usable" means the binary is on PATH and (for docker) `docker info` succeeds. Returns `None` if nothing's available. +The host site's declared `container_runtime` is moved to the front of the +order (`_detection_order()`), but it is only a *hint* — a +missing-from-PATH preference falls through to the next candidate. The one +exception: a site declaring `container_runtime: kubernetes` short-circuits +PATH probing entirely, since there is no binary to find. + ### `load_runtime(*, project_path=None) → RuntimeChoice` Resolve the runtime to use. Reads `container.runtime` from @@ -49,7 +65,7 @@ consulted today. ```python @dataclass(frozen=True) class RuntimeChoice: - runtime: str # docker | podman | podman-hpc | none + runtime: str # docker | podman | podman-hpc | kubernetes | none explicit: bool # True if pinned, False if `auto` produced this ``` @@ -57,24 +73,72 @@ class RuntimeChoice: should warn — that case mismatches the manifest's recorded `container_image` against what actually executed. -## Image tag computation +## Image identity + +There is **one** content-addressed identity, spelled two ways depending +on where the image lives. Both spellings carry the same digest on every +backend, so a `code_version` computed by the Snakefile generator and one +computed by the status walker can never disagree. + +### `image_identity(project_name, containerfile, project_path) → (safe_name, digest)` + +The source of truth. The 12-char sha256 digest covers: + +1. The **Containerfile** contents. +2. Every **dependency file** from `DEPENDENCY_FILES` present at the + project root. +3. The contents of every **`COPY`/`ADD` source** referenced from the + Containerfile — files hashed directly, directories walked recursively + in sorted relative-path order. `COPY .` therefore hashes the whole + build context. + +Each entry is framed with a kind label and its project-relative path +before its bytes, so moving a line between files (or renaming a file) +changes the digest. + +`_parse_copy_sources()` handles backslash continuations and the JSON exec +form (`COPY ["src", "dest"]`), and skips what isn't in the host build +context: `--from=` copies, URLs, and `git@` arguments. Heredoc +`COPY <-`. The hash covers the -Containerfile contents plus every dependency file from `DEPENDENCY_FILES` -that exists at the project root. +The local-image-store spelling: `lc--`. +Sanitization is lowercase + spaces → hyphens. + +### `registry_image_ref(project_name, containerfile, project_path, *, registry) → str` + +The registry spelling: `/lc-:` — the digest moves +into the tag position so one repository per project accumulates its image +history. -Sanitization: lowercase + spaces → hyphens. +### `deployment_registry() → str | None` + +The deployment-injected `LIGHTCONE_REGISTRY` prefix (trailing slash +stripped), or `None` off-deployment. + +### `runtime_registry(runtime) → str | None` + +Which spelling a given runtime resolves identities against: the +deployment registry on `kubernetes`, `None` (local-store tags) +everywhere else. Shared by the Snakefile generator and the status walker. ### `find_dependency_files(project_path) → list[Path]` -Sorted list of dependency files actually present. Used by -`compute_image_tag`. +Sorted list of dependency files actually present. ### `hash_file_contents(files) → str` -Concatenated SHA-256 hex digest of the listed files. Internal helper. +Framed SHA-256 digest over a list of files. A standalone helper — the +image digest uses the richer `image_identity()` path above. ### `is_containerfile(spec, project_path) → bool` @@ -85,9 +149,16 @@ not a registry image). ### `build_image(tag, containerfile, context, *, runtime, build_args=None) → ContainerBuildResult` -Run ` build -t -f [--build-arg …] `. -For `podman-hpc`, also runs `podman-hpc migrate ` so compute nodes -can read the image. Raises `ContainerBuildError` on any failure. +Run ` build -t -f [--build-arg …] ` +against a **staged** context (`_populate_build_context()` mirrors the +Containerfile and its `COPY`/`ADD` sources into a tempdir — NERSC's DVS +mounts don't implement `llistxattr`, which buildah calls unconditionally +on every `COPY` source). For `podman-hpc`, also runs +`podman-hpc migrate ` so compute nodes can read the image. + +Raises `ContainerBuildError` on any failure, and immediately for a +`runtime` outside `RUNTIMES` — `kubernetes` has no local builder and goes +through [`engine.cloudbuild`](cloudbuild.md) instead. ### `pull_image(image, *, runtime) → None` @@ -140,12 +211,13 @@ Snakemake placeholders inside `recipe` (`{output[0]}`, `{input.X}`, `{wildcards.universe}`) are preserved — they substitute through Python's `str.format` at execution time, after wrapping. -### `make_image_tag_resolver(project_path, project_name) → Callable` +### `make_image_tag_resolver(project_path, project_name, *, registry=None) → Callable` Returns a memoizing wrapper around `resolve_image_for_run`. Multiple -outputs typically share a Containerfile; resolving re-hashes the file -plus all dependency files (lockfiles can be megabytes), so caching by -spec string for the lifetime of the caller's loop matters. +outputs typically share a Containerfile; resolving re-hashes the file, +all dependency files (lockfiles can be megabytes), and every `COPY` +source, so caching by spec string for the lifetime of the caller's loop +matters. Pass `registry=runtime_registry(runtime)` to get registry refs. ### `resolve_image_for_run(spec, *, project_path, project_name, registry=None) → str | None` @@ -164,7 +236,10 @@ runtime will execute: ### `get_container_status(spec, project_path, project_name, *, runtime) → ContainerStatus` Without building or pulling, return a `ContainerStatus` describing what -would happen. +would happen. On the `kubernetes` runtime with a deployment registry set, +`exists` comes from +[`cloudbuild.registry_image_exists()`](cloudbuild.md) — a registry HEAD +rather than a local image-store probe — and `image` is the registry ref. ### `ContainerStatus` (dataclass) diff --git a/docs/api/dask_cluster.md b/docs/api/dask_cluster.md index be2aebd2..1f56cb13 100644 --- a/docs/api/dask_cluster.md +++ b/docs/api/dask_cluster.md @@ -5,7 +5,7 @@ four branches, no service to manage. Source: `src/lightcone/engine/dask_cluster.py`. -## `cluster_for_run(*, verbose=False, worker_image=None, max_workers=None) → Iterator[dict[str, str]]` +## `cluster_for_run(*, verbose=False, local_directory=None, worker_image=None, max_workers=None) → Iterator[dict[str, str]]` Yields the env overlay the child snakemake needs to reach the cluster (the executor plugin lives in a different process, so connection info @@ -46,6 +46,29 @@ unschedulable pool), and workers that don't advertise the `cpus`/`memory` resource contract (a deployment that doesn't expose the `environment` option). +`local_directory` is where dask workers stage spilled task data; `lc run` +resolves it under [`engine.scratch`](scratch.md) so on NERSC the spill +lands on Lustre rather than DVS-mounted home/CFS. Ignored by the Gateway +branch, whose workers are pods. + +## `gateway_branch_active() → bool` + +Would `cluster_for_run` take the Gateway branch right now? A pure +function of the environment, in the same priority order as the branches +(`DASK_SCHEDULER_ADDRESS` wins, then `DASK_GATEWAY__ADDRESS`). Exposed so +`lc run` can shape its invocation — notably the one-image-per-run check — +*before* entering the cluster context. + +## Environment contracts + +| Env var | Direction | Meaning | +|---------|-----------|---------| +| `DASK_SCHEDULER_ADDRESS` | in / out | An existing scheduler to attach to; also the overlay yielded by the three address-based branches. | +| `DASK_GATEWAY__ADDRESS` | in | Selects the Gateway branch (injected by the deployment, along with the rest of `DASK_GATEWAY__*` that configures `Gateway()`). | +| `LIGHTCONE_GATEWAY_CLUSTER` | out | `GATEWAY_CLUSTER_ENV` — the created cluster's name, for the executor plugin to rejoin by. Internal parent→child rendezvous, not a user knob. | +| `LIGHTCONE_GATEWAY_WORKER_TIMEOUT` | in | `GATEWAY_WORKER_TIMEOUT_ENV` — seconds to wait for the first worker (default 600; a first-time image pull is minutes). | +| `LIGHTCONE_WORKER_IMAGE` | out | Provisioned into scheduler/worker pods; recorded by the manifest layer as execution ground truth. | + ## Resource keys These string constants form a contract with the executor plugin: @@ -99,6 +122,6 @@ and keeps everything in one process tree. ## Tests -`tests/test_dask_cluster.py` covers the three branches and the +`tests/test_dask_cluster.py` covers the four branches and the resource-advertising contract. The SLURM branch is tested with mocked `subprocess.Popen` plus a stubbed `Client.wait_for_workers`. diff --git a/docs/api/index.md b/docs/api/index.md index 491843b1..9dc3a975 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -7,17 +7,21 @@ is a thin Click wrapper around these modules. | Module | Role | |--------|------| -| [`lightcone.cli.commands`](cli.md) | Click CLI: `init`, `run`, `build`, `status`, `verify`, `setup`. | +| [`lightcone.cli.commands`](cli.md) | Click CLI: `init`, `run`, `status`, `verify`, `build`, `export`, plus `eval` when the extra is installed. | | [`lightcone.engine.manifest`](manifest.md) | Per-output `.lightcone-manifest.json` write/read; `code_version`, `sha256_dir`. The integrity layer. | | [`lightcone.engine.snakefile`](snakefile.md) | Generate `.lightcone/Snakefile` and `snakefile-config.json` from `astra.yaml`. | -| [`lightcone.engine.container`](container.md) | Runtime detection, content-addressed image tags, `wrap_recipe`. | -| [`lightcone.engine.dask_cluster`](dask_cluster.md) | Cluster lifecycle for `lc run` (local / SLURM / external). | +| [`lightcone.engine.runner`](runner.md) | `run_rule` — the body of every generated rule; the `SENTINEL` output protocol. | +| [`lightcone.engine.container`](container.md) | Runtime detection, content-addressed image identity, `wrap_recipe`. | +| [`lightcone.engine.cloudbuild`](cloudbuild.md) | GCP Cloud Build backend — remote image builds where no OCI runtime exists. | +| [`lightcone.engine.dask_cluster`](dask_cluster.md) | Cluster lifecycle for `lc run` (external scheduler / Dask Gateway / SLURM / local). | +| [`lightcone.engine.scratch`](scratch.md) | Resolve the scratch root; per-run dirs, the run lock, the `.snakemake` symlink. | | [`lightcone.engine.status`](status.md) | Manifest-driven status walker. | | [`lightcone.engine.verify`](verify.md) | Recompute hashes; walk the input chain. | | [`lightcone.engine.tree`](tree.md) | Sub-analysis tree helpers — outputs, decisions, `from:` resolution. | | [`lightcone.engine.validation`](validation.md) | Post-recipe sanity checks (empty dir, all-NaN columns, …). | +| [`lightcone.engine.wrroc`](wrroc.md) | Workflow Run RO-Crate exporter behind `lc export wrroc`. | +| [`lightcone.engine.site_registry`](site_registry.md) | Known-site defaults; `detect_current_site()` drives scratch and runtime selection. | | [`snakemake_executor_plugin_dask`](dask_executor.md) | Snakemake executor plugin → `dask.distributed`. | -| `lightcone.engine.site_registry` | Vestigial — no active code path imports it. See [api/site_registry](site_registry.md). | ## Common entry points @@ -55,7 +59,7 @@ from lightcone.engine.container import ( build_image, ) -runtime = detect_runtime() # 'podman' / 'docker' / 'podman-hpc' / None +runtime = detect_runtime() # 'podman-hpc' / 'podman' / 'docker' / 'kubernetes' / None tag = compute_image_tag("my-project", Path("Containerfile"), Path(".")) build_image(tag, Path("Containerfile"), Path("."), runtime=runtime) ``` diff --git a/docs/api/manifest.md b/docs/api/manifest.md index 345bc451..2b726329 100644 --- a/docs/api/manifest.md +++ b/docs/api/manifest.md @@ -75,6 +75,20 @@ Required keys in `cfg`: - `code_version` - `git_sha`, `lc_version` +Additive fields the function fills in itself, from the environment rather +than from `cfg`: + +- `host`, `finished_at`, `slurm_job_id` (`SLURM_JOB_ID`). +- `worker_image` — `LIGHTCONE_WORKER_IMAGE`, provisioned by `lc run` into + every scheduler/worker pod on a Dask Gateway deployment with the image + the cluster was actually started from. `container_image` is what the + spec *declared*; this is the pod-reported ground truth of what + executed. `None` on every other backend. +- `git_remote` — the origin URL at materialization time, surfaced by + `lc export wrroc` as a `CodeRepository`. + +Older manifests without these fields still parse. + `inputs` is a `dict[str, Path]` mapping declared input id → filesystem path. For each input, the function reads the upstream manifest if present and records its `data_version`; otherwise falls back to @@ -94,10 +108,12 @@ the rule. "code_version": "sha256:…", "data_version": "sha256:…", "container_image": "lc-myproject-abc123", + "worker_image": null, // pod image on a Dask Gateway run; null elsewhere "recipe": "python scripts/eval.py", "decisions": { "scaling": "standard", "use_pca": "no" }, "input_versions": { "features": "sha256:…", "labels": "mtime-size:…-…" }, "git_sha": "...", + "git_remote": "https://github.com/org/myproject.git", "lc_version": "0.4.0", "host": "saul01", "slurm_job_id": "1234567", diff --git a/docs/api/runner.md b/docs/api/runner.md index 8efc1cc6..9c03951a 100644 --- a/docs/api/runner.md +++ b/docs/api/runner.md @@ -1,14 +1,69 @@ -# lightcone.engine.runner (removed) +# lightcone.engine.runner -The pluggable runner (`docker`, `venv`, `local`, `slurm`) was replaced by -two thinner pieces: +The per-rule execution helper invoked from the generated Snakefile. Each +rule's `run:` block boils down to one call to `run_rule()`. -- The Snakefile generator at [engine/snakefile](snakefile.md) wraps each - recipe in a ` run --rm ...` invocation at generation time - (or leaves it bare when no container is configured). -- The Dask cluster manager at [engine/dask_cluster](dask_cluster.md) - decides whether the run is local, SLURM-backed via `srun`, or attached - to an external scheduler. +Source: `src/lightcone/engine/runner.py`. -There is no longer a single "backend" abstraction — those two -modules together cover what the runner used to do. +!!! note "Not the old pluggable runner" + + An earlier module of this name was a backend abstraction (`docker`, + `venv`, `local`, `slurm`). That is gone: containers are resolved at + Snakefile-generation time by [engine/container](container.md), and + *where* a rule runs is decided by + [engine/dask_cluster](dask_cluster.md). Today's `runner` is the thin + body of a generated rule. + +## `SENTINEL` + +```python +SENTINEL = "__LCSTREAM__::" +``` + +Every line the runner emits carries this prefix. It is the entire +mechanism behind `lc run`'s clean narrative output: the dask executor's +`_run_shell` keeps sentinel-prefixed lines from the worker subprocess and +drops everything else — snakemake bootstrap, dask logs, stray prints — +so nothing has to be filtered against a moving target of upstream log +strings. `_run_snakemake` in the CLI strips the prefix on the way to the +terminal. + +Chosen to be vanishingly unlikely in real recipe output: printable +ASCII, column-0 anchored, distinctive, and short enough that capture cost +stays negligible. + +## `run_rule(*, rule_key, universe, output_dir, inputs, cfg) → None` + +In order: + +1. Emit a `▶ []` header. +2. Run `cfg["shell_command"]` — already template-substituted and + container-wrapped by [engine/snakefile](snakefile.md) — with stdout + and stderr captured. +3. Re-emit the recipe's captured output, indented, stdout then stderr. + (Snakemake's own capture interleaves the same way.) +4. On **non-zero exit**: emit a `✗` trailer carrying the exit code and + raise `subprocess.CalledProcessError`, so Snakemake records the job as + failed and halts the DAG. The manifest is **not** written — `lc + verify` must never see a manifest pointing at incomplete data. +5. On success: [`write_manifest`](manifest.md), then + [`validate_output`](validation.md), emitting any warnings as `⚠` + lines. +6. Emit a `✓ [] ` trailer. + +`inputs` is keyed by the *raw* declared input id (`sub.real`), not the +Snakemake-safe identifier (`sub__real`) — the raw ids are what +`write_manifest` records in `input_versions` and what `lc verify` walks. + +## `_emit(line="")` (private) + +Writes one sentinel-prefixed line to stdout and **flushes**. The flush +matters: the runner executes inside a child snakemake subprocess whose +stdout is captured by the worker's `_run_shell`, and without it Python +may block-buffer the recipe output until after the rule's trailer. + +## Public surface + +```python +__all__ = ["SENTINEL", "run_rule"] +``` diff --git a/docs/api/scratch.md b/docs/api/scratch.md new file mode 100644 index 00000000..3fa16806 --- /dev/null +++ b/docs/api/scratch.md @@ -0,0 +1,113 @@ +# lightcone.engine.scratch + +One concept: **where lightcone keeps its operational state** — Snakemake +metadata, Dask worker spill, the run-exclusion lock. Resolved at the +start of every `lc run`. + +Source: `src/lightcone/engine/scratch.py`. + +## Why it exists + +On NERSC, `$HOME` and `/global/cfs` are mounted on compute nodes via Cray +DVS, which [does not support file +locking](https://docs.nersc.gov/performance/io/dvs/). Snakemake's +workflow lock, our run-exclusion lock, and any future coordination +primitive silently no-op there, and small-file I/O is slow. `$SCRATCH` is +Lustre, which works correctly. On a JupyterHub deployment the problem is +different but the answer is the same shape: a pod's `/tmp` is pod-local, +so the state has to live on the NFS `$HOME` every worker pod mounts. + +## `resolve_scratch_root(project_path) → Path` + +First hit wins: + +1. **`LIGHTCONE_SCRATCH`** env var (`LIGHTCONE_SCRATCH_ENV`) — the escape + hatch / CI override. +2. **`scratch_root`** in `/.lightcone/lightcone.yaml` — the + per-project pin, written by `lc init --scratch`. An unparseable YAML + file is treated as empty rather than raising. +3. **`scratch_root`** from the detected site + ([`site_registry.detect_current_site()`](site_registry.md)), stored as + a shell expression (`$SCRATCH`, `$HOME`) and expanded with + `os.path.expandvars`. A `$VAR` that survives expansion means the env + wasn't set — that falls through rather than writing to a literal path + named `$SCRATCH`. +4. **`tempfile.gettempdir()`** — single-node fallback. + +Always returns a `Path`; never raises. The path itself is not created — +use `prepare_run_dirs()` for that. + +## `prepare_run_dirs(project_path, *, run_id=None) → RunDirs` + +Creates and returns the per-run sub-directories. `run_id` defaults to the +current PID — unique per `lc run`, easily mapped back to a process. + +```python +@dataclass(frozen=True) +class RunDirs: + root: Path # /.lightcone + snakemake_state: Path # /.lightcone/snakemake//.snakemake + dask_local: Path # /.lightcone/dask/ + run_lock_path: Path # /.lightcone/locks/.run-lock +``` + +Snakemake state is **project-scoped** (persistent across invocations); +the dask-local dir and the lock are **run-scoped**. The lockfile is +touched on creation so `flock` never races on `O_CREAT`. + +### `project_hash(project_path) → str` + +12 hex chars of `sha256(absolute project path)`. Namespaces state under a +shared scratch root: two projects on one `$SCRATCH` get separate +`.snakemake/` dirs, and the same project moved to a different machine +gets fresh state. + +## `ensure_snakemake_symlink(project_path, snakemake_state) → None` + +Repoints `/.snakemake` at `snakemake_state`, so Snakemake finds +its state at the canonical path while the bytes live on a filesystem that +honours `flock`. + +- Symlink already pointing there → no-op. +- Symlink pointing elsewhere → replaced. +- A **real** directory (from a prior direct `snakemake` invocation) → + renamed to `.snakemake.legacy` rather than deleted. Losing somebody + else's job metadata silently is a worse failure than leaving a backup. + If a backup already exists it is kept, and the current directory is + removed instead. + +## `acquire_run_lock(rundirs)` (context manager) + +Holds an exclusive, **non-blocking** `flock` on +`/.lightcone/locks/.run-lock` for the duration of +the run. A concurrent `lc run` on the same project raises +`RunLockBusyError` rather than queueing silently; `lc run` surfaces it as +a `ClickException`. + +The kernel releases the lock when the holding process exits — clean +shutdown, crash, or SIGKILL — so a previously killed run can't deadlock +the next one. Once the lock is held, any leftover sentinel files under +`/locks/` are cleared: those aren't tied to a process +and would otherwise refuse the next workflow start. Safe to do here +precisely because we've just proven we're alone on the project. + +### `RunLockBusyError` + +Subclass of `RuntimeError`. Message names the lockfile path. + +## Tests + +`tests/test_scratch.py` walks the resolution chain (env var wins, +project config, site default including the fall-through when `$SCRATCH` +isn't set, tempdir fallback), the `RunDirs` layout and its per-project +separation, the three `ensure_snakemake_symlink` cases including the +`.snakemake.legacy` backup, and the run lock — stale-lock clearing and +rejection of a concurrent holder. + +## Where it's called + +`lc run` (`cli/commands.py`) resolves scratch before anything else: +`prepare_run_dirs()` → `ensure_snakemake_symlink()` → … → +`acquire_run_lock()` around the cluster context. `rundirs.dask_local` +becomes `cluster_for_run(local_directory=...)`, and `rundirs.root` is +where a failed run's snakemake stderr log is written. diff --git a/docs/api/site_registry.md b/docs/api/site_registry.md index 09429604..a0c7c381 100644 --- a/docs/api/site_registry.md +++ b/docs/api/site_registry.md @@ -1,48 +1,114 @@ # lightcone.engine.site_registry -> **Status: orphaned.** Nothing in the active code path imports this -> module. It's residue from the (now-removed) target system. Keep -> reading if you want to know what's still there; otherwise skip to -> [api/container](container.md) and [api/dask_cluster](dask_cluster.md) -> for what actually drives execution today. +Known-site defaults. One question — *which site are we on?* — answered in +one place, so "the site asks for X" features don't each re-derive it. Source: `src/lightcone/engine/site_registry.py`. -## What's still in the file +## Who calls it -The module exposes: +| Caller | What it reads | +|--------|---------------| +| `lc init` (`cli/commands.py`) | `display_name` + `scratch_root`, to surface where `lc run` will keep its state | +| [`engine.scratch.resolve_scratch_root`](scratch.md) | `scratch_root`, as the third step of the resolution chain | +| `engine.container._site_preferred_runtime` | `container_runtime`, moved to the front of the detection order — and the only way `kubernetes` is ever selected automatically | -- `SITE_DEFAULTS` — a dict mapping site keys (`"perlmutter"`, `"local"`) - to a structured defaults dict (display name, hostname patterns, - suggested QoS / constraint / time-limit options, scratch deny paths, - container runtime). -- `detect_site(hostname_or_name) → str | None` -- `get_site_defaults(site_key) → dict | None` -- `list_known_sites() → list[tuple[str, str]]` -- `get_site_scratch_deny_rules(site_key) → list[str]` +## `SITE_DEFAULTS` -The functions still work on their own; they just don't have a caller -inside lightcone-cli right now. +A dict of site key → declared defaults. Recognised keys: -## What's actually used today +| Key | Meaning | +|-----|---------| +| `hostname_patterns` | Substrings matched against the hostname by `detect_site()`. | +| `env_markers` | Env var names that must *all* be set for `detect_site_from_env()` to match. | +| `display_name` | Human-readable name (`HostSite.display_name`). | +| `backend` | `slurm` / `kubernetes` / `local`. Sites with `backend: local` are skipped by hostname detection. | +| `container_runtime` | The runtime `engine.container` prefers here. | +| `scratch_root` | Shell expression (`$SCRATCH`, `$HOME`) expanded at run time. | +| `scratch_paths` | Shared filesystems the agent shouldn't edit; `get_site_scratch_deny_rules()` turns them into `Edit(...)` rules. | +| `suggested_options`, `cache_key_overrides`, `connection` | Perlmutter scheduler detail. | -The Perlmutter scratch deny rules used to be merged into -`.claude/settings.json` automatically when a non-local target was -configured. With the target system gone, the equivalent rules are -hard-coded inline in `PERMISSION_TIERS` (see -`src/lightcone/cli/commands.py`): +Three sites ship today: + +- **`perlmutter`** — matched by hostname (`perlmutter`, `saul`). + `container_runtime: podman-hpc`, `scratch_root: $SCRATCH` (NERSC's + `$HOME` and CFS are DVS-mounted and silently swallow `flock`). +- **`jupyterhub`** — a JupyterHub deployment with Dask Gateway. Pod + hostnames are meaningless, so it declares **no** hostname patterns and + is matched by `env_markers: ["DASK_GATEWAY__ADDRESS"]` instead. + `container_runtime: kubernetes` (the worker pod *is* the container — + see [api/container](container.md)); `scratch_root: $HOME`, because the + `.snakemake` state the driver symlinks into scratch has to live on the + NFS home every worker pod mounts, not in a pod-local `/tmp`. +- **`local`** — the fallback; no detection, no declared runtime. + +## `detect_current_site() → HostSite` + +The entry point for the rest of the codebase. **Env markers win over +hostname patterns** — a pod's hostname is noise, the injected env is the +signal: + +```python +key = detect_site_from_env() or detect_site(socket.gethostname()) +``` + +Returns a falsy `HostSite` (`key is None`, empty defaults) when nothing +matches — it never raises, so callers can treat it as an optional lookup. + +### `HostSite` (frozen dataclass) + +```python +@dataclass(frozen=True) +class HostSite: + key: str | None + defaults: Mapping[str, Any] + + def __bool__(self) -> bool: ... # True when a site matched + @property + def display_name(self) -> str: ... # declared name, else key, else "unknown" + def get(self, name, default=None): ... # read a declared field +``` + +Typical use: ```python -"Edit(//scratch/**)", -"Edit(//pscratch/**)", +from lightcone.engine.site_registry import detect_current_site + +site = detect_current_site() +if site: + print(site.display_name, site.get("scratch_root")) ``` -If you want richer per-site rules without rebuilding the target -system, point `lc init`'s `_install_claude_plugin` at -`get_site_scratch_deny_rules(detect_site(socket.gethostname()))` and -merge the result into the deny list. Two lines of code. +## Other functions + +### `detect_site(hostname_or_name) → str | None` + +Substring match of a hostname (or a site name typed by a user) against +each site's key and `hostname_patterns`. Sites with `backend: local` are +skipped, so a machine literally named `local` doesn't match. + +### `detect_site_from_env() → str | None` + +Returns the first site whose declared `env_markers` are *all* present and +non-empty in the environment. Sites with no markers never match here. + +### `get_site_defaults(site_key) → dict | None` + +Raw defaults dict for a key. + +### `list_known_sites() → list[tuple[str, str]]` + +`(site_key, display_name)` for every entry. + +### `get_site_scratch_deny_rules(site_key) → list[str]` + +`Edit()` rules built from the site's `scratch_paths`. Not wired +into `lc init` today — the equivalent patterns are listed under `ask` in +`PERMISSION_TIERS` (`src/lightcone/cli/commands.py`). -## Recommendation +## Adding a site -Either delete this module (no callers) or revive it for the use case -above. Leaving it as-is encourages the drift this audit is fighting. +Append an entry to `SITE_DEFAULTS`. Declare `hostname_patterns` for a +machine you can recognise by name, or `env_markers` for a deployment you +can only recognise by the environment it injects. See +[Adding an HPC Site](../contributing/hpc-sites.md). diff --git a/docs/api/snakefile.md b/docs/api/snakefile.md index ed8f3b43..3f87f643 100644 --- a/docs/api/snakefile.md +++ b/docs/api/snakefile.md @@ -21,10 +21,13 @@ Reads `astra.yaml`, resolves the analysis tree, and writes: Returns the two paths. -`runtime` is one of `docker | podman | podman-hpc | none` and is used to -wrap each recipe at generation time (see -[engine.container.wrap_recipe](container.md#wrap_recipe)). Resolution is -done once here, not per rule, so all rules use a consistent runtime. +`runtime` is one of `docker | podman | podman-hpc | kubernetes | none` +and is used to wrap each recipe at generation time (see +[engine/container](container.md); on `kubernetes` the wrap is a +passthrough — the worker pod is the container). It also selects the image +spelling via `runtime_registry()`: local-store tags, or registry refs on +a deployment. Resolution is done once here, not per rule, so all rules +use a consistent runtime. ## `discover_universes(project_path) → list[str]` @@ -45,17 +48,27 @@ rule : params: cfg=lambda wc: CFG[""][wc.universe], run: - shell('printf "▶ [%s]\\n" "{wildcards.universe}" >&2') - shell(params.cfg["shell_command"]) - write_manifest( + run_rule( + rule_key="", + universe=wildcards.universe, output_dir=Path(output.data), inputs={"": Path(input.), ...}, - cfg=params.cfg, + cfg=dict(params.cfg), ) - for _w in validate_output(Path(output.data), params.cfg.get("output_type"), params.cfg["output_id"]): - print(f"\033[33m⚠\033[0m {_w}", file=sys.stderr) ``` +The body is a thin call into [`engine.runner.run_rule`](runner.md), +which runs the pre-rendered shell command, emits the sentinel-framed +narrative lines, writes the manifest on success, and runs the validation +hook. Keeping the logic in an importable module rather than inline in the +generated file means it is testable and the generated Snakefile stays +readable. + +The `inputs` dict literal is what bridges the two spellings of an input +id: Snakemake's input directive needs an identifier (`sub__real`), while +`write_manifest`'s `input_versions` — and `lc verify`'s chain walk — are +keyed by the raw declared id (`sub.real`). + ## `cfg` content Per-`(rule_key, universe)` entry written into diff --git a/docs/api/targets.md b/docs/api/targets.md deleted file mode 100644 index 4f29d61b..00000000 --- a/docs/api/targets.md +++ /dev/null @@ -1,11 +0,0 @@ -# lightcone.engine.targets (removed) - -The target configuration module is gone. The only remaining global config -is `~/.lightcone/config.yaml`, which today carries one key: - -```yaml -container: - runtime: auto # auto | docker | podman | podman-hpc | none -``` - -It is read by [`lightcone.engine.container.load_runtime`](container.md). diff --git a/docs/api/wrroc.md b/docs/api/wrroc.md new file mode 100644 index 00000000..5d260eb8 --- /dev/null +++ b/docs/api/wrroc.md @@ -0,0 +1,108 @@ +# lightcone.engine.wrroc + +The Workflow Run RO-Crate exporter behind +[`lc export wrroc`](../cli/export.md). Walks a project's +`.lightcone-manifest.json` sidecars and emits an +[RO-Crate](https://www.researchobject.org/workflow-run-crate/) bundle +suitable for WorkflowHub, Zenodo, or any RO-Crate-aware archive. + +Source: `src/lightcone/engine/wrroc.py`. + +The manifest layer stays the canonical internal format — WRROC is the +**publication** view, generated on demand. The exporter is one-shot: no +live crate is maintained. + +## Profiles + +We target the deepest of the three WRROC profiles, *Provenance Run +Crate*, because the manifest layer already captures the per-step data it +requires. All three are declared on the root's `conformsTo` (and, since +validators expect it, added to the graph as `CreativeWork` entities): + +```python +PROCESS_RUN_CRATE_PROFILE = "https://w3id.org/ro/wfrun/process/0.5" +WORKFLOW_RUN_CRATE_PROFILE = "https://w3id.org/ro/wfrun/workflow/0.5" +PROVENANCE_RUN_CRATE_PROFILE = "https://w3id.org/ro/wfrun/provenance/0.5" +``` + +## Entity mapping + +| lightcone concept | WRROC entity | +|-------------------|--------------| +| `astra.yaml` | `ComputationalWorkflow` | +| each universe | `PropertyValue` set on the workflow | +| each materialized output dir | `Dataset` (with the data files inside) | +| each recipe execution | `CreateAction` — `object` = upstream Datasets / external Files, `result` = the output Dataset, `instrument` = the recipe `SoftwareApplication`, `agent` = the `Person` | +| each container image | `SoftwareApplication` | +| each decision value | `PropertyValue` on the workflow | + +## `export_wrroc(project_path, output_path, *, universes=None, author=None, license=None, zip_bundle=False, include_data=True) → ExportResult` + +The public entry point. + +- **`universes`** — `None` discovers every universe from `universes/`. + A universe with no materialized output contributes nothing and is left + out of `universes_included`. +- **`author`** — `"Name "`. Falls back to `git config + user.name`/`user.email`, then the `LIGHTCONE_AUTHOR` env var. +- **`license`** — URL or SPDX identifier. Required by the Workflow + RO-Crate profile; falls back to `astra.yaml`'s `license`, then to + `DEFAULT_LICENSE` (CC-BY-4.0). +- **`include_data=False`** — bundle manifests, `astra.yaml`, and the + universe files only. Useful for archiving provenance without + re-uploading large data. +- **`zip_bundle`** — `output_path` is the `.zip` file rather than the + bundle directory. + +Raises `FileNotFoundError` when the project has no `astra.yaml`, and +`FileExistsError` rather than clobbering a non-empty output directory (or +an existing directory named as a zip target). + +`rocrate` is imported lazily inside the function, so the module stays +importable without the optional dependency. + +```python +@dataclass +class ExportResult: + bundle_path: Path + runs_included: int # CreateActions emitted + universes_included: list[str] + is_zip: bool +``` + +`runs_included == 0` is not an error — the bundle then holds only the +workflow definition — but `lc export wrroc` warns, because that shape +will not pass strict Provenance Run Crate validation. + +## `WRROCBuilder` + +Accumulates lightcone state into one `ROCrate` instance. It owns the +`@id` minting strategy and de-duplicates repeated entities (the same +recipe `SoftwareApplication` is shared by every `CreateAction` that used +it). + +| Method | Role | +|--------|------| +| `add_workflow() → str` | Add `astra.yaml` as the `ComputationalWorkflow`; returns its `@id`. | +| `add_universe_runs(universe_id, tree_outputs) → int` | Add `Dataset` + `CreateAction` for every materialized output in the universe; returns the count of actions added. | + +## Public surface + +```python +__all__ = [ + "ExportResult", + "PROCESS_RUN_CRATE_PROFILE", + "PROVENANCE_RUN_CRATE_PROFILE", + "WORKFLOW_RUN_CRATE_PROFILE", + "WRROCBuilder", + "export_wrroc", +] +``` + +## Tests + +`tests/test_wrroc.py` builds bundles against temporary projects and +asserts the graph shape — profile conformance, workflow-as-main-entity, +the preserved input chain across two steps, decision `PropertyValue`s — +plus the zip, author, universe-filter, and `--metadata-only` paths, and a +round-trip load back through `rocrate-py`. diff --git a/docs/architecture.md b/docs/architecture.md index 504fbb7b..a53d5732 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -13,7 +13,8 @@ Snakemake that owns provenance.** This page expands that sentence. contract lives here. 3. **Cluster management** — `lc run` always dispatches through a Dask scheduler whose lifetime equals the run's lifetime. The cluster - manager picks the right shape (local / SLURM / external) on the fly. + manager picks the right shape (external scheduler / Dask Gateway / + SLURM / local) on the fly. The Claude Code plugin (skills + hooks + agents) is the agentic surface layered on top. @@ -37,12 +38,19 @@ rule : params: cfg=lambda wc: CFG[""][wc.universe], run: - shell('printf "▶ [%s]\\n" "{wildcards.universe}" >&2') - shell(params.cfg["shell_command"]) # the recipe (already container-wrapped) - write_manifest(output_dir=Path(output.data), inputs={...}, cfg=params.cfg) - for w in validate_output(...): print(f"⚠ {w}", file=sys.stderr) + run_rule( # engine.runner + rule_key="", + universe=wildcards.universe, + output_dir=Path(output.data), + inputs={...}, + cfg=dict(params.cfg), + ) ``` +`run_rule` runs the pre-rendered (already container-wrapped) shell +command, emits the sentinel-framed narrative lines `lc run` forwards, +writes the manifest on success, and runs the validation hook. + ### What goes in `cfg` `snakefile-config.json` is keyed by ` → → cfg` where @@ -98,10 +106,12 @@ re-runs the rule on the next `lc run`. "code_version": "sha256:…", "data_version": "sha256:…", "container_image": "lc-myproject-abc123" , + "worker_image": null, "recipe": "python scripts/compute.py", "decisions": {...}, "input_versions": { "": "sha256:…" }, "git_sha": "...", + "git_remote": "...", "lc_version": "...", "host": "...", "slurm_job_id": "...", @@ -109,6 +119,11 @@ re-runs the rule on the next `lc run`. } ``` +`container_image` is what the spec *declared*. `worker_image` is the +image the Dask Gateway pod that actually ran the recipe was started from +(`LIGHTCONE_WORKER_IMAGE`, provisioned by `lc run`) — `null` on every +other backend. Both fields are additive; older manifests still parse. + ### `data_version` exclusions `sha256_dir()` skips two filenames: `.lightcone-manifest.json` (chicken @@ -152,22 +167,39 @@ directory required, works on a fresh clone or frozen archive. Module: [`lightcone.engine.dask_cluster`](api/dask_cluster.md). -`cluster_for_run()` is the only entry point. It is a context manager -that yields a Dask scheduler address valid for the duration of the run, -across three branches: +`cluster_for_run()` is the only entry point. It is a context manager that +yields the **env overlay** the child snakemake needs to reach the cluster +(the executor plugin lives in a different process), valid for the +duration of the run, across four branches: 1. `DASK_SCHEDULER_ADDRESS` already set → yield as-is. We don't own the cluster, we don't tear it down. -2. `SLURM_JOB_ID` set → start an in-process scheduler bound to the +2. `DASK_GATEWAY__ADDRESS` set (a JupyterHub / Dask Gateway deployment) + → **create** a run-scoped Gateway cluster running the project's + image, scale it adaptively up to the job bound, and shut it down on + exit. Create-per-run is what makes image updates seamless: a Gateway + cluster's image is fixed at creation. Gateway schedulers speak a + `gateway://` comm scheme a bare `Client` can't dial, so this branch + yields the *cluster name* (`LIGHTCONE_GATEWAY_CLUSTER`) and the + executor rejoins through the authenticated Gateway API. +3. `SLURM_JOB_ID` set → start an in-process scheduler bound to the driver hostname (`SLURMD_NODENAME` or `gethostname()`), then `srun` one `dask worker` per node across the allocation. Workers advertise the node's resources via Dask abstract resources (`cpus`, `memory`, `gpus`). The Snakemake executor plugin maps per-rule `cpus_per_task` / `mem_mb` / `gpus_per_task` to per-task constraints. -3. Neither → `LocalCluster()` sized to the local machine. +4. None of the above → `LocalCluster()` sized to the local machine. + +Outside the Gateway branch the scheduler is always in-process so its +lifetime equals the run's lifetime: no service to manage, no orphaned +schedulers. The Gateway branch enforces the same contract server-side — +created per run, culled on exit, with the deployment's idle timeout as +the backstop. -The scheduler is always in-process so its lifetime equals the run's -lifetime: no service to manage, no orphaned schedulers. +Because the Gateway branch realizes containers as *pod images* rather +than by wrapping recipes, one run can honour only one image: `lc run` +rejects a spec resolving to more than one distinct container image there +(`gateway_branch_active()` is the pre-flight predicate). ### The Snakemake executor @@ -190,26 +222,67 @@ generation time, so the worker just runs them. --- +## Scratch & run locking + +Module: [`lightcone.engine.scratch`](api/scratch.md). + +Before anything else, `lc run` resolves a **scratch root** — where +Snakemake metadata, Dask spill, and the run lock live — in this order: +`LIGHTCONE_SCRATCH`, then `scratch_root` in +`.lightcone/lightcone.yaml`, then the detected site's declared +`scratch_root` (a shell expression like `$SCRATCH`), then the tempdir. + +`/.snakemake` is then repointed there by symlink. This matters +on NERSC, where `$HOME` and CFS are DVS-mounted and silently swallow +`flock`, and on a JupyterHub deployment, where a pod's `/tmp` is +pod-local while every worker pod mounts the same NFS `$HOME`. + +A non-blocking `flock` on `/.lightcone/locks/.run-lock` +is held for the whole run: a second `lc run` on the same project fails +cleanly instead of interleaving Snakemake state updates, and acquiring +it clears any workflow lock a previously crashed run left behind. + +--- + ## Container layer Module: [`lightcone.engine.container`](api/container.md). Two surfaces: -- **Build** — `compute_image_tag()` + `build_image()`. Tags are - `lc--` over the Containerfile and dependency - files (`requirements.txt`, `pyproject.toml`, `poetry.lock`, - `Pipfile.lock`, …). Rebuilds happen only when the hash changes. +- **Build** — `image_identity()` + `build_image()`. The 12-char sha256 + digest covers the Containerfile, every dependency file + (`requirements.txt`, `pyproject.toml`, `poetry.lock`, `uv.lock`, + `environment.yml`, …), **and the contents of every `COPY`/`ADD` source + the Containerfile references** — files hashed directly, directories + walked recursively (skipping `.git`, `.venv`, `results/`, caches, …). + A `COPY .` therefore hashes the whole build context. Rebuilds happen + only when the digest changes. - **Run-time wrap** — `wrap_recipe()` produces the command string that the Snakefile generator embeds into each rule. +One identity, two spellings: `lc--` in a local image +store (`compute_image_tag()`), `/lc-:` in a +registry (`registry_image_ref()`). `runtime_registry()` decides which +one a runtime uses, so the Snakefile generator and the status walker can +never disagree about an image identity. + Runtime resolution: `~/.lightcone/config.yaml` carries -`container.runtime` (`auto | docker | podman | podman-hpc | none`). -`auto` picks the first usable in `(podman, docker, podman-hpc)`, -skipping docker if its daemon is unreachable. `none` is an explicit -opt-out — recipes run on the host. When `auto` falls back to `none` -silently, `lc run` warns that the manifest's `container_image` field -will misrepresent what actually executed. +`container.runtime` +(`auto | docker | podman | podman-hpc | kubernetes | none`). `auto` picks +the first usable in `(podman-hpc, podman, docker)`, skipping docker if +its daemon is unreachable, with the detected site's declared +`container_runtime` moved to the front. `none` is an explicit opt-out — +recipes run on the host. When `auto` falls back to `none` silently, +`lc run` warns that the manifest's `container_image` field will +misrepresent what actually executed. + +`kubernetes` is the odd one out and never comes from PATH detection: it +is selected by the `jupyterhub` site (or pinned explicitly), the worker +pod *is* the container so `wrap_recipe()` is a passthrough, images +resolve to registry refs, and builds go through +[`engine.cloudbuild`](api/cloudbuild.md) — GCP Cloud Build, authenticated +by the pod's Workload Identity — instead of a local OCI CLI. For `podman-hpc`, the build path also runs `podman-hpc migrate ` so compute nodes can read the image without a registry. @@ -275,13 +348,16 @@ src/lightcone/ # PEP 420 namespace package — NO __init__.py ├── engine/ # execution substrate │ ├── manifest.py # write_manifest, sha256_dir, code_version │ ├── snakefile.py # generate .lightcone/Snakefile from astra.yaml -│ ├── container.py # docker/podman/podman-hpc build + recipe wrap -│ ├── dask_cluster.py # cluster lifecycle (local/SLURM/external) +│ ├── container.py # image identity, local build + recipe wrap +│ ├── cloudbuild.py # GCP Cloud Build backend (no local OCI runtime) +│ ├── dask_cluster.py # cluster lifecycle (external/Gateway/SLURM/local) +│ ├── scratch.py # scratch root, per-run dirs, run lock │ ├── status.py # manifest-driven status walker (no Snakemake) │ ├── verify.py # recompute hashes, walk the chain │ ├── tree.py # sub-analysis tree helpers │ ├── validation.py # post-recipe output sanity checks -│ └── site_registry.py # vestigial; not imported by active code +│ ├── wrroc.py # Workflow Run RO-Crate exporter +│ └── site_registry.py # known-site defaults; drives scratch + runtime └── eval/ # evaluation harness for the agent loop ├── cli.py harness.py sandbox.py graders.py build.py report.py models.py @@ -319,12 +395,18 @@ astra.yaml ── snakefile.generate() ──► .lightcone/Snakefile + .lightco DAG resolution per-rule run: dask scheduler (Snakemake) shell(recipe) (LocalCluster / + write_manifest() SLURM-srun / + Dask Gateway / external) │ └─► results///data results///.lightcone-manifest.json ``` +The Gateway branch differs in where the container comes from: the worker +pod is started *from* the project image, so the recipe is not wrapped — +`lc run` builds the image (Cloud Build), creates the cluster with it, and +the pod executes the recipe natively. + What Snakemake owns (we don't write it): DAG construction, topological execution, parallelism, dry-run, locking, retry, log capture, per-rule resources, `--rerun-triggers` for staleness detection. @@ -343,8 +425,8 @@ each rule to a Dask scheduler. | `astra.yaml` | Project | The spec. Inputs, outputs, recipes, decisions, sub-analyses. | | `.lightcone/Snakefile` | Project (generated) | Auto-generated by `lc run`. Don't edit. | | `.lightcone/snakefile-config.json` | Project (generated) | Per-`(rule, universe)` config. | -| `.lightcone/lightcone.yaml` | Project | Tiny scratchpad — currently writes only `target: local`. Not consumed by today's code. | -| `~/.lightcone/config.yaml` | User | `container.runtime`. | +| `.lightcone/lightcone.yaml` | Project | Written by `lc init` (`target: local`, plus `scratch_root` when `--scratch` is passed). `scratch_root` is read by [`engine.scratch`](api/scratch.md) on every `lc run`; `target` is inert. | +| `~/.lightcone/config.yaml` | User | `container.runtime`. Auto-created with `auto` on the first `lc` invocation. | | `.claude/settings.json` | Project | Claude Code permissions. | The `dagster.yaml` and `~/.lightcone/targets/*.yaml` files referenced in diff --git a/docs/cli/build.md b/docs/cli/build.md index d53445cf..9e93603e 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -49,30 +49,59 @@ nothing to build. lc-- ``` -The hash covers the Containerfile contents plus any of these dependency -files at the project root: - -- `requirements.txt` -- `requirements-dev.txt` -- `requirements-test.txt` -- `pyproject.toml` -- `setup.py` -- `setup.cfg` -- `poetry.lock` -- `Pipfile.lock` - -Edit any one of those and the tag changes. That, in turn, changes +(On the `kubernetes` runtime the same identity is spelled +`/lc-:` — the digest is +identical, it just moves into the tag position.) + +The hash covers three things: + +1. The **Containerfile** contents. +2. Any of these **dependency files** at the project root: + `requirements.txt`, `requirements-dev.txt`, `requirements-test.txt`, + `pyproject.toml`, `setup.py`, `setup.cfg`, `poetry.lock`, + `Pipfile.lock`, `uv.lock`, `conda-lock.yml`, `environment.yml`, + `environment.yaml`. +3. Every **`COPY` / `ADD` source** referenced from the Containerfile: + files are hashed directly, directories are walked recursively. + `COPY .` therefore covers the whole project tree, minus a fixed + exclude list (`.git`, `.venv`, `venv`, `__pycache__`, `results`, + `.lightcone`, `.snakemake`, `node_modules`, and the usual tool + caches) so the tag doesn't churn every time you run the pipeline. + +Sources behind `--from=`, and URL / git `ADD` arguments, are +skipped — they aren't part of the host build context. + +Edit any of the above and the tag changes. That, in turn, changes `code_version` in every recipe that uses the image, which marks all downstream outputs `stale` in `lc status`. +!!! note "New in v0.4.0" + Point 3 is new. Projects whose Containerfile copies source code + (`COPY src/ /app/src`, `COPY . /app`) will see one extra rebuild + the first time they run under v0.4.0, and will now correctly + rebuild when that copied code changes — which previously went + unnoticed. + ## Examples ```bash lc build # build / pull whatever's missing lc build --force # rebuild / re-pull everything lc build --runtime podman-hpc # force the HPC runtime +lc build --runtime kubernetes # force the Cloud Build path (JupyterHub) ``` +`--runtime kubernetes` is normally unnecessary: on a lightcone +JupyterHub the site is detected from `DASK_GATEWAY__ADDRESS` and the +runtime resolves to `kubernetes` on its own. Pass it explicitly to pin +the behaviour, or to build against the deployment registry from a +session where detection would have picked something else. It requires +the Cloud Build contract in the environment — `LIGHTCONE_REGISTRY` and +`LIGHTCONE_BUILD_BUCKET` (plus the optional +`LIGHTCONE_BUILD_SERVICE_ACCOUNT`) — and a metadata-served GCP identity; +without those `lc build` errors with *"No image build backend on this +host"*. + ## Pre-staging for HPC On a login node: diff --git a/docs/cli/eval.md b/docs/cli/eval.md new file mode 100644 index 00000000..258b77a0 --- /dev/null +++ b/docs/cli/eval.md @@ -0,0 +1,100 @@ +# lc eval + +Evaluate the lightcone-cli build loop against seed tasks. + +This is a **maintainer tool**, not part of the analysis workflow. It +drives a coding agent through a scripted task inside an ephemeral +sandbox, then scores the resulting project. Nothing here touches your +own `astra.yaml`. + +## Availability + +`lc eval` only registers when its dependencies are installed. They live +in a PEP 735 `[dependency-groups]` entry, **not** a packaging extra — +`pip install lightcone-cli[eval]` does not work. From a source checkout: + +```bash +git clone https://github.com/LightconeResearch/lightcone-cli +cd lightcone-cli +uv sync --group eval +uv run lc eval --help +``` + +Without the group, `lc eval` fails with `Error: No such command 'eval'`. + +The harness also needs credentials in the environment (a `.env` file at +the project root is loaded automatically): + +- `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN` — passed into the + sandbox so the agent can run. +- Daytona credentials — sandboxes are Daytona-hosted. + +## Synopsis + +```text +lc eval run [OPTIONS] CONFIG_PATH +lc eval report [OPTIONS] RESULTS_PATH +lc eval compare RESULTS1 RESULTS2 +``` + +## `lc eval run` + +Run an eval suite from a config file. + +| Option | Default | Effect | +|--------|---------|--------| +| `--concurrency`, `-c N` | from the config's `max_concurrency` | Max parallel sandboxes. | +| `--num-trials`, `-n N` | from the config's `num_trials` | Override the trial count. | +| `--dry-run` | off | Print the trial schedule and exit without running anything. | +| `--evals-dir PATH` | `evals/` in the cwd | Where task definitions live. | + +A config is a small YAML file naming the tasks to run: + +```yaml +id: build +tasks: + - snae +num_trials: 3 +max_concurrency: 2 +output_dir: "eval-results" +``` + +Each task is a directory under `evals/tasks//` holding a `task.yaml` +(the prompt and its graders), a target `astra.yaml`, and any fixture +`data/`. Graders come in three shapes: a shell command (exit 0 = pass), +a `status` grader scoring the fraction of declared outputs that +materialized, and a custom script whose last stdout line is the score. +The composite score is their weighted mean. + +Results are written to `/-/results.json`. + +```bash +lc eval run evals/build.yaml +lc eval run evals/build.yaml --dry-run +lc eval run evals/build.yaml --num-trials 1 --concurrency 2 +``` + +## `lc eval report` + +Re-display a previous run's results as a table. + +| Option | Effect | +|--------|--------| +| `--json` | Emit the raw JSON summary instead of the table. | + +```bash +lc eval report eval-results/build-20260315/results.json +``` + +## `lc eval compare` + +Print two runs side by side — the fastest way to see whether a skill or +prompt change moved the needle. + +```bash +lc eval compare eval-results/run1.json eval-results/run2.json +``` + +The implementation lives in `src/lightcone/eval/` — `cli.py` for the +command surface, with `harness.py`, `sandbox.py`, `graders.py`, and +`report.py` behind it. diff --git a/docs/cli/index.md b/docs/cli/index.md index 33ffba0b..4ad257ce 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -21,6 +21,7 @@ Code skills, with the CLI as the durable, scriptable backstop. | [`lc status`](status.md) | Manifest-driven status report. No Snakemake import needed. | | [`lc verify`](verify.md) | Recompute hashes, walk the input chain, surface tampering. | | [`lc export`](export.md) | Emit interoperable bundles (Workflow Run RO-Crate) for publication. | +| [`lc eval`](eval.md) | Maintainer harness: score the agentic build loop against seed tasks. | ## Global options @@ -35,7 +36,11 @@ Options: ## Removed commands For historical context: `lc dev`, `lc setup`, `lc target`, and `lc update` no -longer exist as explicit commands. `lc eval` is available when the `eval` -extra is installed (`pip install lightcone-cli[eval]`); without it the import -is silently skipped and `lc eval` will fail with "No such command". See the -removal pages for details. +longer exist as explicit commands. See the removal pages for details. + +`lc eval` is not a removed command — it is registered only when its +dependency group is installed. It is a PEP 735 `[dependency-groups]` +entry, not a packaging extra, so `pip install lightcone-cli[eval]` does +*not* work. From a source checkout: `uv sync --group eval`. Without it +the import is silently skipped and `lc eval` fails with "No such +command". See [`lc eval`](eval.md). diff --git a/docs/cli/init.md b/docs/cli/init.md index 045d7a50..f7d0844b 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -15,21 +15,58 @@ lc init [OPTIONS] [DIRECTORY] Inside `DIRECTORY` (creating it if needed): ```text -astra.yaml # tiny boilerplate spec with one example output +astra.yaml # tiny boilerplate spec with two example outputs CLAUDE.md # short note pointing future agents at the project -.gitignore # Python + lightcone state +myst.yml # MyST config for the report (MySTRA plugin, book theme) +index.md # template report, TODO-driven +Containerfile # build instructions for the analysis image +requirements.txt # Python dependencies baked into that image +.gitignore # Python + lightcone state + `_build/` .lightcone/ - lightcone.yaml # currently a stub: { target: local } + lightcone.yaml # project state (scratch root, …) results/ # placeholder; populated by `lc run` -universes/ # placeholder; populate via `astra universe generate -n …` +src/ # placeholder; where recipe scripts go +universes/ + baseline.yaml # the default universe; add more via `astra universe generate -n …` .claude/ # bundled Claude Code plugin - skills/, agents/, hooks/, scripts/, templates/ - settings.json # the chosen permission tier + skills/, agents/, scripts/, templates/ + settings.json # the chosen permission tier + session hooks .venv/ # Python venv (skipped with --no-venv) +.git/ # git repository (skipped with --no-git) ``` +The spec scaffold (`astra.yaml`, `universes/baseline.yaml`, the base +`.gitignore`, `src/`) is delegated to `astra init`; everything else is +layered on by `lc init`. + `lc init` refuses to run if `DIRECTORY/astra.yaml` already exists. +## The report scaffold + +`myst.yml` + `index.md` are a template [MyST](https://mystmd.org/) +report, pre-wired to the **MySTRA** plugin. The plugin resolves +references *against `astra.yaml`*, so the prose never hard-codes a +measured value: + +- `{astra}` as a role — an inline mention, e.g. + ``{astra}`decisions.example_method` ``. +- `{astra}` as a directive — a block embed, e.g. `:::{astra} outputs` + to render the analysis's outputs. +- `{astra:value}` — a live number pulled from a materialized output, + e.g. ``{astra:value}`outputs.main_result` ``. + +The skeleton has Introduction / Methods / Results sections marked +`TODO:`, referencing the boilerplate ids `example_method` and +`main_result`. Preview it with: + +```bash +npm i -g mystmd # once +myst start +``` + +Builds land in `_build/`, which the scaffolded `.gitignore` already +excludes. + ## Options | Option | Default | Effect | @@ -37,19 +74,24 @@ universes/ # placeholder; populate via `astra universe genera | `--no-git` | off | Skip `git init`. | | `--no-venv` | off | Skip `python -m venv .venv`. | | `--permissions {yolo,recommended,minimal}` | `recommended` | Which `.claude/settings.json` permission tier to install. | +| `--scratch PATH` | the site default (e.g. `$SCRATCH` on Perlmutter) | Scratch root for Snakemake state, Dask spill, and run locks. Shell expressions like `$SCRATCH` are kept verbatim in `.lightcone/lightcone.yaml` and expanded at run time. | > The historical `--target`, `--existing-project`, and `--sub-analysis` -> flags have been removed; today's `lc init` only knows the three flags -> above. For migrating an existing project, run `lc init` in a fresh -> directory and use the `/lc-from-code` skill from inside Claude Code. +> flags have been removed. For migrating an existing project, run +> `lc init` in a fresh directory and use the `/lc-from-code` skill from +> inside Claude Code. ## Permission tiers -| Tier | Allowed | Denied | -|------|---------|--------| -| `yolo` | `Bash(*)`, `Edit`, `Read`, `Write`, `WebSearch`, `WebFetch`, `mcp__*` | — | -| `recommended` | `Read`, `Edit`, `Write`, `Bash(*)`, `WebSearch`, `WebFetch` | Edits to `~/.ssh`, `~/.aws`, `~/.gnupg`, `/scratch`, `/pscratch`; `sudo`, `rm -rf`, `git push`. | -| `minimal` | `Read` | Everything else. | +| Tier | Allowed | Ask first | Denied | +|------|---------|-----------|--------| +| `yolo` | `Bash(*)`, `Edit`, `Read`, `Write`, `WebSearch`, `WebFetch`, `mcp__*` | — | — | +| `recommended` | `Read`, `Edit`, `Write`, `Bash(*)`, `WebSearch`, `WebFetch` | `Edit`/`Write` under `/scratch` and `/pscratch` | Edits to `~/.ssh`, `~/.aws`, `~/.gnupg`; `sudo`, `rm -rf`, `rm -fr`, `git push`. | +| `minimal` | `Read` | — | Everything else. | + +Scratch paths are `ask`, not `deny`: on HPC the project itself often +lives in `$SCRATCH`, so the agent legitimately needs to write there — +you just get a prompt first. The tiers are defined as `PERMISSION_TIERS` in `src/lightcone/cli/commands.py` — adjust there if you want to add a tier @@ -62,6 +104,7 @@ lc init # scaffold in cwd, recommended tier lc init my-analysis # scaffold in ./my-analysis lc init my-analysis --no-git --no-venv # bare bones lc init . --permissions yolo # for autonomous loops you trust +lc init my-analysis --scratch '$SCRATCH' # pin the scratch root (HPC) ``` ## Next steps @@ -74,3 +117,9 @@ claude # open Claude Code # Then ask the agent to implement the spec. # It will run lc run, watch lc status, then validate and verify. ``` + +To write up the result, edit `index.md` and preview it live: + +```bash +myst start # requires the MyST CLI: npm i -g mystmd +``` diff --git a/docs/cli/run.md b/docs/cli/run.md index 3e311be9..0a62f2bc 100644 --- a/docs/cli/run.md +++ b/docs/cli/run.md @@ -30,18 +30,42 @@ everything (Snakemake's `rule all`). `lightcone.engine.container.load_runtime`. If `auto` falls back to `none` while the spec declares containers, print a loud provenance warning. -4. Generate `.lightcone/Snakefile` and +4. **Pre-flight the images.** Every distinct `container:` in the spec is + built (Containerfile) or pulled (registry ref) before the DAG starts, + so the first run after a Containerfile edit doesn't die mid-DAG on a + missing image. This is the same pass `lc build` runs — already-present + images are skipped. +5. Generate `.lightcone/Snakefile` and `.lightcone/snakefile-config.json` for the selected universes. -5. Translate any explicit `OUTPUTS` into Snakemake target paths +6. Translate any explicit `OUTPUTS` into Snakemake target paths (`/.lightcone-manifest.json`) — this is what tells Snakemake "build that specific output." -6. Open a Dask cluster context (`local`, `srun`-backed inside - `SLURM_JOB_ID`, or external if `DASK_SCHEDULER_ADDRESS` is set). -7. Spawn `snakemake -s … -d … --cores N --jobs N --executor dask - --rerun-triggers …` with `DASK_SCHEDULER_ADDRESS` in the environment. -8. In the default (non-verbose) path, filter the executor's banner +7. Open a Dask cluster context. Four branches, checked in this order: + + | Condition | Cluster | + |---|---| + | `DASK_SCHEDULER_ADDRESS` set | Attach to that scheduler; never torn down. | + | `DASK_GATEWAY__ADDRESS` set | Create a **run-scoped Dask Gateway cluster** with the project's image, adaptive from 1 to `--jobs` workers, culled on exit. | + | `SLURM_JOB_ID` set | In-process scheduler + one `dask worker` per allocated node via `srun`. | + | none of the above | `LocalCluster` sized to the machine. | + +8. Spawn `snakemake -s … -d … --cores N --jobs N --executor dask + --shared-fs-usage persistence input-output sources storage-local-copies + source-cache --rerun-triggers …`, with the cluster's connection info + in the environment (`DASK_SCHEDULER_ADDRESS`, or the cluster name for + the Gateway branch, which is rejoined through the authenticated + Gateway API rather than dialled by address). +9. In the default (non-verbose) path, filter the executor's banner chatter so the output reads as lightcone's, not Snakemake's. Real - error content always passes through. + error content always passes through; on failure the last of + snakemake's stderr is written to `snakemake-stderr-.log` under + the scratch root. + +`--shared-fs-usage` deliberately omits `software-deployment`. With it, +spawned job commands would embed the *driver's* `sys.executable` — a +path that doesn't exist inside a Dask Gateway worker image. Without it, +workers invoke plain `python` from their own environment, which is +correct on every backend. ## Output qualification @@ -53,11 +77,23 @@ lc run inference # error if 'inference' is ambiguous lc run hod_fitting.inference # disambiguated ``` -Each rule's body wraps the recipe in a ` run --rm --pull=never --v "$PWD":"$PWD" -w "$PWD" bash -c ''` shell when a -container is configured. After the recipe shell exits, the Snakefile -calls `write_manifest()` host-side and the validation snippet emits -warnings for empty / all-NaN / wrong-extension outputs. +## How recipes get containerized + +On `docker`, `podman`, and `podman-hpc`, each rule's body wraps the +recipe in a ` run --rm --pull=never -v "$PWD":"$PWD" -w "$PWD" + bash -c ''` shell. + +On the `kubernetes` runtime this wrapping is a **passthrough**: the Dask +worker pod executing the recipe was already started *from* the project's +image, so wrapping would containerize twice. The image still flows into +`code_version` and into the manifest — and the pod reports back the +image it actually ran as the manifest's `worker_image` field. With +`runtime: none`, or a recipe with no `container:`, the recipe also runs +unwrapped. + +After the recipe shell exits, the Snakefile calls `write_manifest()` +host-side and the validation snippet emits warnings for empty / +all-NaN / wrong-extension outputs. ## Examples @@ -84,6 +120,38 @@ Workers advertise `cpus`, `memory`, and `gpus` resources. Per-rule resource hints (`cpus_per_task`, `mem_mb`, `gpus_per_task`) constrain which workers can pick up which jobs. +## Inside a JupyterHub deployment + +On a lightcone JupyterHub (Kubernetes + Dask Gateway), `lc run` needs no +arguments beyond the usual — the deployment injects +`DASK_GATEWAY__ADDRESS`, `LIGHTCONE_REGISTRY`, and +`LIGHTCONE_BUILD_BUCKET` into your session and `lc` picks them up: + +```bash +lc run --universe baseline -j 8 +``` + +- The pre-flight image pass builds through **GCP Cloud Build** (there is + no local OCI runtime in your session) and pushes + `/lc-:`. Unchanged files never rebuild — + freshness is a single registry check. +- A run-scoped Gateway cluster is created **with that image**, adaptive + from 1 to `--jobs` workers, and culled when the run finishes. A + Gateway cluster's image is fixed at creation, so create-per-run is + what makes image updates seamless. +- `lc run` waits for the first worker before dispatching, bounded by + `LIGHTCONE_GATEWAY_WORKER_TIMEOUT` (seconds, default `600`) — an + unpullable image would otherwise hang at zero workers forever. It + then asserts that workers advertise the `cpus`+`memory` resource + contract, since Dask would silently never schedule a rule otherwise. + +!!! warning "One image per run on this backend" + The worker pod *is* the container, so the whole run executes in a + single image. If `astra.yaml` resolves to more than one distinct + container image, `lc run` refuses to start and asks you to + consolidate on a single Containerfile (or one shared prebuilt + image). Every other backend wraps per rule and is unaffected. + ## Provenance gotcha If `~/.lightcone/config.yaml` says `runtime: auto` and no runtime is diff --git a/docs/contributing/backends.md b/docs/contributing/backends.md index 5c651e2d..cfd9ac79 100644 --- a/docs/contributing/backends.md +++ b/docs/contributing/backends.md @@ -1,17 +1,17 @@ -# Adding an Execution Backend (rewritten) +# Adding an Execution Backend The `ASTRAContainerRunner` plugin point is gone. Execution is structured -quite differently now, and "adding a backend" decomposes into one or both +quite differently now, and "adding a backend" decomposes into one or more of these: ## Adding a container runtime -The supported runtimes are `docker`, `podman`, and `podman-hpc` (plus the -`none` no-op). They are listed in -`src/lightcone/engine/container.py::RUNTIMES`. To add a new one: +The OCI runtimes are `podman-hpc`, `podman`, and `docker`, listed in +`src/lightcone/engine/container.py::RUNTIMES` in detection-priority +order (plus the `none` no-op). To add another OCI CLI: -1. Append the binary name to `RUNTIMES` (detection priority is the tuple - order). +1. Append the binary name to `RUNTIMES` (the tuple order *is* the + detection priority). 2. If detection needs a probe (like the docker-daemon ping), extend `detect_runtime()`. 3. If `wrap_recipe()` needs different flags for the runtime, branch on @@ -21,22 +21,59 @@ The supported runtimes are `docker`, `podman`, and `podman-hpc` (plus the `pull_image()`. 5. Add tests in `tests/test_container.py`. +### `kubernetes` is deliberately *not* in `RUNTIMES` + +`KUBERNETES = "kubernetes"` is a separate constant on purpose, and +appending it to `RUNTIMES` would be wrong: `RUNTIMES` means "a binary we +can find on PATH and shell out to", and the guards in `build_image()`, +`pull_image()`, and `wrap_recipe()` reject anything outside it. + +The `kubernetes` path is structurally different at every step: + +| Step | `RUNTIMES` member | `kubernetes` | +|------|-------------------|--------------| +| Selection | PATH probe, or explicit pin | site detection (`jupyterhub`) or explicit pin — never PATH | +| Build | ` build` locally | [`engine.cloudbuild.ensure_image()`](../api/cloudbuild.md) | +| Image identity | `lc--` local tag | `/lc-:` registry ref | +| Freshness | local image-store inspect | registry HEAD | +| Run | `wrap_recipe()` wraps the recipe | passthrough — the worker pod *is* the container | + +A genuinely new *non-CLI* backend therefore follows the `kubernetes` +shape instead: its own constant, its own branches in `detect_runtime()` / +`load_runtime()` / `runtime_registry()` / `get_container_status()`, and a +passthrough (or bespoke) `wrap_recipe()` case. + ## Adding a Dask cluster shape -Today the cluster manager has three branches: existing scheduler, SLURM -allocation, local. To add a fourth (for example, a custom GPU farm): +The cluster manager has four branches, checked in this order: existing +scheduler (`DASK_SCHEDULER_ADDRESS`), Dask Gateway +(`DASK_GATEWAY__ADDRESS`), SLURM allocation (`SLURM_JOB_ID`), local. To +add a fifth (for example, a custom GPU farm): 1. Add a branch to `cluster_for_run()` in - `src/lightcone/engine/dask_cluster.py`. -2. Make sure it advertises the same resource keys (`cpus`, `memory`, + `src/lightcone/engine/dask_cluster.py`, in the right priority + position. +2. Yield the env overlay the child snakemake needs to reach it. + Address-based branches yield `{"DASK_SCHEDULER_ADDRESS": addr}`; a + scheduler that can't be dialled directly needs its own rendezvous + variable (the Gateway branch yields `LIGHTCONE_GATEWAY_CLUSTER` and + the executor rejoins by name through the Gateway API). +3. Make sure workers advertise the same resource keys (`cpus`, `memory`, `gpus`) so the [Snakemake executor plugin](../api/dask_executor.md) - can match. -3. Add tests in `tests/test_dask_cluster.py`. + can match. Dask schedules a task only on workers advertising *every* + requested key, so a missing key hangs silently instead of erroring — + fail fast on it, as `_assert_worker_resources()` does. +4. Own the lifetime: if you create the cluster, tear it down on exit. +5. If `lc run` must know about the branch *before* entering the context + (as it does for the Gateway one-image-per-run check), expose a pure + predicate alongside `gateway_branch_active()`. +6. Add tests in `tests/test_dask_cluster.py`. ## Adding a non-Snakemake executor In principle Snakemake supports multiple executors and we ship one (`snakemake_executor_plugin_dask`). If you need a different scheduler, you can write another Snakemake executor plugin and pass it through -`lc run --executor ` — but that flag does not exist today and would -need to be added to `src/lightcone/cli/commands.py::run`. +`lc run --executor ` — but that flag does not exist today: the +executor is hard-coded in +`src/lightcone/cli/commands.py::_build_snakemake_cmd`. diff --git a/docs/contributing/hpc-sites.md b/docs/contributing/hpc-sites.md index 94df4d40..f33ef38a 100644 --- a/docs/contributing/hpc-sites.md +++ b/docs/contributing/hpc-sites.md @@ -1,21 +1,83 @@ -# Adding an HPC Site (deprecated) - -There is no longer a meaningful concept of "adding an HPC site" — the -target system that this used to feed is gone. The `site_registry` module -is still present in the source tree but unused. See -[api/site_registry](../api/site_registry.md). - -If you want lightcone-cli to behave well on a new cluster, what you -actually need is: - -1. **A container runtime that works on compute nodes.** `podman-hpc` is - the supported case. Wire it up via `~/.lightcone/config.yaml`. -2. **Dask workers reachable from the scheduler.** `lc run` already does - the right thing inside an `salloc`/`sbatch` allocation — the cluster - manager binds the scheduler to the SLURM canonical hostname and - launches one worker per node via `srun`. See - [api/dask_cluster](../api/dask_cluster.md). -3. **A scratch path that the agent should not edit.** Today's permission - tiers hard-code Perlmutter scratch deny rules; if you add a new - site, update `PERMISSION_TIERS` in - `src/lightcone/cli/commands.py`. +# Adding a Site + +A *site* is a machine or deployment lightcone-cli can recognise and adapt +to. Sites are declared in `SITE_DEFAULTS` in +`src/lightcone/engine/site_registry.py` — see +[api/site_registry](../api/site_registry.md) for the full field +reference. Three consumers read them today: `lc init` (surfacing the +scratch root), [`engine.scratch`](../api/scratch.md) (resolving it), and +`engine.container` (picking the runtime). + +## 1. Declare the site + +Append an entry to `SITE_DEFAULTS`. How it gets *detected* is the first +decision: + +- **A machine you can recognise by name** → `hostname_patterns`, + substring-matched against `socket.gethostname()`. Perlmutter uses + `["perlmutter", "saul"]`. +- **A deployment you can only recognise by its environment** → + `env_markers`, a list of env var names that must all be set. The + `jupyterhub` site uses `["DASK_GATEWAY__ADDRESS"]`, because pod + hostnames carry no information. + +Env markers win over hostname patterns in `detect_current_site()`. + +## 2. Pick a container runtime + +`container_runtime` is moved to the front of the detection order, so it +wins on this site whenever the binary exists (it's a hint — a missing +binary falls through). `podman-hpc` is the supported HPC case: rootless, +and its `migrate` step makes images readable from compute nodes without a +registry. A Kubernetes deployment with a remote builder declares +`kubernetes` — see [Adding an Execution Backend](backends.md) for why +that one is not a normal runtime. Users can still override in +`~/.lightcone/config.yaml`. + +## 3. Declare a scratch root + +`scratch_root` is where `lc run` keeps Snakemake metadata, Dask spill, +and the run lock. Store it as a **shell expression** (`$SCRATCH`, +`$HOME`): it is expanded with `os.path.expandvars` at run time, and if +the variable isn't set, resolution falls through rather than writing to a +literal path named `$SCRATCH`. + +What it must be is a filesystem that honours `flock` and is visible to +every process in the run. On NERSC that rules out DVS-mounted `$HOME` and +CFS, leaving Lustre `$SCRATCH`. On a JupyterHub deployment it rules out +the pod-local `/tmp`, leaving the shared NFS `$HOME`. + +## 4. Guard shared filesystems from the agent + +`scratch_paths` feeds `get_site_scratch_deny_rules()`, which isn't wired +into `lc init` today — the equivalent patterns are listed inline in +`PERMISSION_TIERS` in `src/lightcone/cli/commands.py`. Note they sit +under `ask` (prompt before writing), not `deny`: projects on HPC often +legitimately live in `$SCRATCH`, so an outright block would be wrong. + +```python +"ask": [ + "Edit(//scratch/**)", + "Edit(//pscratch/**)", + "Write(//scratch/**)", + "Write(//pscratch/**)", +], +``` + +Add your site's paths there — or wire `get_site_scratch_deny_rules()` +into `_install_claude_plugin()` and merge the result, which is the +version that scales past two sites. + +## 5. Check the cluster shape + +`lc run` already does the right thing inside an `salloc`/`sbatch` +allocation: the cluster manager binds the scheduler to the SLURM +canonical hostname and launches one worker per node via `srun`. See +[api/dask_cluster](../api/dask_cluster.md). If your site needs a shape +that isn't one of the four existing branches, see +[Adding an Execution Backend](backends.md). + +## 6. Test it + +`tests/test_site_registry.py` covers detection precedence and the +declared-field lookups; add a case for the new entry. diff --git a/docs/hpc/containers.md b/docs/hpc/containers.md deleted file mode 100644 index 786dc86d..00000000 --- a/docs/hpc/containers.md +++ /dev/null @@ -1,52 +0,0 @@ -# Container Builds for HPC - -HPC nodes generally cannot reach a docker daemon, so lightcone-cli ships -support for `podman-hpc` (NERSC Perlmutter and friends). The build/migrate -workflow is owned by `lightcone.engine.container`. - -## podman-hpc workflow - -`podman-hpc` is rootless and HPC-aware. After a build, the image must be -*migrated* into the per-node container cache so compute nodes can read it -without a registry. - -```bash -# On a login node, with podman-hpc on PATH: -lc setup # writes ~/.lightcone/config.yaml -$EDITOR ~/.lightcone/config.yaml # set container.runtime: podman-hpc -lc build # builds + migrates each image -``` - -`lc build` checks for cached tags and skips rebuilds. Use `--force` to -rebuild everything. - -## Tag computation - -Tags are content-addressed: - -``` -lc-- -``` - -The hash covers the Containerfile contents plus any of these dependency -files found at the project root: -`requirements.txt`, `requirements-dev.txt`, `requirements-test.txt`, -`pyproject.toml`, `setup.py`, `setup.cfg`, `poetry.lock`, `Pipfile.lock`. - -## At run time - -`lc run` does **not** re-shell into Snakemake's `container:` directive or -`--sdm apptainer`. The Snakefile generator wraps each rule's recipe in: - -```bash -podman-hpc run --rm --pull=never -v "$PWD":"$PWD" -w "$PWD" \ - bash -c '' -``` - -`--pull=never` is critical: short-name resolution would otherwise try -`unqualified-search-registries` for tags like `lc-myproject-abc123` and -fail. Pre-pulling registry images via `lc build` (or pre-staging -Containerfile images via `lc build`) is therefore mandatory. - -See also: [api/container](../api/container.md) for the implementation, -and [`lc build`](../cli/build.md) for the user-facing command. diff --git a/docs/hpc/index.md b/docs/hpc/index.md deleted file mode 100644 index 672e3fc1..00000000 --- a/docs/hpc/index.md +++ /dev/null @@ -1,17 +0,0 @@ -# HPC & SLURM (consolidated) - -The standalone HPC subsystem (target files, site registry, sbatch -generation) is gone. SLURM execution is now handled by Dask: when `lc run` -is invoked inside an existing SLURM allocation, the cluster manager -launches one `dask worker` per allocated node via `srun` and Snakemake -dispatches each rule across them. - -For the user-facing flow, see [Running on a Cluster](../user/cluster.md). - -For maintainer detail: - -- [api/dask_cluster](../api/dask_cluster.md) — the three-branch decision - (existing scheduler / SLURM allocation / local). -- [api/dask_executor](../api/dask_executor.md) — the Snakemake executor - plugin that turns each rule into a `client.submit(...)` call. -- [api/container](../api/container.md) — `podman-hpc` build & migrate. diff --git a/docs/hpc/site-registry.md b/docs/hpc/site-registry.md deleted file mode 100644 index 49dd39e7..00000000 --- a/docs/hpc/site-registry.md +++ /dev/null @@ -1,6 +0,0 @@ -# Site Registry (orphaned) - -The `lightcone.engine.site_registry` module still exists but is not imported -by any active code path. It carries Perlmutter scheduler defaults that used -to feed the wizard for the (now removed) target system. See -[api/site_registry](../api/site_registry.md) for the current state. diff --git a/docs/hpc/targets.md b/docs/hpc/targets.md deleted file mode 100644 index 2db4548c..00000000 --- a/docs/hpc/targets.md +++ /dev/null @@ -1,3 +0,0 @@ -# Target Configuration (removed) - -The per-machine target system is gone. See [`lc target`](../cli/target.md). diff --git a/docs/maintainer.md b/docs/maintainer.md index d7671c1c..ac9a8f15 100644 --- a/docs/maintainer.md +++ b/docs/maintainer.md @@ -18,10 +18,9 @@ If you're looking for the user-facing docs, the signatures, common entry points, and module responsibilities. - [Skills](skills/index.md) — what each `/lc-*` Claude Code skill does, including the full `/lc-from-paper` reproduction bundle. -- [HPC & SLURM](hpc/index.md) — how the Dask cluster manager adapts to local, - SLURM, and external schedulers. - [Contributing](contributing/setup.md) — clone, install, run the test suite, - lint, and build the docs locally. + lint, and build the docs locally; plus how to add a container runtime, a + Dask cluster shape, or a new site. ## Get started in three commands @@ -44,5 +43,6 @@ we do not replicate any of that. The parts that are ours: - **Snakefile generator** — translates `astra.yaml` into `.lightcone/Snakefile`. - **Manifest layer** — writes and verifies `.lightcone-manifest.json` per output. -- **Cluster manager** — picks local / SLURM / external Dask shape at runtime. +- **Cluster manager** — picks the Dask shape at runtime: external scheduler, + Dask Gateway, SLURM, or local. - **Claude Code plugin** — skills, hooks, and agents bundled into the wheel. diff --git a/docs/skills/authoring.md b/docs/skills/authoring.md index 5fbfbf72..7dc1e792 100644 --- a/docs/skills/authoring.md +++ b/docs/skills/authoring.md @@ -92,7 +92,21 @@ from lightcone.eval.cli import run_cmd ## Installing changes into an existing project -`lc init` copies the plugin once and refuses to run a second time on -the same directory. See [Updating an existing project](../cli/update.md) -for the Python heredoc that resyncs all the plugin subdirs (`skills`, -`agents`, `scripts`, `guides`, `templates`) into an existing project. +`lc init` copies the plugin once and refuses to run a second time on the +same directory (it aborts when `astra.yaml` already exists). To resync +all the plugin subdirs (`skills`, `agents`, `scripts`, `guides`, +`templates`) plus `.claude/settings.json` into an existing project, call +the installer directly: + +```bash +python - <<'PY' +from pathlib import Path +from lightcone.cli.commands import _install_claude_plugin +from lightcone.cli.plugin import get_plugin_source_dir + +_install_claude_plugin(Path("."), get_plugin_source_dir(), "recommended") +PY +``` + +Existing subdirectories are removed before copying, so this is a +replace — not a merge. Local edits under `.claude/skills/` are lost. diff --git a/docs/skills/figure-comparison.md b/docs/skills/figure-comparison.md index c5b65fd6..1283f515 100644 --- a/docs/skills/figure-comparison.md +++ b/docs/skills/figure-comparison.md @@ -31,10 +31,14 @@ to run `lc run` first and stops. `comparison-report.yaml` names another universe or the user supplied one. Checks `ls results//`. 3. **Locate the paper reference substrate.** In order: a path passed as - an argument, then `work/reference/` from lc-from-paper's layout - (`source/` for arXiv TeX, `document.md` for the Docling fallback, - plus extracted `figures/` and `tables/`). Legacy locations are - tried only after lc-from-paper paths fail. + an argument, then `work/reference/` from lc-from-paper's layout — + `index.json` (the canonical structural index of figures and tables, + written by `/paper-extraction` on both paths), plus `source/` for + arXiv TeX or `document.md` for the Docling fallback, plus the + extracted `figures/` and `tables/`. Legacy locations are tried only + after lc-from-paper paths fail. A stray `metadata.json` from the + Docling path is ignored — its content is already folded into + `index.json`. ## Scope resolution @@ -50,12 +54,26 @@ The skill picks its target set in priority order: builds a best-effort report from `astra.yaml`'s `description` and `findings:` plus `work/reference/`. +## Resolving the reproduced side + +Every `lc run` output is a **directory** — +`results///` containing the artifact file(s) plus +`.lightcone-manifest.json`. The skill globs *inside* that directory for +the first suitable type-specific extension (images, tables, values), +always ignoring `.lightcone-manifest.json` and `.snakemake_timestamp`. +An explicit `reproduced_file` from the scope file wins; filename-stem +similarity across `results//*/` is the last resort, and an +unmatched target renders as a red `NOT PRODUCED` panel rather than being +guessed at. + ## Output A single `.lightcone/comparison.html` with paper artifacts on the left and reproduced artifacts on the right. Helper scripts and intermediate manifests also live under `.lightcone/` so they don't pollute the -baseline results. +baseline results. The skill never creates `.lightcone/` — it exists in +any `lc init`-ed project, and its absence means the project isn't +initialized. The HTML embeds figure images as base64 — paste it into email, drop it on a shared drive, or send it through Slack without breaking links. @@ -71,8 +89,9 @@ it on a shared drive, or send it through Slack without breaking links. - **Read-only over build artifacts.** Never run the pipeline; if outputs are missing, stop and ask the user to build first. - **Don't compare directly against a whole PDF.** When only - `work/reference/paper.pdf` exists, ask the user to run paper - extraction first. + `work/reference/paper.pdf` exists, ask the user to run + `/paper-extraction` first (in lc-from-paper projects this happens + during ORIENT). - **Preserve scope ordering.** `comparison-report.yaml` wins over `targets/targets.md` wins over the default flow. diff --git a/docs/skills/index.md b/docs/skills/index.md index 6e367fba..4579e170 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -48,7 +48,7 @@ Not entry points. Other skills invoke them — or Claude does, when a deeper ref | Skill | Command | Purpose | |-------|---------|---------| | `astra` | `/astra` | Reference for the `astra.yaml` spec: structure, decisions, options, prior insights, findings, evidence, sub-analyses, composition mechanics. | -| `lc-cli` | `/lc-cli` | Reference for `lc` workflow: commands, the Spec-Code Invariant, status interpretation, failure diagnosis, multiverse runs, publishing via WRROC. | +| `lc-cli` | `/lc-cli` | Reference for `lc` workflow: commands, the Spec-Code Invariant, status interpretation, failure diagnosis, multiverse runs, scratch overrides for HPC, JupyterHub/Dask Gateway deployments, publishing via WRROC. | These intentionally stay out of the top-level README. Researchers use the project-lifecycle skills directly; the reference skills are infrastructure. @@ -67,32 +67,39 @@ argument-hint: "[DESCRIPTION]" --- ``` -The frontmatter tells Claude Code which tools the skill may invoke -and what the slash command's argument hint looks like. The body is the -prompt itself: phase definitions, rules, references to guide files, -anti-patterns. Skills bundle their own helper scripts under `scripts/` -and longer prompt fragments under `assets/` when relevant. +`name` and `description` are required; `allowed-tools` (which tools the +skill may invoke) and `argument-hint` are optional and only some skills +declare them. The body is the prompt itself: phase definitions, rules, +references to guide files, anti-patterns. Skills bundle their own helper +scripts under `scripts/`, longer prompt fragments under `references/`, +and starter files under `templates/` when relevant. ## Plugin layout ```text claude/lightcone/ ├── skills/ -│ ├── lc-new/{SKILL.md, references/*.md} +│ ├── README.md # bundle map + co-location rationale +│ ├── lc-new/SKILL.md │ ├── lc-from-code/SKILL.md │ ├── lc-from-paper/{SKILL.md, references/*.md, templates/{constitution.md, CLAUDE.md}} │ ├── lc-feedback/SKILL.md │ ├── ralph/{SKILL.md, references/*.md, scripts/ralph} -│ ├── paper-extraction/{SKILL.md, scripts/*.py} -│ ├── figure-comparison/{SKILL.md, scripts/*.py} +│ ├── paper-extraction/{SKILL.md, references/*.md, scripts/extract-paper-substrate.py} +│ ├── figure-comparison/SKILL.md │ ├── check-sentence-by-sentence/SKILL.md │ ├── astra/SKILL.md # reference: astra.yaml spec │ └── lc-cli/SKILL.md # reference: lc workflow ├── agents/lc-extractor.md # literature subagent for /lc-new -├── templates/CLAUDE.md # the project CLAUDE.md template +├── templates/CLAUDE.md # copied to the project's .claude/templates/ └── scripts/*.sh # session lifecycle hooks (incl. session-start primer) ``` +`lc init` copies `skills/`, `agents/`, `scripts/`, `guides/`, and +`templates/` into the project's `.claude/`. The project's **root** +`CLAUDE.md` is not one of them — it is written from the `_PROJECT_CLAUDE_MD` +string in `src/lightcone/cli/commands.py`. + The plugin is force-included into the wheel via `pyproject.toml::tool.hatch.build.targets.wheel.force-include`, so `lc init` finds it whether you're running from source or PyPI. diff --git a/docs/skills/lc-feedback.md b/docs/skills/lc-feedback.md index ab79f8c1..95fda6bb 100644 --- a/docs/skills/lc-feedback.md +++ b/docs/skills/lc-feedback.md @@ -31,9 +31,9 @@ create`. - Default to `lightcone-cli` if ambiguous. 4. **Collect environment** silently: ```bash - python3 -c "import astra; print(astra.__version__)" 2>/dev/null || echo "n/a" - python3 -c "import lightcone.cli; print(lightcone.cli.__version__)" 2>/dev/null || echo "n/a" - python3 --version 2>&1 + astra --version 2>/dev/null || echo "n/a" + lc --version 2>/dev/null || echo "n/a" + python --version 2>&1 uname -s -r ``` 5. **Confirm** via `AskUserQuestion`: show target repo, title, body. diff --git a/docs/skills/lc-from-code.md b/docs/skills/lc-from-code.md index f0e2a31f..833cb65b 100644 --- a/docs/skills/lc-from-code.md +++ b/docs/skills/lc-from-code.md @@ -58,6 +58,9 @@ Hard conventions enforced by the prompt: inside it (`{output}/data.parquet`). - Don't refactor, restructure, or "improve" existing code — parameter plumbing only. +- Prefer a single shared container image unless recipes genuinely need + different stacks: on a JupyterHub/Dask Gateway deployment, `lc run` + rejects a spec resolving to more than one image per run. ### Phase 3 — Run & debug @@ -66,7 +69,8 @@ shows every output `ok`. If the scan surfaced existing results elsewhere in the project, compare them against the new `results/baseline//` to confirm the migration preserved behavior. Re-validate with `astra validate astra.yaml` and present -the summary. +the summary — flagging, for an `lc init`-scaffolded project, that the +MyST report (`index.md`) still references the boilerplate element ids. ## Hard rules diff --git a/docs/skills/lc-from-paper.md b/docs/skills/lc-from-paper.md index aaa1f4ae..238c7662 100644 --- a/docs/skills/lc-from-paper.md +++ b/docs/skills/lc-from-paper.md @@ -94,7 +94,8 @@ ORIENT drafts two files in the reproduction workdir; every iteration picks them up on launch. - **`constitution.md`** — the ralph loop's driving document, *task-bound*. - YAML frontmatter declares `status: active`. Goal (carrying the + YAML frontmatter declares `status: active` (the launcher accepts `open` + or `active`; `active` is the convention here). Goal (carrying the **fidelity intent** — the user's own "what do you want out of this stretch, given what you have to spend on it"), Scope (in/out), Quality bar, Evidence (paper DOI, arXiv ID, code repo URL), Open diff --git a/docs/skills/lc-new.md b/docs/skills/lc-new.md index 4196f3cc..3ada0d24 100644 --- a/docs/skills/lc-new.md +++ b/docs/skills/lc-new.md @@ -11,7 +11,7 @@ Source: [`claude/lightcone/skills/lc-new/SKILL.md`](https://github.com/Lightcone ```text Read, Write(astra.yaml), Write(universes/*), Write(CLAUDE.md), -Edit(astra.yaml), Edit(universes/*), Edit(CLAUDE.md), +Edit(astra.yaml), Edit(universes/*), Edit(CLAUDE.md), Edit(index.md), Glob, Grep, Bash(astra:*), Bash(lc:*), WebSearch, WebFetch, AskUserQuestion, Agent ``` @@ -38,9 +38,13 @@ files. The `lc-extractor` subagent is dispatched via `Agent`. 4. **Finalize.** `astra validate astra.yaml`; `astra validate --verify-evidence` if quotes exist; `astra universe generate -n baseline`. Author a short `description:` on the root analysis (one or - two paragraphs orienting a reader), then fill the `## Working Notes` - section of `CLAUDE.md` with conversational context the spec doesn't - carry. + two paragraphs orienting a reader), then append a `## Project Notes` + section to `CLAUDE.md` with conversational context the spec doesn't + carry, and update the scaffolded intro's "has not been scoped yet" + framing. The scaffolded MyST report (`index.md`) still references the + boilerplate ids (`example_method`, `main_result`); swap those for a + real decision ID and output ID from the finalized spec so the report + doesn't ship with dangling references. Writes happen at the end of each phase, not in bulk — the user always has something visible to review. @@ -48,8 +52,8 @@ has something visible to review. ## Hard restrictions (from the SKILL.md) - Specification agent only. No Python, no R, no implementation code. -- Touchable files: `astra.yaml`, `universes/*.yaml`, and `CLAUDE.md` - (Finalize only). +- Touchable files: `astra.yaml`, `universes/*.yaml`, `CLAUDE.md` + (Finalize only), and `index.md` (Finalize only, reference IDs only). - Quotes are never fabricated; every evidence entry must pass `astra validate --verify-evidence`. - PDFs stay inside `lc-extractor` subagents — the main agent never diff --git a/docs/user/cluster.md b/docs/user/cluster.md index d2659e75..84f7d2d6 100644 --- a/docs/user/cluster.md +++ b/docs/user/cluster.md @@ -51,6 +51,24 @@ A cluster's image is fixed at creation, so create-per-run is also what keeps the environment fresh: edit the Containerfile, `lc run`, and the next cluster runs the rebuilt image. +The cluster scales adaptively from 1 up to your `--jobs` bound, and +`lc run` waits for the first worker before dispatching — bounded by +`LIGHTCONE_GATEWAY_WORKER_TIMEOUT` (seconds, default `600`), so an image +that can't be pulled fails with a message instead of hanging at zero +workers. + +!!! warning "One image per run on this backend" + Because the worker pod *is* the container, the whole run executes in + a single image — there are no nested containers to wrap per rule. If + `astra.yaml` resolves to more than one distinct container image, + `lc run` refuses to start and asks you to consolidate on a single + Containerfile (or one shared prebuilt image). SLURM and local runs + wrap per rule and are unaffected. + +Every manifest written on this backend records a `worker_image` field — +the image the pod reported it was actually running, alongside the +`container_image` the spec declared. + ## Pre-flight: pick the right container runtime On most HPC sites, docker isn't available on compute nodes. Most @@ -264,6 +282,10 @@ scratch_root: $SCRATCH - `PermissionError` reading another user's symlinked `results/`. Cross-user scratch path without group ACLs — request access from the data owner, or copy the manifests into your own scratch. +- On JupyterHub: no worker becomes ready, a Cloud Build fails, or + `lc run` rejects a multi-image spec. See the + [Troubleshooting](troubleshooting.md) page, which has a section for + each. For the wiring detail, see [engine/dask_cluster](../api/dask_cluster.md) in the maintainer docs. diff --git a/docs/user/getting-started.md b/docs/user/getting-started.md index cfe2085b..323e4367 100644 --- a/docs/user/getting-started.md +++ b/docs/user/getting-started.md @@ -33,6 +33,8 @@ r2-decision-demo/ ├── .lightcone/ # internal scratchpad — don't edit by hand ├── Containerfile # build instructions for a local testing container ├── requirements.txt # software dependencies +├── myst.yml # config for the report (see below) +├── index.md # the report itself — a template to fill in ├── universes/ ├── src/ └── results/ @@ -50,6 +52,20 @@ materialize, but not yet a real analysis. skills will update this as you go (filling in working notes, design context). You can edit it by hand whenever you want. +And one you'll come back to at the end: + +**`index.md`** — a template [MyST](https://mystmd.org/) report. It's a +TODO-driven skeleton (Introduction / Methods / Results) that references +your analysis *by path* rather than by value: `{astra}` for inline +mentions and block embeds, `{astra:value}` for live numbers pulled from +materialized outputs. That way a re-run updates the write-up instead of +silently contradicting it. Preview it any time with: + +```bash +npm i -g mystmd # one-off, if you don't have it +myst start +``` + ## 2. Open Claude Code ```bash @@ -201,6 +217,20 @@ declare whether anything has been tampered with since materialization. Useful pre-publication, when archiving a project, or any time you want a stronger guarantee than `lc status`. +## 7. Write it up + +Now that the outputs exist, fill in `index.md` and watch it render: + +```bash +myst start # requires the MyST CLI: npm i -g mystmd +``` + +Replace the `TODO:` prose with your own, and reference the analysis +instead of retyping it — `{astra}` to mention a decision or embed the +outputs, `{astra:value}` to pull a materialized number straight out of +`results/`. Re-run the analysis and the report follows along. Builds +land in `_build/`, which is already git-ignored. + ## What just happened - `astra.yaml` was the only file you "wrote" — and the agent did most of the diff --git a/docs/user/index.md b/docs/user/index.md index 5d7a64ec..486214eb 100644 --- a/docs/user/index.md +++ b/docs/user/index.md @@ -18,7 +18,8 @@ No need to write code by hand, **you stay in charge of the scientific choices**, `/lc-from-code`, `/lc-from-paper`, and `/lc-feedback` — what each command does and when to reach for it. - [Running on a Cluster](cluster.md) — taking your analysis to a SLURM - HPC system, including Perlmutter-specific notes. + HPC system (with Perlmutter-specific notes) or to a lightcone + JupyterHub deployment on Kubernetes. - [Troubleshooting](troubleshooting.md) — common issues and how to unstick them. - [Glossary](glossary.md) — the terms that show up everywhere diff --git a/docs/user/install.md b/docs/user/install.md index 9ea81ff2..a03308cc 100644 --- a/docs/user/install.md +++ b/docs/user/install.md @@ -194,6 +194,26 @@ have either, you can still use `lc` — set `runtime: none` in `~/.lightcone/config.yaml` and recipes will run on the host without isolation. +## 6. (Optional) The MyST CLI + +`lc init` scaffolds a template report (`myst.yml` + `index.md`). To +preview or build it you need [MyST](https://mystmd.org/), which ships as +an npm package: + +```bash +npm i -g mystmd +myst --version +``` + +Then, from inside a project: + +```bash +myst start # live preview at http://localhost:3000 +``` + +You only need this when you want to *read* the report — `lc run`, +`lc status`, and `lc verify` don't touch it. + ## Sanity check lc --help diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md index b7978b3a..efee14ee 100644 --- a/docs/user/troubleshooting.md +++ b/docs/user/troubleshooting.md @@ -62,6 +62,78 @@ of `docker`, `podman`, or `podman-hpc`. Two options: image that didn't actually run — fine for development, not fine for archival. +## `lc run` on JupyterHub: "No Dask Gateway worker became ready within 600s" + +`lc run` creates a Gateway cluster and waits for its first worker before +dispatching anything. If none appears, the run stops rather than hanging +forever at zero workers. In practice this is almost always **an image +the cluster can't pull**: + +- Run `lc build` on its own and read the error — a failed Cloud Build + shows up there much more clearly. +- Check the cluster's state in the JupyterLab Dask panel; a worker pod + stuck in `ImagePullBackOff` confirms it. +- Less often it's capacity: the node pool has no room (or quota) to + schedule a worker. +- If your image is genuinely huge and the first cold pull is just slow, + raise the bound: + ```bash + export LIGHTCONE_GATEWAY_WORKER_TIMEOUT=1800 + ``` + +## "No image build backend on this host" + +You're on the `kubernetes` runtime (no local docker/podman) and the +environment doesn't carry the Cloud Build contract — `LIGHTCONE_REGISTRY` +and `LIGHTCONE_BUILD_BUCKET` (plus the optional +`LIGHTCONE_BUILD_SERVICE_ACCOUNT`). On a lightcone JupyterHub these are +injected into every user pod, so a missing one means the deployment is +misconfigured; ask your hub admin. If you landed on the `kubernetes` +runtime by accident, pin a real one in `~/.lightcone/config.yaml` +(`container: {runtime: podman}`) or pass `lc build --runtime …`. + +## A Cloud Build image build fails + +`lc build` tails the failed build's log. Read it first — the common +causes are ordinary Dockerfile problems (a package that doesn't exist +for the base image, a `COPY` of a path that isn't in the project). Two +that aren't: + +- **Permission errors from GCP.** The pod's identity needs + `cloudbuild.builds.editor`, `iam.serviceAccountUser` on the build + service account, object create/view on the build bucket, and + `artifactregistry.reader` for the freshness check. A deployment issue, + not a project one. +- **Nothing happens and it says the image is cached.** That's success: + the content hash matched an image already in the registry, so there + was nothing to build. Force it with `lc build --force`. + +## `lc run` refuses: several container images on this deployment + +On a JupyterHub deployment the worker pod *is* the container, so one run +executes in exactly one image. If `astra.yaml` declares more than one +distinct `container:` (root, sub-analysis, or recipe level), `lc run` +stops before starting and lists them. + +Consolidate on a single Containerfile that has everything, or a single +shared prebuilt image. This restriction is specific to that backend — +locally and on SLURM each rule is wrapped individually and any number of +images is fine. + +## Every rule hangs on JupyterHub, with no error + +If `lc run` reports that workers "do not advertise the lightcone +resource contract", stop there: Dask only schedules a task on a worker +advertising *every* resource key the task requests (`cpus` for every +rule, `memory` for any rule with `mem_mb`), so a worker missing them +would sit idle forever. + +`lc` provisions those keys through the gateway's standard `environment` +cluster option. Seeing this error means the deployment doesn't expose +that option, or strips it. Ask the hub admin to expose the standard +`image` / `worker_cores` / `worker_memory` / `environment` cluster +options. + ## `lc run` says "Workflow defines that rule … but no input" This is Snakemake speak. It usually means: @@ -112,14 +184,19 @@ Fix: `lc run` the downstream output. The chain will re-anchor. ## Claude Code says it can't write a file -The default permission tier (`recommended`) blocks edits to a few -sensitive places: `~/.ssh`, `~/.aws`, `~/.gnupg`, `/scratch`, -`/pscratch`, plus `sudo`, `git push`, `rm -rf`, … +The default permission tier (`recommended`) **blocks** edits to a few +sensitive places — `~/.ssh`, `~/.aws`, `~/.gnupg` — plus `sudo`, +`git push`, `rm -rf`. + +It also **prompts** (rather than blocks) before writing under `/scratch` +or `/pscratch`: on HPC your project often lives there, so the agent +legitimately needs to write to it, but a stray edit would be expensive. +Approve the prompt and it proceeds. -If the file you're trying to edit isn't in those, check -`.claude/settings.json`. If it is — your `recommended` tier is doing -its job. Either move the work elsewhere or, knowing what you're doing, -invoke `lc init … --permissions yolo` next time. +If the file you're trying to edit isn't in either list, check +`.claude/settings.json`. If it's in the deny list — your `recommended` +tier is doing its job. Either move the work elsewhere or, knowing what +you're doing, invoke `lc init … --permissions yolo` next time. ## I deleted `.claude/` by accident diff --git a/zensical.toml b/zensical.toml index 34661408..9a89cc06 100644 --- a/zensical.toml +++ b/zensical.toml @@ -30,18 +30,23 @@ nav = [ {"lc status" = "cli/status.md"}, {"lc verify" = "cli/verify.md"}, {"lc export" = "cli/export.md"}, + {"lc eval" = "cli/eval.md"}, ]}, {"Python API" = [ {"Overview" = "api/index.md"}, {"cli/commands" = "api/cli.md"}, {"engine/manifest" = "api/manifest.md"}, {"engine/snakefile" = "api/snakefile.md"}, + {"engine/runner" = "api/runner.md"}, {"engine/container" = "api/container.md"}, {"engine/cloudbuild" = "api/cloudbuild.md"}, + {"engine/scratch" = "api/scratch.md"}, + {"engine/site_registry" = "api/site_registry.md"}, {"engine/status" = "api/status.md"}, {"engine/verify" = "api/verify.md"}, {"engine/tree" = "api/tree.md"}, {"engine/validation" = "api/validation.md"}, + {"engine/wrroc" = "api/wrroc.md"}, {"engine/dask_cluster" = "api/dask_cluster.md"}, {"snakemake_executor_plugin_dask" = "api/dask_executor.md"}, ]}, @@ -60,6 +65,8 @@ nav = [ {"Contributing" = [ {"Development Setup" = "contributing/setup.md"}, {"Testing" = "contributing/testing.md"}, + {"Adding an Execution Backend" = "contributing/backends.md"}, + {"Adding a Site" = "contributing/hpc-sites.md"}, ]}, ]}, {"ASTRA docs" = "https://astra-spec.org/latest/"},