From a0a7587be0a5cc4b822a8d3987a7529cae46f730 Mon Sep 17 00:00:00 2001 From: Giuseppe Infusini Date: Fri, 11 Sep 2026 00:52:13 +1000 Subject: [PATCH] examples: four walkthrough notebooks for the common API journeys Each one takes something you would normally do by clicking in the app and does it from the API instead, explaining the wire formats that are not guessable from the client and what happens when you get them wrong. md_dr_pairwise_walkthrough upload, then pairwise + dose response md_workspace_walkthrough workspaces, tabs, registry, the grid md_qc_report_automation the QC Report automation, rebuilt md_pairwise_ora_walkthrough saved entity lists and pathway enrichment All four run end to end against dev. Checked in without saved outputs so readers see their own results. The ORA notebook additionally needs pandas and requests, noted in the folder README. --- examples/walkthroughs/README.md | 55 ++ .../md_dr_pairwise_walkthrough.ipynb | 570 +++++++++++++++++ .../md_pairwise_ora_walkthrough.ipynb | 568 +++++++++++++++++ .../md_qc_report_automation.ipynb | 577 +++++++++++++++++ .../md_workspace_walkthrough.ipynb | 580 ++++++++++++++++++ 5 files changed, 2350 insertions(+) create mode 100644 examples/walkthroughs/README.md create mode 100644 examples/walkthroughs/md_dr_pairwise_walkthrough.ipynb create mode 100644 examples/walkthroughs/md_pairwise_ora_walkthrough.ipynb create mode 100644 examples/walkthroughs/md_qc_report_automation.ipynb create mode 100644 examples/walkthroughs/md_workspace_walkthrough.ipynb diff --git a/examples/walkthroughs/README.md b/examples/walkthroughs/README.md new file mode 100644 index 0000000..29ca0bf --- /dev/null +++ b/examples/walkthroughs/README.md @@ -0,0 +1,55 @@ +# Walkthroughs + +Four notebooks that each take one thing you would normally do by clicking in the app and do it from +the API instead. They are meant to be read as much as run: every non-obvious wire format is +explained where it appears, along with what happens if you get it wrong. + +Read them in this order if you are new to the client. Each one assumes the one before it. + +| notebook | what it covers | +|---|---| +| `md_dr_pairwise_walkthrough.ipynb` | Upload data, then run a pairwise comparison and a per-compound dose response from a single upload | +| `md_workspace_walkthrough.ipynb` | Workspaces, tabs, the module registry and the 12-column grid | +| `md_qc_report_automation.ipynb` | The in-app Quality Control Report automation, rebuilt from the API | +| `md_pairwise_ora_walkthrough.ipynb` | Saved entity lists and pathway enrichment, starting from a pairwise result | + +## Running them + +Each notebook has a single configuration cell near the top. Change the ids there and run the cells +in order; nothing else needs editing. + +You will need an `MD_AUTH_TOKEN` in a `.env` file next to the notebook, or in your environment: + +``` +MD_AUTH_TOKEN=... +``` + +Beyond `md-python` itself, they need `python-dotenv`. `md_pairwise_ora_walkthrough.ipynb` also needs +`pandas` and `requests`, because it downloads a result table and decides significance locally: + +``` +pip install python-dotenv pandas requests +``` + +The notebooks are checked in without saved outputs, so you see your own results rather than someone +else's. + +## What they create + +All four write to the API. Between them they create uploads, datasets, analysis jobs, workspaces, +tabs, modules and entity lists in whichever organisation your token belongs to. Point them at a +development environment first, and delete what you no longer need. + +## Two things worth knowing before you start + +**Module settings are validated for shape, not for meaning.** The API checks that a settings hash +has the right keys and the right types, and returns a 422 when it does not. It does not check that a +value makes sense. A metadata column name that does not exist, or an enum value with the wrong +capitalisation, is accepted, stored, and only shows up later as an empty widget on a tab. The QC +report notebook builds a small validator for the checks you can make up front, and is explicit about +the ones you cannot. + +**A required setting can still be empty.** Several modules declare a setting as required and give it +a default of `[]`. An empty list satisfies the requirement, so the module is created and then +renders nothing. Where that applies it is called out in the notebook, but it is the single easiest +way to build a report that looks broken for no visible reason. diff --git a/examples/walkthroughs/md_dr_pairwise_walkthrough.ipynb b/examples/walkthroughs/md_dr_pairwise_walkthrough.ipynb new file mode 100644 index 0000000..cf4adc2 --- /dev/null +++ b/examples/walkthroughs/md_dr_pairwise_walkthrough.ipynb @@ -0,0 +1,570 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cell00", + "metadata": {}, + "source": [ + "# One upload → pairwise + dose-response\n", + "\n", + "Upload a proteomics dataset to Mass Dynamics once, then run **both** analyses off it:\n", + "\n", + "1. **Pairwise**: every compound@dose compared against the vehicle, in a *single* job.\n", + "2. **Dose-response**: a 4-parameter curve per protein, in *one job per compound*.\n", + "\n", + "Edit the **Configuration** cell, then run the notebook top to bottom.\n", + "\n", + "### Before you start\n", + "\n", + "```bash\n", + "pip install \"git+https://github.com/MassDynamics/md-python.git\" python-dotenv\n", + "```\n", + "\n", + "and put your credentials in a `.env` file next to this notebook:\n", + "\n", + "```\n", + "MD_API_BASE_URL=https://app.massdynamics.com/api\n", + "MD_AUTH_TOKEN=\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "cell01", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell02", + "metadata": {}, + "outputs": [], + "source": [ + "import csv\n", + "import os\n", + "from pathlib import Path\n", + "\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient, Dataset, Upload, ExperimentDesign, SampleMetadata\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)\n", + "print(\"Health:\", client.health.check())" + ] + }, + { + "cell_type": "markdown", + "id": "cell03", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "The only cell you need to edit." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell04", + "metadata": {}, + "outputs": [], + "source": [ + "# Folder with your data files, and which files to upload from it.\n", + "DATA_DIR = Path(\"/Your/Path/To/Data\")\n", + "FILENAMES = [\"YOUR_PROTEIN_FILE.tsv\", \"YOUR_PEPTIDE_FILE.tsv\"]\n", + "\n", + "# Sample metadata CSV: sample_name, condition, dose\n", + "METADATA_CSV = Path(\"/Your/Path/To/Data/sample_metadata.csv\")\n", + "\n", + "# Upload name. Must be unique within your organisation.\n", + "UPLOAD_NAME = \"YOUR UPLOAD NAME\"\n", + "\n", + "# spectronaut | diann_tabular | tims_diann | maxquant | md_format | md_format_gene\n", + "SOURCE = \"md_format\"\n", + "\n", + "# Which condition is the vehicle / control. Must match the CSV exactly.\n", + "CONTROL_CONDITION = \"YOUR_CONTROL_CONDITION\"\n", + "\n", + "# Set this to an existing upload id to skip the upload and re-run the analyses.\n", + "EXISTING_UPLOAD_ID = None" + ] + }, + { + "cell_type": "markdown", + "id": "cell05", + "metadata": {}, + "source": [ + "## 1. Read the sample metadata\n", + "\n", + "Your CSV needs one row per sample and three columns:\n", + "\n", + "| column | meaning |\n", + "|---|---|\n", + "| `sample_name` | must match the run names inside your data file exactly (for Spectronaut, `R.FileName`) |\n", + "| `condition` | the **compound**, not compound+dose. Every dose of a compound shares it. |\n", + "| `dose` | a number. The vehicle is `0`. |\n", + "\n", + "```\n", + "sample_name,condition,dose\n", + "SAMPLE_01,DRUG_1,5\n", + "SAMPLE_02,DRUG_1,2\n", + "SAMPLE_03,CONTROL,0\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell06", + "metadata": {}, + "outputs": [], + "source": [ + "with METADATA_CSV.open(newline=\"\", encoding=\"utf-8-sig\") as fh:\n", + " rows = list(csv.DictReader(fh))\n", + "\n", + "for col in (\"sample_name\", \"condition\", \"dose\"):\n", + " assert col in rows[0], f\"{METADATA_CSV} is missing a '{col}' column\"\n", + "\n", + "conditions = sorted({r[\"condition\"] for r in rows})\n", + "print(f\"{len(rows)} samples, {len(conditions)} conditions: {conditions}\")\n", + "rows[:3]" + ] + }, + { + "cell_type": "markdown", + "id": "cell07", + "metadata": {}, + "source": [ + "## 2. Build the two metadata tables\n", + "\n", + "**`experiment_design`** maps data-file run names to samples: `[filename, sample_name, condition]`.\n", + "For LFQ (one run per sample) `filename` is just the sample name.\n", + "\n", + "**`sample_metadata`** carries the per-sample variables the analyses use. We add a derived\n", + "**`condition_dose`** column (e.g. `DRUG_1@5`).\n", + "\n", + "That derived column matters: the pairwise step groups on it so **each dose gets its own comparison\n", + "against the vehicle**. Grouping on `condition` alone would pool every dose of a compound into one\n", + "group, which is almost never what you want from a dose series." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell08", + "metadata": {}, + "outputs": [], + "source": [ + "experiment_design = ExperimentDesign(\n", + " data=[[\"filename\", \"sample_name\", \"condition\"]]\n", + " + [[r[\"sample_name\"], r[\"sample_name\"], r[\"condition\"]] for r in rows]\n", + ")\n", + "\n", + "sample_metadata = SampleMetadata(\n", + " data=[[\"sample_name\", \"condition\", \"dose\", \"condition_dose\"]]\n", + " + [\n", + " [r[\"sample_name\"], r[\"condition\"], r[\"dose\"], f\"{r['condition']}@{r['dose']}\"]\n", + " for r in rows\n", + " ]\n", + ")\n", + "\n", + "print(experiment_design)\n", + "print()\n", + "print(sample_metadata)" + ] + }, + { + "cell_type": "markdown", + "id": "cell09", + "metadata": {}, + "source": [ + "## 3. Upload\n", + "\n", + "`client.uploads.create()` blocks while the files transfer, then starts ingestion.\n", + "\n", + "Skipped automatically if you set `EXISTING_UPLOAD_ID` in the config cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell10", + "metadata": {}, + "outputs": [], + "source": [ + "if EXISTING_UPLOAD_ID:\n", + " upload_id = EXISTING_UPLOAD_ID\n", + " print(\"Reusing upload:\", upload_id)\n", + "else:\n", + " for name in FILENAMES:\n", + " assert (DATA_DIR / name).is_file(), f\"Missing data file: {DATA_DIR / name}\"\n", + "\n", + " upload = Upload(\n", + " name=UPLOAD_NAME,\n", + " source=SOURCE,\n", + " experiment_design=experiment_design,\n", + " sample_metadata=sample_metadata,\n", + " file_location=str(DATA_DIR),\n", + " filenames=FILENAMES,\n", + " )\n", + " print(f\"Uploading {len(FILENAMES)} file(s)...\")\n", + " upload_id = client.uploads.create(upload)\n", + " print(\"Upload ID:\", upload_id)" + ] + }, + { + "cell_type": "markdown", + "id": "cell11", + "metadata": {}, + "source": [ + "## 4. Wait for the upload **and** for its dataset\n", + "\n", + "Two waits, not one. The second is the one people miss.\n", + "\n", + "The upload reporting `COMPLETED` does not mean its intensity dataset is `COMPLETED`; they flip a\n", + "moment apart. Submit a job into that gap and the pairwise job is rejected outright, while the\n", + "dose-response job is *accepted* and then dies server-side with a confusing `NoneType` error. So wait\n", + "for the dataset itself before going any further." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell12", + "metadata": {}, + "outputs": [], + "source": [ + "client.uploads.wait_until_complete(upload_id, poll_s=15, timeout_s=7200)\n", + "print(\"Upload ingested.\")\n", + "\n", + "dataset = client.datasets.find_initial_dataset(upload_id)\n", + "dataset_id = str(dataset.id)\n", + "\n", + "state = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=dataset_id, poll_s=15, timeout_s=7200\n", + ")\n", + "print(\"Intensity dataset:\", dataset_id, \"->\", state.state)\n", + "assert str(state.state) == \"COMPLETED\"" + ] + }, + { + "cell_type": "markdown", + "id": "cell13", + "metadata": {}, + "source": [ + "## 5. What do the analysis jobs accept?\n", + "\n", + "`GET /jobs` returns each job's parameter definition: field names, types, defaults, and which are\n", + "required. This is the source of truth for `job_run_params`, and it is worth a look before you submit:\n", + "these definitions change, and what you see here always matches the server you are talking to.\n", + "\n", + "Two notes on reading it:\n", + "\n", + "* `input_datasets` is listed but does **not** go in `job_run_params`. The server fills it in from\n", + " the dataset id you pass separately.\n", + "* Some fields only apply in combination, e.g. `filter_threshold_percentage` applies when\n", + " `filter_values_criteria` is `\"percentage\"`. Send the one that matches.\n", + "\n", + "The `job_get` helper below exists because `jobs.list()` returns plain dicts on some md-python\n", + "versions and `Job` objects on others (and spells the published flag `isPublished` in one,\n", + "`is_published` in the other). It reads either." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a5ac9d66", + "metadata": {}, + "outputs": [], + "source": [ + "# Every job slug this API exposes, grouped by run type.\n", + "from collections import defaultdict\n", + "\n", + "jobs = client.jobs.list()\n", + "\n", + "by_type = defaultdict(list)\n", + "for job in jobs:\n", + " by_type[str(job.run_type)].append(job)\n", + "\n", + "published = sum(1 for j in jobs if j.is_published)\n", + "print(f\"{len(jobs)} jobs on {client.base_url} ({published} published)\\n\")\n", + "\n", + "for run_type in sorted(by_type):\n", + " print(run_type)\n", + " for job in sorted(by_type[run_type], key=lambda j: j.slug or \"\"):\n", + " flag = \"\" if job.is_published else \" <- not published\"\n", + " print(f\" {str(job.slug):<32} {len(job.properties or {}):>3} params {job.name}{flag}\")\n", + " print()\n", + "\n", + "# Bare list. Paste into a comment, or use it to validate a job_slug before submitting\n", + "print(sorted(j.slug for j in jobs if j.is_published and j.slug))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell14", + "metadata": {}, + "outputs": [], + "source": [ + "def show_params(slug):\n", + " \"\"\"Print the parameters a job accepts. Fetches its own job list, so this cell\n", + " works on its own, with no dependency on an earlier cell having run.\"\"\"\n", + " job = next((j for j in client.jobs.list() if j.slug == slug), None)\n", + " assert job, f\"No job with slug {slug!r}\"\n", + "\n", + " props = job.properties or {}\n", + " print(f\"{slug}: {len(props)} parameters\\n\")\n", + " for name, spec in props.items():\n", + " if not isinstance(spec, dict):\n", + " print(f\" {name}\")\n", + " continue\n", + " bits = []\n", + " if spec.get(\"fieldType\"):\n", + " bits.append(str(spec[\"fieldType\"]))\n", + " if \"default\" in spec:\n", + " bits.append(f\"default={spec['default']!r}\")\n", + " if any(r.get(\"name\") == \"is_required\" for r in spec.get(\"rules\", [])):\n", + " bits.append(\"REQUIRED\")\n", + " print(f\" {name:<32} {' '.join(bits)}\")\n", + "\n", + "show_params(\"pairwise_comparison\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "cell16", + "metadata": {}, + "source": [ + "## 6. Pairwise: every compound@dose vs the vehicle\n", + "\n", + "All the comparisons go into **one** job on purpose: limma then computes a single shared FDR\n", + "correction across them. Do not loop and submit one job per pair." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell17", + "metadata": {}, + "outputs": [], + "source": [ + "design_columns = sample_metadata.to_columns()\n", + "CONDITION_COLUMN = \"condition_dose\" # or \"condition\" to pool doses\n", + "\n", + "groups = []\n", + "for value in design_columns[CONDITION_COLUMN]:\n", + " if value not in groups:\n", + " groups.append(value)\n", + "\n", + "control_groups = {\n", + " g for g, cond in zip(design_columns[CONDITION_COLUMN], design_columns[\"condition\"])\n", + " if cond == CONTROL_CONDITION\n", + "}\n", + "assert control_groups, f\"No samples with condition == {CONTROL_CONDITION!r}\"\n", + "assert len(control_groups) == 1, (\n", + " f\"The vehicle spans several groups: {sorted(control_groups)}. \"\n", + " \"Give it a single dose value, or set CONDITION_COLUMN = 'condition'.\"\n", + ")\n", + "control = control_groups.pop()\n", + "\n", + "comparisons = [[g, control] for g in groups if g != control]\n", + "print(f\"{len(comparisons)} comparisons against {control!r}\")\n", + "comparisons[:5]" + ] + }, + { + "cell_type": "markdown", + "id": "cell18", + "metadata": {}, + "source": [ + "The two parameters below are the ones that are easy to get wrong. Both were taken from the job\n", + "spec printed above, not guessed:\n", + "\n", + "* **`filter_values_criteria`** is a **string** (`\"percentage\"` or `\"count\"`), *not* a dict, and it\n", + " needs a matching sibling: `filter_threshold_percentage` or `filter_threshold_count`.\n", + "* **`de_method_`** is **required** and is keyed by entity type, so protein data needs\n", + " `de_method_protein`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell19", + "metadata": {}, + "outputs": [], + "source": [ + "ENTITY_TYPE = \"protein\" # protein | peptide | gene\n", + "\n", + "pairwise_params = {\n", + " \"condition_column\": CONDITION_COLUMN,\n", + " \"condition_comparisons\": {\"condition_comparison_pairs\": comparisons},\n", + " \"experiment_design\": design_columns,\n", + " \"entity_type\": ENTITY_TYPE,\n", + " f\"de_method_{ENTITY_TYPE}\": \"limma\",\n", + " \"filter_values_criteria\": \"percentage\",\n", + " \"filter_threshold_percentage\": 0.5,\n", + " \"filter_valid_values_logic\": \"at least one condition\",\n", + " \"fit_separate_models\": True,\n", + " \"limma_trend\": True,\n", + " \"robust_empirical_bayes\": True,\n", + "}\n", + "\n", + "pairwise_id = client.datasets.create(\n", + " Dataset(\n", + " input_dataset_ids=[dataset_id],\n", + " name=f\"{UPLOAD_NAME} - pairwise vs control\",\n", + " job_slug=\"pairwise_comparison\",\n", + " job_run_params=pairwise_params,\n", + " )\n", + ")\n", + "print(\"Pairwise dataset:\", pairwise_id)" + ] + }, + { + "cell_type": "markdown", + "id": "cell20", + "metadata": {}, + "source": [ + "## 7. Dose-response: one job per compound\n", + "\n", + "The opposite fan-out to pairwise, and for a concrete reason: the dose-response pipeline fits one\n", + "curve per protein and groups samples by **dose only**. It has no notion of which compound a sample\n", + "came from, so putting two compounds in one job silently pools them into a single meaningless curve.\n", + "\n", + "Each job therefore gets one compound's dose series **plus the vehicle samples** as the dose-0 anchor.\n", + "\n", + "The platform requires at least **5 samples** and **3 distinct dose levels** per job (the vehicle's 0\n", + "counts). Compounds that fall short are reported and skipped rather than failing the run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell21", + "metadata": {}, + "outputs": [], + "source": [ + "control_rows = [r for r in rows if r[\"condition\"] == CONTROL_CONDITION]\n", + "compounds = [c for c in conditions if c != CONTROL_CONDITION]\n", + "\n", + "dose_response_ids = {}\n", + "for compound in compounds:\n", + " subset = [r for r in rows if r[\"condition\"] in (compound, CONTROL_CONDITION)]\n", + " sample_names = [r[\"sample_name\"] for r in subset]\n", + " doses = [float(r[\"dose\"]) for r in subset] # must be numbers, not strings\n", + " distinct = sorted(set(doses))\n", + "\n", + " if len(sample_names) < 5 or len(distinct) < 3:\n", + " print(f\" SKIP {compound}: {len(sample_names)} samples, {len(distinct)} dose levels\")\n", + " continue\n", + "\n", + " # experiment_design is the ONLY thing that scopes the job.\n", + " params = {\n", + " \"experiment_design\": {\"sample_name\": sample_names, \"dose\": doses},\n", + " \"control_samples\": [r[\"sample_name\"] for r in control_rows],\n", + " \"log_intensities\": True,\n", + " \"use_imputed_intensities\": False,\n", + " \"normalise\": \"none\",\n", + " \"span_rollmean_k\": 1,\n", + " \"prop_required_in_protein\": 0.5,\n", + " }\n", + "\n", + " ds_id = client.datasets.create(\n", + " Dataset(\n", + " input_dataset_ids=[dataset_id],\n", + " name=f\"{UPLOAD_NAME} - DR {compound}\",\n", + " job_slug=\"dose_response\",\n", + " job_run_params=params,\n", + " )\n", + " )\n", + " dose_response_ids[compound] = ds_id\n", + " print(f\" {compound}: {len(sample_names)} samples, doses {distinct} -> {ds_id}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell22", + "metadata": {}, + "source": [ + "## 8. Wait for everything" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell23", + "metadata": {}, + "outputs": [], + "source": [ + "pw_state = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=pairwise_id, poll_s=15, timeout_s=7200\n", + ")\n", + "print(\"pairwise:\", pw_state.state)\n", + "\n", + "for compound, ds_id in dose_response_ids.items():\n", + " st = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=ds_id, poll_s=15, timeout_s=7200\n", + " )\n", + " print(f\"{compound}: {st.state}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell24", + "metadata": {}, + "source": [ + "## Results\n", + "\n", + "Everything is now visible in the Mass Dynamics web app under this upload. To pull a table down\n", + "programmatically:\n", + "\n", + "```python\n", + "client.datasets.list_by_upload(upload_id) # what exists\n", + "[t.name for t in client.datasets.get_by_id(pairwise_id).tables] # what tables it has\n", + "url = client.datasets.download_table_url(pairwise_id, \"output_comparisons\", format=\"csv\")\n", + "```\n", + "\n", + "| dataset type | tables |\n", + "|---|---|\n", + "| `PAIRWISE` | `output_comparisons`, `runtime_metadata` |\n", + "| `DOSE_RESPONSE` | `output_curves`, `output_volcanoes`, `input_drc`, `runtime_metadata` |\n", + "\n", + "In `output_volcanoes` the half-maximal dose column is **`ED50`** (the `drc` package's terminology),\n", + "not `EC50`.\n", + "\n", + "### If a job is rejected\n", + "\n", + "Re-run the `show_params(...)` cell and compare the field list against what you are sending. The\n", + "API is the source of truth and these definitions do change." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "md-python (3.13.13.final.0)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/walkthroughs/md_pairwise_ora_walkthrough.ipynb b/examples/walkthroughs/md_pairwise_ora_walkthrough.ipynb new file mode 100644 index 0000000..1c4ae6b --- /dev/null +++ b/examples/walkthroughs/md_pairwise_ora_walkthrough.ipynb @@ -0,0 +1,568 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Saved lists and pathway enrichment from a pairwise result\n", + "\n", + "The app has three automations that do this by hand: **Pairwise Analysis Volcano**, **Pairwise\n", + "Analysis Generate Up/Down Regulated Lists**, and the **Reactome ORA Template**. You run them one\n", + "after the other, answering a form each time.\n", + "\n", + "This notebook does the same chain from the API, with no clicking. It starts from a pairwise dataset\n", + "you already have:\n", + "\n", + "1. pull the pairwise result table down and decide up and down regulated **in this script**\n", + "2. save those as entity lists in the workspace\n", + "3. build a tab with the volcanoes, a selection-driven table and an abundance plot\n", + "4. run a pathway enrichment job on each up-regulated list and plot the result on a second tab\n", + "\n", + "Creating the pairwise dataset itself is a separate job and is covered by the pairwise walkthrough\n", + "notebook. Here you just need its id.\n", + "\n", + "### The one thing that is genuinely different over the API\n", + "\n", + "The in-app automation calls `PairwiseGetUpRegulated` and `PairwiseGetDownRegulated`. Those are\n", + "**browser instructions**. They read the volcano module's own thresholds and split the proteins\n", + "client-side. There is no endpoint for them, so you cannot ask the API \"which proteins are up in this\n", + "volcano\".\n", + "\n", + "You do not need one. The pairwise result table is downloadable, it carries an adjusted p-value and\n", + "a log2 fold change per comparison, and the split is two lines of pandas. **When you script this, the\n", + "significance call is yours to make, in your own code, on data you can see.** That is arguably the\n", + "point: the threshold stops being a setting buried in a plot and becomes a line you can read, version\n", + "and change." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import io\n", + "import os\n", + "\n", + "import pandas as pd\n", + "import requests\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient, Dataset\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "Everything you would otherwise answer in three automation forms. One id and a handful of numbers." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# The PAIRWISE dataset to work from. Make one with the pairwise walkthrough\n", + "# notebook or in the app, then paste its id here.\n", + "PAIRWISE_ID = \"00000000-0000-0000-0000-000000000000\"\n", + "\n", + "ENTITY_TYPE = \"protein\" # lowercase, always\n", + "CONDITION_COLUMN = \"condition\" # the sample metadata column holding the groups\n", + "\n", + "# Significance. The volcano module defaults to 0.05; we tighten it and, crucially,\n", + "# apply the SAME numbers here in Python when we build the lists, so the plot and\n", + "# the saved list agree.\n", + "FDR = 0.01\n", + "FOLD_CHANGE = 2 # linear, so |log2FC| >= 1\n", + "\n", + "# Pathway enrichment. The ORA job takes a species name, not a taxon id.\n", + "# human | mouse | yeast | chinese_hamster\n", + "SPECIES = \"human\"\n", + "\n", + "# Gene set database. Options depend on the species; for human they are Reactome,\n", + "# the three GO branches, and the MSigDB collections.\n", + "DATABASE = \"Reactome\"\n", + "\n", + "TOP_N = 15 # pathways shown in the dot plot\n", + "\n", + "WORKSPACE_NAME = \"Pairwise + ORA (API)\"\n", + "ANALYSIS_TAB = \"Differential expression\"\n", + "ORA_TAB = \"Pathway enrichment (ORA)\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Read the analysis off the pairwise dataset\n", + "\n", + "A pairwise dataset already records what it was run on and what it compared, so there is nothing to\n", + "retype. Three things come straight off it:\n", + "\n", + "* the **intensity dataset** it was computed from, which the entity lists and the abundance plot both\n", + " need\n", + "* the **contrasts** it ran, as pairs of condition values\n", + "* through the intensity dataset, the **upload** and its sample metadata\n", + "\n", + "Taking the contrasts from the job rather than typing them matters. Every downstream step looks up\n", + "columns named after the pair, like `Log2FC Treated - Control`. A pair typed by hand that does not\n", + "match the job exactly fails several cells later with a missing column, and the name it complains\n", + "about looks correct at a glance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import math\n", + "\n", + "pairwise = client.datasets.get_by_id(PAIRWISE_ID)\n", + "assert pairwise, f\"No dataset {PAIRWISE_ID!r}\"\n", + "assert str(pairwise.type) == \"PAIRWISE\", f\"{pairwise.name} is a {pairwise.type} dataset\"\n", + "print(f\"{pairwise.name} ({pairwise.type}, {pairwise.state})\")\n", + "\n", + "# The intensity dataset this pairwise was computed from.\n", + "DATASET_ID = str(pairwise.input_dataset_ids[0])\n", + "dataset = client.datasets.get_by_id(DATASET_ID)\n", + "print(f\"computed from: {dataset.name} ({DATASET_ID})\")\n", + "\n", + "# The contrasts the job actually ran. (test, reference): positive log2FC means\n", + "# higher in the test group.\n", + "params = pairwise.job_run_params or {}\n", + "CONTRASTS = [tuple(p) for p in params[\"condition_comparisons\"][\"condition_comparison_pairs\"]]\n", + "print(f\"\\n{len(CONTRASTS)} contrast(s):\")\n", + "for left, right in CONTRASTS:\n", + " print(f\" {left} vs {right}\")\n", + "\n", + "# The upload behind the intensity dataset, for its sample metadata. The groups\n", + "# are used by the abundance plot at the end.\n", + "upload_id = None\n", + "for u in client.uploads.query(search=dataset.name)[\"data\"]:\n", + " if DATASET_ID in [str(d.id) for d in client.datasets.list_by_upload(u[\"id\"])]:\n", + " upload_id = u[\"id\"]\n", + " break\n", + "assert upload_id, \"Could not find the upload that owns the intensity dataset\"\n", + "\n", + "rows = client.uploads.get_sample_metadata(upload_id).data\n", + "header, body = rows[0], rows[1:]\n", + "groups = sorted({r[header.index(CONDITION_COLUMN)] for r in body})\n", + "print(f\"\\nupload {upload_id}\\n{CONDITION_COLUMN}: {groups}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Download the table and make the call yourself\n", + "\n", + "`download_table_url` returns a presigned S3 URL for one table of the dataset. `output_comparisons`\n", + "is the one with the statistics.\n", + "\n", + "Its columns are per contrast and named after the pair: `Log2FC Treated - Control`,\n", + "`AdjPValue Treated - Control`, and so on. Alongside them sit the identity columns every downstream\n", + "step needs: `ProteinIds`, `GeneNames`, `Description` and `GroupId`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "url = client.datasets.download_table_url(PAIRWISE_ID, \"output_comparisons\")\n", + "raw = requests.get(url if isinstance(url, str) else url[\"url\"]).content\n", + "comparisons = pd.read_csv(io.BytesIO(raw))\n", + "\n", + "print(f\"{len(comparisons)} entities x {len(comparisons.columns)} columns\")\n", + "print(\"identity columns:\", [c for c in (\"ProteinIds\", \"GeneNames\", \"Description\", \"GroupId\")\n", + " if c in comparisons.columns])\n", + "print(\"\\nper-contrast columns:\")\n", + "for left, right in CONTRASTS:\n", + " pair = f\"{left} - {right}\"\n", + " have = [f\"{p} {pair}\" in comparisons.columns for p in (\"Log2FC\", \"AdjPValue\")]\n", + " print(f\" {pair:<28} Log2FC {have[0]} AdjPValue {have[1]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "LOG2_FC = math.log2(FOLD_CHANGE)\n", + "\n", + "\n", + "def split_regulated(df, left, right, fdr=FDR, log2fc=LOG2_FC):\n", + " \"\"\"Up and down regulated entities for one contrast.\n", + "\n", + " This is the whole 'significance call', in the open. Change the comparison\n", + " operators or the columns and you change what gets saved, with no hidden\n", + " module setting involved.\n", + " \"\"\"\n", + " pair = f\"{left} - {right}\"\n", + " fc, p = df[f\"Log2FC {pair}\"], df[f\"AdjPValue {pair}\"]\n", + " significant = p < fdr\n", + " return (df[significant & (fc >= log2fc)].copy(),\n", + " df[significant & (fc <= -log2fc)].copy())\n", + "\n", + "\n", + "regulated = {}\n", + "print(f\"FDR < {FDR}, |log2FC| >= {LOG2_FC:g} (fold change {FOLD_CHANGE}x)\\n\")\n", + "for left, right in CONTRASTS:\n", + " up, down = split_regulated(comparisons, left, right)\n", + " regulated[(left, right)] = {\"Up\": up, \"Down\": down}\n", + " print(f\" {left} vs {right:<12} up {len(up):>4} down {len(down):>4}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Save them as entity lists\n", + "\n", + "`client.workspaces.entity_lists.create()`. Lists live under a workspace, which is why the resource\n", + "hangs off `client.workspaces` rather than sitting at the top level of the client.\n", + "\n", + "Each item needs an `entity_id`, a `group_id` and a `dataset_id`, which is exactly the `ProteinIds` /\n", + "`GroupId` pair from the table plus the dataset they belong to.\n", + "\n", + "The non-obvious part is which dataset. It has to be the **intensity** dataset, not the pairwise one\n", + "the numbers came from: the server checks that every referenced dataset is `INTENSITY` and holds a\n", + "`Protein_Metadata` table, and a pairwise dataset is neither. Pass the pairwise id and you get a 400\n", + "that names the type. `GroupId` is safe to carry across, because a pairwise dataset inherits the\n", + "group ids of the intensity dataset it was computed from.\n", + "\n", + "The names follow the automation's convention, `{left}_vs_{right}_Up_Reg`, so lists made this way sit\n", + "alongside lists made by hand in the app and sort together." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "workspace = client.workspaces.create(\n", + " name=WORKSPACE_NAME, description=\"Pairwise, saved lists and ORA, built through the API\"\n", + ")\n", + "WORKSPACE_ID = str(workspace.id)\n", + "print(\"workspace:\", WORKSPACE_ID)\n", + "\n", + "\n", + "def create_entity_list(name, frame):\n", + " \"\"\"Create an entity list from rows of the comparisons table.\"\"\"\n", + " entity_list = client.workspaces.entity_lists.create(\n", + " workspace_id=WORKSPACE_ID,\n", + " name=name,\n", + " entity_type=ENTITY_TYPE,\n", + " items=[\n", + " {\"entity_id\": str(r.ProteinIds), \"group_id\": int(r.GroupId), \"dataset_id\": DATASET_ID}\n", + " for r in frame.itertuples()\n", + " ],\n", + " )\n", + " return str(entity_list.id)\n", + "\n", + "\n", + "lists = {}\n", + "for (left, right), sides in regulated.items():\n", + " for direction, frame in sides.items():\n", + " if frame.empty:\n", + " print(f\" skipped {left}_vs_{right}_{direction}_Reg (empty)\")\n", + " continue\n", + " name = f\"{left}_vs_{right}_{direction}_Reg\"\n", + " lists[(left, right, direction)] = create_entity_list(name, frame)\n", + " print(f\" {name:<40} {len(frame):>4} entities\")\n", + "\n", + "print(f\"\\n{len(lists)} lists created\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. The analysis tab\n", + "\n", + "Three volcanoes, one per contrast, each carrying **its own** up and down list as a highlight. The\n", + "highlight shape comes from the automation's `SetProteinListModule` instruction: a list of\n", + "`{\"id\": ...}` per entity list.\n", + "\n", + "The automation also offers `showAnnotation` with an `annotationLabel` of `\"gene\"`, and it is left\n", + "off here on purpose. These lists run to hundreds of proteins, and labelling every highlighted point\n", + "produces an unreadable plot. Switch it on in the app for a specific module once you have zoomed in.\n", + "\n", + "`adjustedPValueThreshold` is set to the same `FDR` used to build the lists. If you change one and\n", + "not the other the plot and the saved list quietly disagree, which is the sort of thing nobody\n", + "notices until a reviewer does.\n", + "\n", + "Under them go two selection-driven modules. Click points on a volcano and both follow:\n", + "\n", + "* a **dataset table** with `source: \"selection\"`, showing only the identity columns\n", + "* an **entity abundance plot** on violin, with `conditionCol` set to the grouping column, so you see\n", + " the per-sample intensities of the selected protein split by group\n", + "\n", + "The table reads the intensity dataset's `Protein_Metadata`, deliberately not the pairwise\n", + "`output_comparisons`. Pointing a `dataset_table` at a pairwise table puts it in comparison mode,\n", + "where `conditionPairs` becomes required and the module refuses to render until you name a pair.\n", + "That is a different tool for a different job. Here we only want to know what got selected." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "68bb6acb", + "metadata": {}, + "outputs": [], + "source": [ + "PAIRWISE_SEARCH = {\"individualResults\": [{\"id\": PAIRWISE_ID, \"name\": pairwise.name}]}\n", + "INTENSITY_SEARCH = {\"individualResults\": [{\"id\": DATASET_ID, \"name\": dataset.name}]}\n", + "\n", + "\n", + "def highlight(left, right):\n", + " \"\"\"The up and down lists for this contrast, in SetProteinListModule's shape.\n", + "\n", + " No `showAnnotation`. The automation offers a gene-name label and it is\n", + " tempting, but a list of a few hundred proteins puts a few hundred labels on\n", + " the plot and you cannot read any of them. Turn it on per module in the app\n", + " once you have zoomed into something worth naming.\n", + " \"\"\"\n", + " return [\n", + " {\"id\": lists[(left, right, d)]}\n", + " for d in (\"Down\", \"Up\") if (left, right, d) in lists\n", + " ]\n", + "\n", + "\n", + "def volcano(left, right):\n", + " return {\n", + " \"datasetsSearch\": PAIRWISE_SEARCH,\n", + " \"experimentAndConditionComparison\": {\n", + " \"comparison\": {\"conditionPair\": f\"{left} - {right}\", \"left\": left, \"right\": right}\n", + " },\n", + " \"significantThresholdMethod\": \"fdr\",\n", + " \"adjustedPValueThreshold\": FDR,\n", + " \"foldChangeThreshold\": FOLD_CHANGE,\n", + " \"proteinLists\": highlight(left, right),\n", + " \"displayLegend\": True,\n", + " }\n", + "\n", + "\n", + "IDENTITY_COLUMNS = [{\"name\": c, \"value\": c} for c in (\"ProteinIds\", \"GeneNames\", \"Description\")]\n", + "\n", + "ANALYSIS = [(\"heading\", 0, 0, 12, 3, {\"text\": f\"Differential expression, FDR < {FDR}\"})]\n", + "for i, (left, right) in enumerate(CONTRASTS):\n", + " ANALYSIS.append((\"pairwise_volcano_plot\", (i % 3) * 4, 3, 4, 18, volcano(left, right)))\n", + "\n", + "ANALYSIS += [\n", + " # Text modules take PLAIN TEXT. HTML tags are not allowed in a workspace.\n", + " (\"text\", 0, 21, 12, 4, {\"text\": \"Selected proteins\\n\"\n", + " \"Click points on any volcano above. \"\n", + " \"Both modules below follow the selection.\"}),\n", + " # Point this at the INTENSITY dataset's metadata table, not at the pairwise\n", + " # output. A dataset_table over a pairwise table turns on the comparison-pair\n", + " # machinery and then refuses to render until `conditionPairs` names at least\n", + " # one pair, which is not what we want here: we want the identity of whatever\n", + " # is selected, not its statistics.\n", + " (\"dataset_table\", 0, 25, 12, 14, {\n", + " \"datasetsSearch\": INTENSITY_SEARCH,\n", + " \"entityType\": ENTITY_TYPE,\n", + " \"tableName\": \"Protein_Metadata\",\n", + " \"source\": \"selection\",\n", + " \"datasetTableValues\": IDENTITY_COLUMNS,\n", + " \"selectableEntities\": True,\n", + " }),\n", + " (\"entity_abundance_plot\", 0, 39, 12, 18, {\n", + " \"datasetsSearch\": INTENSITY_SEARCH,\n", + " \"entityType\": ENTITY_TYPE,\n", + " \"selectionType\": \"selection\",\n", + " \"plotType\": \"violin\",\n", + " \"conditionCol\": CONDITION_COLUMN,\n", + " \"conditions\": groups,\n", + " \"yScale\": \"intensity\",\n", + " \"titleField\": \"protein_name\",\n", + " \"numberOfColumns\": 3,\n", + " }),\n", + "]\n", + "\n", + "tab = client.workspaces.tabs.create(workspace_id=WORKSPACE_ID, name=ANALYSIS_TAB)\n", + "ANALYSIS_TAB_ID = str(tab.id)\n", + "\n", + "for item_id, x, y, w, h, settings in ANALYSIS:\n", + " client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=ANALYSIS_TAB_ID, item_id=item_id,\n", + " x=x, y=y, width=w, height=h, settings=settings,\n", + " )\n", + " print(f\" ok {item_id:<24} x={x:<3} y={y:<3} w={w:<3} h={h}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Pathway enrichment: run the analyses, then plot them\n", + "\n", + "The Reactome ORA template places three modules that each call out to Reactome live. That works, but\n", + "it is not an analysis you own: there is no dataset, nothing to download, and nothing to point a\n", + "second plot at.\n", + "\n", + "There is a real ORA job on the platform, slug `ora`, and the API can run it. It takes an intensity\n", + "dataset, a foreground list of entity ids, a species and a gene-set database, and it produces an\n", + "`ENRICHMENT` dataset like any other job output. That dataset is then what the `ora_dot_plot` module\n", + "reads.\n", + "\n", + "**Submit everything first, then wait.** The jobs run on the platform, not here, so they run\n", + "concurrently once submitted. Creating a job returns its dataset id immediately. If you submit and\n", + "wait one at a time, your wall clock is the sum of every job; if you submit them all and then wait,\n", + "it is the length of the slowest one. With three contrasts that is a nuisance, and with thirty it is\n", + "the difference between a coffee and an afternoon. The cell below is in three plain parts: submit\n", + "all, wait for all, then build the tab.\n", + "\n", + "Two parameters are worth pausing on:\n", + "\n", + "`foreground_ids` is a **flat list of entity ids**, not an entity list id. The app's field resolves\n", + "the list you pick down to its ids before submitting, so from a script you pass the accessions\n", + "straight through. We already have them in the dataframe.\n", + "\n", + "`background` decides what the foreground is tested against. `\"Detected features in this dataset\"`\n", + "uses the proteins actually measured in this experiment, which is the honest choice for a\n", + "hypergeometric test. `\"Selected Database\"` uses everything in the database and will inflate your\n", + "enrichments, because the proteome you could not detect is not a fair reference universe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- 1. submit every job, waiting for none of them ---------------------------\n", + "\n", + "submitted = []\n", + "for left, right in CONTRASTS:\n", + " up = regulated[(left, right)][\"Up\"]\n", + " if up.empty:\n", + " print(f\" skipped {left} vs {right}: no up-regulated entities\")\n", + " continue\n", + "\n", + " foreground = up[\"ProteinIds\"].astype(str).tolist()\n", + " ora_id = str(\n", + " client.datasets.create(\n", + " Dataset(\n", + " input_dataset_ids=[DATASET_ID],\n", + " name=f\"ORA {left} vs {right} up ({DATABASE})\",\n", + " job_slug=\"ora\",\n", + " job_run_params={\n", + " \"entity_type\": ENTITY_TYPE,\n", + " \"foreground_ids\": foreground,\n", + " \"species\": SPECIES,\n", + " \"database\": DATABASE,\n", + " \"background\": \"Detected features in this dataset\",\n", + " \"min_gene_set_size\": 1,\n", + " \"max_gene_set_size\": 500,\n", + " },\n", + " )\n", + " )\n", + " )\n", + " submitted.append((left, right, ora_id))\n", + " print(f\" queued {left} vs {right}: {len(foreground)} foreground -> {ora_id}\")\n", + "\n", + "print(f\"\\n{len(submitted)} job(s) running on the platform\")\n", + "\n", + "\n", + "# --- 2. now wait for them all ------------------------------------------------\n", + "\n", + "completed = []\n", + "for left, right, ora_id in submitted:\n", + " state = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=ora_id, poll_s=10, timeout_s=1800\n", + " )\n", + " print(f\" {left} vs {right}: {state.state}\")\n", + " if str(state.state) == \"COMPLETED\":\n", + " completed.append((left, right, ora_id))\n", + " else:\n", + " print(f\" {state.error_message}\")\n", + "\n", + "\n", + "# --- 3. build the tab from whatever finished ---------------------------------\n", + "\n", + "ora_tab = client.workspaces.tabs.create(workspace_id=WORKSPACE_ID, name=ORA_TAB)\n", + "ORA_TAB_ID = str(ora_tab.id)\n", + "\n", + "y = 0\n", + "for left, right, ora_id in completed:\n", + " ora = client.datasets.get_by_id(ora_id)\n", + " client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=ORA_TAB_ID, item_id=\"heading\",\n", + " x=0, y=y, width=12, height=2,\n", + " settings={\"text\": f\"{left} vs {right}, up-regulated, {DATABASE}\"},\n", + " )\n", + " client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=ORA_TAB_ID, item_id=\"ora_dot_plot\",\n", + " x=0, y=y + 2, width=12, height=16,\n", + " settings={\n", + " \"datasetsSearch\": {\"individualResults\": [{\"id\": ora_id, \"name\": ora.name}]},\n", + " \"topN\": TOP_N,\n", + " \"xAxis\": \"GeneRatio\",\n", + " \"entityListForSelection\": \"measured_in_dataset\",\n", + " },\n", + " )\n", + " print(f\" ok {left} vs {right}\")\n", + " y += 18\n", + "\n", + "print(f\"\\nOpen it: {client.base_url.replace('/api', '')}/w/{WORKSPACE_ID}/dashboard\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## What to take from this\n", + "\n", + "**The threshold lives in your script, not in a plot.** `FDR` and `FOLD_CHANGE` are set once at the\n", + "top and used twice: to build the lists and to configure the volcanoes. Over the API that duplication\n", + "is visible and checkable. In the app the same two numbers live in a form and in a module's settings\n", + "panel, and nothing tells you when they drift apart.\n", + "\n", + "**The lists are the join.** Once the up and down sets exist as entity lists they are ordinary\n", + "workspace objects: highlighted on the volcano, selectable in the app, and used as the foreground of\n", + "the ORA job. Nothing downstream re-derives them, so everything is looking at the same proteins.\n", + "\n", + "**Enrichment is a job, not a widget.** Running `ora` gives you an `ENRICHMENT` dataset you can\n", + "download, diff between runs, and point more than one module at. The Reactome ORA modules call the\n", + "service live and leave nothing behind, which is fine for a look and poor for a pipeline.\n", + "\n", + "**One trap worth knowing.** `entity_abundance_plot.conditions` is a required setting whose registry\n", + "default is an empty list. It validates clean and then renders empty, the same shape of problem as\n", + "the QC summary table's Table Grouping, so it is filled in explicitly above." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/walkthroughs/md_qc_report_automation.ipynb b/examples/walkthroughs/md_qc_report_automation.ipynb new file mode 100644 index 0000000..ad4de26 --- /dev/null +++ b/examples/walkthroughs/md_qc_report_automation.ipynb @@ -0,0 +1,577 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c00", + "metadata": {}, + "source": [ + "# A QC report, built by API instead of by automation\n", + "\n", + "The app ships a **Quality Control Report** automation. You pick it from the Automations menu, it\n", + "asks four questions, and it lays out a full QC report on the tab.\n", + "\n", + "This notebook does the same thing from the API. Same modules, same grid positions, same settings,\n", + "no clicking. Which means you can attach it to the end of a pipeline and have the report waiting when\n", + "the data lands.\n", + "\n", + "### What the in-app automation actually is\n", + "\n", + "Automations are declarative workflows that run **in the browser**\n", + "(`workflow/app/javascript/workspaces/lib/automations/QCReport.js`). There is no automations endpoint\n", + "on the API, so you cannot trigger one remotely. But you do not need to: an automation is a list of\n", + "`AddModuleToTab` steps, and adding a module to a tab is a plain API call.\n", + "\n", + "The real automation asks for four things once:\n", + "\n", + "| it asks | we set once, below |\n", + "|---|---|\n", + "| `datasetsSearch` | the intensity dataset |\n", + "| `entityType` | Protein or Peptide |\n", + "| `groupBy` | the metadata column to colour and group by |\n", + "| `xAxis` | the metadata column to order samples along |\n", + "\n", + "then places **29 modules** using them. This notebook does exactly that." + ] + }, + { + "cell_type": "markdown", + "id": "c01", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c02", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)" + ] + }, + { + "cell_type": "markdown", + "id": "c03", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "The four answers the automation would ask for, plus where to put the report.\n", + "\n", + "### `GROUP_BY` and `X_AXIS` are sample metadata column names\n", + "\n", + "These two are not free text and they are not settings of the module. They are the names of columns\n", + "in the **sample metadata of the upload the dataset came from**, and almost every data module on the\n", + "tab reads one of them:\n", + "\n", + "| module | setting | value |\n", + "|---|---|---|\n", + "| dimensionality reduction (x5) | `colourBy` | `GROUP_BY` |\n", + "| CV distribution, CV violin | `xAxis` | `GROUP_BY` |\n", + "| box plot | `xAxis` | `X_AXIS` |\n", + "| intensity distribution | `groupByColumn` | `GROUP_BY` |\n", + "| QC summary table (x3) | `xAxis`, labelled Table Grouping | `GROUP_BY` |\n", + "\n", + "For this dataset the grouping column is `condition`, so that is what `GROUP_BY` is set to below.\n", + "`sample_name` is the one column you can rely on always being there, which is why it is the default\n", + "for ordering.\n", + "\n", + "Nothing checks these for you. The registry cannot publish a list of valid values, because the valid\n", + "values are whatever columns that particular upload happens to carry, so a column name that does not\n", + "exist is stored without complaint and the module comes back empty. This is the same class of problem\n", + "as the empty Table Grouping described in section 3.\n", + "\n", + "**What that means if you are automating this.** A script like this one is only reusable across\n", + "uploads if those uploads share a sample metadata schema. If every upload you produce carries a\n", + "column called `condition`, this notebook runs unchanged on all of them. If one pipeline writes\n", + "`condition`, another writes `treatment` and a third writes `Condition`, then the same script builds\n", + "a report of blank tiles on two of the three and tells you nothing went wrong. Deciding on the\n", + "metadata column names once, upstream at upload time, is what makes the report automatable at all." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c04", + "metadata": {}, + "outputs": [], + "source": [ + "# The intensity dataset to report on.\n", + "DATASET_ID = \"00000000-0000-0000-0000-000000000000\"\n", + "\n", + "# LOWERCASE. protein | peptide | gene | metabolite | ptm\n", + "# Capitalised values are accepted by the API and then fail to render.\n", + "ENTITY_TYPE = \"protein\"\n", + "\n", + "# Sample metadata columns of the upload behind DATASET_ID, not module settings.\n", + "# GROUP_BY colours and groups; X_AXIS orders the samples. A name that is not a\n", + "# real column is accepted and renders empty, so check these against your upload.\n", + "GROUP_BY = \"condition\"\n", + "X_AXIS = \"sample_name\"\n", + "\n", + "# Scaling for the dimensionality reduction and heatmap. none | zscore | centered\n", + "# The in-app automation uses zscore. Leaving this at the \"none\" default is\n", + "# almost never what you want: unscaled PCA is dominated by the most abundant\n", + "# proteins.\n", + "SCALING_METHOD = \"zscore\"\n", + "\n", + "WORKSPACE_NAME = \"QC report (API)\"\n", + "TAB_NAME = \"Quality control\"" + ] + }, + { + "cell_type": "markdown", + "id": "c05", + "metadata": {}, + "source": [ + "## 1. A workspace and a tab to put it on\n", + "\n", + "Skip this if you already have one: set `WORKSPACE_ID` and `TAB_ID` yourself." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c06", + "metadata": {}, + "outputs": [], + "source": [ + "workspace = client.workspaces.create(\n", + " name=WORKSPACE_NAME,\n", + " description=\"QC report generated through the API\",\n", + ")\n", + "WORKSPACE_ID = str(workspace.id)\n", + "\n", + "tab = client.workspaces.tabs.create(workspace_id=WORKSPACE_ID, name=TAB_NAME)\n", + "TAB_ID = str(tab.id)\n", + "\n", + "print(\"workspace:\", WORKSPACE_ID)\n", + "print(\"tab :\", TAB_ID)" + ] + }, + { + "cell_type": "markdown", + "id": "c07", + "metadata": {}, + "source": [ + "## 2. The shared settings\n", + "\n", + "Every data module in the report gets the same dataset and entity type. This is the automation's\n", + "\"ask once, use everywhere\" made literal.\n", + "\n", + "`datasetsSearch` is a structured field: it needs `{\"individualResults\": [{\"id\", \"name\"}]}` with\n", + "**both** id and name, not a bare id." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c08", + "metadata": {}, + "outputs": [], + "source": [ + "dataset = client.datasets.get_by_id(DATASET_ID)\n", + "assert dataset, f\"No dataset {DATASET_ID!r}\"\n", + "print(f\"{dataset.name} ({dataset.type}, {dataset.state})\")\n", + "\n", + "COMMON = {\n", + " \"datasetsSearch\": {\n", + " \"individualResults\": [{\"id\": DATASET_ID, \"name\": dataset.name}]\n", + " },\n", + " \"entityType\": ENTITY_TYPE,\n", + "}\n", + "\n", + "X_AXIS_ORDERED = [{\"field\": X_AXIS, \"order\": \"none\"}]\n", + "\n", + "# Grouping by condition, in the shape the automation uses everywhere it groups:\n", + "# the CV plots and the QC summary tables, where it is labelled Table Grouping.\n", + "# Two things to note. The summary table's grouping key is called `xAxis`, and\n", + "# its registry default is an empty list, so a table placed without it comes out\n", + "# blank. And the CV plots must be grouped by GROUP_BY rather than by X_AXIS: a\n", + "# coefficient of variation needs at least two replicates per group, so grouping\n", + "# by sample_name gives one sample per group and the render fails outright.\n", + "GROUPING = [{\"field\": GROUP_BY, \"order\": \"asc\", \"type\": \"categorical\"}]\n", + "\n", + "# Fetch the registry once; the validator below reads it.\n", + "REGISTRY = {str(m.id): m for m in client.module_registry.list()}" + ] + }, + { + "cell_type": "markdown", + "id": "valmd", + "metadata": {}, + "source": [ + "## 3. Validate before you build\n", + "\n", + "The API will accept settings that do not work. A value outside a field's allowed list is stored\n", + "happily and only shows up later as an empty dropdown in the app and a 500 from the visualisation\n", + "endpoint. That is what broke the first version of this notebook: `entityType` was sent as\n", + "`\"Protein\"` instead of `\"protein\"`.\n", + "\n", + "Three things are checkable up front:\n", + "\n", + "| check | how |\n", + "|---|---|\n", + "| keys the module does not declare | `module.validate_settings_keys(settings)` |\n", + "| required keys with no value | `module.missing_required_keys({**defaults, **settings})` |\n", + "| values outside an allowed list | `spec[\"parameters\"][\"options\"]` |\n", + "\n", + "That third one is the important one and it is easy to miss: **142 of the 625 settings across the\n", + "registry publish an options list**, under `parameters.options`, not at the top level of the spec.\n", + "\n", + "There is a fourth mistake none of the three catch: a **required key whose default is empty**.\n", + "`qc_summary_table`'s Table Grouping (`xAxis`) is required, and the registry default is `[]`. An\n", + "empty list is a value, so `missing_required_keys` is satisfied, the module is created, and the table\n", + "renders with nothing to group by. Required and populated are not the same check.\n", + "\n", + "`entityType` is the awkward exception. It publishes no options, because its valid values depend on\n", + "the dataset and the organisation's feature flags, so you have to know them:\n", + "`protein`, `peptide`, `gene`, `metabolite`, `ptm`, all lowercase." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "valcode", + "metadata": {}, + "outputs": [], + "source": [ + "# Entity types, lowercase. The registry does NOT publish these as options\n", + "# (fieldType \"EntityType\" is resolved from the dataset at render time), so this\n", + "# is the one list you have to know rather than look up.\n", + "# Source: workflow app/javascript/workspaces/lib/helpers/EntityListHelper.js\n", + "ENTITY_TYPES = {\"protein\", \"peptide\", \"gene\", \"metabolite\", \"ptm\"}\n", + "\n", + "\n", + "def check_settings(item_id, settings):\n", + " \"\"\"Return a list of problems with `settings` for `item_id`, empty if fine.\n", + "\n", + " Catches three classes of mistake before anything is sent:\n", + "\n", + " 1. keys the module does not declare (server rejects these outright)\n", + " 2. required keys with no value (module renders broken)\n", + " 3. values outside a declared options list (silently accepted, then fails)\n", + "\n", + " (3) is the dangerous one. The API accepts any string for an enum field, so a\n", + " wrong value is only visible as an empty dropdown in the app and a 500 from\n", + " the visualisation endpoint.\n", + " \"\"\"\n", + " module = REGISTRY.get(item_id)\n", + " if module is None:\n", + " return [f\"{item_id!r} is not in the module registry\"]\n", + "\n", + " problems = []\n", + " schema = module.input_settings or {}\n", + " specs = schema if isinstance(schema, dict) else {\n", + " s.get(\"key\"): s for s in schema if isinstance(s, dict)\n", + " }\n", + "\n", + " unknown = module.validate_settings_keys(settings)\n", + " if unknown:\n", + " problems.append(f\"keys not declared by this module: {list(unknown)}\")\n", + "\n", + " missing = module.missing_required_keys({**module.defaults(), **settings})\n", + " if missing:\n", + " problems.append(f\"required with no value: {missing}\")\n", + "\n", + " for key, value in settings.items():\n", + " spec = specs.get(key)\n", + " if not isinstance(spec, dict):\n", + " continue\n", + " options = (spec.get(\"parameters\") or {}).get(\"options\")\n", + " if options:\n", + " allowed = [o.get(\"value\") for o in options if isinstance(o, dict)]\n", + " if allowed and value not in allowed:\n", + " problems.append(f\"{key}={value!r} not one of {allowed}\")\n", + "\n", + " if settings.get(\"entityType\") not in (None, *ENTITY_TYPES):\n", + " problems.append(\n", + " f\"entityType={settings['entityType']!r} is not one of \"\n", + " f\"{sorted(ENTITY_TYPES)} (lowercase)\"\n", + " )\n", + " return problems\n", + "\n", + "\n", + "# Prove it catches the mistake that broke this notebook the first time.\n", + "bad = {**COMMON, \"entityType\": \"Protein\"}\n", + "print(\"Checking a deliberately wrong entityType:\")\n", + "for p in check_settings(\"dimensionality_reduction_plot\", bad):\n", + " print(\" \", p)\n", + "print(\"\\nChecking the real settings:\",\n", + " check_settings(\"dimensionality_reduction_plot\", COMMON) or \"no problems\")" + ] + }, + { + "cell_type": "markdown", + "id": "c09", + "metadata": {}, + "source": [ + "## 4. The report, as data\n", + "\n", + "The automation is a chain of builder calls. Here the same thing is a list: one row per module, with\n", + "its grid position and whatever it needs on top of `COMMON`.\n", + "\n", + "Positions are taken from the automation itself, so the result matches what the app produces. The\n", + "grid is 12 columns wide, and `y` runs down the page.\n", + "\n", + "Because reading 29 rows of chained builder code is nobody's idea of fun, this is where the API\n", + "version is arguably nicer than the automation: the report is a table you can edit, reorder, or\n", + "generate." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c10", + "metadata": {}, + "outputs": [], + "source": [ + "def note(title, body):\n", + " \"\"\"Workspace text modules take PLAIN TEXT. HTML tags are not allowed.\"\"\"\n", + " return {\"text\": f\"{title}\\n{body}\"}\n", + "\n", + "\n", + "# Settings shared by the five dimensionality reduction plots. The in-app\n", + "# automation sets scaling and colours by the grouping column rather than\n", + "# leaving the sample_name default.\n", + "DR = {\"scalingMethod\": SCALING_METHOD, \"colourBy\": GROUP_BY, \"drMethod\": \"pca\"}\n", + "\n", + "REPORT = [\n", + " # item_id, x, y, w, h, extra settings\n", + " (\"heading\", 0, 0, 12, 3, {\"text\": \"Quality Control Report\"}),\n", + " (\"text\", 0, 3, 12, 6,\n", + " note(\"Dimensionality reduction\",\n", + " \"How samples relate to one another. Replicates should cluster.\")),\n", + "\n", + " # Variance explained, then PC1 against PC2, PC3, PC4, PC5.\n", + " (\"dimensionality_reduction_plot\", 0, 9, 6, 20, {**DR, \"drSubtype\": \"pca_variance\"}),\n", + " (\"dimensionality_reduction_plot\", 6, 9, 6, 20,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC2\"}}),\n", + " (\"dimensionality_reduction_plot\", 0, 29, 4, 12,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC3\"}}),\n", + " (\"dimensionality_reduction_plot\", 4, 29, 4, 12,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC4\"}}),\n", + " (\"dimensionality_reduction_plot\", 8, 29, 4, 12,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC5\"}}),\n", + "\n", + " (\"text\", 0, 41, 12, 5,\n", + " note(\"Variability\", \"Coefficient of variation within each group.\")),\n", + " (\"cv_distribution_violin_plot\", 0, 46, 12, 15, {\"xAxis\": GROUPING}),\n", + " (\"cv_distribution_plot\", 0, 61, 12, 14, {\"xAxis\": GROUPING}),\n", + " (\"qc_summary_table\", 0, 75, 12, 15, {\"xAxis\": GROUPING}),\n", + "\n", + " (\"text\", 0, 90, 12, 5,\n", + " note(\"Sample correlation\", \"Correlation between samples, clustered.\")),\n", + " # The heatmap has no scalingMethod. Its equivalent is normalisationMethod,\n", + " # which already defaults to zscore. The QC report uses it as a sample\n", + " # CORRELATION heatmap, which is a different dataType.\n", + " (\"heatmap\", 0, 95, 12, 21,\n", + " {\"dataType\": \"correlation\", \"correlationAxis\": \"samples\",\n", + " \"correlationMethod\": \"pearson\", \"showSampleNamesCorrelationHeatmap\": True}),\n", + "\n", + " (\"text\", 0, 116, 12, 5,\n", + " note(\"Missing values\", \"Where the gaps are, by feature and by sample.\")),\n", + " (\"missing_values_by_feature_plot\", 0, 121, 6, 19, {}),\n", + " (\"missing_values_by_sample_plot\", 6, 121, 6, 19, {}),\n", + " (\"missing_values_heatmap\", 0, 140, 12, 25, {}),\n", + " (\"qc_summary_table\", 0, 165, 12, 15, {\"xAxis\": GROUPING}),\n", + "\n", + " (\"text\", 0, 180, 12, 5,\n", + " note(\"Intensity distributions\", \"Per-sample intensity spread.\")),\n", + " (\"box_plot\", 0, 185, 12, 16, {\"xAxis\": X_AXIS_ORDERED}),\n", + " (\"intensity_distribution_plot\", 0, 201, 12, 15, {\"groupByColumn\": GROUP_BY}),\n", + " (\"qc_summary_table\", 0, 216, 12, 15, {\"xAxis\": GROUPING}),\n", + "]\n", + "\n", + "print(f\"{len(REPORT)} modules, {max(y + h for _, _, y, _, h, _ in REPORT)} grid rows tall\")" + ] + }, + { + "cell_type": "markdown", + "id": "c11", + "metadata": {}, + "source": [ + "## 5. Build it\n", + "\n", + "One loop. `create_with_defaults()` fills in every registry default, so each row only carries what\n", + "differs from the default.\n", + "\n", + "If a module rejects the settings it is reported and skipped rather than aborting the run, so you get\n", + "as much of the report as the data supports." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c12", + "metadata": {}, + "outputs": [], + "source": [ + "LAYOUT_ONLY = {\"heading\", \"text\"}\n", + "\n", + "# Validate everything BEFORE creating anything. A bad enum value is accepted by\n", + "# the API and only shows up as a broken widget, so catching it here is the\n", + "# difference between a report and a page of error tiles.\n", + "problems = []\n", + "for item_id, x, y, w, h, extra in REPORT:\n", + " settings = dict(extra) if item_id in LAYOUT_ONLY else {**COMMON, **extra}\n", + " for p in check_settings(item_id, settings):\n", + " problems.append(f\"{item_id} (y={y}): {p}\")\n", + "\n", + "if problems:\n", + " print(f\"{len(problems)} problem(s) found, nothing was created:\\n\")\n", + " for p in problems:\n", + " print(\" \", p)\n", + " raise SystemExit(\"Fix the settings above and re-run.\")\n", + "\n", + "print(\"All settings validate. Building.\\n\")\n", + "\n", + "placed = []\n", + "for item_id, x, y, w, h, extra in REPORT:\n", + " settings = dict(extra) if item_id in LAYOUT_ONLY else {**COMMON, **extra}\n", + " module = client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=TAB_ID, item_id=item_id,\n", + " x=x, y=y, width=w, height=h, settings=settings,\n", + " )\n", + " placed.append(module)\n", + " print(f\" ok {item_id:<32} x={x:<3} y={y:<4} w={w:<3} h={h}\")\n", + "\n", + "print(f\"\\n{len(placed)} modules placed\")" + ] + }, + { + "cell_type": "markdown", + "id": "c13", + "metadata": {}, + "source": [ + "## 6. What ended up on the tab" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c14", + "metadata": {}, + "outputs": [], + "source": [ + "for m in sorted(client.workspaces.modules.list(WORKSPACE_ID, TAB_ID),\n", + " key=lambda m: (m.y, m.x)):\n", + " print(f\" {str(m.item_id):<32} x={m.x:<3} y={m.y:<4} w={m.width:<3} h={m.height}\")\n", + "\n", + "print(f\"\\nOpen it: {client.base_url.replace('/api', '')}/w/{WORKSPACE_ID}/dashboard\")" + ] + }, + { + "cell_type": "markdown", + "id": "c15", + "metadata": {}, + "source": [ + "## 7. Pull a figure back out\n", + "\n", + "The report is now in the app. If you also want the figures headlessly, render any module to its\n", + "Plotly payload and write it out. There is no server-side image export, so rendering to PNG happens\n", + "locally.\n", + "\n", + "```python\n", + "result = client.workspaces.modules.visualize(WORKSPACE_ID, TAB_ID, module_id)\n", + "# result == {\"data\": [...], \"layout\": {...}}\n", + "\n", + "import plotly.io as pio\n", + "pio.write_image(result, \"figure.png\") # needs kaleido\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c16", + "metadata": {}, + "outputs": [], + "source": [ + "# Not every module renders server-side, and some currently error. Report the\n", + "# reason per module and keep going: the point is the report, which is already\n", + "# built and visible in the app either way.\n", + "candidates = [m for m in placed\n", + " if str(m.item_id) not in {\"heading\", \"text\", \"qc_summary_table\"}]\n", + "\n", + "rendered = None\n", + "for target in candidates:\n", + " try:\n", + " result = client.workspaces.modules.visualize(\n", + " WORKSPACE_ID, TAB_ID, str(target.id), poll_s=3, timeout_s=120\n", + " )\n", + " except Exception as exc:\n", + " reason = \"no server-side render\" if \"not supported\" in str(exc) else \\\n", + " f\"error: {str(exc).splitlines()[0][:70]}\"\n", + " print(f\" {str(target.item_id):<32} {reason}\")\n", + " continue\n", + "\n", + " if isinstance(result, dict) and \"data\" in result:\n", + " rendered = (target, result)\n", + " break\n", + " print(f\" {str(target.item_id):<32} still rendering\")\n", + "\n", + "if rendered:\n", + " target, result = rendered\n", + " print(f\"\\nRendered {target.item_id}: {len(result['data'])} trace(s)\")\n", + " for t in result[\"data\"][:4]:\n", + " print(f\" {t.get('type', '?'):<10} {str(t.get('name')):<26} \"\n", + " f\"{len(t.get('x', []) or [])} points\")\n", + "else:\n", + " print(\"\\nNothing rendered headlessly this time. The report itself is fine \"\n", + " \"and is visible in the app.\")" + ] + }, + { + "cell_type": "markdown", + "id": "c17", + "metadata": {}, + "source": [ + "## Making it a real automation\n", + "\n", + "Nothing here needs a notebook. Drop sections 2 to 4 into a `.py` file, take `DATASET_ID` from an\n", + "argument or an environment variable, and call it at the end of your pipeline:\n", + "\n", + "```python\n", + "build_qc_report(dataset_id=sys.argv[1])\n", + "```\n", + "\n", + "The report exists before anyone opens the app. That is the part the in-app automation cannot do:\n", + "it needs a person and a browser.\n", + "\n", + "Two honest limits:\n", + "\n", + "* **You cannot trigger an in-app automation over the API.** They are frontend workflows. What you\n", + " can do is reproduce their output, which is what this notebook is.\n", + "* **`quality_control_report_classic`** is a single module that renders an entire QC report on its\n", + " own, and it only needs `{\"experimentId\": }`. If that is all you want, place that one\n", + " module instead of the 22 here. This notebook builds the report out of individual modules because\n", + " that is what the modern automation does, and because it is the version you can edit." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/walkthroughs/md_workspace_walkthrough.ipynb b/examples/walkthroughs/md_workspace_walkthrough.ipynb new file mode 100644 index 0000000..50143b6 --- /dev/null +++ b/examples/walkthroughs/md_workspace_walkthrough.ipynb @@ -0,0 +1,580 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cell00", + "metadata": {}, + "source": [ + "# Build a workspace through the API\n", + "\n", + "Create a workspace, add tabs, lay modules out on the grid, and render a plot.\n", + "\n", + "Edit the **Configuration** cell, then run top to bottom.\n", + "\n", + "The visual side of the app is `Workspace -> Tab -> Module`. A workspace is purely a container: it\n", + "does not own or store data. Modules reference existing uploads and datasets by id, so there is no\n", + "such thing as uploading *into* a workspace.\n", + "\n", + "### Before you start\n", + "\n", + "```bash\n", + "pip install \"git+https://github.com/MassDynamics/md-python.git\" python-dotenv\n", + "```\n", + "\n", + "with credentials in a `.env` file next to this notebook:\n", + "\n", + "```\n", + "MD_API_BASE_URL=https://app.massdynamics.com/api\n", + "MD_AUTH_TOKEN=\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "cell01", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell02", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "import os\n", + "from collections import defaultdict\n", + "\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)\n", + "print(\"Health:\", client.health.check())" + ] + }, + { + "cell_type": "markdown", + "id": "cell03", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "The only cell you need to edit." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell04", + "metadata": {}, + "outputs": [], + "source": [ + "WORKSPACE_NAME = \"YOUR WORKSPACE NAME\"\n", + "WORKSPACE_DESCRIPTION = \"Built through the API\"\n", + "\n", + "TAB_NAMES = [\"Overview\", \"Differential abundance\"]\n", + "\n", + "# Part of a dataset name to search for in section 2.\n", + "DATASET_SEARCH = \"pairwise\"\n", + "\n", + "# A PAIRWISE dataset to plot, and which comparison to show.\n", + "# Leave as None to build the layout without a data module.\n", + "PAIRWISE_DATASET_ID = None\n", + "COMPARISON_LEFT = \"TREATED_GROUP\"\n", + "COMPARISON_RIGHT = \"CONTROL_GROUP\"" + ] + }, + { + "cell_type": "markdown", + "id": "cell05", + "metadata": {}, + "source": [ + "## 1. Create the workspace and its tabs\n", + "\n", + "Both take a name and little else." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell06", + "metadata": {}, + "outputs": [], + "source": [ + "workspace = client.workspaces.create(\n", + " name=WORKSPACE_NAME,\n", + " description=WORKSPACE_DESCRIPTION,\n", + ")\n", + "workspace_id = str(workspace.id)\n", + "print(\"Workspace:\", workspace_id)\n", + "\n", + "tab_ids = {}\n", + "for name in TAB_NAMES:\n", + " tab = client.workspaces.tabs.create(workspace_id=workspace_id, name=name)\n", + " tab_ids[name] = str(tab.id)\n", + " print(f\" tab {name!r}: {tab.id}\")\n", + "\n", + "tab_id = tab_ids[TAB_NAMES[0]]" + ] + }, + { + "cell_type": "markdown", + "id": "new0", + "metadata": {}, + "source": [ + "## 2. Find the data you want to plot\n", + "\n", + "Modules reference datasets by id, so before you can place a plot you need to find the right dataset.\n", + "\n", + "`client.datasets.query()` searches by **partial name** and filters by type and state. It returns\n", + "`{\"data\": [...], \"pagination\": {...}}`, 50 per page.\n", + "\n", + "```python\n", + "client.datasets.query(search=\"my experiment\") # name contains\n", + "client.datasets.query(type=[\"PAIRWISE\"], state=[\"COMPLETED\"])\n", + "client.datasets.query(upload_id=...) # everything from one upload\n", + "```\n", + "\n", + "Valid `type` values are `DEMO`, `DOSE_RESPONSE`, `DOSE_RESPONSE_AGGREGATE`, `ENRICHMENT`,\n", + "`IMPUTATION`, `INTENSITY`, `NORMALISATION_AND_IMPUTATION` and `PAIRWISE`. Anything else is a 400.\n", + "Note `ANOVA` datasets exist but cannot be filtered for, so find those with `search=` instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "new1", + "metadata": {}, + "outputs": [], + "source": [ + "result = client.datasets.query(search=DATASET_SEARCH, page=1)\n", + "found = result[\"data\"]\n", + "print(f\"{result['pagination']['total_count']} datasets matching {DATASET_SEARCH!r}\"\n", + " f\" (showing {len(found)})\\n\")\n", + "\n", + "for d in found[:15]:\n", + " print(f\" {str(d.get('type')):<18} {str(d.get('state')):<10} \"\n", + " f\"{d.get('id')} {d.get('name')}\")" + ] + }, + { + "cell_type": "markdown", + "id": "new2", + "metadata": {}, + "source": [ + "### Narrow it to what you can actually plot\n", + "\n", + "A volcano needs a `PAIRWISE` dataset that has finished. Filtering on both saves you from picking\n", + "one that is still processing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "new3", + "metadata": {}, + "outputs": [], + "source": [ + "pairwise_datasets = client.datasets.query(\n", + " type=[\"PAIRWISE\"], state=[\"COMPLETED\"], page=1\n", + ")[\"data\"]\n", + "\n", + "print(f\"{len(pairwise_datasets)} completed PAIRWISE datasets on this page:\\n\")\n", + "for d in pairwise_datasets[:10]:\n", + " print(f\" {d['id']} {d['name']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "new4", + "metadata": {}, + "source": [ + "### What does one of them look like?\n", + "\n", + "`get_by_id()` returns the full record. Three parts are worth reading:\n", + "\n", + "| | |\n", + "|---|---|\n", + "| `tables` | what you can download. A PAIRWISE dataset carries `output_comparisons`. |\n", + "| `input_dataset_ids` | what it was computed from, so you can walk back to the intensity dataset |\n", + "| `job_run_params` | the settings it was run with |\n", + "\n", + "For a pairwise dataset `job_run_params` is especially useful: `condition_comparisons` lists every\n", + "contrast the dataset actually contains, and `condition_column` names the metadata column they were\n", + "grouped on. Those are exactly the values the volcano module needs later, so read them from here\n", + "rather than guessing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "new5", + "metadata": {}, + "outputs": [], + "source": [ + "if pairwise_datasets:\n", + " inspect_id = pairwise_datasets[0][\"id\"]\n", + " ds = client.datasets.get_by_id(inspect_id)\n", + "\n", + " print(f\"name : {ds.name}\")\n", + " print(f\"id : {ds.id}\")\n", + " print(f\"type : {ds.type} state: {ds.state}\")\n", + " print(f\"tables : {[t.name for t in (ds.tables or [])]}\")\n", + " print(f\"inputs : {[str(i) for i in ds.input_dataset_ids]}\")\n", + "\n", + " params = ds.job_run_params or {}\n", + " print(f\"\\ncondition_column: {params.get('condition_column')}\")\n", + " pairs = (params.get(\"condition_comparisons\") or {}).get(\n", + " \"condition_comparison_pairs\", []\n", + " )\n", + " print(f\"{len(pairs)} comparison(s) in this dataset:\")\n", + " for left, right in pairs[:8]:\n", + " print(f\" {left} vs {right}\")\n", + "\n", + " if pairs:\n", + " left, right = pairs[0]\n", + " print(\"\\nTo plot the first one, put these in the Configuration cell:\")\n", + " print(f' PAIRWISE_DATASET_ID = \"{ds.id}\"')\n", + " print(f' COMPARISON_LEFT = \"{left}\"')\n", + " print(f' COMPARISON_RIGHT = \"{right}\"')\n", + "else:\n", + " print(\"No completed PAIRWISE datasets found. Run a pairwise comparison first.\")" + ] + }, + { + "cell_type": "markdown", + "id": "new6", + "metadata": {}, + "source": [ + "### Reading the numbers\n", + "\n", + "To pull the results themselves, get a presigned URL for one of the dataset's tables:\n", + "\n", + "```python\n", + "url = client.datasets.download_table_url(dataset_id, \"output_comparisons\", format=\"csv\")\n", + "\n", + "import pandas as pd\n", + "df = pd.read_csv(url) # one row per entity, one column trio per comparison\n", + "```\n", + "\n", + "`output_comparisons` has `AdjPValue - `, `PValue ...` and `MaxLog2FoldChange ...`\n", + "columns per comparison, plus a bare aggregate `AdjPValue`. Dose-response datasets expose\n", + "`output_curves` and `output_volcanoes` instead, where the half-maximal dose column is `ED50`." + ] + }, + { + "cell_type": "markdown", + "id": "cell07", + "metadata": {}, + "source": [ + "## 3. Which modules can I place?\n", + "\n", + "`client.module_registry.list()` returns every module the app can render. The `<- needs no settings`\n", + "marker flags the handful you can drop onto a tab with no configuration at all, which is useful when\n", + "you want to build the page shape first and wire up data later." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell08", + "metadata": {}, + "outputs": [], + "source": [ + "modules = client.module_registry.list()\n", + "\n", + "by_group = defaultdict(list)\n", + "for m in modules:\n", + " by_group[str(m.group)].append(m)\n", + "\n", + "print(f\"{len(modules)} modules\\n\")\n", + "for group in sorted(by_group):\n", + " print(group)\n", + " for m in sorted(by_group[group], key=lambda x: str(x.id)):\n", + " no_config = not m.missing_required_keys(m.defaults())\n", + " print(f\" {str(m.id):<40} {m.name}{' <- needs no settings' if no_config else ''}\")\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "cell09", + "metadata": {}, + "source": [ + "## 4. What parameters does a module take?\n", + "\n", + "`client.module_registry.get(item_id)` returns the module's `input_settings`: field name, type,\n", + "default, and whether it is required. This is the source of truth. The server rejects any settings\n", + "key the module does not declare.\n", + "\n", + "**Requirements cascade.** A field carrying a `when` clause only becomes required once the field it\n", + "depends on is set, so the \"you must supply\" list grows as you fill it in. Fill in what it asks for,\n", + "then run this again." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell10", + "metadata": {}, + "outputs": [], + "source": [ + "def show_module(item_id):\n", + " \"\"\"Print one module's parameters. Fetches its own registry entry, so this\n", + " cell works on its own.\"\"\"\n", + " m = client.module_registry.get(item_id)\n", + " assert m, f\"{item_id!r} is not in the module registry\"\n", + "\n", + " schema = m.input_settings or {}\n", + " items = list(schema.items()) if isinstance(schema, dict) else [\n", + " (s.get(\"key\"), s) for s in schema if isinstance(s, dict)\n", + " ]\n", + " print(f\"{m.id} ({m.group}) - {len(items)} settings\\n\")\n", + " for key, spec in items:\n", + " if not isinstance(spec, dict):\n", + " print(f\" {key}\")\n", + " continue\n", + " bits = []\n", + " if spec.get(\"fieldType\"):\n", + " bits.append(str(spec[\"fieldType\"]))\n", + " if spec.get(\"default\") is not None:\n", + " bits.append(f\"default={spec['default']!r}\")\n", + " if spec.get(\"required\") is True or any(\n", + " r.get(\"name\") == \"is_required\" for r in (spec.get(\"rules\") or [])\n", + " ):\n", + " bits.append(\"REQUIRED\")\n", + " if spec.get(\"when\"):\n", + " bits.append(f\"when={spec['when']}\")\n", + " print(f\" {str(key):<34} {' '.join(bits)}\")\n", + "\n", + " missing = m.missing_required_keys(m.defaults())\n", + " print()\n", + " print(f\"You must supply: {missing}\" if missing\n", + " else \"Nothing required. Can be placed with settings={}.\")\n", + "\n", + "show_module(\"pairwise_volcano_plot\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell11", + "metadata": {}, + "source": [ + "## 5. The grid\n", + "\n", + "A tab is a [react-grid-layout](https://github.com/jbaysolutions/vue-grid-layout) with **12 columns**\n", + "(`DashboardTabModules.vue`, `:col-num=\"12\"`). Every module is placed with four integers:\n", + "\n", + "| | |\n", + "|---|---|\n", + "| `x` | column it starts at, `0` to `11` |\n", + "| `y` | row it starts at, `0` upwards |\n", + "| `width` | how many of the 12 columns it spans |\n", + "| `height` | how many rows tall |\n", + "\n", + "Width is the number that carries the meaning. 12 is full width, so:\n", + "\n", + "```\n", + "width 12 -> one per row\n", + "width 6 -> two side by side (x = 0, 6)\n", + "width 4 -> three side by side (x = 0, 4, 8)\n", + "width 3 -> four side by side (x = 0, 3, 6, 9)\n", + "```\n", + "\n", + "`x + width` must not exceed 12 or the module wraps. Height is in grid rows, not pixels (a row is\n", + "roughly 30px), so a plot usually wants 8 to 12. Step `y` down by the height of the row above or\n", + "modules overlap.\n", + "\n", + "```\n", + "y=0 [ heading, width 12 ]\n", + "y=2 [ panel w=4 ][ panel w=4 ][ panel w=4 ]\n", + "y=8 [ half w=6 ][ half w=6 ]\n", + "y=14 [ volcano, width 12 ]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "cell12", + "metadata": {}, + "source": [ + "### Placing them\n", + "\n", + "`create_with_defaults()` fetches the registry entry, fills in every declared default, merges your\n", + "settings on top, and fails fast if a required setting has no default and you did not supply it.\n", + "\n", + "Prefer it over `create()`: the API does not merge defaults server-side, so a partial settings hash\n", + "renders as a broken widget in the app." + ] + }, + { + "cell_type": "markdown", + "id": "texthtml", + "metadata": {}, + "source": [ + "### The `text` module takes plain text\n", + "\n", + "Write the words you want to see and nothing else. **Workspaces do not allow HTML tags**, and\n", + "markdown is not rendered either, so `\"## Heading\"` shows up literally. When you want a heading, use\n", + "the `heading` module, which exists for exactly that.\n", + "\n", + "Worth knowing because it will mislead you: the registry still describes the field as *\"HTML and\n", + "embedded base64 imgs\"* and the app renders it with `v-html`. Send HTML on the strength of that and\n", + "it will store without complaint. Workspaces are plain text by policy, so send plain text.\n", + "\n", + "There is also **no settings button** on a text module in the app, by design. Its instruction\n", + "declares `INPUT_PROPERTIES = {}`, so there is no settings form to render. You edit it inline\n", + "instead: click into the module and a WYSIWYG toolbar appears in place. The `text` key exists in the\n", + "registry manifest purely so the API can set the body, which is what we do here.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell13", + "metadata": {}, + "outputs": [], + "source": [ + "def place(item_id, x, y, width, height, settings=None):\n", + " assert x + width <= 12, f\"x({x}) + width({width}) exceeds the 12-column grid\"\n", + " m = client.workspaces.modules.create_with_defaults(\n", + " workspace_id=workspace_id, tab_id=tab_id, item_id=item_id,\n", + " x=x, y=y, width=width, height=height, settings=settings or {},\n", + " )\n", + " print(f\" {item_id:<26} x={x:<3} y={y:<3} w={width:<3} h={height:<3} -> {m.id}\")\n", + " return m\n", + "\n", + "\n", + "print(\"row 1: full width\")\n", + "place(\"text\", 0, 0, 12, 2, {\"text\": \"## Results\\nBuilt through the API.\"})\n", + "\n", + "print(\"\\nrow 2: three across, width 4\")\n", + "for i, x in enumerate([0, 4, 8], start=1):\n", + " place(\"text\", x, 2, 4, 6, {\"text\": f\"Panel {i}\\n\\nx={x}, width=4\"})\n", + "\n", + "print(\"\\nrow 3: two across, width 6\")\n", + "for i, x in enumerate([0, 6], start=1):\n", + " place(\"text\", x, 8, 6, 6, {\"text\": f\"Half {i}\\n\\nx={x}, width=6\"})" + ] + }, + { + "cell_type": "markdown", + "id": "cell14", + "metadata": {}, + "source": [ + "## 6. A data module\n", + "\n", + "Structured field types are not plain ids. Two you will meet immediately:\n", + "\n", + "```python\n", + "DatasetsSelect {\"individualResults\": [{\"id\": ..., \"name\": ...}]} # id AND name\n", + "ConditionComparison {\"comparison\": {\"conditionPair\": \" - \",\n", + " \"left\": ..., \"right\": ...}}\n", + "```\n", + "\n", + "`conditionPair` must be exactly `\"{left} - {right}\"`. Send a bare list of ids and the API answers\n", + "*\"DatasetsSelect must be an object with individualResults {id,name}\"*.\n", + "\n", + "Source: `workflow/app/models/module_registry/structured_field_violations.rb`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell15", + "metadata": {}, + "outputs": [], + "source": [ + "if PAIRWISE_DATASET_ID:\n", + " dataset = client.datasets.get_by_id(PAIRWISE_DATASET_ID)\n", + " assert dataset, f\"No dataset {PAIRWISE_DATASET_ID!r}\"\n", + "\n", + " print(\"row 4: volcano, full width\")\n", + " volcano = place(\n", + " \"pairwise_volcano_plot\", 0, 14, 12, 12,\n", + " settings={\n", + " \"datasetsSearch\": {\n", + " \"individualResults\": [\n", + " {\"id\": PAIRWISE_DATASET_ID, \"name\": dataset.name}\n", + " ]\n", + " },\n", + " \"experimentAndConditionComparison\": {\n", + " \"comparison\": {\n", + " \"conditionPair\": f\"{COMPARISON_LEFT} - {COMPARISON_RIGHT}\",\n", + " \"left\": COMPARISON_LEFT,\n", + " \"right\": COMPARISON_RIGHT,\n", + " }\n", + " },\n", + " },\n", + " )\n", + "else:\n", + " volcano = None\n", + " print(\"Skipped: set PAIRWISE_DATASET_ID to place a volcano plot.\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell16", + "metadata": {}, + "source": [ + "### What is on the tab now" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell17", + "metadata": {}, + "outputs": [], + "source": [ + "for m in sorted(client.workspaces.modules.list(workspace_id, tab_id),\n", + " key=lambda m: (m.y, m.x)):\n", + " print(f\" {str(m.item_id):<26} x={m.x:<3} y={m.y:<3} w={m.width:<3} h={m.height}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell20", + "metadata": {}, + "source": [ + "## Where to go next\n", + "\n", + "The workspace is now live in the app. To keep working with it:\n", + "\n", + "```python\n", + "client.workspaces.list_all() # find it again\n", + "client.workspaces.tabs.list_all(workspace_id) # its tabs\n", + "client.workspaces.modules.list(workspace_id, tab_id) # its modules\n", + "client.workspaces.modules.update(...) # move or resize\n", + "client.workspaces.modules.delete(workspace_id, tab_id, module_id)\n", + "```\n", + "\n", + "`client.workspaces.modules.create_text(...)` is a shortcut for text modules if you are writing a lot\n", + "of narrative.\n", + "\n", + "If a module is rejected, re-run `show_module(\"\")` and compare the field list against what\n", + "you are sending. The registry is the source of truth and it does change." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}