diff --git a/examples/walkthroughs/README.md b/examples/walkthroughs/README.md new file mode 100644 index 0000000..29ca0bf --- /dev/null +++ b/examples/walkthroughs/README.md @@ -0,0 +1,55 @@ +# Walkthroughs + +Four notebooks that each take one thing you would normally do by clicking in the app and do it from +the API instead. They are meant to be read as much as run: every non-obvious wire format is +explained where it appears, along with what happens if you get it wrong. + +Read them in this order if you are new to the client. Each one assumes the one before it. + +| notebook | what it covers | +|---|---| +| `md_dr_pairwise_walkthrough.ipynb` | Upload data, then run a pairwise comparison and a per-compound dose response from a single upload | +| `md_workspace_walkthrough.ipynb` | Workspaces, tabs, the module registry and the 12-column grid | +| `md_qc_report_automation.ipynb` | The in-app Quality Control Report automation, rebuilt from the API | +| `md_pairwise_ora_walkthrough.ipynb` | Saved entity lists and pathway enrichment, starting from a pairwise result | + +## Running them + +Each notebook has a single configuration cell near the top. Change the ids there and run the cells +in order; nothing else needs editing. + +You will need an `MD_AUTH_TOKEN` in a `.env` file next to the notebook, or in your environment: + +``` +MD_AUTH_TOKEN=... +``` + +Beyond `md-python` itself, they need `python-dotenv`. `md_pairwise_ora_walkthrough.ipynb` also needs +`pandas` and `requests`, because it downloads a result table and decides significance locally: + +``` +pip install python-dotenv pandas requests +``` + +The notebooks are checked in without saved outputs, so you see your own results rather than someone +else's. + +## What they create + +All four write to the API. Between them they create uploads, datasets, analysis jobs, workspaces, +tabs, modules and entity lists in whichever organisation your token belongs to. Point them at a +development environment first, and delete what you no longer need. + +## Two things worth knowing before you start + +**Module settings are validated for shape, not for meaning.** The API checks that a settings hash +has the right keys and the right types, and returns a 422 when it does not. It does not check that a +value makes sense. A metadata column name that does not exist, or an enum value with the wrong +capitalisation, is accepted, stored, and only shows up later as an empty widget on a tab. The QC +report notebook builds a small validator for the checks you can make up front, and is explicit about +the ones you cannot. + +**A required setting can still be empty.** Several modules declare a setting as required and give it +a default of `[]`. An empty list satisfies the requirement, so the module is created and then +renders nothing. Where that applies it is called out in the notebook, but it is the single easiest +way to build a report that looks broken for no visible reason. diff --git a/examples/walkthroughs/md_dr_pairwise_walkthrough.ipynb b/examples/walkthroughs/md_dr_pairwise_walkthrough.ipynb new file mode 100644 index 0000000..cf4adc2 --- /dev/null +++ b/examples/walkthroughs/md_dr_pairwise_walkthrough.ipynb @@ -0,0 +1,570 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cell00", + "metadata": {}, + "source": [ + "# One upload → pairwise + dose-response\n", + "\n", + "Upload a proteomics dataset to Mass Dynamics once, then run **both** analyses off it:\n", + "\n", + "1. **Pairwise**: every compound@dose compared against the vehicle, in a *single* job.\n", + "2. **Dose-response**: a 4-parameter curve per protein, in *one job per compound*.\n", + "\n", + "Edit the **Configuration** cell, then run the notebook top to bottom.\n", + "\n", + "### Before you start\n", + "\n", + "```bash\n", + "pip install \"git+https://github.com/MassDynamics/md-python.git\" python-dotenv\n", + "```\n", + "\n", + "and put your credentials in a `.env` file next to this notebook:\n", + "\n", + "```\n", + "MD_API_BASE_URL=https://app.massdynamics.com/api\n", + "MD_AUTH_TOKEN=\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "cell01", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell02", + "metadata": {}, + "outputs": [], + "source": [ + "import csv\n", + "import os\n", + "from pathlib import Path\n", + "\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient, Dataset, Upload, ExperimentDesign, SampleMetadata\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)\n", + "print(\"Health:\", client.health.check())" + ] + }, + { + "cell_type": "markdown", + "id": "cell03", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "The only cell you need to edit." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell04", + "metadata": {}, + "outputs": [], + "source": [ + "# Folder with your data files, and which files to upload from it.\n", + "DATA_DIR = Path(\"/Your/Path/To/Data\")\n", + "FILENAMES = [\"YOUR_PROTEIN_FILE.tsv\", \"YOUR_PEPTIDE_FILE.tsv\"]\n", + "\n", + "# Sample metadata CSV: sample_name, condition, dose\n", + "METADATA_CSV = Path(\"/Your/Path/To/Data/sample_metadata.csv\")\n", + "\n", + "# Upload name. Must be unique within your organisation.\n", + "UPLOAD_NAME = \"YOUR UPLOAD NAME\"\n", + "\n", + "# spectronaut | diann_tabular | tims_diann | maxquant | md_format | md_format_gene\n", + "SOURCE = \"md_format\"\n", + "\n", + "# Which condition is the vehicle / control. Must match the CSV exactly.\n", + "CONTROL_CONDITION = \"YOUR_CONTROL_CONDITION\"\n", + "\n", + "# Set this to an existing upload id to skip the upload and re-run the analyses.\n", + "EXISTING_UPLOAD_ID = None" + ] + }, + { + "cell_type": "markdown", + "id": "cell05", + "metadata": {}, + "source": [ + "## 1. Read the sample metadata\n", + "\n", + "Your CSV needs one row per sample and three columns:\n", + "\n", + "| column | meaning |\n", + "|---|---|\n", + "| `sample_name` | must match the run names inside your data file exactly (for Spectronaut, `R.FileName`) |\n", + "| `condition` | the **compound**, not compound+dose. Every dose of a compound shares it. |\n", + "| `dose` | a number. The vehicle is `0`. |\n", + "\n", + "```\n", + "sample_name,condition,dose\n", + "SAMPLE_01,DRUG_1,5\n", + "SAMPLE_02,DRUG_1,2\n", + "SAMPLE_03,CONTROL,0\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell06", + "metadata": {}, + "outputs": [], + "source": [ + "with METADATA_CSV.open(newline=\"\", encoding=\"utf-8-sig\") as fh:\n", + " rows = list(csv.DictReader(fh))\n", + "\n", + "for col in (\"sample_name\", \"condition\", \"dose\"):\n", + " assert col in rows[0], f\"{METADATA_CSV} is missing a '{col}' column\"\n", + "\n", + "conditions = sorted({r[\"condition\"] for r in rows})\n", + "print(f\"{len(rows)} samples, {len(conditions)} conditions: {conditions}\")\n", + "rows[:3]" + ] + }, + { + "cell_type": "markdown", + "id": "cell07", + "metadata": {}, + "source": [ + "## 2. Build the two metadata tables\n", + "\n", + "**`experiment_design`** maps data-file run names to samples: `[filename, sample_name, condition]`.\n", + "For LFQ (one run per sample) `filename` is just the sample name.\n", + "\n", + "**`sample_metadata`** carries the per-sample variables the analyses use. We add a derived\n", + "**`condition_dose`** column (e.g. `DRUG_1@5`).\n", + "\n", + "That derived column matters: the pairwise step groups on it so **each dose gets its own comparison\n", + "against the vehicle**. Grouping on `condition` alone would pool every dose of a compound into one\n", + "group, which is almost never what you want from a dose series." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell08", + "metadata": {}, + "outputs": [], + "source": [ + "experiment_design = ExperimentDesign(\n", + " data=[[\"filename\", \"sample_name\", \"condition\"]]\n", + " + [[r[\"sample_name\"], r[\"sample_name\"], r[\"condition\"]] for r in rows]\n", + ")\n", + "\n", + "sample_metadata = SampleMetadata(\n", + " data=[[\"sample_name\", \"condition\", \"dose\", \"condition_dose\"]]\n", + " + [\n", + " [r[\"sample_name\"], r[\"condition\"], r[\"dose\"], f\"{r['condition']}@{r['dose']}\"]\n", + " for r in rows\n", + " ]\n", + ")\n", + "\n", + "print(experiment_design)\n", + "print()\n", + "print(sample_metadata)" + ] + }, + { + "cell_type": "markdown", + "id": "cell09", + "metadata": {}, + "source": [ + "## 3. Upload\n", + "\n", + "`client.uploads.create()` blocks while the files transfer, then starts ingestion.\n", + "\n", + "Skipped automatically if you set `EXISTING_UPLOAD_ID` in the config cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell10", + "metadata": {}, + "outputs": [], + "source": [ + "if EXISTING_UPLOAD_ID:\n", + " upload_id = EXISTING_UPLOAD_ID\n", + " print(\"Reusing upload:\", upload_id)\n", + "else:\n", + " for name in FILENAMES:\n", + " assert (DATA_DIR / name).is_file(), f\"Missing data file: {DATA_DIR / name}\"\n", + "\n", + " upload = Upload(\n", + " name=UPLOAD_NAME,\n", + " source=SOURCE,\n", + " experiment_design=experiment_design,\n", + " sample_metadata=sample_metadata,\n", + " file_location=str(DATA_DIR),\n", + " filenames=FILENAMES,\n", + " )\n", + " print(f\"Uploading {len(FILENAMES)} file(s)...\")\n", + " upload_id = client.uploads.create(upload)\n", + " print(\"Upload ID:\", upload_id)" + ] + }, + { + "cell_type": "markdown", + "id": "cell11", + "metadata": {}, + "source": [ + "## 4. Wait for the upload **and** for its dataset\n", + "\n", + "Two waits, not one. The second is the one people miss.\n", + "\n", + "The upload reporting `COMPLETED` does not mean its intensity dataset is `COMPLETED`; they flip a\n", + "moment apart. Submit a job into that gap and the pairwise job is rejected outright, while the\n", + "dose-response job is *accepted* and then dies server-side with a confusing `NoneType` error. So wait\n", + "for the dataset itself before going any further." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell12", + "metadata": {}, + "outputs": [], + "source": [ + "client.uploads.wait_until_complete(upload_id, poll_s=15, timeout_s=7200)\n", + "print(\"Upload ingested.\")\n", + "\n", + "dataset = client.datasets.find_initial_dataset(upload_id)\n", + "dataset_id = str(dataset.id)\n", + "\n", + "state = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=dataset_id, poll_s=15, timeout_s=7200\n", + ")\n", + "print(\"Intensity dataset:\", dataset_id, \"->\", state.state)\n", + "assert str(state.state) == \"COMPLETED\"" + ] + }, + { + "cell_type": "markdown", + "id": "cell13", + "metadata": {}, + "source": [ + "## 5. What do the analysis jobs accept?\n", + "\n", + "`GET /jobs` returns each job's parameter definition: field names, types, defaults, and which are\n", + "required. This is the source of truth for `job_run_params`, and it is worth a look before you submit:\n", + "these definitions change, and what you see here always matches the server you are talking to.\n", + "\n", + "Two notes on reading it:\n", + "\n", + "* `input_datasets` is listed but does **not** go in `job_run_params`. The server fills it in from\n", + " the dataset id you pass separately.\n", + "* Some fields only apply in combination, e.g. `filter_threshold_percentage` applies when\n", + " `filter_values_criteria` is `\"percentage\"`. Send the one that matches.\n", + "\n", + "The `job_get` helper below exists because `jobs.list()` returns plain dicts on some md-python\n", + "versions and `Job` objects on others (and spells the published flag `isPublished` in one,\n", + "`is_published` in the other). It reads either." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a5ac9d66", + "metadata": {}, + "outputs": [], + "source": [ + "# Every job slug this API exposes, grouped by run type.\n", + "from collections import defaultdict\n", + "\n", + "jobs = client.jobs.list()\n", + "\n", + "by_type = defaultdict(list)\n", + "for job in jobs:\n", + " by_type[str(job.run_type)].append(job)\n", + "\n", + "published = sum(1 for j in jobs if j.is_published)\n", + "print(f\"{len(jobs)} jobs on {client.base_url} ({published} published)\\n\")\n", + "\n", + "for run_type in sorted(by_type):\n", + " print(run_type)\n", + " for job in sorted(by_type[run_type], key=lambda j: j.slug or \"\"):\n", + " flag = \"\" if job.is_published else \" <- not published\"\n", + " print(f\" {str(job.slug):<32} {len(job.properties or {}):>3} params {job.name}{flag}\")\n", + " print()\n", + "\n", + "# Bare list. Paste into a comment, or use it to validate a job_slug before submitting\n", + "print(sorted(j.slug for j in jobs if j.is_published and j.slug))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell14", + "metadata": {}, + "outputs": [], + "source": [ + "def show_params(slug):\n", + " \"\"\"Print the parameters a job accepts. Fetches its own job list, so this cell\n", + " works on its own, with no dependency on an earlier cell having run.\"\"\"\n", + " job = next((j for j in client.jobs.list() if j.slug == slug), None)\n", + " assert job, f\"No job with slug {slug!r}\"\n", + "\n", + " props = job.properties or {}\n", + " print(f\"{slug}: {len(props)} parameters\\n\")\n", + " for name, spec in props.items():\n", + " if not isinstance(spec, dict):\n", + " print(f\" {name}\")\n", + " continue\n", + " bits = []\n", + " if spec.get(\"fieldType\"):\n", + " bits.append(str(spec[\"fieldType\"]))\n", + " if \"default\" in spec:\n", + " bits.append(f\"default={spec['default']!r}\")\n", + " if any(r.get(\"name\") == \"is_required\" for r in spec.get(\"rules\", [])):\n", + " bits.append(\"REQUIRED\")\n", + " print(f\" {name:<32} {' '.join(bits)}\")\n", + "\n", + "show_params(\"pairwise_comparison\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "cell16", + "metadata": {}, + "source": [ + "## 6. Pairwise: every compound@dose vs the vehicle\n", + "\n", + "All the comparisons go into **one** job on purpose: limma then computes a single shared FDR\n", + "correction across them. Do not loop and submit one job per pair." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell17", + "metadata": {}, + "outputs": [], + "source": [ + "design_columns = sample_metadata.to_columns()\n", + "CONDITION_COLUMN = \"condition_dose\" # or \"condition\" to pool doses\n", + "\n", + "groups = []\n", + "for value in design_columns[CONDITION_COLUMN]:\n", + " if value not in groups:\n", + " groups.append(value)\n", + "\n", + "control_groups = {\n", + " g for g, cond in zip(design_columns[CONDITION_COLUMN], design_columns[\"condition\"])\n", + " if cond == CONTROL_CONDITION\n", + "}\n", + "assert control_groups, f\"No samples with condition == {CONTROL_CONDITION!r}\"\n", + "assert len(control_groups) == 1, (\n", + " f\"The vehicle spans several groups: {sorted(control_groups)}. \"\n", + " \"Give it a single dose value, or set CONDITION_COLUMN = 'condition'.\"\n", + ")\n", + "control = control_groups.pop()\n", + "\n", + "comparisons = [[g, control] for g in groups if g != control]\n", + "print(f\"{len(comparisons)} comparisons against {control!r}\")\n", + "comparisons[:5]" + ] + }, + { + "cell_type": "markdown", + "id": "cell18", + "metadata": {}, + "source": [ + "The two parameters below are the ones that are easy to get wrong. Both were taken from the job\n", + "spec printed above, not guessed:\n", + "\n", + "* **`filter_values_criteria`** is a **string** (`\"percentage\"` or `\"count\"`), *not* a dict, and it\n", + " needs a matching sibling: `filter_threshold_percentage` or `filter_threshold_count`.\n", + "* **`de_method_`** is **required** and is keyed by entity type, so protein data needs\n", + " `de_method_protein`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell19", + "metadata": {}, + "outputs": [], + "source": [ + "ENTITY_TYPE = \"protein\" # protein | peptide | gene\n", + "\n", + "pairwise_params = {\n", + " \"condition_column\": CONDITION_COLUMN,\n", + " \"condition_comparisons\": {\"condition_comparison_pairs\": comparisons},\n", + " \"experiment_design\": design_columns,\n", + " \"entity_type\": ENTITY_TYPE,\n", + " f\"de_method_{ENTITY_TYPE}\": \"limma\",\n", + " \"filter_values_criteria\": \"percentage\",\n", + " \"filter_threshold_percentage\": 0.5,\n", + " \"filter_valid_values_logic\": \"at least one condition\",\n", + " \"fit_separate_models\": True,\n", + " \"limma_trend\": True,\n", + " \"robust_empirical_bayes\": True,\n", + "}\n", + "\n", + "pairwise_id = client.datasets.create(\n", + " Dataset(\n", + " input_dataset_ids=[dataset_id],\n", + " name=f\"{UPLOAD_NAME} - pairwise vs control\",\n", + " job_slug=\"pairwise_comparison\",\n", + " job_run_params=pairwise_params,\n", + " )\n", + ")\n", + "print(\"Pairwise dataset:\", pairwise_id)" + ] + }, + { + "cell_type": "markdown", + "id": "cell20", + "metadata": {}, + "source": [ + "## 7. Dose-response: one job per compound\n", + "\n", + "The opposite fan-out to pairwise, and for a concrete reason: the dose-response pipeline fits one\n", + "curve per protein and groups samples by **dose only**. It has no notion of which compound a sample\n", + "came from, so putting two compounds in one job silently pools them into a single meaningless curve.\n", + "\n", + "Each job therefore gets one compound's dose series **plus the vehicle samples** as the dose-0 anchor.\n", + "\n", + "The platform requires at least **5 samples** and **3 distinct dose levels** per job (the vehicle's 0\n", + "counts). Compounds that fall short are reported and skipped rather than failing the run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell21", + "metadata": {}, + "outputs": [], + "source": [ + "control_rows = [r for r in rows if r[\"condition\"] == CONTROL_CONDITION]\n", + "compounds = [c for c in conditions if c != CONTROL_CONDITION]\n", + "\n", + "dose_response_ids = {}\n", + "for compound in compounds:\n", + " subset = [r for r in rows if r[\"condition\"] in (compound, CONTROL_CONDITION)]\n", + " sample_names = [r[\"sample_name\"] for r in subset]\n", + " doses = [float(r[\"dose\"]) for r in subset] # must be numbers, not strings\n", + " distinct = sorted(set(doses))\n", + "\n", + " if len(sample_names) < 5 or len(distinct) < 3:\n", + " print(f\" SKIP {compound}: {len(sample_names)} samples, {len(distinct)} dose levels\")\n", + " continue\n", + "\n", + " # experiment_design is the ONLY thing that scopes the job.\n", + " params = {\n", + " \"experiment_design\": {\"sample_name\": sample_names, \"dose\": doses},\n", + " \"control_samples\": [r[\"sample_name\"] for r in control_rows],\n", + " \"log_intensities\": True,\n", + " \"use_imputed_intensities\": False,\n", + " \"normalise\": \"none\",\n", + " \"span_rollmean_k\": 1,\n", + " \"prop_required_in_protein\": 0.5,\n", + " }\n", + "\n", + " ds_id = client.datasets.create(\n", + " Dataset(\n", + " input_dataset_ids=[dataset_id],\n", + " name=f\"{UPLOAD_NAME} - DR {compound}\",\n", + " job_slug=\"dose_response\",\n", + " job_run_params=params,\n", + " )\n", + " )\n", + " dose_response_ids[compound] = ds_id\n", + " print(f\" {compound}: {len(sample_names)} samples, doses {distinct} -> {ds_id}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell22", + "metadata": {}, + "source": [ + "## 8. Wait for everything" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell23", + "metadata": {}, + "outputs": [], + "source": [ + "pw_state = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=pairwise_id, poll_s=15, timeout_s=7200\n", + ")\n", + "print(\"pairwise:\", pw_state.state)\n", + "\n", + "for compound, ds_id in dose_response_ids.items():\n", + " st = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=ds_id, poll_s=15, timeout_s=7200\n", + " )\n", + " print(f\"{compound}: {st.state}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell24", + "metadata": {}, + "source": [ + "## Results\n", + "\n", + "Everything is now visible in the Mass Dynamics web app under this upload. To pull a table down\n", + "programmatically:\n", + "\n", + "```python\n", + "client.datasets.list_by_upload(upload_id) # what exists\n", + "[t.name for t in client.datasets.get_by_id(pairwise_id).tables] # what tables it has\n", + "url = client.datasets.download_table_url(pairwise_id, \"output_comparisons\", format=\"csv\")\n", + "```\n", + "\n", + "| dataset type | tables |\n", + "|---|---|\n", + "| `PAIRWISE` | `output_comparisons`, `runtime_metadata` |\n", + "| `DOSE_RESPONSE` | `output_curves`, `output_volcanoes`, `input_drc`, `runtime_metadata` |\n", + "\n", + "In `output_volcanoes` the half-maximal dose column is **`ED50`** (the `drc` package's terminology),\n", + "not `EC50`.\n", + "\n", + "### If a job is rejected\n", + "\n", + "Re-run the `show_params(...)` cell and compare the field list against what you are sending. The\n", + "API is the source of truth and these definitions do change." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "md-python (3.13.13.final.0)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/walkthroughs/md_pairwise_ora_walkthrough.ipynb b/examples/walkthroughs/md_pairwise_ora_walkthrough.ipynb new file mode 100644 index 0000000..1c4ae6b --- /dev/null +++ b/examples/walkthroughs/md_pairwise_ora_walkthrough.ipynb @@ -0,0 +1,568 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Saved lists and pathway enrichment from a pairwise result\n", + "\n", + "The app has three automations that do this by hand: **Pairwise Analysis Volcano**, **Pairwise\n", + "Analysis Generate Up/Down Regulated Lists**, and the **Reactome ORA Template**. You run them one\n", + "after the other, answering a form each time.\n", + "\n", + "This notebook does the same chain from the API, with no clicking. It starts from a pairwise dataset\n", + "you already have:\n", + "\n", + "1. pull the pairwise result table down and decide up and down regulated **in this script**\n", + "2. save those as entity lists in the workspace\n", + "3. build a tab with the volcanoes, a selection-driven table and an abundance plot\n", + "4. run a pathway enrichment job on each up-regulated list and plot the result on a second tab\n", + "\n", + "Creating the pairwise dataset itself is a separate job and is covered by the pairwise walkthrough\n", + "notebook. Here you just need its id.\n", + "\n", + "### The one thing that is genuinely different over the API\n", + "\n", + "The in-app automation calls `PairwiseGetUpRegulated` and `PairwiseGetDownRegulated`. Those are\n", + "**browser instructions**. They read the volcano module's own thresholds and split the proteins\n", + "client-side. There is no endpoint for them, so you cannot ask the API \"which proteins are up in this\n", + "volcano\".\n", + "\n", + "You do not need one. The pairwise result table is downloadable, it carries an adjusted p-value and\n", + "a log2 fold change per comparison, and the split is two lines of pandas. **When you script this, the\n", + "significance call is yours to make, in your own code, on data you can see.** That is arguably the\n", + "point: the threshold stops being a setting buried in a plot and becomes a line you can read, version\n", + "and change." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import io\n", + "import os\n", + "\n", + "import pandas as pd\n", + "import requests\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient, Dataset\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "Everything you would otherwise answer in three automation forms. One id and a handful of numbers." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# The PAIRWISE dataset to work from. Make one with the pairwise walkthrough\n", + "# notebook or in the app, then paste its id here.\n", + "PAIRWISE_ID = \"00000000-0000-0000-0000-000000000000\"\n", + "\n", + "ENTITY_TYPE = \"protein\" # lowercase, always\n", + "CONDITION_COLUMN = \"condition\" # the sample metadata column holding the groups\n", + "\n", + "# Significance. The volcano module defaults to 0.05; we tighten it and, crucially,\n", + "# apply the SAME numbers here in Python when we build the lists, so the plot and\n", + "# the saved list agree.\n", + "FDR = 0.01\n", + "FOLD_CHANGE = 2 # linear, so |log2FC| >= 1\n", + "\n", + "# Pathway enrichment. The ORA job takes a species name, not a taxon id.\n", + "# human | mouse | yeast | chinese_hamster\n", + "SPECIES = \"human\"\n", + "\n", + "# Gene set database. Options depend on the species; for human they are Reactome,\n", + "# the three GO branches, and the MSigDB collections.\n", + "DATABASE = \"Reactome\"\n", + "\n", + "TOP_N = 15 # pathways shown in the dot plot\n", + "\n", + "WORKSPACE_NAME = \"Pairwise + ORA (API)\"\n", + "ANALYSIS_TAB = \"Differential expression\"\n", + "ORA_TAB = \"Pathway enrichment (ORA)\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Read the analysis off the pairwise dataset\n", + "\n", + "A pairwise dataset already records what it was run on and what it compared, so there is nothing to\n", + "retype. Three things come straight off it:\n", + "\n", + "* the **intensity dataset** it was computed from, which the entity lists and the abundance plot both\n", + " need\n", + "* the **contrasts** it ran, as pairs of condition values\n", + "* through the intensity dataset, the **upload** and its sample metadata\n", + "\n", + "Taking the contrasts from the job rather than typing them matters. Every downstream step looks up\n", + "columns named after the pair, like `Log2FC Treated - Control`. A pair typed by hand that does not\n", + "match the job exactly fails several cells later with a missing column, and the name it complains\n", + "about looks correct at a glance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import math\n", + "\n", + "pairwise = client.datasets.get_by_id(PAIRWISE_ID)\n", + "assert pairwise, f\"No dataset {PAIRWISE_ID!r}\"\n", + "assert str(pairwise.type) == \"PAIRWISE\", f\"{pairwise.name} is a {pairwise.type} dataset\"\n", + "print(f\"{pairwise.name} ({pairwise.type}, {pairwise.state})\")\n", + "\n", + "# The intensity dataset this pairwise was computed from.\n", + "DATASET_ID = str(pairwise.input_dataset_ids[0])\n", + "dataset = client.datasets.get_by_id(DATASET_ID)\n", + "print(f\"computed from: {dataset.name} ({DATASET_ID})\")\n", + "\n", + "# The contrasts the job actually ran. (test, reference): positive log2FC means\n", + "# higher in the test group.\n", + "params = pairwise.job_run_params or {}\n", + "CONTRASTS = [tuple(p) for p in params[\"condition_comparisons\"][\"condition_comparison_pairs\"]]\n", + "print(f\"\\n{len(CONTRASTS)} contrast(s):\")\n", + "for left, right in CONTRASTS:\n", + " print(f\" {left} vs {right}\")\n", + "\n", + "# The upload behind the intensity dataset, for its sample metadata. The groups\n", + "# are used by the abundance plot at the end.\n", + "upload_id = None\n", + "for u in client.uploads.query(search=dataset.name)[\"data\"]:\n", + " if DATASET_ID in [str(d.id) for d in client.datasets.list_by_upload(u[\"id\"])]:\n", + " upload_id = u[\"id\"]\n", + " break\n", + "assert upload_id, \"Could not find the upload that owns the intensity dataset\"\n", + "\n", + "rows = client.uploads.get_sample_metadata(upload_id).data\n", + "header, body = rows[0], rows[1:]\n", + "groups = sorted({r[header.index(CONDITION_COLUMN)] for r in body})\n", + "print(f\"\\nupload {upload_id}\\n{CONDITION_COLUMN}: {groups}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Download the table and make the call yourself\n", + "\n", + "`download_table_url` returns a presigned S3 URL for one table of the dataset. `output_comparisons`\n", + "is the one with the statistics.\n", + "\n", + "Its columns are per contrast and named after the pair: `Log2FC Treated - Control`,\n", + "`AdjPValue Treated - Control`, and so on. Alongside them sit the identity columns every downstream\n", + "step needs: `ProteinIds`, `GeneNames`, `Description` and `GroupId`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "url = client.datasets.download_table_url(PAIRWISE_ID, \"output_comparisons\")\n", + "raw = requests.get(url if isinstance(url, str) else url[\"url\"]).content\n", + "comparisons = pd.read_csv(io.BytesIO(raw))\n", + "\n", + "print(f\"{len(comparisons)} entities x {len(comparisons.columns)} columns\")\n", + "print(\"identity columns:\", [c for c in (\"ProteinIds\", \"GeneNames\", \"Description\", \"GroupId\")\n", + " if c in comparisons.columns])\n", + "print(\"\\nper-contrast columns:\")\n", + "for left, right in CONTRASTS:\n", + " pair = f\"{left} - {right}\"\n", + " have = [f\"{p} {pair}\" in comparisons.columns for p in (\"Log2FC\", \"AdjPValue\")]\n", + " print(f\" {pair:<28} Log2FC {have[0]} AdjPValue {have[1]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "LOG2_FC = math.log2(FOLD_CHANGE)\n", + "\n", + "\n", + "def split_regulated(df, left, right, fdr=FDR, log2fc=LOG2_FC):\n", + " \"\"\"Up and down regulated entities for one contrast.\n", + "\n", + " This is the whole 'significance call', in the open. Change the comparison\n", + " operators or the columns and you change what gets saved, with no hidden\n", + " module setting involved.\n", + " \"\"\"\n", + " pair = f\"{left} - {right}\"\n", + " fc, p = df[f\"Log2FC {pair}\"], df[f\"AdjPValue {pair}\"]\n", + " significant = p < fdr\n", + " return (df[significant & (fc >= log2fc)].copy(),\n", + " df[significant & (fc <= -log2fc)].copy())\n", + "\n", + "\n", + "regulated = {}\n", + "print(f\"FDR < {FDR}, |log2FC| >= {LOG2_FC:g} (fold change {FOLD_CHANGE}x)\\n\")\n", + "for left, right in CONTRASTS:\n", + " up, down = split_regulated(comparisons, left, right)\n", + " regulated[(left, right)] = {\"Up\": up, \"Down\": down}\n", + " print(f\" {left} vs {right:<12} up {len(up):>4} down {len(down):>4}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Save them as entity lists\n", + "\n", + "`client.workspaces.entity_lists.create()`. Lists live under a workspace, which is why the resource\n", + "hangs off `client.workspaces` rather than sitting at the top level of the client.\n", + "\n", + "Each item needs an `entity_id`, a `group_id` and a `dataset_id`, which is exactly the `ProteinIds` /\n", + "`GroupId` pair from the table plus the dataset they belong to.\n", + "\n", + "The non-obvious part is which dataset. It has to be the **intensity** dataset, not the pairwise one\n", + "the numbers came from: the server checks that every referenced dataset is `INTENSITY` and holds a\n", + "`Protein_Metadata` table, and a pairwise dataset is neither. Pass the pairwise id and you get a 400\n", + "that names the type. `GroupId` is safe to carry across, because a pairwise dataset inherits the\n", + "group ids of the intensity dataset it was computed from.\n", + "\n", + "The names follow the automation's convention, `{left}_vs_{right}_Up_Reg`, so lists made this way sit\n", + "alongside lists made by hand in the app and sort together." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "workspace = client.workspaces.create(\n", + " name=WORKSPACE_NAME, description=\"Pairwise, saved lists and ORA, built through the API\"\n", + ")\n", + "WORKSPACE_ID = str(workspace.id)\n", + "print(\"workspace:\", WORKSPACE_ID)\n", + "\n", + "\n", + "def create_entity_list(name, frame):\n", + " \"\"\"Create an entity list from rows of the comparisons table.\"\"\"\n", + " entity_list = client.workspaces.entity_lists.create(\n", + " workspace_id=WORKSPACE_ID,\n", + " name=name,\n", + " entity_type=ENTITY_TYPE,\n", + " items=[\n", + " {\"entity_id\": str(r.ProteinIds), \"group_id\": int(r.GroupId), \"dataset_id\": DATASET_ID}\n", + " for r in frame.itertuples()\n", + " ],\n", + " )\n", + " return str(entity_list.id)\n", + "\n", + "\n", + "lists = {}\n", + "for (left, right), sides in regulated.items():\n", + " for direction, frame in sides.items():\n", + " if frame.empty:\n", + " print(f\" skipped {left}_vs_{right}_{direction}_Reg (empty)\")\n", + " continue\n", + " name = f\"{left}_vs_{right}_{direction}_Reg\"\n", + " lists[(left, right, direction)] = create_entity_list(name, frame)\n", + " print(f\" {name:<40} {len(frame):>4} entities\")\n", + "\n", + "print(f\"\\n{len(lists)} lists created\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. The analysis tab\n", + "\n", + "Three volcanoes, one per contrast, each carrying **its own** up and down list as a highlight. The\n", + "highlight shape comes from the automation's `SetProteinListModule` instruction: a list of\n", + "`{\"id\": ...}` per entity list.\n", + "\n", + "The automation also offers `showAnnotation` with an `annotationLabel` of `\"gene\"`, and it is left\n", + "off here on purpose. These lists run to hundreds of proteins, and labelling every highlighted point\n", + "produces an unreadable plot. Switch it on in the app for a specific module once you have zoomed in.\n", + "\n", + "`adjustedPValueThreshold` is set to the same `FDR` used to build the lists. If you change one and\n", + "not the other the plot and the saved list quietly disagree, which is the sort of thing nobody\n", + "notices until a reviewer does.\n", + "\n", + "Under them go two selection-driven modules. Click points on a volcano and both follow:\n", + "\n", + "* a **dataset table** with `source: \"selection\"`, showing only the identity columns\n", + "* an **entity abundance plot** on violin, with `conditionCol` set to the grouping column, so you see\n", + " the per-sample intensities of the selected protein split by group\n", + "\n", + "The table reads the intensity dataset's `Protein_Metadata`, deliberately not the pairwise\n", + "`output_comparisons`. Pointing a `dataset_table` at a pairwise table puts it in comparison mode,\n", + "where `conditionPairs` becomes required and the module refuses to render until you name a pair.\n", + "That is a different tool for a different job. Here we only want to know what got selected." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "68bb6acb", + "metadata": {}, + "outputs": [], + "source": [ + "PAIRWISE_SEARCH = {\"individualResults\": [{\"id\": PAIRWISE_ID, \"name\": pairwise.name}]}\n", + "INTENSITY_SEARCH = {\"individualResults\": [{\"id\": DATASET_ID, \"name\": dataset.name}]}\n", + "\n", + "\n", + "def highlight(left, right):\n", + " \"\"\"The up and down lists for this contrast, in SetProteinListModule's shape.\n", + "\n", + " No `showAnnotation`. The automation offers a gene-name label and it is\n", + " tempting, but a list of a few hundred proteins puts a few hundred labels on\n", + " the plot and you cannot read any of them. Turn it on per module in the app\n", + " once you have zoomed into something worth naming.\n", + " \"\"\"\n", + " return [\n", + " {\"id\": lists[(left, right, d)]}\n", + " for d in (\"Down\", \"Up\") if (left, right, d) in lists\n", + " ]\n", + "\n", + "\n", + "def volcano(left, right):\n", + " return {\n", + " \"datasetsSearch\": PAIRWISE_SEARCH,\n", + " \"experimentAndConditionComparison\": {\n", + " \"comparison\": {\"conditionPair\": f\"{left} - {right}\", \"left\": left, \"right\": right}\n", + " },\n", + " \"significantThresholdMethod\": \"fdr\",\n", + " \"adjustedPValueThreshold\": FDR,\n", + " \"foldChangeThreshold\": FOLD_CHANGE,\n", + " \"proteinLists\": highlight(left, right),\n", + " \"displayLegend\": True,\n", + " }\n", + "\n", + "\n", + "IDENTITY_COLUMNS = [{\"name\": c, \"value\": c} for c in (\"ProteinIds\", \"GeneNames\", \"Description\")]\n", + "\n", + "ANALYSIS = [(\"heading\", 0, 0, 12, 3, {\"text\": f\"Differential expression, FDR < {FDR}\"})]\n", + "for i, (left, right) in enumerate(CONTRASTS):\n", + " ANALYSIS.append((\"pairwise_volcano_plot\", (i % 3) * 4, 3, 4, 18, volcano(left, right)))\n", + "\n", + "ANALYSIS += [\n", + " # Text modules take PLAIN TEXT. HTML tags are not allowed in a workspace.\n", + " (\"text\", 0, 21, 12, 4, {\"text\": \"Selected proteins\\n\"\n", + " \"Click points on any volcano above. \"\n", + " \"Both modules below follow the selection.\"}),\n", + " # Point this at the INTENSITY dataset's metadata table, not at the pairwise\n", + " # output. A dataset_table over a pairwise table turns on the comparison-pair\n", + " # machinery and then refuses to render until `conditionPairs` names at least\n", + " # one pair, which is not what we want here: we want the identity of whatever\n", + " # is selected, not its statistics.\n", + " (\"dataset_table\", 0, 25, 12, 14, {\n", + " \"datasetsSearch\": INTENSITY_SEARCH,\n", + " \"entityType\": ENTITY_TYPE,\n", + " \"tableName\": \"Protein_Metadata\",\n", + " \"source\": \"selection\",\n", + " \"datasetTableValues\": IDENTITY_COLUMNS,\n", + " \"selectableEntities\": True,\n", + " }),\n", + " (\"entity_abundance_plot\", 0, 39, 12, 18, {\n", + " \"datasetsSearch\": INTENSITY_SEARCH,\n", + " \"entityType\": ENTITY_TYPE,\n", + " \"selectionType\": \"selection\",\n", + " \"plotType\": \"violin\",\n", + " \"conditionCol\": CONDITION_COLUMN,\n", + " \"conditions\": groups,\n", + " \"yScale\": \"intensity\",\n", + " \"titleField\": \"protein_name\",\n", + " \"numberOfColumns\": 3,\n", + " }),\n", + "]\n", + "\n", + "tab = client.workspaces.tabs.create(workspace_id=WORKSPACE_ID, name=ANALYSIS_TAB)\n", + "ANALYSIS_TAB_ID = str(tab.id)\n", + "\n", + "for item_id, x, y, w, h, settings in ANALYSIS:\n", + " client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=ANALYSIS_TAB_ID, item_id=item_id,\n", + " x=x, y=y, width=w, height=h, settings=settings,\n", + " )\n", + " print(f\" ok {item_id:<24} x={x:<3} y={y:<3} w={w:<3} h={h}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Pathway enrichment: run the analyses, then plot them\n", + "\n", + "The Reactome ORA template places three modules that each call out to Reactome live. That works, but\n", + "it is not an analysis you own: there is no dataset, nothing to download, and nothing to point a\n", + "second plot at.\n", + "\n", + "There is a real ORA job on the platform, slug `ora`, and the API can run it. It takes an intensity\n", + "dataset, a foreground list of entity ids, a species and a gene-set database, and it produces an\n", + "`ENRICHMENT` dataset like any other job output. That dataset is then what the `ora_dot_plot` module\n", + "reads.\n", + "\n", + "**Submit everything first, then wait.** The jobs run on the platform, not here, so they run\n", + "concurrently once submitted. Creating a job returns its dataset id immediately. If you submit and\n", + "wait one at a time, your wall clock is the sum of every job; if you submit them all and then wait,\n", + "it is the length of the slowest one. With three contrasts that is a nuisance, and with thirty it is\n", + "the difference between a coffee and an afternoon. The cell below is in three plain parts: submit\n", + "all, wait for all, then build the tab.\n", + "\n", + "Two parameters are worth pausing on:\n", + "\n", + "`foreground_ids` is a **flat list of entity ids**, not an entity list id. The app's field resolves\n", + "the list you pick down to its ids before submitting, so from a script you pass the accessions\n", + "straight through. We already have them in the dataframe.\n", + "\n", + "`background` decides what the foreground is tested against. `\"Detected features in this dataset\"`\n", + "uses the proteins actually measured in this experiment, which is the honest choice for a\n", + "hypergeometric test. `\"Selected Database\"` uses everything in the database and will inflate your\n", + "enrichments, because the proteome you could not detect is not a fair reference universe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- 1. submit every job, waiting for none of them ---------------------------\n", + "\n", + "submitted = []\n", + "for left, right in CONTRASTS:\n", + " up = regulated[(left, right)][\"Up\"]\n", + " if up.empty:\n", + " print(f\" skipped {left} vs {right}: no up-regulated entities\")\n", + " continue\n", + "\n", + " foreground = up[\"ProteinIds\"].astype(str).tolist()\n", + " ora_id = str(\n", + " client.datasets.create(\n", + " Dataset(\n", + " input_dataset_ids=[DATASET_ID],\n", + " name=f\"ORA {left} vs {right} up ({DATABASE})\",\n", + " job_slug=\"ora\",\n", + " job_run_params={\n", + " \"entity_type\": ENTITY_TYPE,\n", + " \"foreground_ids\": foreground,\n", + " \"species\": SPECIES,\n", + " \"database\": DATABASE,\n", + " \"background\": \"Detected features in this dataset\",\n", + " \"min_gene_set_size\": 1,\n", + " \"max_gene_set_size\": 500,\n", + " },\n", + " )\n", + " )\n", + " )\n", + " submitted.append((left, right, ora_id))\n", + " print(f\" queued {left} vs {right}: {len(foreground)} foreground -> {ora_id}\")\n", + "\n", + "print(f\"\\n{len(submitted)} job(s) running on the platform\")\n", + "\n", + "\n", + "# --- 2. now wait for them all ------------------------------------------------\n", + "\n", + "completed = []\n", + "for left, right, ora_id in submitted:\n", + " state = client.datasets.wait_until_complete(\n", + " upload_id=upload_id, dataset_id=ora_id, poll_s=10, timeout_s=1800\n", + " )\n", + " print(f\" {left} vs {right}: {state.state}\")\n", + " if str(state.state) == \"COMPLETED\":\n", + " completed.append((left, right, ora_id))\n", + " else:\n", + " print(f\" {state.error_message}\")\n", + "\n", + "\n", + "# --- 3. build the tab from whatever finished ---------------------------------\n", + "\n", + "ora_tab = client.workspaces.tabs.create(workspace_id=WORKSPACE_ID, name=ORA_TAB)\n", + "ORA_TAB_ID = str(ora_tab.id)\n", + "\n", + "y = 0\n", + "for left, right, ora_id in completed:\n", + " ora = client.datasets.get_by_id(ora_id)\n", + " client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=ORA_TAB_ID, item_id=\"heading\",\n", + " x=0, y=y, width=12, height=2,\n", + " settings={\"text\": f\"{left} vs {right}, up-regulated, {DATABASE}\"},\n", + " )\n", + " client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=ORA_TAB_ID, item_id=\"ora_dot_plot\",\n", + " x=0, y=y + 2, width=12, height=16,\n", + " settings={\n", + " \"datasetsSearch\": {\"individualResults\": [{\"id\": ora_id, \"name\": ora.name}]},\n", + " \"topN\": TOP_N,\n", + " \"xAxis\": \"GeneRatio\",\n", + " \"entityListForSelection\": \"measured_in_dataset\",\n", + " },\n", + " )\n", + " print(f\" ok {left} vs {right}\")\n", + " y += 18\n", + "\n", + "print(f\"\\nOpen it: {client.base_url.replace('/api', '')}/w/{WORKSPACE_ID}/dashboard\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## What to take from this\n", + "\n", + "**The threshold lives in your script, not in a plot.** `FDR` and `FOLD_CHANGE` are set once at the\n", + "top and used twice: to build the lists and to configure the volcanoes. Over the API that duplication\n", + "is visible and checkable. In the app the same two numbers live in a form and in a module's settings\n", + "panel, and nothing tells you when they drift apart.\n", + "\n", + "**The lists are the join.** Once the up and down sets exist as entity lists they are ordinary\n", + "workspace objects: highlighted on the volcano, selectable in the app, and used as the foreground of\n", + "the ORA job. Nothing downstream re-derives them, so everything is looking at the same proteins.\n", + "\n", + "**Enrichment is a job, not a widget.** Running `ora` gives you an `ENRICHMENT` dataset you can\n", + "download, diff between runs, and point more than one module at. The Reactome ORA modules call the\n", + "service live and leave nothing behind, which is fine for a look and poor for a pipeline.\n", + "\n", + "**One trap worth knowing.** `entity_abundance_plot.conditions` is a required setting whose registry\n", + "default is an empty list. It validates clean and then renders empty, the same shape of problem as\n", + "the QC summary table's Table Grouping, so it is filled in explicitly above." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/walkthroughs/md_qc_report_automation.ipynb b/examples/walkthroughs/md_qc_report_automation.ipynb new file mode 100644 index 0000000..ad4de26 --- /dev/null +++ b/examples/walkthroughs/md_qc_report_automation.ipynb @@ -0,0 +1,577 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c00", + "metadata": {}, + "source": [ + "# A QC report, built by API instead of by automation\n", + "\n", + "The app ships a **Quality Control Report** automation. You pick it from the Automations menu, it\n", + "asks four questions, and it lays out a full QC report on the tab.\n", + "\n", + "This notebook does the same thing from the API. Same modules, same grid positions, same settings,\n", + "no clicking. Which means you can attach it to the end of a pipeline and have the report waiting when\n", + "the data lands.\n", + "\n", + "### What the in-app automation actually is\n", + "\n", + "Automations are declarative workflows that run **in the browser**\n", + "(`workflow/app/javascript/workspaces/lib/automations/QCReport.js`). There is no automations endpoint\n", + "on the API, so you cannot trigger one remotely. But you do not need to: an automation is a list of\n", + "`AddModuleToTab` steps, and adding a module to a tab is a plain API call.\n", + "\n", + "The real automation asks for four things once:\n", + "\n", + "| it asks | we set once, below |\n", + "|---|---|\n", + "| `datasetsSearch` | the intensity dataset |\n", + "| `entityType` | Protein or Peptide |\n", + "| `groupBy` | the metadata column to colour and group by |\n", + "| `xAxis` | the metadata column to order samples along |\n", + "\n", + "then places **29 modules** using them. This notebook does exactly that." + ] + }, + { + "cell_type": "markdown", + "id": "c01", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c02", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)" + ] + }, + { + "cell_type": "markdown", + "id": "c03", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "The four answers the automation would ask for, plus where to put the report.\n", + "\n", + "### `GROUP_BY` and `X_AXIS` are sample metadata column names\n", + "\n", + "These two are not free text and they are not settings of the module. They are the names of columns\n", + "in the **sample metadata of the upload the dataset came from**, and almost every data module on the\n", + "tab reads one of them:\n", + "\n", + "| module | setting | value |\n", + "|---|---|---|\n", + "| dimensionality reduction (x5) | `colourBy` | `GROUP_BY` |\n", + "| CV distribution, CV violin | `xAxis` | `GROUP_BY` |\n", + "| box plot | `xAxis` | `X_AXIS` |\n", + "| intensity distribution | `groupByColumn` | `GROUP_BY` |\n", + "| QC summary table (x3) | `xAxis`, labelled Table Grouping | `GROUP_BY` |\n", + "\n", + "For this dataset the grouping column is `condition`, so that is what `GROUP_BY` is set to below.\n", + "`sample_name` is the one column you can rely on always being there, which is why it is the default\n", + "for ordering.\n", + "\n", + "Nothing checks these for you. The registry cannot publish a list of valid values, because the valid\n", + "values are whatever columns that particular upload happens to carry, so a column name that does not\n", + "exist is stored without complaint and the module comes back empty. This is the same class of problem\n", + "as the empty Table Grouping described in section 3.\n", + "\n", + "**What that means if you are automating this.** A script like this one is only reusable across\n", + "uploads if those uploads share a sample metadata schema. If every upload you produce carries a\n", + "column called `condition`, this notebook runs unchanged on all of them. If one pipeline writes\n", + "`condition`, another writes `treatment` and a third writes `Condition`, then the same script builds\n", + "a report of blank tiles on two of the three and tells you nothing went wrong. Deciding on the\n", + "metadata column names once, upstream at upload time, is what makes the report automatable at all." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c04", + "metadata": {}, + "outputs": [], + "source": [ + "# The intensity dataset to report on.\n", + "DATASET_ID = \"00000000-0000-0000-0000-000000000000\"\n", + "\n", + "# LOWERCASE. protein | peptide | gene | metabolite | ptm\n", + "# Capitalised values are accepted by the API and then fail to render.\n", + "ENTITY_TYPE = \"protein\"\n", + "\n", + "# Sample metadata columns of the upload behind DATASET_ID, not module settings.\n", + "# GROUP_BY colours and groups; X_AXIS orders the samples. A name that is not a\n", + "# real column is accepted and renders empty, so check these against your upload.\n", + "GROUP_BY = \"condition\"\n", + "X_AXIS = \"sample_name\"\n", + "\n", + "# Scaling for the dimensionality reduction and heatmap. none | zscore | centered\n", + "# The in-app automation uses zscore. Leaving this at the \"none\" default is\n", + "# almost never what you want: unscaled PCA is dominated by the most abundant\n", + "# proteins.\n", + "SCALING_METHOD = \"zscore\"\n", + "\n", + "WORKSPACE_NAME = \"QC report (API)\"\n", + "TAB_NAME = \"Quality control\"" + ] + }, + { + "cell_type": "markdown", + "id": "c05", + "metadata": {}, + "source": [ + "## 1. A workspace and a tab to put it on\n", + "\n", + "Skip this if you already have one: set `WORKSPACE_ID` and `TAB_ID` yourself." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c06", + "metadata": {}, + "outputs": [], + "source": [ + "workspace = client.workspaces.create(\n", + " name=WORKSPACE_NAME,\n", + " description=\"QC report generated through the API\",\n", + ")\n", + "WORKSPACE_ID = str(workspace.id)\n", + "\n", + "tab = client.workspaces.tabs.create(workspace_id=WORKSPACE_ID, name=TAB_NAME)\n", + "TAB_ID = str(tab.id)\n", + "\n", + "print(\"workspace:\", WORKSPACE_ID)\n", + "print(\"tab :\", TAB_ID)" + ] + }, + { + "cell_type": "markdown", + "id": "c07", + "metadata": {}, + "source": [ + "## 2. The shared settings\n", + "\n", + "Every data module in the report gets the same dataset and entity type. This is the automation's\n", + "\"ask once, use everywhere\" made literal.\n", + "\n", + "`datasetsSearch` is a structured field: it needs `{\"individualResults\": [{\"id\", \"name\"}]}` with\n", + "**both** id and name, not a bare id." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c08", + "metadata": {}, + "outputs": [], + "source": [ + "dataset = client.datasets.get_by_id(DATASET_ID)\n", + "assert dataset, f\"No dataset {DATASET_ID!r}\"\n", + "print(f\"{dataset.name} ({dataset.type}, {dataset.state})\")\n", + "\n", + "COMMON = {\n", + " \"datasetsSearch\": {\n", + " \"individualResults\": [{\"id\": DATASET_ID, \"name\": dataset.name}]\n", + " },\n", + " \"entityType\": ENTITY_TYPE,\n", + "}\n", + "\n", + "X_AXIS_ORDERED = [{\"field\": X_AXIS, \"order\": \"none\"}]\n", + "\n", + "# Grouping by condition, in the shape the automation uses everywhere it groups:\n", + "# the CV plots and the QC summary tables, where it is labelled Table Grouping.\n", + "# Two things to note. The summary table's grouping key is called `xAxis`, and\n", + "# its registry default is an empty list, so a table placed without it comes out\n", + "# blank. And the CV plots must be grouped by GROUP_BY rather than by X_AXIS: a\n", + "# coefficient of variation needs at least two replicates per group, so grouping\n", + "# by sample_name gives one sample per group and the render fails outright.\n", + "GROUPING = [{\"field\": GROUP_BY, \"order\": \"asc\", \"type\": \"categorical\"}]\n", + "\n", + "# Fetch the registry once; the validator below reads it.\n", + "REGISTRY = {str(m.id): m for m in client.module_registry.list()}" + ] + }, + { + "cell_type": "markdown", + "id": "valmd", + "metadata": {}, + "source": [ + "## 3. Validate before you build\n", + "\n", + "The API will accept settings that do not work. A value outside a field's allowed list is stored\n", + "happily and only shows up later as an empty dropdown in the app and a 500 from the visualisation\n", + "endpoint. That is what broke the first version of this notebook: `entityType` was sent as\n", + "`\"Protein\"` instead of `\"protein\"`.\n", + "\n", + "Three things are checkable up front:\n", + "\n", + "| check | how |\n", + "|---|---|\n", + "| keys the module does not declare | `module.validate_settings_keys(settings)` |\n", + "| required keys with no value | `module.missing_required_keys({**defaults, **settings})` |\n", + "| values outside an allowed list | `spec[\"parameters\"][\"options\"]` |\n", + "\n", + "That third one is the important one and it is easy to miss: **142 of the 625 settings across the\n", + "registry publish an options list**, under `parameters.options`, not at the top level of the spec.\n", + "\n", + "There is a fourth mistake none of the three catch: a **required key whose default is empty**.\n", + "`qc_summary_table`'s Table Grouping (`xAxis`) is required, and the registry default is `[]`. An\n", + "empty list is a value, so `missing_required_keys` is satisfied, the module is created, and the table\n", + "renders with nothing to group by. Required and populated are not the same check.\n", + "\n", + "`entityType` is the awkward exception. It publishes no options, because its valid values depend on\n", + "the dataset and the organisation's feature flags, so you have to know them:\n", + "`protein`, `peptide`, `gene`, `metabolite`, `ptm`, all lowercase." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "valcode", + "metadata": {}, + "outputs": [], + "source": [ + "# Entity types, lowercase. The registry does NOT publish these as options\n", + "# (fieldType \"EntityType\" is resolved from the dataset at render time), so this\n", + "# is the one list you have to know rather than look up.\n", + "# Source: workflow app/javascript/workspaces/lib/helpers/EntityListHelper.js\n", + "ENTITY_TYPES = {\"protein\", \"peptide\", \"gene\", \"metabolite\", \"ptm\"}\n", + "\n", + "\n", + "def check_settings(item_id, settings):\n", + " \"\"\"Return a list of problems with `settings` for `item_id`, empty if fine.\n", + "\n", + " Catches three classes of mistake before anything is sent:\n", + "\n", + " 1. keys the module does not declare (server rejects these outright)\n", + " 2. required keys with no value (module renders broken)\n", + " 3. values outside a declared options list (silently accepted, then fails)\n", + "\n", + " (3) is the dangerous one. The API accepts any string for an enum field, so a\n", + " wrong value is only visible as an empty dropdown in the app and a 500 from\n", + " the visualisation endpoint.\n", + " \"\"\"\n", + " module = REGISTRY.get(item_id)\n", + " if module is None:\n", + " return [f\"{item_id!r} is not in the module registry\"]\n", + "\n", + " problems = []\n", + " schema = module.input_settings or {}\n", + " specs = schema if isinstance(schema, dict) else {\n", + " s.get(\"key\"): s for s in schema if isinstance(s, dict)\n", + " }\n", + "\n", + " unknown = module.validate_settings_keys(settings)\n", + " if unknown:\n", + " problems.append(f\"keys not declared by this module: {list(unknown)}\")\n", + "\n", + " missing = module.missing_required_keys({**module.defaults(), **settings})\n", + " if missing:\n", + " problems.append(f\"required with no value: {missing}\")\n", + "\n", + " for key, value in settings.items():\n", + " spec = specs.get(key)\n", + " if not isinstance(spec, dict):\n", + " continue\n", + " options = (spec.get(\"parameters\") or {}).get(\"options\")\n", + " if options:\n", + " allowed = [o.get(\"value\") for o in options if isinstance(o, dict)]\n", + " if allowed and value not in allowed:\n", + " problems.append(f\"{key}={value!r} not one of {allowed}\")\n", + "\n", + " if settings.get(\"entityType\") not in (None, *ENTITY_TYPES):\n", + " problems.append(\n", + " f\"entityType={settings['entityType']!r} is not one of \"\n", + " f\"{sorted(ENTITY_TYPES)} (lowercase)\"\n", + " )\n", + " return problems\n", + "\n", + "\n", + "# Prove it catches the mistake that broke this notebook the first time.\n", + "bad = {**COMMON, \"entityType\": \"Protein\"}\n", + "print(\"Checking a deliberately wrong entityType:\")\n", + "for p in check_settings(\"dimensionality_reduction_plot\", bad):\n", + " print(\" \", p)\n", + "print(\"\\nChecking the real settings:\",\n", + " check_settings(\"dimensionality_reduction_plot\", COMMON) or \"no problems\")" + ] + }, + { + "cell_type": "markdown", + "id": "c09", + "metadata": {}, + "source": [ + "## 4. The report, as data\n", + "\n", + "The automation is a chain of builder calls. Here the same thing is a list: one row per module, with\n", + "its grid position and whatever it needs on top of `COMMON`.\n", + "\n", + "Positions are taken from the automation itself, so the result matches what the app produces. The\n", + "grid is 12 columns wide, and `y` runs down the page.\n", + "\n", + "Because reading 29 rows of chained builder code is nobody's idea of fun, this is where the API\n", + "version is arguably nicer than the automation: the report is a table you can edit, reorder, or\n", + "generate." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c10", + "metadata": {}, + "outputs": [], + "source": [ + "def note(title, body):\n", + " \"\"\"Workspace text modules take PLAIN TEXT. HTML tags are not allowed.\"\"\"\n", + " return {\"text\": f\"{title}\\n{body}\"}\n", + "\n", + "\n", + "# Settings shared by the five dimensionality reduction plots. The in-app\n", + "# automation sets scaling and colours by the grouping column rather than\n", + "# leaving the sample_name default.\n", + "DR = {\"scalingMethod\": SCALING_METHOD, \"colourBy\": GROUP_BY, \"drMethod\": \"pca\"}\n", + "\n", + "REPORT = [\n", + " # item_id, x, y, w, h, extra settings\n", + " (\"heading\", 0, 0, 12, 3, {\"text\": \"Quality Control Report\"}),\n", + " (\"text\", 0, 3, 12, 6,\n", + " note(\"Dimensionality reduction\",\n", + " \"How samples relate to one another. Replicates should cluster.\")),\n", + "\n", + " # Variance explained, then PC1 against PC2, PC3, PC4, PC5.\n", + " (\"dimensionality_reduction_plot\", 0, 9, 6, 20, {**DR, \"drSubtype\": \"pca_variance\"}),\n", + " (\"dimensionality_reduction_plot\", 6, 9, 6, 20,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC2\"}}),\n", + " (\"dimensionality_reduction_plot\", 0, 29, 4, 12,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC3\"}}),\n", + " (\"dimensionality_reduction_plot\", 4, 29, 4, 12,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC4\"}}),\n", + " (\"dimensionality_reduction_plot\", 8, 29, 4, 12,\n", + " {**DR, \"drSubtype\": \"pca\", \"nComponents\": {\"xAxis\": \"PC1\", \"yAxis\": \"PC5\"}}),\n", + "\n", + " (\"text\", 0, 41, 12, 5,\n", + " note(\"Variability\", \"Coefficient of variation within each group.\")),\n", + " (\"cv_distribution_violin_plot\", 0, 46, 12, 15, {\"xAxis\": GROUPING}),\n", + " (\"cv_distribution_plot\", 0, 61, 12, 14, {\"xAxis\": GROUPING}),\n", + " (\"qc_summary_table\", 0, 75, 12, 15, {\"xAxis\": GROUPING}),\n", + "\n", + " (\"text\", 0, 90, 12, 5,\n", + " note(\"Sample correlation\", \"Correlation between samples, clustered.\")),\n", + " # The heatmap has no scalingMethod. Its equivalent is normalisationMethod,\n", + " # which already defaults to zscore. The QC report uses it as a sample\n", + " # CORRELATION heatmap, which is a different dataType.\n", + " (\"heatmap\", 0, 95, 12, 21,\n", + " {\"dataType\": \"correlation\", \"correlationAxis\": \"samples\",\n", + " \"correlationMethod\": \"pearson\", \"showSampleNamesCorrelationHeatmap\": True}),\n", + "\n", + " (\"text\", 0, 116, 12, 5,\n", + " note(\"Missing values\", \"Where the gaps are, by feature and by sample.\")),\n", + " (\"missing_values_by_feature_plot\", 0, 121, 6, 19, {}),\n", + " (\"missing_values_by_sample_plot\", 6, 121, 6, 19, {}),\n", + " (\"missing_values_heatmap\", 0, 140, 12, 25, {}),\n", + " (\"qc_summary_table\", 0, 165, 12, 15, {\"xAxis\": GROUPING}),\n", + "\n", + " (\"text\", 0, 180, 12, 5,\n", + " note(\"Intensity distributions\", \"Per-sample intensity spread.\")),\n", + " (\"box_plot\", 0, 185, 12, 16, {\"xAxis\": X_AXIS_ORDERED}),\n", + " (\"intensity_distribution_plot\", 0, 201, 12, 15, {\"groupByColumn\": GROUP_BY}),\n", + " (\"qc_summary_table\", 0, 216, 12, 15, {\"xAxis\": GROUPING}),\n", + "]\n", + "\n", + "print(f\"{len(REPORT)} modules, {max(y + h for _, _, y, _, h, _ in REPORT)} grid rows tall\")" + ] + }, + { + "cell_type": "markdown", + "id": "c11", + "metadata": {}, + "source": [ + "## 5. Build it\n", + "\n", + "One loop. `create_with_defaults()` fills in every registry default, so each row only carries what\n", + "differs from the default.\n", + "\n", + "If a module rejects the settings it is reported and skipped rather than aborting the run, so you get\n", + "as much of the report as the data supports." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c12", + "metadata": {}, + "outputs": [], + "source": [ + "LAYOUT_ONLY = {\"heading\", \"text\"}\n", + "\n", + "# Validate everything BEFORE creating anything. A bad enum value is accepted by\n", + "# the API and only shows up as a broken widget, so catching it here is the\n", + "# difference between a report and a page of error tiles.\n", + "problems = []\n", + "for item_id, x, y, w, h, extra in REPORT:\n", + " settings = dict(extra) if item_id in LAYOUT_ONLY else {**COMMON, **extra}\n", + " for p in check_settings(item_id, settings):\n", + " problems.append(f\"{item_id} (y={y}): {p}\")\n", + "\n", + "if problems:\n", + " print(f\"{len(problems)} problem(s) found, nothing was created:\\n\")\n", + " for p in problems:\n", + " print(\" \", p)\n", + " raise SystemExit(\"Fix the settings above and re-run.\")\n", + "\n", + "print(\"All settings validate. Building.\\n\")\n", + "\n", + "placed = []\n", + "for item_id, x, y, w, h, extra in REPORT:\n", + " settings = dict(extra) if item_id in LAYOUT_ONLY else {**COMMON, **extra}\n", + " module = client.workspaces.modules.create_with_defaults(\n", + " workspace_id=WORKSPACE_ID, tab_id=TAB_ID, item_id=item_id,\n", + " x=x, y=y, width=w, height=h, settings=settings,\n", + " )\n", + " placed.append(module)\n", + " print(f\" ok {item_id:<32} x={x:<3} y={y:<4} w={w:<3} h={h}\")\n", + "\n", + "print(f\"\\n{len(placed)} modules placed\")" + ] + }, + { + "cell_type": "markdown", + "id": "c13", + "metadata": {}, + "source": [ + "## 6. What ended up on the tab" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c14", + "metadata": {}, + "outputs": [], + "source": [ + "for m in sorted(client.workspaces.modules.list(WORKSPACE_ID, TAB_ID),\n", + " key=lambda m: (m.y, m.x)):\n", + " print(f\" {str(m.item_id):<32} x={m.x:<3} y={m.y:<4} w={m.width:<3} h={m.height}\")\n", + "\n", + "print(f\"\\nOpen it: {client.base_url.replace('/api', '')}/w/{WORKSPACE_ID}/dashboard\")" + ] + }, + { + "cell_type": "markdown", + "id": "c15", + "metadata": {}, + "source": [ + "## 7. Pull a figure back out\n", + "\n", + "The report is now in the app. If you also want the figures headlessly, render any module to its\n", + "Plotly payload and write it out. There is no server-side image export, so rendering to PNG happens\n", + "locally.\n", + "\n", + "```python\n", + "result = client.workspaces.modules.visualize(WORKSPACE_ID, TAB_ID, module_id)\n", + "# result == {\"data\": [...], \"layout\": {...}}\n", + "\n", + "import plotly.io as pio\n", + "pio.write_image(result, \"figure.png\") # needs kaleido\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c16", + "metadata": {}, + "outputs": [], + "source": [ + "# Not every module renders server-side, and some currently error. Report the\n", + "# reason per module and keep going: the point is the report, which is already\n", + "# built and visible in the app either way.\n", + "candidates = [m for m in placed\n", + " if str(m.item_id) not in {\"heading\", \"text\", \"qc_summary_table\"}]\n", + "\n", + "rendered = None\n", + "for target in candidates:\n", + " try:\n", + " result = client.workspaces.modules.visualize(\n", + " WORKSPACE_ID, TAB_ID, str(target.id), poll_s=3, timeout_s=120\n", + " )\n", + " except Exception as exc:\n", + " reason = \"no server-side render\" if \"not supported\" in str(exc) else \\\n", + " f\"error: {str(exc).splitlines()[0][:70]}\"\n", + " print(f\" {str(target.item_id):<32} {reason}\")\n", + " continue\n", + "\n", + " if isinstance(result, dict) and \"data\" in result:\n", + " rendered = (target, result)\n", + " break\n", + " print(f\" {str(target.item_id):<32} still rendering\")\n", + "\n", + "if rendered:\n", + " target, result = rendered\n", + " print(f\"\\nRendered {target.item_id}: {len(result['data'])} trace(s)\")\n", + " for t in result[\"data\"][:4]:\n", + " print(f\" {t.get('type', '?'):<10} {str(t.get('name')):<26} \"\n", + " f\"{len(t.get('x', []) or [])} points\")\n", + "else:\n", + " print(\"\\nNothing rendered headlessly this time. The report itself is fine \"\n", + " \"and is visible in the app.\")" + ] + }, + { + "cell_type": "markdown", + "id": "c17", + "metadata": {}, + "source": [ + "## Making it a real automation\n", + "\n", + "Nothing here needs a notebook. Drop sections 2 to 4 into a `.py` file, take `DATASET_ID` from an\n", + "argument or an environment variable, and call it at the end of your pipeline:\n", + "\n", + "```python\n", + "build_qc_report(dataset_id=sys.argv[1])\n", + "```\n", + "\n", + "The report exists before anyone opens the app. That is the part the in-app automation cannot do:\n", + "it needs a person and a browser.\n", + "\n", + "Two honest limits:\n", + "\n", + "* **You cannot trigger an in-app automation over the API.** They are frontend workflows. What you\n", + " can do is reproduce their output, which is what this notebook is.\n", + "* **`quality_control_report_classic`** is a single module that renders an entire QC report on its\n", + " own, and it only needs `{\"experimentId\": }`. If that is all you want, place that one\n", + " module instead of the 22 here. This notebook builds the report out of individual modules because\n", + " that is what the modern automation does, and because it is the version you can edit." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/walkthroughs/md_workspace_walkthrough.ipynb b/examples/walkthroughs/md_workspace_walkthrough.ipynb new file mode 100644 index 0000000..50143b6 --- /dev/null +++ b/examples/walkthroughs/md_workspace_walkthrough.ipynb @@ -0,0 +1,580 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cell00", + "metadata": {}, + "source": [ + "# Build a workspace through the API\n", + "\n", + "Create a workspace, add tabs, lay modules out on the grid, and render a plot.\n", + "\n", + "Edit the **Configuration** cell, then run top to bottom.\n", + "\n", + "The visual side of the app is `Workspace -> Tab -> Module`. A workspace is purely a container: it\n", + "does not own or store data. Modules reference existing uploads and datasets by id, so there is no\n", + "such thing as uploading *into* a workspace.\n", + "\n", + "### Before you start\n", + "\n", + "```bash\n", + "pip install \"git+https://github.com/MassDynamics/md-python.git\" python-dotenv\n", + "```\n", + "\n", + "with credentials in a `.env` file next to this notebook:\n", + "\n", + "```\n", + "MD_API_BASE_URL=https://app.massdynamics.com/api\n", + "MD_AUTH_TOKEN=\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "cell01", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell02", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "import os\n", + "from collections import defaultdict\n", + "\n", + "from dotenv import load_dotenv\n", + "from md_python import MDClient\n", + "\n", + "load_dotenv()\n", + "assert os.getenv(\"MD_AUTH_TOKEN\"), \"MD_AUTH_TOKEN is not set (check your .env)\"\n", + "\n", + "client = MDClient(version=\"v2\")\n", + "print(\"API:\", client.base_url)\n", + "print(\"Health:\", client.health.check())" + ] + }, + { + "cell_type": "markdown", + "id": "cell03", + "metadata": {}, + "source": [ + "## Configuration\n", + "\n", + "The only cell you need to edit." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell04", + "metadata": {}, + "outputs": [], + "source": [ + "WORKSPACE_NAME = \"YOUR WORKSPACE NAME\"\n", + "WORKSPACE_DESCRIPTION = \"Built through the API\"\n", + "\n", + "TAB_NAMES = [\"Overview\", \"Differential abundance\"]\n", + "\n", + "# Part of a dataset name to search for in section 2.\n", + "DATASET_SEARCH = \"pairwise\"\n", + "\n", + "# A PAIRWISE dataset to plot, and which comparison to show.\n", + "# Leave as None to build the layout without a data module.\n", + "PAIRWISE_DATASET_ID = None\n", + "COMPARISON_LEFT = \"TREATED_GROUP\"\n", + "COMPARISON_RIGHT = \"CONTROL_GROUP\"" + ] + }, + { + "cell_type": "markdown", + "id": "cell05", + "metadata": {}, + "source": [ + "## 1. Create the workspace and its tabs\n", + "\n", + "Both take a name and little else." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell06", + "metadata": {}, + "outputs": [], + "source": [ + "workspace = client.workspaces.create(\n", + " name=WORKSPACE_NAME,\n", + " description=WORKSPACE_DESCRIPTION,\n", + ")\n", + "workspace_id = str(workspace.id)\n", + "print(\"Workspace:\", workspace_id)\n", + "\n", + "tab_ids = {}\n", + "for name in TAB_NAMES:\n", + " tab = client.workspaces.tabs.create(workspace_id=workspace_id, name=name)\n", + " tab_ids[name] = str(tab.id)\n", + " print(f\" tab {name!r}: {tab.id}\")\n", + "\n", + "tab_id = tab_ids[TAB_NAMES[0]]" + ] + }, + { + "cell_type": "markdown", + "id": "new0", + "metadata": {}, + "source": [ + "## 2. Find the data you want to plot\n", + "\n", + "Modules reference datasets by id, so before you can place a plot you need to find the right dataset.\n", + "\n", + "`client.datasets.query()` searches by **partial name** and filters by type and state. It returns\n", + "`{\"data\": [...], \"pagination\": {...}}`, 50 per page.\n", + "\n", + "```python\n", + "client.datasets.query(search=\"my experiment\") # name contains\n", + "client.datasets.query(type=[\"PAIRWISE\"], state=[\"COMPLETED\"])\n", + "client.datasets.query(upload_id=...) # everything from one upload\n", + "```\n", + "\n", + "Valid `type` values are `DEMO`, `DOSE_RESPONSE`, `DOSE_RESPONSE_AGGREGATE`, `ENRICHMENT`,\n", + "`IMPUTATION`, `INTENSITY`, `NORMALISATION_AND_IMPUTATION` and `PAIRWISE`. Anything else is a 400.\n", + "Note `ANOVA` datasets exist but cannot be filtered for, so find those with `search=` instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "new1", + "metadata": {}, + "outputs": [], + "source": [ + "result = client.datasets.query(search=DATASET_SEARCH, page=1)\n", + "found = result[\"data\"]\n", + "print(f\"{result['pagination']['total_count']} datasets matching {DATASET_SEARCH!r}\"\n", + " f\" (showing {len(found)})\\n\")\n", + "\n", + "for d in found[:15]:\n", + " print(f\" {str(d.get('type')):<18} {str(d.get('state')):<10} \"\n", + " f\"{d.get('id')} {d.get('name')}\")" + ] + }, + { + "cell_type": "markdown", + "id": "new2", + "metadata": {}, + "source": [ + "### Narrow it to what you can actually plot\n", + "\n", + "A volcano needs a `PAIRWISE` dataset that has finished. Filtering on both saves you from picking\n", + "one that is still processing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "new3", + "metadata": {}, + "outputs": [], + "source": [ + "pairwise_datasets = client.datasets.query(\n", + " type=[\"PAIRWISE\"], state=[\"COMPLETED\"], page=1\n", + ")[\"data\"]\n", + "\n", + "print(f\"{len(pairwise_datasets)} completed PAIRWISE datasets on this page:\\n\")\n", + "for d in pairwise_datasets[:10]:\n", + " print(f\" {d['id']} {d['name']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "new4", + "metadata": {}, + "source": [ + "### What does one of them look like?\n", + "\n", + "`get_by_id()` returns the full record. Three parts are worth reading:\n", + "\n", + "| | |\n", + "|---|---|\n", + "| `tables` | what you can download. A PAIRWISE dataset carries `output_comparisons`. |\n", + "| `input_dataset_ids` | what it was computed from, so you can walk back to the intensity dataset |\n", + "| `job_run_params` | the settings it was run with |\n", + "\n", + "For a pairwise dataset `job_run_params` is especially useful: `condition_comparisons` lists every\n", + "contrast the dataset actually contains, and `condition_column` names the metadata column they were\n", + "grouped on. Those are exactly the values the volcano module needs later, so read them from here\n", + "rather than guessing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "new5", + "metadata": {}, + "outputs": [], + "source": [ + "if pairwise_datasets:\n", + " inspect_id = pairwise_datasets[0][\"id\"]\n", + " ds = client.datasets.get_by_id(inspect_id)\n", + "\n", + " print(f\"name : {ds.name}\")\n", + " print(f\"id : {ds.id}\")\n", + " print(f\"type : {ds.type} state: {ds.state}\")\n", + " print(f\"tables : {[t.name for t in (ds.tables or [])]}\")\n", + " print(f\"inputs : {[str(i) for i in ds.input_dataset_ids]}\")\n", + "\n", + " params = ds.job_run_params or {}\n", + " print(f\"\\ncondition_column: {params.get('condition_column')}\")\n", + " pairs = (params.get(\"condition_comparisons\") or {}).get(\n", + " \"condition_comparison_pairs\", []\n", + " )\n", + " print(f\"{len(pairs)} comparison(s) in this dataset:\")\n", + " for left, right in pairs[:8]:\n", + " print(f\" {left} vs {right}\")\n", + "\n", + " if pairs:\n", + " left, right = pairs[0]\n", + " print(\"\\nTo plot the first one, put these in the Configuration cell:\")\n", + " print(f' PAIRWISE_DATASET_ID = \"{ds.id}\"')\n", + " print(f' COMPARISON_LEFT = \"{left}\"')\n", + " print(f' COMPARISON_RIGHT = \"{right}\"')\n", + "else:\n", + " print(\"No completed PAIRWISE datasets found. Run a pairwise comparison first.\")" + ] + }, + { + "cell_type": "markdown", + "id": "new6", + "metadata": {}, + "source": [ + "### Reading the numbers\n", + "\n", + "To pull the results themselves, get a presigned URL for one of the dataset's tables:\n", + "\n", + "```python\n", + "url = client.datasets.download_table_url(dataset_id, \"output_comparisons\", format=\"csv\")\n", + "\n", + "import pandas as pd\n", + "df = pd.read_csv(url) # one row per entity, one column trio per comparison\n", + "```\n", + "\n", + "`output_comparisons` has `AdjPValue - `, `PValue ...` and `MaxLog2FoldChange ...`\n", + "columns per comparison, plus a bare aggregate `AdjPValue`. Dose-response datasets expose\n", + "`output_curves` and `output_volcanoes` instead, where the half-maximal dose column is `ED50`." + ] + }, + { + "cell_type": "markdown", + "id": "cell07", + "metadata": {}, + "source": [ + "## 3. Which modules can I place?\n", + "\n", + "`client.module_registry.list()` returns every module the app can render. The `<- needs no settings`\n", + "marker flags the handful you can drop onto a tab with no configuration at all, which is useful when\n", + "you want to build the page shape first and wire up data later." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell08", + "metadata": {}, + "outputs": [], + "source": [ + "modules = client.module_registry.list()\n", + "\n", + "by_group = defaultdict(list)\n", + "for m in modules:\n", + " by_group[str(m.group)].append(m)\n", + "\n", + "print(f\"{len(modules)} modules\\n\")\n", + "for group in sorted(by_group):\n", + " print(group)\n", + " for m in sorted(by_group[group], key=lambda x: str(x.id)):\n", + " no_config = not m.missing_required_keys(m.defaults())\n", + " print(f\" {str(m.id):<40} {m.name}{' <- needs no settings' if no_config else ''}\")\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "cell09", + "metadata": {}, + "source": [ + "## 4. What parameters does a module take?\n", + "\n", + "`client.module_registry.get(item_id)` returns the module's `input_settings`: field name, type,\n", + "default, and whether it is required. This is the source of truth. The server rejects any settings\n", + "key the module does not declare.\n", + "\n", + "**Requirements cascade.** A field carrying a `when` clause only becomes required once the field it\n", + "depends on is set, so the \"you must supply\" list grows as you fill it in. Fill in what it asks for,\n", + "then run this again." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell10", + "metadata": {}, + "outputs": [], + "source": [ + "def show_module(item_id):\n", + " \"\"\"Print one module's parameters. Fetches its own registry entry, so this\n", + " cell works on its own.\"\"\"\n", + " m = client.module_registry.get(item_id)\n", + " assert m, f\"{item_id!r} is not in the module registry\"\n", + "\n", + " schema = m.input_settings or {}\n", + " items = list(schema.items()) if isinstance(schema, dict) else [\n", + " (s.get(\"key\"), s) for s in schema if isinstance(s, dict)\n", + " ]\n", + " print(f\"{m.id} ({m.group}) - {len(items)} settings\\n\")\n", + " for key, spec in items:\n", + " if not isinstance(spec, dict):\n", + " print(f\" {key}\")\n", + " continue\n", + " bits = []\n", + " if spec.get(\"fieldType\"):\n", + " bits.append(str(spec[\"fieldType\"]))\n", + " if spec.get(\"default\") is not None:\n", + " bits.append(f\"default={spec['default']!r}\")\n", + " if spec.get(\"required\") is True or any(\n", + " r.get(\"name\") == \"is_required\" for r in (spec.get(\"rules\") or [])\n", + " ):\n", + " bits.append(\"REQUIRED\")\n", + " if spec.get(\"when\"):\n", + " bits.append(f\"when={spec['when']}\")\n", + " print(f\" {str(key):<34} {' '.join(bits)}\")\n", + "\n", + " missing = m.missing_required_keys(m.defaults())\n", + " print()\n", + " print(f\"You must supply: {missing}\" if missing\n", + " else \"Nothing required. Can be placed with settings={}.\")\n", + "\n", + "show_module(\"pairwise_volcano_plot\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell11", + "metadata": {}, + "source": [ + "## 5. The grid\n", + "\n", + "A tab is a [react-grid-layout](https://github.com/jbaysolutions/vue-grid-layout) with **12 columns**\n", + "(`DashboardTabModules.vue`, `:col-num=\"12\"`). Every module is placed with four integers:\n", + "\n", + "| | |\n", + "|---|---|\n", + "| `x` | column it starts at, `0` to `11` |\n", + "| `y` | row it starts at, `0` upwards |\n", + "| `width` | how many of the 12 columns it spans |\n", + "| `height` | how many rows tall |\n", + "\n", + "Width is the number that carries the meaning. 12 is full width, so:\n", + "\n", + "```\n", + "width 12 -> one per row\n", + "width 6 -> two side by side (x = 0, 6)\n", + "width 4 -> three side by side (x = 0, 4, 8)\n", + "width 3 -> four side by side (x = 0, 3, 6, 9)\n", + "```\n", + "\n", + "`x + width` must not exceed 12 or the module wraps. Height is in grid rows, not pixels (a row is\n", + "roughly 30px), so a plot usually wants 8 to 12. Step `y` down by the height of the row above or\n", + "modules overlap.\n", + "\n", + "```\n", + "y=0 [ heading, width 12 ]\n", + "y=2 [ panel w=4 ][ panel w=4 ][ panel w=4 ]\n", + "y=8 [ half w=6 ][ half w=6 ]\n", + "y=14 [ volcano, width 12 ]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "cell12", + "metadata": {}, + "source": [ + "### Placing them\n", + "\n", + "`create_with_defaults()` fetches the registry entry, fills in every declared default, merges your\n", + "settings on top, and fails fast if a required setting has no default and you did not supply it.\n", + "\n", + "Prefer it over `create()`: the API does not merge defaults server-side, so a partial settings hash\n", + "renders as a broken widget in the app." + ] + }, + { + "cell_type": "markdown", + "id": "texthtml", + "metadata": {}, + "source": [ + "### The `text` module takes plain text\n", + "\n", + "Write the words you want to see and nothing else. **Workspaces do not allow HTML tags**, and\n", + "markdown is not rendered either, so `\"## Heading\"` shows up literally. When you want a heading, use\n", + "the `heading` module, which exists for exactly that.\n", + "\n", + "Worth knowing because it will mislead you: the registry still describes the field as *\"HTML and\n", + "embedded base64 imgs\"* and the app renders it with `v-html`. Send HTML on the strength of that and\n", + "it will store without complaint. Workspaces are plain text by policy, so send plain text.\n", + "\n", + "There is also **no settings button** on a text module in the app, by design. Its instruction\n", + "declares `INPUT_PROPERTIES = {}`, so there is no settings form to render. You edit it inline\n", + "instead: click into the module and a WYSIWYG toolbar appears in place. The `text` key exists in the\n", + "registry manifest purely so the API can set the body, which is what we do here.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell13", + "metadata": {}, + "outputs": [], + "source": [ + "def place(item_id, x, y, width, height, settings=None):\n", + " assert x + width <= 12, f\"x({x}) + width({width}) exceeds the 12-column grid\"\n", + " m = client.workspaces.modules.create_with_defaults(\n", + " workspace_id=workspace_id, tab_id=tab_id, item_id=item_id,\n", + " x=x, y=y, width=width, height=height, settings=settings or {},\n", + " )\n", + " print(f\" {item_id:<26} x={x:<3} y={y:<3} w={width:<3} h={height:<3} -> {m.id}\")\n", + " return m\n", + "\n", + "\n", + "print(\"row 1: full width\")\n", + "place(\"text\", 0, 0, 12, 2, {\"text\": \"## Results\\nBuilt through the API.\"})\n", + "\n", + "print(\"\\nrow 2: three across, width 4\")\n", + "for i, x in enumerate([0, 4, 8], start=1):\n", + " place(\"text\", x, 2, 4, 6, {\"text\": f\"Panel {i}\\n\\nx={x}, width=4\"})\n", + "\n", + "print(\"\\nrow 3: two across, width 6\")\n", + "for i, x in enumerate([0, 6], start=1):\n", + " place(\"text\", x, 8, 6, 6, {\"text\": f\"Half {i}\\n\\nx={x}, width=6\"})" + ] + }, + { + "cell_type": "markdown", + "id": "cell14", + "metadata": {}, + "source": [ + "## 6. A data module\n", + "\n", + "Structured field types are not plain ids. Two you will meet immediately:\n", + "\n", + "```python\n", + "DatasetsSelect {\"individualResults\": [{\"id\": ..., \"name\": ...}]} # id AND name\n", + "ConditionComparison {\"comparison\": {\"conditionPair\": \" - \",\n", + " \"left\": ..., \"right\": ...}}\n", + "```\n", + "\n", + "`conditionPair` must be exactly `\"{left} - {right}\"`. Send a bare list of ids and the API answers\n", + "*\"DatasetsSelect must be an object with individualResults {id,name}\"*.\n", + "\n", + "Source: `workflow/app/models/module_registry/structured_field_violations.rb`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell15", + "metadata": {}, + "outputs": [], + "source": [ + "if PAIRWISE_DATASET_ID:\n", + " dataset = client.datasets.get_by_id(PAIRWISE_DATASET_ID)\n", + " assert dataset, f\"No dataset {PAIRWISE_DATASET_ID!r}\"\n", + "\n", + " print(\"row 4: volcano, full width\")\n", + " volcano = place(\n", + " \"pairwise_volcano_plot\", 0, 14, 12, 12,\n", + " settings={\n", + " \"datasetsSearch\": {\n", + " \"individualResults\": [\n", + " {\"id\": PAIRWISE_DATASET_ID, \"name\": dataset.name}\n", + " ]\n", + " },\n", + " \"experimentAndConditionComparison\": {\n", + " \"comparison\": {\n", + " \"conditionPair\": f\"{COMPARISON_LEFT} - {COMPARISON_RIGHT}\",\n", + " \"left\": COMPARISON_LEFT,\n", + " \"right\": COMPARISON_RIGHT,\n", + " }\n", + " },\n", + " },\n", + " )\n", + "else:\n", + " volcano = None\n", + " print(\"Skipped: set PAIRWISE_DATASET_ID to place a volcano plot.\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell16", + "metadata": {}, + "source": [ + "### What is on the tab now" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell17", + "metadata": {}, + "outputs": [], + "source": [ + "for m in sorted(client.workspaces.modules.list(workspace_id, tab_id),\n", + " key=lambda m: (m.y, m.x)):\n", + " print(f\" {str(m.item_id):<26} x={m.x:<3} y={m.y:<3} w={m.width:<3} h={m.height}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell20", + "metadata": {}, + "source": [ + "## Where to go next\n", + "\n", + "The workspace is now live in the app. To keep working with it:\n", + "\n", + "```python\n", + "client.workspaces.list_all() # find it again\n", + "client.workspaces.tabs.list_all(workspace_id) # its tabs\n", + "client.workspaces.modules.list(workspace_id, tab_id) # its modules\n", + "client.workspaces.modules.update(...) # move or resize\n", + "client.workspaces.modules.delete(workspace_id, tab_id, module_id)\n", + "```\n", + "\n", + "`client.workspaces.modules.create_text(...)` is a shortcut for text modules if you are writing a lot\n", + "of narrative.\n", + "\n", + "If a module is rejected, re-run `show_module(\"\")` and compare the field list against what\n", + "you are sending. The registry is the source of truth and it does change." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}