Skip to content

ADD cau-qc: the quality-control pass over an assembled channel - #20

Merged
emattei merged 31 commits into
mainfrom
feat/qc-figures
Nov 9, 2026
Merged

emattei merged 31 commits into
mainfrom
feat/qc-figures

Conversation

@emattei

@emattei emattei commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

crispr_analysis_utils.qc, with its skill. One pass over a channel: find the
floor, measure the cells, cut, write, then look at the guides.

data = md.read_h5mu("results/mudata/mudata.h5mu")
qc.publish("results/qc/knee", *qc.knee(data))
cells, _ = qc.filter_cells(data, min_umis=1000)       # read off that knee
fractions = qc.cell_fractions(cells)
qc.publish("results/qc/metrics", *qc.metric_distributions(fractions))
passes, rules = qc.cell_filters(fractions, nmads=5)
clean = qc.apply_filters(cells, passes, rules, path="results/qc/filtered.h5mu")
qc.publish("results/qc/guides", *qc.guide_abundance(qc.guide_counts(clean)))

Every figure is three files: the PDF at 300 dpi, a tsv.gz of the numbers
it was drawn from, and a caption generated with it so the two cannot drift.
publish refuses a figure with an empty table or a blank caption.

The panels draw the rules the file was cut on. metric_distributions
calls cell_filters for them, so passing both the same nmads, limits and
band makes it impossible for the figure to disagree with the object. Each
line is labelled with its rule and value, and each panel titled with how many
cells it put beyond them -- a cut that removes a handful leaves no grey tall
enough to see, and without the count a one-colour panel looks like a panel
that failed to say anything.

apply_filters writes the record. uns["qc"] carries the floor, every
threshold with its source, and how many cells each rejected. It refuses a
frame that does not cover exactly the object's cells: a cell no rule saw
would otherwise pass every one of them.

Classes come from the contig, not the symbol. MT also names the nuclear
metallothioneins and the MTRNR2L family; on GENCODE v43 the contig gives 37
genes and the prefix 566. Ribosomal pseudogenes are counted apart from the
~80 real ribosomal proteins they share a prefix with.

The guide side reports what a channel saw. The guides never seen get a bar
of their own on the linear strip of a symlog axis, since a log scale has no
place for a zero and leaving them out answers a different question; the
caption says plainly that a channel holds far fewer cells than a library holds
guides, so a guide missing from one is not a guide missing from it.

Plots are Okabe-Ito for categories and cividis for continuous scales, per
CLAUDE.md. mudata_builder gains the uns the QC pass reads.

811 tests.

🤖 Generated with Claude Code

emattei and others added 30 commits October 15, 2026 12:33
Every figure writes three files under one stem -- the PDF at 300 dpi, the
numbers it was drawn from, and its caption -- so a reader can find the
number behind a claim without rerunning anything.

Covers the knee plot, the gene classes (mito by contig, ribosomal split
from its pseudogenes, the readout panel), per-cell fractions over scanpy,
a UMI floor, and MAD thresholds read off the channel's own spread.

The mudata builder gains --readout-genes, so a targeted panel marks its
own genes in gene.var and the targeted fraction is computable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scale="normal" rescales the MAD by 1.4826, so nmads reads as standard
deviations rather than raw MADs and the same number cuts a third less
tightly. log defaults off, since a count wants it and a percentage does
not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It said the MAD collapses to zero on an unfiltered barcode space. It does
not: on a real channel it is 0.33 on log10, and the band simply centres on
ambient, reaching 131 UMIs so that every real cell is an outlier. The
reason to filter first is the population, not a degenerate statistic.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drawn with matplotlib directly. Counts go on a log axis and fractions on a
linear one, and an optional flagged mask stacks the rejected cells on the
kept ones, so a threshold is read against the distribution it came from.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
np.nanmin(..., initial=1.0) takes the minimum of the array and the initial
value, so the floor was always 1. On filtered cells that put most of the
range under the data, leaving the panel empty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The kept cells carry the colour and the removed ones recede, so the eye
goes to what the analysis goes on with. The median line turns black so it
is not read as a third grey category.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
logspace rounds its endpoints, so a cell sitting exactly on the smallest
or largest value fell outside its own range: two of 10,296 went missing on
a real channel. The outer edges are pinned back onto the data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stairs fills between its baseline and its values, so passing the flagged
count against a baseline of the kept count filled downward across the
whole bar: every panel came out grey. The stacked layer is kept + flagged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The line was the median and the colouring came from a mask passed in, so
every panel greyed the same union of outliers -- in the total_counts panel
the grey cells were the ones the mitochondrial test rejected. Now each
panel computes its own MAD band through the new mad_bounds, draws it, and
colours blue within and grey beyond, so the line on the figure is the cut.

mad_outlier keeps its behaviour and is now a thin wrapper over the bounds,
which is what stops the two from drifting apart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A cell can be poor on either metric alone and the pair says which, so the
thresholds are drawn as lines and the quadrant counts go in the caption.
The thresholds are arguments rather than derived: a MAD band is one way to
reach them and a closely QCed library is another.

The points are rasterised and the text is not, which is what keeps the PDF
openable: the same figure drawn in R came to 39 MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A single-entry legend labelling every point is noise, and it landed on
the threshold line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A threshold set against another analysis's definition of a class only
applies to that definition. Passing the mask measures it alongside the
built-in classes instead of silently comparing against a different number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A targeted assay works when its panel dominates, and comparing genes rather
than cells is what separates enrichment across the whole panel from a few
genes carrying it. mean_upm does the normalisation on the sparse matrix,
since counts_per_million is for the dense guide-by-sample table of a bulk
screen and would densify a 10,000 by 62,757 one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Half an unfiltered GENCODE reference is never seen, so the background
median is zero and a ratio says nothing. The caption now gives how many
background genes are seen at all, which is the informative number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The same figure -- two measures, thresholds, quadrant counts -- compares
genes against a reference as readily as it compares cells. Only the
caption and the index column named a cell.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
matplotlib drops a non-positive point on a log axis silently. It now stays
in the data file marked by a drawn column, still counts in its quadrant,
and the caption reports how many there were.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Points sized for ten thousand cells are invisible for a few hundred genes,
and the same figure now serves both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The edit that was meant to add it matched a pre-formatter version of the
call and never applied, so the test landed without the behaviour. Points
are now sized by how many there are: invisible at 2pt for a few hundred
genes, a solid block at 14pt for ten thousand cells.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comparing panel genes against a reference filters nothing, so labelling
the points below a line "removed" was wrong. With groups the lines stay as
markers of where things were expected to fall and the colour carries what
each point is, with unlabelled points grey and behind the rest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A label now keeps its colour across figures: a Categorical sets its own
order and anything else is sorted, since row order is not a meaning. The
class that is Day 0 in one figure was a different colour in the next.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The alpha that keeps a cloud of ten thousand cells readable made a handful
of labelled genes barely visible. The background keeps it; the classes
being compared no longer do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Okabe-Ito's yellow is a pale fill on a white page, so a Housekeeping gene
was hard to find even drawn solid. The outline gives every labelled point
an edge of its own; the see-through background keeps none, since an edge
on ten thousand points is mud.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A line with no label cannot be checked, and a cut that removes a handful
of cells leaves no grey tall enough to see, so a panel of one colour read
as a panel saying nothing. Each line now carries the rule and the value,
each panel is titled with how many cells it put beyond the band, and the
legend that named only the blue is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The upper threshold is usually a few points from the panel's right edge,
so a label drawn to its right lost half its characters. Both labels now
sit on the kept side of their line, where there is room.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Over about a decade matplotlib labels the minor ticks too, and four
superscripts inside an inch ran together into one smear. Label 1, 2 and 5
per decade, in the same digits the caption uses. The panel's own count
now comes from the bins, like the table, rather than from the mask.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The module drew the figures and stopped short of the thing they are drawn
for. `cell_filters` turns the metric frame into one column per rule, a
band measured from the channel or a limit someone decided; `apply_filters`
keeps the cells that pass every one and writes the h5mu, with every
threshold that decided it under `uns["qc"]`, so the file says how it was
made. The floor writes itself down in `filter_cells` rather than being
restated later.

Both the panel and the filter now reach `mad_bounds` through one helper,
which also drops the NaNs that would otherwise make the band NaN and fail
every cell.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every column by default, but not every metric earns a cut: a ribosomal
fraction with a long tail loses real cells to a band nobody chose.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The figure measured its own band while the filter applied bands and
limits, so four of seven panels drew a line the file was never cut on:
a ribosomal panel reported 25 cells beyond a band nobody filtered on
while the file dropped 2 on a limit of 6. `metric_distributions` now
calls `cell_filters` for its rules and takes the same arguments, so a
panel can draw several lines, each labelled with the rule that set it.

The guard against applying no rule at all moves to `apply_filters`,
where keeping every cell and calling it filtered is the danger; drawing
a metric no rule covers is not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
guide_counts gives every guide its UMIs and the cells it reached;
guide_abundance bins them with the never-seen in a bar of their own,
on the linear strip of a symlog axis, since a log scale has no place
for a zero and dropping them would answer a different question.

The bin edges are rounded to integers: a log bin narrower than a count
leaves every other one empty and draws a comb.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@emattei
emattei merged commit 7ea3134 into main Nov 9, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant