Give the Console a way into internal-standard normalization - #795
Merged
Conversation
…tation-pipeline # Conflicts: # tests/MSDIAL5/MsdialCoreTestApp/Parser/ConfigParser.cs # tests/MSDIAL5/MsdialCoreTestAppTests/Parser/ConfigParserTests.cs
The algorithm has been in MsdialCore all along -- Normalization.SplashNormalize is what the graphical application calls -- but nothing outside the GUI could reach it. A pipeline that produces an aligned result therefore had to stop there and hand the project to a person. MSDIALCUI normalize reads a saved project and its alignment, builds the standard table, calls the same function, and exports the matrix. The one thing the GUI can take for granted and a command line cannot is which aligned peak each standard actually is. StandardCompound identifies it by PeakID, which is an alignment ID: it belongs to one alignment and to no other, so a table written for one run points at unrelated peaks in the next, and does so silently, normalizing whole lipid classes against whatever happened to land on that ID. So a standard may name itself instead and be resolved against the annotations of the alignment being normalized, and a table that does carry an ID is checked against the annotation on it and says so when the two disagree. A standard that resolves to nothing stops the run, because the alternative is an unnormalized lipid class that looks normalized. MS-DIAL reports a lipid at two resolutions separated by a bar, so a standard table may name either one and both are matched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two faults that only a real run could show, both of the same kind: something that looked like it worked and quietly did not. The data storage is half a project. Loading it straight from MessagePack leaves the annotation databases null, and that surfaces much later as a null reference inside the match evaluator, pointing nowhere near the cause. The project is now loaded the way the application loads it, through MsdialIntegrateSerializer over a directory stream manager, so the databases and the mapper come with it. Then the export asked for "Height", which is the raw peak height. Normalizing does not touch that field -- it writes to a separate one -- so the exported matrix was identical to the input, down to the byte, while every log line said the normalization had succeeded. It asks for "Normalized height" now. Also: the parameter file's thread count was written but never read back, so a method file could describe a thread count it could never request and every Console run stayed on the default of two. Verified against the reference lipidomics dataset, seven plasma samples in both polarities: 13 of 13 internal standards in positive mode and 10 of 11 in negative normalize to exactly the concentration the lookup table gives them, flat across all seven samples, which is the invariant the graphical application's output shows. The eleventh is FA 16:0(d3), which that table assigns to no class of its own and which is therefore normalized as an ordinary fatty acid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An independent audit found that --allow-unresolved-standards did not do what its own message said. It promised the affected lipid classes "would be left unnormalized"; in fact SplashNormalize falls through to the "Any others" standard, so a class whose own standard was missing was quantified against a compound of an entirely different one -- a cardiolipin divided by a lysophosphatidylcholine -- and written out in the same unit, with the same comment, as a properly quantified row. Nothing in the file told them apart. On the audited dataset thirty-seven standard-to-class assignments were unresolved and eleven rows carried such numbers. None reached the merged lipidome, but only because the laboratory's rule table happens to quantify every affected class in the other polarity. That is the rule table's doing, not a check. Those rows now keep their identity and lose their numbers, and say why in place of them: "NOT QUANTIFIED: the CL standard PG 15:0_18:1(d7) did not resolve in this alignment". The raw height stays in the Height matrix beside it, so nothing is hidden -- only the invalid concentration is withheld. An annotated wrong number is still read by the next script; an empty cell is not. The message says this now. Two more from the same audit. A standard whose alignment ID holds a different compound used to warn and carry on, exit 0, with sphingomyelin quantified against a triacylglycerol peak. The posture was inverted: a standard that cannot be found stopped the run, while one demonstrably pointing at the wrong compound did not. It stops too, unless --allow-mismatched-peak-ids says the annotation is wrong rather than the table. Passing the .mdproject instead of the .mddata produced fifteen frames of MessagePack internals. It now names the file that cannot be read and the one to pass instead. The resolution report repeated a standard's own resolution once per class it covered -- a hundred and sixty lines, with the three that needed a decision somewhere in the middle. Each standard is reported once with the classes it covers, and the ambiguous and unresolved ones come last. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The exporter writes the annotator identifier into every row's Comment column as "Annotation method: ...". The LC-MS and LC-IMMS Console paths passed the library's file path as that identifier, so every exported matrix carried an absolute local directory -- into an artifact whose whole purpose is to be shared, and for a library that may be a private one. The DIMS and IMMS paths already passed a name, so the two halves of the same program disagreed. The identifier is now the library's file stem, which is what a reader needs: a laboratory library is date-stamped, and knowing which one annotated a row is the point. What pins it exactly is the checksum in the run manifest, not a path repeated in every cell. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parameter file offers a family of matrix-export flags and is portable into the GUI, where each means what it says. The Console read exactly one of them, IsHeightMatrixExport, and used it to gate an unrelated artifact: a run declaring "Height matrix export: True" produced a long-format quality-assurance table and no height matrix, and said nothing about either. The wide matrices the lipidomics workflow needs existed only as a side effect of the normalize verb. Height, normalized height, area, retention time, mass and signal-to-noise are now each written when their own flag is set. The quality-assurance matrix still follows the height request, because no parameter names it; it is written beside the height matrix rather than instead of it, and both are announced. Verified on one SCIEX file: with Height matrix export: True the run writes AlignResult-*_Height.txt (156 rows) and AlignResult-*.qa.tsv, exit 0, and no drive path appears anywhere in the exported matrix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The same setting is written down twice with two different values: the method file's annotation block and the annotator settings table. Reading the code settles it -- a table row starts from the method block and overrides, column by column, whatever it supplies -- but neither file says so, and a reader of the retained artifacts has no way to tell which number governed the annotations. Each MSP and text annotator now prints its resolved RT, MS1, MS2 tolerances and total score cutoff as it is built, so the run log carries the answer. Verified on one file with a text library whose table says RT tolerance 0.5 and cutoff 0.8 against a method block saying 0.1 and 0.85: the log reads 0.5 and 0.8, exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Normalization.SplashNormalizehas been in MsdialCore all along — it is what the graphical application calls — but nothing outside the GUI could reach it. A pipeline that produces an aligned result had to stop there and hand the project to a person.It reads the saved project and its alignment, builds the standard table, calls the same function, and exports the matrix.
Which peak is the standard
The one thing the GUI can take for granted and a command line cannot.
StandardCompoundidentifies its peak byPeakID, which is an alignment ID: it belongs to one alignment and to no other. A table written for one run points at unrelated peaks in the next — and does so silently, normalizing whole lipid classes against whatever happened to land on that ID.So a standard may name itself and be resolved against the annotations of the alignment actually being normalized:
A table that does carry an ID is checked against the annotation on it and says so when the two disagree. A standard that resolves to nothing stops the run, because the alternative is an unnormalized lipid class that looks normalized;
--allow-unresolved-standardscontinues and names what was dropped.MS-DIAL reports a lipid at two resolutions separated by a bar (
PC 33:1(d7)|PC 15:0_18:1(d7)), so a standard table may name either one and both are matched.Standards table
Tab- or comma-separated, columns matched by name:
StandardName,TargetClass,Concentrationrequired;PeakID,DilutionRate,MolecularWeightoptional.TargetClassofAny otherscovers every class the table does not name — the same convention the GUI uses.