What happens
A query that binds a variable and returns scannum reports the same scan many times over —
once per candidate mass that matched it — so the reported match count is a large multiple of
the number of scans that actually matched.
Measured on a 48-spectrum Thermo file with 14 MS1 scans:
QUERY scannum(MS1DATA) WHERE MS1MZ=X
returns 2,783 rows over those 14 scans. The distinct set of scan numbers is correct; it
is the multiplicity that is wrong.
Why
A variable query is executed once per candidate mass and the per-candidate results are
concatenated (msql_engine.py, the comment column is set to the candidate's m/z at
line ~342). The de-duplication that follows is guarded on that column:
if "comment" in collated_df:
collated_df["truncated_comment"] = collated_df["comment"].astype(float).astype(int)
collated_df = collated_df.drop_duplicates(subset=["scan", "truncated_comment"])
Two things combine:
- the subset is
["scan", "truncated_comment"], so it de-duplicates per (scan, candidate)
rather than per scan — by design, since a scaninfo result wants one row per candidate; and
- the
scannum result frame is rebuilt from scratch and carries only a scan column
(msql_engine.py:606-614), so "comment" in collated_df is False and no de-duplication
runs at all.
So for scannum specifically, every candidate that matched a scan contributes another copy
of that scan.
Suggested fix
scannum's contract is a list of scan numbers, so de-duplicating it unconditionally on
scan after collation would match what the function already does within a single execution
(list(set(...)) at line 610/612).
Note
scaninfo is not affected in the same way — its frame does carry comment.
What happens
A query that binds a variable and returns
scannumreports the same scan many times over —once per candidate mass that matched it — so the reported match count is a large multiple of
the number of scans that actually matched.
Measured on a 48-spectrum Thermo file with 14 MS1 scans:
returns 2,783 rows over those 14 scans. The distinct set of scan numbers is correct; it
is the multiplicity that is wrong.
Why
A variable query is executed once per candidate mass and the per-candidate results are
concatenated (
msql_engine.py, thecommentcolumn is set to the candidate's m/z atline ~342). The de-duplication that follows is guarded on that column:
Two things combine:
["scan", "truncated_comment"], so it de-duplicates per (scan, candidate)rather than per scan — by design, since a
scaninforesult wants one row per candidate; andscannumresult frame is rebuilt from scratch and carries only ascancolumn(
msql_engine.py:606-614), so"comment" in collated_dfis False and no de-duplicationruns at all.
So for
scannumspecifically, every candidate that matched a scan contributes another copyof that scan.
Suggested fix
scannum's contract is a list of scan numbers, so de-duplicating it unconditionally onscanafter collation would match what the function already does within a single execution(
list(set(...))at line 610/612).Note
scaninfois not affected in the same way — its frame does carrycomment.