[TejusK123]

B3: signed 3' overhangs at both duplex ends, from output.mrd (duplex rule)

Implements B3 in review/REVISIONS.md, with Tejus's answer to Q3 (18 Oct 2026, written under Q3).

What changes

Before: folding_properties.py computed "3' Overhang Bot" as abs(right - left), the unsigned difference between the unpaired 5' and 3' ends of AutoDeep's own RNAfold structure of the precursor, and "3' Overhang Top" with a padding heuristic. Both were computed even for loci without star reads.

Now: both come from miRDeep2's output.mrd (mrd.overhangs), by the duplex rule:

• Bot is the Drosha end (base of the stem) and Top the Dicer end (loop side). The column names stay until B5 renames them.
• The mature is miRDeep2's M region and the star is the observed star (the obs line, the most abundant star read), which is the star result.csv reports. The structure is pri_struct, the one miRDeep2 used. Nothing is refolded (Tejus: miRDeep2's ViennaRNA is current).
• Only base pairs between the mature and the star count, because Drosha and Dicer cut off the flanks and the loop. At the Drosha end the outermost of these pairs is used, at the Dicer end the innermost. The overhang is the number of unpaired nucleotides beyond that pair on the strand whose 3' end is there, minus the number on the strand whose 5' end is there.
• Signed: +2 is canonical, 0 blunt, negative a 5' overhang.
• Blank when the locus has no star reads (miRDeep2's star is then its own prediction, 2/2 by construction) or when the mature and star don't pair. XGBoost reads blanks as missing values.

Why not miRDeep2's expected-star rule. The first version of this PR used miRDeep2's rule (find_star: 2 plus the observed star's offset from the star miRDeep2 expects). Tejus chose the duplex rule instead. The reasons, numbers and caveats for the paper are in review/METHODS_CHANGES.md (B3, "For the paper"). In short:

• miRDeep2's rule anchors on fixed mature positions, which can pair with the loop.
• It changes when the mature call flips to the other arm.
• It is harder to state in Methods without sounding circular.

The two rules differ on 10 of 148 Pomonella loci with star reads; those loci are listed in METHODS_CHANGES.md.

Effect

C. pomonella run (212 loci; AutoDeep infer on a copy, compared with the pre-B3 run of the same data):

• All other features are identical, and the columns are in the same order.
• The 64 loci without star reads now have blank overhangs.
• For the 148 loci with star reads: Drosha end -3 to 8 (86 at +2), Dicer end -2 to 7 (102 at +2).
• With the shipped model, 29 of 212 labels change: 28 Candidate → falsepositive and 1 Confident → falsepositive. All 29 are loci without star reads, so this is the shipped model's response to blank values it never saw in training. No locus with star reads changes label (the same result as with miRDeep2's rule). This goes away when B8 rebuilds the training data and D1 retrains the model. Until then, predictions on refactoring are worse for loci without star reads.

miRDeep2 tutorial example: its one novel locus has star reads and stays 2/2, so examples/mirdeep2_tutorial/expected/xgboost_judgement.csv is unchanged.

Reanalysis (review/reanalysis.py on training_data_pre/ and the shipped model): unchanged, identical to review/reanalysis_osx-arm64.txt. That is expected: the training CSVs are not rebuilt until B8.

Training data: training_data_pre/ keeps the old values until B8. Until then, AutoDeep train -t on new runs mixes the old and new definitions.

Tests

These were written first and failed on the earlier code:

• tests/test_overhangs_pipeline.py: runs folding_properties.py and inference on two synthetic loci with a known structure. The pre-B3 code wrote 1, 1 for a star that leaves 1-nt 5' overhangs (correct: -1, -1) and 2, 2 for a locus without star reads (correct: blank). Inference accepts the blanks.
• tests/test_mrd_overhangs.py, duplex rule. These three failed on the miRDeep2-rule version:
  • a mature whose 5' end pairs with the loop (as in QFTL02000135.1_4908): miRDeep2's rule gave -1, the duplex rule 3;
  • the same duplex with the mature on either arm gives the same values (miRDeep2's rule gave (2, -1) and (2, 3));
  • a bulge next to the end (as in QFTL02000007.1_25533): miRDeep2's rule gave 2, the duplex rule -3.

The rest of tests/test_mrd_overhangs.py:

• hand-written hairpins: +2/+2 with the mature on either arm, +3, +1, blunt, -1, mixed, unpaired end nucleotides, no star reads, no mature-star pair, missing structure, and a frayed end (documented limit: it reads as 0);
• ten MirGeneDB 3.0 human precursors chosen in advance (tests/data/mirgenedb_hsa_ten.csv, ViennaRNA 2.7.0 folds). Seven give 2 at both ends. Let-7-P2a1 and Mir-145 give 1 at the Drosha end and Mir-21 gives 3 at the Dicer end, as their folds show. The values are the same under both rules;
• on the tutorial run (CI) and on Pomonella (local): the observed star equals result.csv's consensus star, and the overhangs are blank exactly when the star read count is 0.

Other test files:

• tests/test_mrd.py: parse_mrd keeps obs, pri_seq and pri_struct.
• tests/test_mrd_parity.py (Pomonella, local): the other features still equal the training data; the overhangs the pipeline writes equal mrd.overhangs; the predictions equal the shipped model's on the training features with the new overhangs.

Local run with AUTODEEP_TEST_RUN set to the Pomonella run: 130 passed.

For Tejus

• review/METHODS_CHANGES.md: paste the old sentences that define the overhangs (Methods and Table 1). The new text and the "For the paper" notes are written.
• tests/data/mirgenedb_hsa_ten.csv holds ten MirGeneDB sequences and their arm annotations. MirGeneDB's site states no licence; please confirm they can be in a GPL-3.0 repository.

🤖 Generated with Claude Code