[TejusK123]

B2: 5' homogeneity, the unbiased Simpson index of the mature-arm 5' starts

The bug (from the review). 5' homogeneity divided the reads that start where the most abundant read starts by every read that starts no more than 20 nt after it. When the mature is on the 3' arm, the star and loop reads start upstream, so they entered the denominator and pulled the value down. That is the case for 1,585 of the 2,907 training loci. The manuscript's description ("top n permutations") matched neither the old code nor the new.

The new definition (Tejus, Q2, 18 Oct 2026). The unbiased Simpson index of the 5' start positions of the reads that miRDeep2 assigns to the mature arm, the same reads as result.csv's mature_read_count: sum n_i(n_i - 1) / (N(N - 1)). It is the probability that two mature-arm reads start at the same nucleotide.

• Reads are grouped by start position whatever their first base (reasoning in METHODS_CHANGES.md).
• With fewer than 2 mature-arm reads, the value is blank.
• num_locis is unchanged. The column name stays the same until B5, because the model selects features by name.

Code

• mrd.py: reads_by_arm() assigns each read to an arm by miRDeep2's rule and is now shared by arm_counts() and five_prime_features(). five_prime_features() takes the whole locus, because it needs the exp line.
• folding_properties.py: passes the locus.

Tests (written first; all failed before the fix)

• tests/test_five_prime_pipeline.py: the acceptance test. Adding star and loop reads to a synthetic 3'-arm miRNA leaves homogeneity unchanged; before the fix, it dropped from 0.90 to 0.70.
• tests/test_mrd.py: the index worked out by hand; reads outside miRDeep2's mature window don't count; a mismatched first base counts at its position; blank with fewer than 2 mature-arm reads.
• The Pomonella and FAW tests now compare homogeneity with output.mrd rather than the training CSVs, which keep the old values until B8. num_locis must still equal the training rows.
• The tutorial's expected xgboost_judgement.csv was regenerated with the pipeline. Its homogeneity goes from 0.988 to 0.981 and the label stays Confident.

Checks

• reanalysis.py on training_data_pre/: identical to review/reanalysis_osx-arm64.txt. The training data are not rebuilt until B8; the reference differs from reanalysis_baseline.txt by up to 0.01 in the XGBoost rows, as before.
• reanalysis.py on a scratch copy with only homogeneity recomputed: XGBoost accuracy is 0.864 in random splits (unchanged) and 0.818 leave-one-species-out (0.812 before). That is within noise.
• Shipped model, not retrained, with only this feature changed: 88 of 2,907 training labels change (47 Candidate to falsepositive, 25 the other way; Pomonella 8 of 212). The Simpson index is roughly the square of the old share, so the model's thresholds fall in different places until D1 retrains it.
• review/homogeneity_measures.py (new) compares the candidate measures on the training loci; its output is review/homogeneity_measures.txt. METHODS_CHANGES.md (B2) cites these numbers: old and new Methods text, and the reasons for the choice.

Results. Locally, with the testing folder: 139 passed. CI: see the checks below.

🤖 Generated with Claude Code