[SC-18206] Fix tests broken by Plotly 7 and NumPy 2 upstream removals

[juanmleng]

Pull Request Description

What and why?

Plotly 7.0.0 (released 2026-08-25) removed create_annotated_heatmap and create_distplot. We allow plotly (>=6.0.0) with no upper bound, so any environment built since then gets Plotly 7 and five tests fail with AttributeError: ConfusionMatrix, RegressionResidualsPlot, TimeSeriesMissingValues, LaggedCorrelationHeatmap and TargetPredictionDistributionPlot. Older environments are unaffected, which is why this appeared with no change on our side.

The three heatmaps now use go.Heatmap with text/texttemplate. The two KDE plots call scipy.stats.gaussian_kde directly — the same thing create_distplot did internally, so the curves are identical to floating point. The plots look the same as before.

No version cap is added. The code runs on Plotly 6 and 7, and texttemplate has existed since Plotly 5.5, so a cap would restrict users without protecting them.

Two smaller things came along: RegressionResidualsPlot now raises MissingDependencyError for scipy like the other statistical tests instead of a bare ImportError, and GiniCoefficient used np.trapz, which NumPy 2 removed — that broke it on Python 3.14, where we resolve numpy>=2.3.

How to test

The bug is invisible on Plotly 6, so run the suite on both:

uv run python -m tests.test_unit_tests      # green
uv run pip install "plotly==7.0.0"
uv run python -m tests.test_unit_tests      # green here, 14 errors on main
uv run pip install "plotly==6.7.0"

Four new test files cover ConfusionMatrix, RegressionResidualsPlot, TargetPredictionDistributionPlot and GiniCoefficient — none had any test, which is why only one of the five failures was spotted originally. Every figure was also rendered to PNG before and after and compared.

What needs special review?

The confusion matrix orientation. create_annotated_heatmap and px.imshow disagree about which corner is the origin, so porting via px.imshow would silently transpose the matrix — TN and TP swapped, no error. This uses go.Heatmap, which keeps z as given, and the new tests assert each quadrant's label.

Also: the choice not to cap Plotly, and whether the NumPy fix belongs here — it is a different upstream and could be split out. It is included because it is one line and the same kind of failure.

Dependencies, breaking changes, and deployment notes

No dependency changes, no breaking changes. Until this ships, affected users can run pip install "plotly<7" and restart their kernel.

Release notes

Fixed five tests that failed with an AttributeError on Plotly 7.0.0 and later, which removed the helpers they used to draw heatmaps and distribution curves: ConfusionMatrix, RegressionResidualsPlot, TimeSeriesMissingValues, LaggedCorrelationHeatmap and TargetPredictionDistributionPlot. The plots are unchanged and now work on both Plotly 6 and 7, so no version pin is needed. Also fixed GiniCoefficient, which failed on Python 3.14 because of a renamed NumPy function.

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)