[TejusK123]

D1: final fit on all rows, early stopping, search space

The bugs (from the review)

• The saved model wasn't the one that was scored. After tuning, the best configuration was refit on the first, unstratified 80% split, without the balanced class weights used to score it. The -hp path saved a model fit on a stratified 80% split.
• max_depth could only be 1–5 (tune.randint(1, 6)). The shipped model sits at 5, and the paper's "between 4 and 6" could not happen.
• base_score was tuned although it has no effect on a softmax model.
• n_estimators was tuned blindly (100–1000) alongside learning rates as low as 10⁻⁴.
• The ASHA scheduler never stopped a trial, because each trial reports once.

The fix (boosted_forest_training.py)

• score_config() scores tuned and -hp hyperparameters alike, on the same ten stratified 80/20 splits with balanced weights.
• Early stopping: unless n_estimators is given, a stratified fifth of each 80% is held out. Boosting stops when its weighted log-loss hasn't improved for 50 rounds, up to 3,000 trees. The mean over the ten splits becomes n_estimators.
• Final model: fit_final() refits the chosen configuration on all rows with balanced weights. That's the saved model.
• Search space: max_depth 2–8, eta 0.005–0.3 (log-uniform). No base_score, no n_estimators, no scheduler.
• Records: config.json records the number of trees and how the model was scored and fit. metrics.json holds the ten-split mean and SD for both paths.

Decided with Tejus (19 Oct 2026):

• Retraining the shipped model is split off as D1b, after B8. Retraining now would fit it to the old 5' homogeneity and overhang values.
• alt_model_test.py has the same final-fit pattern; it's left for C1.

Tests (tests/test_final_fit.py, written first; all six failed before the fix)

• For both the tuning and -hp paths, the saved model's trees equal a refit on all rows with balanced weights.
• Early stopping chooses the number of trees, and the saved model has that many.
• -hp with n_estimators uses it as given; without it, early stopping chooses.
• The search space.

The existing seed, output and hyperparameter tests still pass.

Check (bundled data, 10 tuning rounds, seed 42, Apple M4)

	Before	After
Best weighted F1 (ten splits)	0.875 ± 0.013	0.872 ± 0.015
Best settings	depth 5, eta 0.028, 874 trees	depth 8, eta 0.006, 1,186 trees
Range across the 10 trials	0.762–0.875	0.838–0.872
Time	47 s	66 s

• The best scores are within noise. The old search spent most trials on learning rates too small for their trees.
• With early stopping, each split trains on 64% of the data; the saved model uses all of it.
• The best depth sat at the new range's top, but depths 5 and 6 scored 0.870. This should be checked again in D1b's longer run.

Other checks

• reanalysis.py: identical to review/reanalysis_osx-arm64.txt. It doesn't use the training code.
• METHODS_CHANGES.md (D1): the old tuning text (including the CMA-ES claim; the sampler is TPE only), the new text, and this check.
• Help text and README updated.

Results. Locally, with the testing folder: 156 passed. CI: see the checks below.

🤖 Generated with Claude Code