Skip to content

ENH,MAINT: use a single random number generator - #157

Open
sdtemple wants to merge 1 commit into
lanl:mainfrom
sdtemple:sdtemple_issue_126_inconsistent_rng
Open

sdtemple wants to merge 1 commit into
lanl:mainfrom
sdtemple:sdtemple_issue_126_inconsistent_rng

Conversation

@sdtemple

@sdtemple sdtemple commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #126

Related to #153 (prettier code for _initialize)

The best practice for estimators like ours is to use 1 RNG instance and pass it to each helper. This PR resolves an issue where two RNGs were used (1 for the first layer of weights and 1 for all other weights thereafter). This avoids re-seeding overhead and guarantees every draw comes from one continuous, non-overlapping stream.

The previous hack existed to match results from https://github.com/thieu1995/GrafoRVFL in toy datasets. That repository of 1-layer RVFLs created two identical RNGs (1 for biases and 1 for weights). This is why the hack worked for our 1-layer implementation. That is not the best practice for using RNGs for such an RVFL estimator.

In general, we don't want to write estimator code simply to adhere to a test. However, we like the GrafoRVFL method as a regression benchmark from the literature. As a result, I adjusted GrafoRVFL on my fork to determine how to adjust the expected values in the GrafoRVFL-based regressor and classifier tests.

See the instructions.md at https://github.com/sdtemple/GrafoRVFL (commit hash aa56f25602ea2c520b569ed23fb90357f67b80d0)

I had to update some other expected values as well in regression tests, since the initialized weights changed with the move to 1 RNG object.

The test suite passed on my local setup (Mac OS) and a remote Linux box.

SI disclosure: I did not use SI for this commit.

* SI disclosure: I did not use SI for this commit

* Converted to use only 1 RNG

* We create 2 RNGs in .fit and .partial_fit which
	results in a lot of redundancy in code.
	The best practice would be to repeatedly
	use 1 RNG.

* We used 2 RNGs to match to GrafoRVFL. See
	github.com/sdtemple/GrafoRVFL where I
	demonstrate if I hack at GrafoRVFL so
	that their code only uses 1 RNG, then
	I get results consistent with this edit

* Updated expected values in regression tests.
	The edits are overall very small.
	I adjusted the size of a model in
	a Boston housing regression because
	the huge model created ill-conditioned
	matrices. This resulted in a big shift
	in the expected values with the slight
	change to how we do RNG.

* Refer to github.com/sdtemple/GrafoRVFL.
	This fork shows how we updated the values.
	The GrafoRVFL results were made by
	running their package on toy data.
@sdtemple sdtemple self-assigned this Nov 20, 2026
@sdtemple sdtemple added enhancement New feature or request maintenance labels Nov 20, 2026
@sdtemple

sdtemple commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

@eiviani-lanl @aganguly-lab It would be good if one of you could vet my fork https://github.com/sdtemple/GrafoRVFL where I made edits to get them to use a single RNG object. That's where the updated expected values in our regression tests come from.

The edits to model.py are very minimal. The majority of this PR is updating GrafoRVFL reference values. I know in #126 Emma even suggested removing GrafoRVFL, but, here, I favored modifying the GrafoRVFL library so that we can still use its updated reference values.

@sdtemple

Copy link
Copy Markdown
Collaborator Author

In Emma's sklearn PR she has an even simpler for loop that doesn't require manually making the first layer of weights and biases. Everything is in the for loop. That would be available to use now with 1 RNG.

https://github.com/eiviani-lanl/scikit-learn/blob/f69f04f2c447d14b08fdb6b452312a6ea69348a8/sklearn/neural_network/_extreme_learning_machine.py#L96

@tylerjereddy tylerjereddy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a quick manual review without consulting an LLM yet, adding a few review comments.

The majority of this PR is updating GrafoRVFL reference values. I know in #126 Emma even suggested removing GrafoRVFL, but, here, I favored modifying the GrafoRVFL library so that we can still use its updated reference values.

I'm not sure I follow the need for this and vetting your grafo branch. We haven't imported and executed grafo in this library for quite some time now, so as long as we still have the same ballpark performance after changing the seed infrastructure, I don't think we need to go through the trouble of modifying grafo. If you see large swings in performance maybe there's a concern, but small changes to expected values are "ok" now that we're not relying directly on matches to grafo.

activation="tanh",
weight_scheme="uniform",
seed=0,
seed=1,

@tylerjereddy tylerjereddy Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change caught my attention--the test fails with the original seed of 0 now, which I don't really like. It looks like the properties enforced below shouldn't really need to be specific to a given seed.

If the test properties enforced are seed sensitive, a brief explanation of why in the review would be helpful to reviewers. A concise code comment may make sense.

Alternatively, we may need to modify the test a bit so that it is not so sensitive to any given seed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll set it back to seed=0 to shrink the diff. Then, all that would be edited is the updates to the expected values.

The test properties aren't that seed sensitive? There is statistical variation due to seed that is natural and one could demonstrate for #157 or main.

See the following comments, especially the one with plots.

#157 (comment)
#157 (comment)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll set it back to seed=0 to shrink the diff.

I don't know that it shrinks the diff in this case.

The test properties aren't that seed sensitive?

test_soft_and_hard() does fail without the seed change here. There's no way to change the expected value here, if you keep the seed the same, without weakening the test from the perspective of a naive reviewer.

You could do an analysis of behavior over random seeds and link to that alongside a comment above the seed change though.

(None, 1e-15, 0.10833333333333334, 0.5062846049300238),
(None, 1e-3, 0.9555555555555556, 0.9920190654177233),
(5.0, 1e-15, 0.7472222222222222, 0.9544467203797617),
(None, 1e-15, 0.12777777777777777, 0.4983740449299509),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ROC AUC here drops below random chance. That's probably ok, I've seen it many times, but not great in general.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that drop in AUROC is well within expected statistical noise. Actually, all of those values are. I made versions of the first 2 cases with 100 random seeds.

Strip and box plots for `reg_alpha=5.` and ACC and AUROC variance_ensemble_ridge_acc_stripplot variance_ensemble_ridge_auroc_stripplot variance_ensemble_ridge_acc_boxplot variance_ensemble_ridge_auroc_boxplot
Strip and box plots for `rtol=1e-15` and ACC and AUROC variance_ensemble_pseudo_acc_stripplot variance_ensemble_pseudo_auroc_stripplot variance_ensemble_pseudo_acc_boxplot variance_ensemble_pseudo_auroc_boxplot

(GFDLClassifier, None, None, 0.7631578947368421),
(GFDLClassifier, None, 1e-3, 0.9473684210526315),
(GFDLClassifier, None, None, 0.7105263157894737),
(GFDLClassifier, None, 1e-3, 0.956140350877193),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you updated comments below but not above here

hidden_layer_sizes=[64] * 4,
activation="tanh",
weight_scheme="uniform",
weight_scheme="lecun_normal",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this set of changes--it shouldn't be needed when just changing seeding infrastructure. Needs self-review comment explaining why this change is better than keeping the original params.

Why is the performance so much worse now with the original params? (I checked locally, and they are a fair bit worse).

Basically, need clear explanation so reviewer doesn't wonder.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may have hidden some comments about that in your commit message. You should put such explanations in self-review and/or in the PR description proper.

I still don't like the need for the change because of conditioning. If we can't avoid it fine, but before merging I'd probably do a check for even larger network sizes to see if I can find a scenario where a large network still does well. In other words, is this just unstable in general, and it isn't as simple as just needing a smaller network, but rather there's a random component to it that might need a code comment.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can set these all back and only update the expected R2 scores. I guess I just worried how it would look R2 values dropping for that specific seed.

I ran quick analyses fitting the Boston housing dataset with hidden_layer_sizes=[800] * 10 and uniform for 100 random seeds (range(100)).

The drop into the 60% for seed=0 is just an unfortunate seed for the RNG edit. The statistical noise and other metrics (like median) are ostensibly unchanged between main and this PR fixing gh126.

Strip plot variance_rng_stripplot
Box plot variance_main_gh126

@sdtemple sdtemple Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll set the weight scheme back to uniform. It'll make the diff smaller.

I guess in general I don't like this one because in some of our experiments it is the worst, has low extreme values. Moreover, the weight schemes with variance scaled by network size, like lecun variants, appear to be better on average and more stable. We demonstrated this here:

https://lisdi-git.lanl.gov/gfdl_research/research_demo/-/work_items/43#note_172673

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'd prefer to avoid changing the network size or any other parameter apart from the seed then.

Perhaps changing the seed to avoid an unlucky one if necessary seems fine, along with a comment that points to your analysis and concisely explains that seed choice can substantially impact outcome.

As noted at #157 (comment), there may be some value in having some tests check median/mean performance across seeds, but that seems like a separate matter, and hardcoding results for a given seed is still useful.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in general I don't like this one because in some of our experiments it is the worst, has low extreme values.

I don't think that matters much here--we're just looking for correct behavior with a given set of params, and changing those params in a regression test just causes more noise for the reviewer in this case. A code comment linking to the analyses seems fine, along with a seed change if needed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open an issue on regression tests that are based on statistics (median/mean). I bet that'll sit for a bit, though.

@tylerjereddy tylerjereddy added this to the 0.3.0 milestone Nov 20, 2026
@sdtemple

Copy link
Copy Markdown
Collaborator Author

Per #157 (comment)

I'll share some quick plots demonstrating completely natural statistical variation due to seed for this PR and for main. I'll do this because it is expected that the regression values will change by multiple percentage points, both in this PR and main.

The key point to look out for in the plots is that the ostensible statistical behavior, like the range, median, IQR, etc. are the same between the RNG approaches in main versus this PR.

@tylerjereddy

Copy link
Copy Markdown
Collaborator

The key point to look out for in the plots is that the ostensible statistical behavior, like the range, median, IQR, etc. are the same between the RNG approaches in main versus this PR.

Fair enough, then just communicate it clearly in the PR. So if you change a seed, you can put the analysis in a PR comment and then a comment linking to that analysis indicating that the seed choice was selected specifically to get some accuracy value and the range varies are described at the link.

One could argue that we should switch tests of this nature to also look at median/mean performance across many seeds, but that seems like an orthogonal issue to this PR, and there is value in hardcoding the result for a specific seed as well. Perhaps a follow-up issue for adding mean or median performance across a few seeds in relevant tests, but doesn't seem critical for now and could slow testsuite down a fair bit.

@tylerjereddy

Copy link
Copy Markdown
Collaborator

I'll do this because it is expected that the regression values will change by multiple percentage points, both in this PR and main.

That's another reason to avoid changing other params though--if the seed selection causes wide variation in outcome, we don't also want to change other params on top of that. A simple code comment alongside a seed change that links to the expected variation seems fine, but changing other hyperparams can make it more confusing for the reviewer.

@sdtemple

Copy link
Copy Markdown
Collaborator Author

I'll add some code comments and documentation in the next commit, as discussed above.

For posterity, below will be some results about variation we see across seeds between the main right now with 2 RNG objects and the PR here with 1 RNG object.

@sdtemple

sdtemple commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

For the test_regression_against_grafo, I fixed the following and studied variation over 100 seeds.

    N, d = 10000, 10
    RNG = 42
    X, y = make_regression(n_samples=N,
                           n_features=d,
                           n_informative=d,
                           n_targets=1,
                           random_state=RNG)

and

    model = GFDLRegressor(
            hidden_layer_sizes=(100,),
            activation="relu",
            weight_scheme="lecun_normal",
            direct_links=1,
            seed=seed,
            reg_alpha=1.
        )
Strip and box plots for the minimum values variance_grafo_regr_min_stripplot variance_grafo_regr_min_boxplot
Strip and box plots for the median values variance_graf_regr_median_stripplot variance_graf_regr_median_boxplot

The R2 scores aren't that ostensibly relevant. The toy make_regression dataset is so simple that they're all at like > 0.9999.

@sdtemple

Copy link
Copy Markdown
Collaborator Author

For the test_classifier_against_grafo, I fixed the following and studied variation over 100 seeds.

    X, y = make_classification(n_classes=2,
                               n_informative=8, random_state=0)

and

    model = GFDLClassifier(hidden_layer_sizes=(100,),
                 activation="tanh",
                 weight_scheme="lecun_normal",
                 direct_links=1,
                 seed=seed,
                 reg_alpha=1.)
Strip and box plots for the minimum values variance_grafo_clf_min_stripplot variance_grafo_clf_min_boxplot
Strip and box plots for the median values variance_grafo_clf_median_stripplot variance_grafo_clf_median_boxplot

@sdtemple

Copy link
Copy Markdown
Collaborator Author

I don't have Shi 2021 results because I seem unable to fetch the data with my network access today.

@sdtemple

sdtemple commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

The median results for the Grafo classifier experiments are pretty well calibrated with n_classes=2. They're almost always 0.5 or .499999999999999.

So, here is a plot of variation in the predict_proba medians when n_classes=5.

Strip and box plots for `predict_proba` when `n_classes=5` variance_grafo_clf_median_v2_stripplot variance_grafo_clf_median_v2_boxplot

@sdtemple

Copy link
Copy Markdown
Collaborator Author

I'm not sure I follow the need for this and vetting your grafo branch. We haven't imported and executed grafo in this library for quite some time now, so as long as we still have the same ballpark performance after changing the seed infrastructure, I don't think we need to go through the trouble of modifying grafo.

Yes, I am aware that we haven't used the graforvfl dependency in awhile. #9

In terms of ballpark accuracy:

  • The atol values for some of the GFDLRegressor tests would have to be very large to capture changes that are well within statistical noise. This can be demonstrated for 1 example here ENH,MAINT: use a single random number generator #157 (comment).
  • Claude Sonnet said as much that increasing atol dramatically to meet such would render the test meaningless.
  • Current code uses default atol and rtol

The point of my graforvfl fork is to explain how I updated the expected values. I didn't just look at the ACTUAL versus DESIRED for failing tests and updated DESIRED based on GFDL. I made the hack to graforvfl that enabled a single RNG object -- with the same seed as us -- to determine the DESIRED values from graforvfl.

@tylerjereddy

Copy link
Copy Markdown
Collaborator

The point of my graforvfl fork is to explain how I updated the expected values.

I just mean that it would be totally fine to just change the expected to the new actual without hacking grafo at all, as long as the deviations are reasonable. Agreed that you shouldn't mess with tolerances here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MAINT: inconsistent use of rng in classifier objects

2 participants