Repository navigation
Conversation
* SI disclosure: I did not use SI for this commit * Converted to use only 1 RNG * We create 2 RNGs in .fit and .partial_fit which results in a lot of redundancy in code. The best practice would be to repeatedly use 1 RNG. * We used 2 RNGs to match to GrafoRVFL. See github.com/sdtemple/GrafoRVFL where I demonstrate if I hack at GrafoRVFL so that their code only uses 1 RNG, then I get results consistent with this edit * Updated expected values in regression tests. The edits are overall very small. I adjusted the size of a model in a Boston housing regression because the huge model created ill-conditioned matrices. This resulted in a big shift in the expected values with the slight change to how we do RNG. * Refer to github.com/sdtemple/GrafoRVFL. This fork shows how we updated the values. The GrafoRVFL results were made by running their package on toy data.
|
@eiviani-lanl @aganguly-lab It would be good if one of you could vet my fork https://github.com/sdtemple/GrafoRVFL where I made edits to get them to use a single RNG object. That's where the updated expected values in our regression tests come from. The edits to |
|
In Emma's sklearn PR she has an even simpler for loop that doesn't require manually making the first layer of weights and biases. Everything is in the for loop. That would be available to use now with 1 RNG. |
tylerjereddy
left a comment
There was a problem hiding this comment.
I did a quick manual review without consulting an LLM yet, adding a few review comments.
The majority of this PR is updating GrafoRVFL reference values. I know in #126 Emma even suggested removing GrafoRVFL, but, here, I favored modifying the GrafoRVFL library so that we can still use its updated reference values.
I'm not sure I follow the need for this and vetting your grafo branch. We haven't imported and executed grafo in this library for quite some time now, so as long as we still have the same ballpark performance after changing the seed infrastructure, I don't think we need to go through the trouble of modifying grafo. If you see large swings in performance maybe there's a concern, but small changes to expected values are "ok" now that we're not relying directly on matches to grafo.
| activation="tanh", | ||
| weight_scheme="uniform", | ||
| seed=0, | ||
| seed=1, |
There was a problem hiding this comment.
This change caught my attention--the test fails with the original seed of 0 now, which I don't really like. It looks like the properties enforced below shouldn't really need to be specific to a given seed.
If the test properties enforced are seed sensitive, a brief explanation of why in the review would be helpful to reviewers. A concise code comment may make sense.
Alternatively, we may need to modify the test a bit so that it is not so sensitive to any given seed.
There was a problem hiding this comment.
I'll set it back to seed=0 to shrink the diff. Then, all that would be edited is the updates to the expected values.
The test properties aren't that seed sensitive? There is statistical variation due to seed that is natural and one could demonstrate for #157 or main.
See the following comments, especially the one with plots.
There was a problem hiding this comment.
I'll set it back to seed=0 to shrink the diff.
I don't know that it shrinks the diff in this case.
The test properties aren't that seed sensitive?
test_soft_and_hard() does fail without the seed change here. There's no way to change the expected value here, if you keep the seed the same, without weakening the test from the perspective of a naive reviewer.
You could do an analysis of behavior over random seeds and link to that alongside a comment above the seed change though.
| (None, 1e-15, 0.10833333333333334, 0.5062846049300238), | ||
| (None, 1e-3, 0.9555555555555556, 0.9920190654177233), | ||
| (5.0, 1e-15, 0.7472222222222222, 0.9544467203797617), | ||
| (None, 1e-15, 0.12777777777777777, 0.4983740449299509), |
There was a problem hiding this comment.
The ROC AUC here drops below random chance. That's probably ok, I've seen it many times, but not great in general.
There was a problem hiding this comment.
| (GFDLClassifier, None, None, 0.7631578947368421), | ||
| (GFDLClassifier, None, 1e-3, 0.9473684210526315), | ||
| (GFDLClassifier, None, None, 0.7105263157894737), | ||
| (GFDLClassifier, None, 1e-3, 0.956140350877193), |
There was a problem hiding this comment.
you updated comments below but not above here
| hidden_layer_sizes=[64] * 4, | ||
| activation="tanh", | ||
| weight_scheme="uniform", | ||
| weight_scheme="lecun_normal", |
There was a problem hiding this comment.
I don't like this set of changes--it shouldn't be needed when just changing seeding infrastructure. Needs self-review comment explaining why this change is better than keeping the original params.
Why is the performance so much worse now with the original params? (I checked locally, and they are a fair bit worse).
Basically, need clear explanation so reviewer doesn't wonder.
There was a problem hiding this comment.
You may have hidden some comments about that in your commit message. You should put such explanations in self-review and/or in the PR description proper.
I still don't like the need for the change because of conditioning. If we can't avoid it fine, but before merging I'd probably do a check for even larger network sizes to see if I can find a scenario where a large network still does well. In other words, is this just unstable in general, and it isn't as simple as just needing a smaller network, but rather there's a random component to it that might need a code comment.
There was a problem hiding this comment.
I can set these all back and only update the expected R2 scores. I guess I just worried how it would look R2 values dropping for that specific seed.
I ran quick analyses fitting the Boston housing dataset with hidden_layer_sizes=[800] * 10 and uniform for 100 random seeds (range(100)).
The drop into the 60% for seed=0 is just an unfortunate seed for the RNG edit. The statistical noise and other metrics (like median) are ostensibly unchanged between main and this PR fixing gh126.
There was a problem hiding this comment.
I'll set the weight scheme back to uniform. It'll make the diff smaller.
I guess in general I don't like this one because in some of our experiments it is the worst, has low extreme values. Moreover, the weight schemes with variance scaled by network size, like lecun variants, appear to be better on average and more stable. We demonstrated this here:
https://lisdi-git.lanl.gov/gfdl_research/research_demo/-/work_items/43#note_172673
There was a problem hiding this comment.
Yeah, I'd prefer to avoid changing the network size or any other parameter apart from the seed then.
Perhaps changing the seed to avoid an unlucky one if necessary seems fine, along with a comment that points to your analysis and concisely explains that seed choice can substantially impact outcome.
As noted at #157 (comment), there may be some value in having some tests check median/mean performance across seeds, but that seems like a separate matter, and hardcoding results for a given seed is still useful.
There was a problem hiding this comment.
I guess in general I don't like this one because in some of our experiments it is the worst, has low extreme values.
I don't think that matters much here--we're just looking for correct behavior with a given set of params, and changing those params in a regression test just causes more noise for the reviewer in this case. A code comment linking to the analyses seems fine, along with a seed change if needed.
There was a problem hiding this comment.
I'll open an issue on regression tests that are based on statistics (median/mean). I bet that'll sit for a bit, though.
|
Per #157 (comment) I'll share some quick plots demonstrating completely natural statistical variation due to seed for this PR and for The key point to look out for in the plots is that the ostensible statistical behavior, like the range, median, IQR, etc. are the same between the RNG approaches in |
Fair enough, then just communicate it clearly in the PR. So if you change a seed, you can put the analysis in a PR comment and then a comment linking to that analysis indicating that the seed choice was selected specifically to get some accuracy value and the range varies are described at the link. One could argue that we should switch tests of this nature to also look at median/mean performance across many seeds, but that seems like an orthogonal issue to this PR, and there is value in hardcoding the result for a specific seed as well. Perhaps a follow-up issue for adding mean or median performance across a few seeds in relevant tests, but doesn't seem critical for now and could slow testsuite down a fair bit. |
That's another reason to avoid changing other params though--if the seed selection causes wide variation in outcome, we don't also want to change other params on top of that. A simple code comment alongside a seed change that links to the expected variation seems fine, but changing other hyperparams can make it more confusing for the reviewer. |
|
I'll add some code comments and documentation in the next commit, as discussed above. For posterity, below will be some results about variation we see across seeds between the |
|
For the N, d = 10000, 10
RNG = 42
X, y = make_regression(n_samples=N,
n_features=d,
n_informative=d,
n_targets=1,
random_state=RNG)and model = GFDLRegressor(
hidden_layer_sizes=(100,),
activation="relu",
weight_scheme="lecun_normal",
direct_links=1,
seed=seed,
reg_alpha=1.
)The R2 scores aren't that ostensibly relevant. The toy |
|
For the X, y = make_classification(n_classes=2,
n_informative=8, random_state=0)and model = GFDLClassifier(hidden_layer_sizes=(100,),
activation="tanh",
weight_scheme="lecun_normal",
direct_links=1,
seed=seed,
reg_alpha=1.) |
|
I don't have Shi 2021 results because I seem unable to fetch the data with my network access today. |
Yes, I am aware that we haven't used the In terms of ballpark accuracy:
The point of my |
I just mean that it would be totally fine to just change the |




















Fixes #126
Related to #153 (prettier code for
_initialize)The best practice for estimators like ours is to use 1 RNG instance and pass it to each helper. This PR resolves an issue where two RNGs were used (1 for the first layer of weights and 1 for all other weights thereafter). This avoids re-seeding overhead and guarantees every draw comes from one continuous, non-overlapping stream.
The previous hack existed to match results from https://github.com/thieu1995/GrafoRVFL in toy datasets. That repository of 1-layer RVFLs created two identical RNGs (1 for biases and 1 for weights). This is why the hack worked for our 1-layer implementation. That is not the best practice for using RNGs for such an RVFL estimator.
In general, we don't want to write estimator code simply to adhere to a test. However, we like the GrafoRVFL method as a regression benchmark from the literature. As a result, I adjusted GrafoRVFL on my fork to determine how to adjust the expected values in the GrafoRVFL-based regressor and classifier tests.
See the
instructions.mdat https://github.com/sdtemple/GrafoRVFL (commit hash aa56f25602ea2c520b569ed23fb90357f67b80d0)I had to update some other expected values as well in regression tests, since the initialized weights changed with the move to 1 RNG object.
The test suite passed on my local setup (Mac OS) and a remote Linux box.
SI disclosure: I did not use SI for this commit.