The implementation of relative threshold validation doesn't align with the documented/expected behaviour.
From the docs:
Aboslute change is calculated as the value diference between the metrics of the candidate and baseline model, namely, v_c - v_b where v_c denotes the candidate metric value and v_b denotes the baseline value. Relative value is the relative difference between the metric of the candidate and the baseline, namely, v_c/v_b.
However, the implementation instead calculates (v_c - v_b) / v_b, NOT v_c/v_b as the docs very clearly state.
|
ratio = diff / metric_baseline |
The easiest fix would be to update the docs to correctly describe the implementation, but I'm not sure what the desired behaviour actually is.
The implementation of relative threshold validation doesn't align with the documented/expected behaviour.
From the docs:
However, the implementation instead calculates
(v_c - v_b) / v_b, NOTv_c/v_bas the docs very clearly state.model-analysis/tensorflow_model_analysis/evaluators/metrics_validator.py
Line 88 in 422dbd3
The easiest fix would be to update the docs to correctly describe the implementation, but I'm not sure what the desired behaviour actually is.