Conversation
…-Software#436) This is a continuation and refinement of MHKiT-Software#435 Highlights: ## From MHKiT-Software#435 * Actions version updates: * checkout - v4 to v6 * setup-python - v2 to v6 * cdip fixes ## Dependencies ### Module Dependencies * Pin pandas below version 3.0 * There are some subtle bugs related to underlying pandas changes that we should address and possibly fix that are out of scope of this PR. * Refactor dependencies to move xarray, netcdf, and h5 dependencies into `all` * This helps standardize the versions for all modules, and behind the scenes xarray typically requires both netcdf and h5. ### Conda and Conda-Forge Dependency Definitions * Move conda forge environment file into environment-dev.yml and create new conda-forge-build job to test the conda-forge specific no deps build ## Actions ### Linting * Add filter to only run `black` on files that have changed * Add Development Installation section to readme detailing pip, conda, and conda forge installation pathways ### Coveralls * Pin coveralls action at v2 * Add a switch in main.yml to toggle reporting coveralls upload failures as test failures (recommended `fail-on-error: false` fix)
Updates to `io/d3d.py` code - Added new coordinate names for the latest version of Delft3D - new `calculate_grid_convergence_index` function to calculate the grid convergence index between two grid's results basted on the equation [GCI]( https://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html) - Allows xarray or netCDF4 input --------- Co-authored-by: akeeste <akeeste@sandia.gov> Co-authored-by: Simms, Andrew <andrew.simms@nlr.gov>
Opening this to solve Issue MHKiT-Software#427 I changed the source code to return degrees instead of radians, which was a simple fix. I also went ahead and updated the polar plots so that zero degrees is at the top of the figure, and positive degrees runs clockwise. We should update this in other polar plots in MHKiT too. I also updated units to pass CF conventions, whereafter I noticed that our tests for this code check that the units are the same as what NDBC outputs. I'm not sure that's necessary. I also saw that we're not actually testing the value output from the source code, which is important. --------- Co-authored-by: Simms, Andrew <andrew.simms@nlr.gov>
## Pandas 3 Changes * Updated pandas dependency to allow versions >=2.2.2 without an upper limit in both `environment-dev.yml` and `pyproject.toml`, enabling support for pandas 3.x. * Added a new function `replace_pandas_missing_values_with_nan` in `mhkit/wave/io/ndbc.py` to replace missing values with `NaN` in a way that is compatible with both pandas 2.x and 3.x. ## Test Updates * Increased `test_get_buoy_metadata` latitude and longitude assertions delta tolerance to allow for a small amount real world buoy drift. * Fix deprecation in `test_request_parse_workflow_multiyear` (`mhkit/tests/wave/io/test_cdip.py`) by using `"D"` instead of `"d"` in the `floor` method. Pandas deprecated `d` in 3.0+ pandas-dev/pandas#58998. period aliases are here: https://pandas.pydata.org/docs/user_guide/timeseries.html#period-aliases
This PR verifies MHKiT-Python supports Python 3.13. No python code changes were required. README was updated to include Python 3.13 Actions were updated to include 3.13 in test matrices and for notebook tests (single version) Action versions for setup-miniconda and download-artifact were update
Adding visualizations to match the [MHKiT_MATLAB ADCP example](https://github.com/MHKiT-Software/MHKiT-MATLAB/blob/master/examples/acoustics_example.mlx) and addressing several issues: Qhull Error MHKiT-Software#444 D3D coordinate systems MHKiT-Software#442
Enhancements: - Adds ability to convert sound spectral densities to millidecade format (https://doi.org/10.1121/10.0003324) - Adds reader for WISPR (Wideband Intelligent Signal Processing and Recording) system for hydrophones - Updates the export_audio function to input voltage timeseries and allow resampling to speed up recordings Bugfixes - Refactors band-averaging for spectral density levels and sound pressure levels. This refactor avoids losing information at frequencies located at band boundaries, and conducts all of the band-averaging immediately after the PSD calculation. The jupyter notebook has been updated with these changes as well.
…#452) This refactor solves a DOLfYN structural problem between its binning architecture and the traditional sliding-window architecture of Welch's algorithm. Because DOLfYN was using slices of bins to compute each PSD segment, it was impossible to ensure data overlap between one bin and the next, meaning the industry-standard 50% overlap cannot be accomplished. This PR removes the bin structure from the PSD computation and ensures overlap using a "step" argument, which is also now built into the bin structure to properly average dimensions for the PSD. In doing this, DOLfYN's custom PSD code, which was a custom implementation of Welch's algorithm without the ability to overlap FFT segments, has been replaced with scipy.signal.welch. This has one breaking change, in that NaN's are no longer tolerated in the PSD calculation. To avoid breaking changes, the DOLfYN code will default to an overlap of 0%. This ensures array shapes will remain the same. On the other hand, since it's new, the Acoustics module will default to 50%. The second breaking change is that the PSD code now renames the input time dimension to "time_psd". In this way, if a different overlap is used, and the PSD is saved into dataset with an averaged "time" dimension, the PSD will not follow the xarray default and go to nan. Finally, I removed the float32 datatypes in certain functions since sometimes we do need float64 precision. Removing custom code means fft.py no longer exists, and tools/misc.py was renamed to tools.py
# MHKiT-Python v1.1.0 ## Additions * Acoustics: Add millicdecade and WISPR instrument support * Added millidecade spectral conversion * Added a WISPR hydrophone reader * Added a voltage-based `export_audio` resampling option * Refactored band-averaging to avoid losing information at frequency-band boundaries * MHKiT-Software#447 * Author: @jmcvey3 * Reviewer: @simmsa * DOLfYN: Add [Nortek Aquadopp](https://www.nortekgroup.com/oceanography/aquadopp-series) ADCP support * Added DOLfYN support for reading Aquadopp instruments * Cleaned Nortek parsing code * Simplified handling of the non-cabled ADV orientation flag * MHKiT-Software#434 * Author: @jmcvey3 * Reviewer: @akeeste * Examples: Add ADCP waves example: `example/adcp_waves_example.ipynb` * Added an example notebook showing how to ingest and analyze wave measurements from a dual-profile Nortek Signature 250 deployment at PacWave. * MHKiT-Software#430 * Author: @jmcvey3 * Reviewer: @akeeste ## Improvements ### Acoustics - Improved flexibility and robustness of the Acoustics module - Added a configurable FFT length for sound pressure PSDs - Renamed bin/windowing attributes for clarity - Fixed an incorrectly signed gain correction - MHKiT-Software#433 - Author: @jmcvey3 - Reviewer: @akeeste ### DOLfYN - Refactored PSD calculations to use `scipy.signal.welch` - Replaced DOLfYN's custom Welch-like PSD implementation (built each segment from bin slices, could not overlap FFT segments) with `scipy.signal.welch` - Removed the bin-based segment structure and added a `step` argument to control overlap - Dropped float32 casts in some functions in favor of float64 - Removed `fft.py` and renamed `tools/misc.py` to `tools.py` - Breaking Changes: - NaNs are no longer tolerated in PSD calculations - DOLfYN defaults to 0% overlap to preserve existing array shapes - Acoustics defaults to 50% overlap - PSD output time dimension renamed to `time_psd` - MHKiT-Software#452 - Author: @jmcvey3 - Reviewer: @akeeste, @simmsa - Improved handling of "averaged" profiles - Fixed handling of Nortek Signature dual-profile ADCP data by defaulting to "_avg" velocity variables when untagged ones are absent. - MHKiT-Software#430 - Author: @jmcvey3 - Reviewer: @akeeste - Critical PSD bugfix - Fixed a bug where individual FFTs received a 50% overlap twice, which corrupted the first and last spectrum of a timeseries. - MHKiT-Software#430 - Author: @jmcvey3 - Reviewer: @akeeste ### Examples - Added histograms to ADCP example - MHKiT-Software#448 - Author: @browniea - Reviewer: @akeeste ### River/IO - Fixed Qhull interpolation and D3D coordinate-system errors - MHKiT-Software#448 - Author: @browniea - Reviewer: @akeeste - Fixes: MHKiT-Software#442, MHKiT-Software#444 - Delft3D module updates - Added new Delft3D coordinate names - Added a new grid-convergence-index calculation function - Added support for xarray/netCDF4 input in the D3D module - MHKiT-Software#428 - Author: @browniea - Reviewer: @akeeste ### Wave - NDBC Directional Wave Units - Fixed NDBC directional wave spectrum output to return degrees instead of radians - Updated polar plots so 0 deg is at the top and increases clockwise - MHKiT-Software#437 - Author: @jmcvey3 - Reviewer: @akeeste - Fixes: MHKiT-Software#427 ### Wave/Hindcast - Added a `hindcast_guard` exception-handling decorator (`hindcast_exceptions.py`) that surfaces a clear error on HSDS request failures, distinguishing the known NLR HSDS outage (MHKiT-Software#450) from other failures - MHKiT-Software#449 - Author: @simmsa - Reviewer: @akeeste ## Maintenance - Added Python 3.13 support - MHKiT-Software#445 - Author: @simmsa - Reviewer: @akeeste - Fixes: MHKiT-Software#441 - Added [pandas 3](https://pandas.pydata.org/community/blog/pandas-3.0.html) Support - Updated the pandas dependency to allow pandas 3.x - Added a compatibility shim for NDBC missing-value handling related to pandas 3 object to String dtype api changes - Fixed a deprecated period alias in tests - MHKiT-Software#443 - Author: @simmsa - Reviewer: @akeeste - Fixes: MHKiT-Software#440 - Updated GitHub Actions CI, expanded installation/developer documentation, refreshed dev environment, and trimmed dependencies - Refactored optional dependencies - Standardized conda/conda-forge environment builds - Scoped black linting to changed files - MHKiT-Software#436 - Author: @simmsa - Reviewer: @akeeste - Update `rex` dependency to target pypi package to [`NLR-rex[hsds]>=0.5.0`](https://pypi.org/project/NLR-rex/) - MHKiT-Software#449 - Author: @simmsa - Reviewer: @akeeste
Merged
Contributor
Author
|
@jmcvey3, this was my experiment to try to fix the extra commits from #436. I tried a few variants of "scary" git commands and was able to squash my commits into one commit, but this always alters the git history by changing the commit hashes. I'm going to not merge this out of an abundance of caution. In the future we (I) can definitely do a better job trying to catch these types of things earlier. |
Contributor
I agree with this conclusion. let's go forward with the existing history and ensure we squash on PRs into main/develop (but not for develop-->main or vice-versa). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#436 did not follow the convention to do a rebase merge commit. This PR is an attempt to squash all of the commits from 436 into a single commit.