[B200+][NVLink5+] reset GPUs after FABRIC_MODE=1 is set - #909
Conversation
0453b86 to
6f936fb
Compare
| # We reset the GPU devices after Fabric Manager startup to clear stale FM | ||
| # partition data that may have carried over previous driver container runs | ||
| echo "Resetting GPUs to clear stale Fabric Partition states..." | ||
| nvidia-smi -r |
There was a problem hiding this comment.
Instead of a blanket reset, I wonder if we need to check if the fabric state is In Progress first on all GPUs before attempting to reset:
nvidia-smi --query-gpu "fabric.state" --format "csv,noheader"
There was a problem hiding this comment.
Isn't it possible that the Completed state could be stale as well?
There was a problem hiding this comment.
My statement was incorrect here. When switch from FABRIC_MODE 0 to 1, all the GPUs will go to In Progress fabric state until there's a partition activation that uses those GPUs. So its not valid to reset the GPU by checking for the fabric state.
Your current implementation of doing the reset if we're trying to start in FABRIC_MODE=1 makes sense.
There was a problem hiding this comment.
Thanks for confirming! I will resolve this comment.
6f936fb to
c9d2e19
Compare
tariq1890
changed the title
Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com>
c9d2e19 to
88a0ec2
Compare
3 participants
NVLinked systems that use NVLink generation 5 or greater require the GPUs to be reset after Fabric Manager is started up with
FABRIC_MODE=1set in its config.