Skip to content

fix: support run_on_local_node in gemm_throttling - #130

Merged
linamy85 merged 1 commit into
AI-Hypercomputer:chsfrom
simrankaurb:fix-gemm-throttling-local-node
Aug 20, 2026
Merged

fix: support run_on_local_node in gemm_throttling#130
linamy85 merged 1 commit into
AI-Hypercomputer:chsfrom
simrankaurb:fix-gemm-throttling-local-node

Conversation

@simrankaurb

@simrankaurb simrankaurb commented Aug 20, 2026

Copy link
Copy Markdown

Summary

Support configurable mesh locality (run_on_local_node) in gemm_throttling to allow running GEMM thermal soak microbenchmarks on single host devices across multi-node clusters.

Rationale & Context

Following the design in #125, run_on_local_node allows running microbenchmarks per single host (jax.local_devices()) rather than over the entire global cluster slice (jax.devices()).

Changes Included

  1. gemm_throttling(): Added run_on_local_node: bool = False parameter and passed local_mesh=run_on_local_node to create_mesh().
  2. gemm_throttling_calculate_metrics(): Added run_on_local_node: bool = False parameter and set device_count = jax.local_device_count() if run_on_local_node else jax.device_count() when calling handle_based_on_sharding(). This prevents inflated total FLOPs and throughput metrics in multi-node environments.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a run_on_local_node parameter to the gemm_throttling benchmark function to allow running on local node devices. However, the reviewer pointed out a critical issue: introducing this parameter without updating gemm_throttling_calculate_metrics will cause incorrect metrics calculations in multi-node environments, as the global device count will be used instead of the local device count. The reviewer provided a code suggestion to fix this by passing the correct device count based on run_on_local_node.

Comment thread Ironwood/src/benchmark_gemm_throttling.py
@simrankaurb
simrankaurb force-pushed the fix-gemm-throttling-local-node branch from 701ca27 to 9c751be Compare August 20, 2026 06:32
@simrankaurb
simrankaurb force-pushed the fix-gemm-throttling-local-node branch from 9c751be to 637f785 Compare August 20, 2026 06:33
@simrankaurb simrankaurb changed the title fix: support run_on_local_node in gemm_throttling to prevent multi-node OOM fix: support run_on_local_node in gemm_throttling Aug 20, 2026
@linamy85
linamy85 merged commit 80fa0cc into AI-Hypercomputer:chs Aug 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants