[#recipebot] Add DeepSeek-V3 256 GPUs FP8MX GBS4096 recipe on A4X

[Alina-PANG]

Adding a recipe for DeepSeek-V3 256 GPUs FP8MX GBS4096 on A4X using Megatron-Bridge (NeMo 26.06.01).

• Model: DeepSeek-V3
• Hardware: A4X (GB200, 64 nodes / 256 GPUs)
• Framework: Megatron-Bridge / NeMo 26.06.01
• Precision: FP8-MX (fp8_mx)
• Sequence Length: 4096
• Global Batch Size: 4096
• Benchmark Run ID: megatron_bridge_training-nemo260601/deepseek_v3_256gpus_fp8mx_seq4096_gbs4096-2026-08-20_062823-b731e628-5371-49aa-a0e0-86456e2932eb

[gemini-code-assist]

Code Review

This pull request introduces a new GKE recipe for pretraining DeepSeek-V3 using the Nvidia Megatron-Bridge framework, including a Helm chart, launcher script, and documentation. The review feedback highlights several key improvements: securely handling the Hugging Face token via environment variables rather than hardcoding it, aligning NCCL plugin versions between the documentation and configuration, optimizing the launcher script's logging to avoid memory overhead, replacing fragile runtime repository cloning and file modifications with pre-baked images and command-line overrides, and adding safety checks around the dmesg command to prevent permission errors in non-privileged containers.

[gemini-code-assist]

The HF_TOKEN is hardcoded to the placeholder "YOUR_HF_TOKEN" here, which will overwrite any HF_TOKEN environment variable passed to the container from Kubernetes (e.g., via workload.envs or Secrets). To allow passing the token dynamically and securely without modifying the launcher script, use a default fallback pattern.

Suggested change

	export HF_TOKEN="YOUR_HF_TOKEN" # Replace with your Hugging Face token.
	export HF_TOKEN="${HF_TOKEN:-YOUR_HF_TOKEN}" # Replace with your Hugging Face token.

[gemini-code-assist]

The HF_TOKEN environment variable exported in the shell is not passed to the Helm chart, meaning the container will not receive it. Add the HF_TOKEN environment variable to the helm install command so it is passed to the workload container.

Suggested change

	helm install $WORKLOAD_NAME . -f values.yaml \
	--set-file workload_launcher=launcher.sh \
	--set workload.image=nvcr.io/nvidia/nemo:26.06.01 \
	--set volumes.gcsMounts[0].bucketName=${GCS_BUCKET} \
	--set volumes.gcsMounts[0].mountPath=/job-logs \
	--set workload.envs[0].value=/job-logs/$WORKLOAD_NAME \
	--set queue=${KUEUE_NAME}
	helm install $WORKLOAD_NAME . -f values.yaml \
	--set-file workload_launcher=launcher.sh \
	--set workload.image=nvcr.io/nvidia/nemo:26.06.01 \
	--set volumes.gcsMounts[0].bucketName=${GCS_BUCKET} \
	--set volumes.gcsMounts[0].mountPath=/job-logs \
	--set workload.envs[0].value=/job-logs/$WORKLOAD_NAME \
	--set workload.envs[2].name=HF_TOKEN \
	--set workload.envs[2].value=${HF_TOKEN} \
	--set queue=${KUEUE_NAME}

[gemini-code-assist]

Pass the HF_TOKEN environment variable to the Helm chart in this example as well.

Suggested change

	helm install $WORKLOAD_NAME . -f values.yaml \
	--set-file workload_launcher=launcher.sh \
	--set workload.image=nvcr.io/nvidia/nemo:26.06.01 \
	--set volumes.gcsMounts[0].bucketName=${GCS_BUCKET} \
	--set volumes.gcsMounts[0].mountPath=/job-logs \
	--set workload.envs[0].value=/job-logs/$WORKLOAD_NAME \
	--set queue=${KUEUE_NAME} \
	--set workload.arguments[0]="trainer.max_steps=100"
	helm install $WORKLOAD_NAME . -f values.yaml \
	--set-file workload_launcher=launcher.sh \
	--set workload.image=nvcr.io/nvidia/nemo:26.06.01 \
	--set volumes.gcsMounts[0].bucketName=${GCS_BUCKET} \
	--set volumes.gcsMounts[0].mountPath=/job-logs \
	--set workload.envs[0].value=/job-logs/$WORKLOAD_NAME \
	--set workload.envs[2].name=HF_TOKEN \
	--set workload.envs[2].value=${HF_TOKEN} \
	--set queue=${KUEUE_NAME} \
	--set workload.arguments[0]="trainer.max_steps=100"

[gemini-code-assist]

The NCCL plugin version in the README (v1.1.0) does not match the version configured in values.yaml (v1.1.2). Update the README to reflect the correct version.

[gemini-code-assist]

Using a list comprehension solely for side effects (like printing/writing to stdout) is an anti-pattern in Python because it builds a full list of None elements in memory. For large log outputs, this can cause significant memory overhead. Use sys.stdout.writelines with a generator expression instead, which is memory-efficient and idiomatic.

Suggested change

	--no-python bash worker_command.sh 2>&1 | python3 -u -c "import sys, time; [sys.stdout.write('[{}] {}'.format(time.strftime('%Y-%m-%d %H:%M:%S'), line)) for line in iter(sys.stdin.readline, '')]" | tee -a "/runtime-logs/${JOBSET_NAME}/logs/nemo_mb_RANK_${JOB_COMPLETION_INDEX}.log"
	--no-python bash worker_command.sh 2>&1 | python3 -u -c "import sys, time; sys.stdout.writelines('[{}] {}'.format(time.strftime('%Y-%m-%d %H:%M:%S'), line) for line in sys.stdin)" | tee -a "/runtime-logs/${JOBSET_NAME}/logs/nemo_mb_RANK_${JOB_COMPLETION_INDEX}.log"

[gemini-code-assist]

Running dmesg -W directly can fail with permission errors if the container is not run in privileged mode (e.g., if hostNetwork is set to false), or if dmesg is not installed in the container image. Wrap it in a check to ensure it is available and runnable to avoid noisy error logs.

                # Stream dmesg logs to stdout if available and permitted.
                if command -v dmesg &> /dev/null && dmesg -T &> /dev/null; then
                  dmesg -W &
                  DMESG_PID=$!
                  trap "kill -9 $DMESG_PID" EXIT
                fi

[gemini-code-assist]

Using sed to dynamically inject Python code into run_script.py at runtime is fragile and prone to breaking if the upstream repository structure or formatting changes. Since Megatron-Bridge/NeMo uses Hydra for configuration, you should be able to pass this parameter directly as a command-line override (e.g., dist.distributed_timeout_minutes=10 or ++dist.distributed_timeout_minutes=10) instead of modifying the source file.

[gemini-code-assist]

Cloning the Megatron-Bridge repository and checking out a specific commit at runtime inside the container is inefficient and introduces a single point of failure (e.g., GitHub network issues, API rate limits). For production and reproducible workloads, it is highly recommended to pre-bake the specific version of the repository into the Docker container image.