Skip to content

Latest commit

 

History

History
361 lines (294 loc) · 17.4 KB

README.en.md

File metadata and controls

361 lines (294 loc) · 17.4 KB

HyperParallel

An Ascend SuperPod-affinity distributed parallel acceleration library that simplifies supernode programming and unleashes computational potential.

HyperParallel provides Ascend SuperPod-affinity distributed parallel acceleration capabilities. Whilst maintaining ease of use, it addresses the architectural characteristics of Ascend SuperPods, including resource pooling, peer-to-peer architecture, hierarchical and diverse network topology, and FP8 low-precision formats. It implements distributed parallelism from cluster level to multi-core parallelism within chips, supports unified pooled management of CPU DRAM and NPU HBM, topology-aware scheduling and communication path planning, and FP8 mixed-precision training amongst other Ascend SuperPod-affinity acceleration capabilities.

Key design principles:

Decoupling of Model and System Optimisation: With the continuous evolution of LLM and multimodal algorithm architectures, performance optimisation techniques have also been advancing. The traditional integrated architecture of algorithm and system optimisation poses challenges for algorithm iteration and long-term system maintenance. HyperParallel supports evolving the programming model from system optimisation embedded within model scripts to a decoupled model and system optimisation approach, with implicit injection of parallelism, recomputation, offload and other system optimisations. It supports the evolution of parallel paradigms from SPMD to MPMD, further supporting coordinated optimisation of cluster MPMD and multi-core MPMD. It supports the evolution of compute-storage relationships from Stateful to Stateless with separated computation and state, as well as large language model training, multimodal large model training, and reinforcement learning capabilities.

End-to-End Determinism: To further ensure training stability and precision reproducibility, HyperParallel supports end-to-end determinism, including high-performance deterministic computation, communication, data preprocessing, and random number determinism, supporting floating-point bitwise alignment. All supported models are validated using determinism. Although there is some performance degradation, enabling determinism during training remains recommended for precision reproducibility, rapid SDC detection, and bug identification.

Unified Training and Inference: As Reasoning RL and Agentic RL tasks become increasingly complex, the training-inference inconsistency problem causing reinforcement learning training convergence difficulties has become more prominent. HyperParallel will explore a unified training-inference architecture, achieving performance optimisation for both training and inference through a single acceleration framework, strengthening training-inference consistency and ensuring RL convergence.

Hybrid Dynamic-Static Execution: Optimisation based on static graphs is an important means of further improving performance. For instance, capabilities such as compute-communication concurrency, memory analysis, and execution sequence orchestration based on static graphs can effectively optimise performance, which are not easily achievable in dynamic graph mode. However, dynamic-to-static compilation support is extremely challenging, and complete dynamic-to-static conversion is not yet achievable. HyperParallel will support partial dynamic-to-static conversion through certain syntax constraints, utilising MindSpore's advanced graph optimisation capabilities to further enhance performance.

Architecture Overview

HyperShard: Programming Model Evolution, System Optimisation Embedded in Model -> Decoupled Model and System Optimisation

  • SuperPod Layout: Unified modelling of tensor sharding, device mapping, and communication paths, achieving single-card abstraction for SuperPods;
  • Declarative HSDP/TP/CP/EP: Implicit injection of optimisations such as parallelism, recomputation, and offload into models, achieving decoupling of model code and system optimisation code, improving algorithm development efficiency;

HyperMPMD: Parallel Paradigm Evolution, SPMD -> Cluster MPMD -> Cluster + Multi-Core MPMD

  • Cluster MPMD: Supports heterogeneous model sharding, supports arbitrary device allocation for model slices;
  • Multimodal MPMD (Mpipe): Supports heterogeneous scheduling for multimodal pipeline parallelism, unlocking SuperPod utilisation for MLLMs;
  • Intra-Chip Multi-Core MPMD: Intra-chip multi-core MPMD parallelism, combined with core-level memory semantic one-sided communication, enhancing compute-communication overlap and MAC utilisation;

HyperOffload: Compute-Storage Relationship Evolution, Stateful -> Stateless Computation-State Separation

  • Unified Programming for Remote and Local Tensors: Supports tensor location allocation, hides remote data transfer, improves cluster memory utilisation;
  • Remote Tensor Prefetching and Caching, Full Model Offload: DDP/HSDP+Offload replaces complex parallel modes such as DP/TP/PP/CP/SP/EP, simplifying system design and improving performance;

Key Features

  • Models

    • DeepSeekV3
    • Qwen3.5-0.8B-Base
    • Qwen3.5-35B-A3B-Base (MoE)
    • Qwen3-VL-30B-A3B-Instruct (MoE)
    • DeepSeekV3.2
    • Qwen3-Omni
  • HyperShard

    • DTensor
      • DTensor basic
      • DTensor redistribute
      • manual_seed (distributed random number seed control)
      • DTensor centric communication
      • Cross Mesh DTensor redistribution
    • HSDP / FSDP
      • Parameter & Optimiser Sharding
      • Parameter & Optimiser & Gradient Sharding
      • Overlap (full overlap mode)
      • gradient_scaling_factor
      • Dynamic-to-Static Conversion
    • Shard / TP
      • Distributed Operator Support List
      • Custom Distributed Operator Registration (YAML registry + Python impl)
      • Custom Shard
      • DFunction (custom distributed autograd functions)
      • Loss Parallel (TP training loss parallelism)
      • TP Styles
        • ColwiseParallel / RowwiseParallel / SequenceParallel
        • parallelize_module (declarative TP interface)
        • 1D
        • Higher-Dimensional TP, 2D/2.5D/3D
      • EP
        • ExpertParallel / ExpertTensorParallel (basic workflow)
        • MoE building blocks (GroupedExperts / TokenChoiceTopKRouter / MoE)
        • Load balancing (expert_bias + aux_loss + AutoScaler)
        • MoE zero-overhead activation storage
        • MoE+EP token dispatch decoupling
        • Dropless basic workflow
        • Compute-Communication Overlap
        • Expert Hot Migration / Hot Expert Replication
      • CP
        • ContextParallel (basic context parallelism)
        • AsyncContextParallel (async context parallelism)
        • DSA series (Indexer / Loss / SparseAttention)
        • TP DTensor local rewrap
        • DeepSpeed Ulysses
        • Ring Attention
        • 3D Sequence Parallelism
      • Overlap
    • Distributed Random Numbers
      • manual_seed (seed control)
      • DropOut
  • HyperMPMD

    • Pipeline Parallelism
      • GPipe
      • 1F1B
      • VPP (ScheduleInterleaved1F1B)
      • PP+FSDP (MetaStep integration)
      • PipelineStage dx/dw computation
      • Compute-Communication Overlap overlap_b_f (CommComputeOverlap dual-thread orchestrator)
      • batched P2P transport / overlap_p2p
      • PP Activation Swap
      • variable-layer + mixed-recompute under overlap_b_f
      • ZBV
      • SeqPP
      • Different Device Allocation per PP Stage
    • Mpipe Multimodal Parallelism
      • Mpipe Transpose Scheduling
      • Mpipe Data Reordering
    • Subgraph Partitioning
      • Multimodal Encoder/Decoder Partitioning to Different Devices
    • Multi-Core Parallelism
      • Multi-Core Parallelism - O0
      • Multi-Core Parallelism - O1
      • MoE Compute-Communication Overlap Optimisation Based on Multi-Core Parallelism
      • PP 1F1B Compute-Communication Overlap Optimisation Based on Multi-Core Parallelism
  • HyperOffload

    • Activation Checkpoint (checkpoint / checkpoint_wrapper / CheckpointPolicy)
    • Activation Swap (swap / swap_wrapper / swap_tensor_wrapper / SwapManager)
    • Activation Swap and Checkpointing Coordinated Configuration
    • Swap Fusion
    • SAS (Selective Activation Swap)
    • SPO (Selective Parameter/Gradient/Optimizer Offload)
    • Memory Semantic-Based Offload
    • Automatic Activation Swap Strategy Generation
  • Optimizer

    • AdamW
    • Muon (momentum-based optimizer)
    • ChainedOptimizer (Muon+AdamW chained combination)
    • get_hyper_optimizer
    • Sharded Optimizer (FSDP/HSDP integration)
    • gradient scaling factor + clip_grad enhancements
  • AutoParallel

    • SAPP-ND: ND Search (memory estimation + performance estimation)
    • SAPP-PPB: Pipeline Parallelism Balancing
    • SAPP-Omni
  • Fast Fault Recovery

    • DCP (Distributed Checkpoint)
      • Distributed checkpoint save/load
      • Async staging save
      • Offline format transform
      • Huggingface Format Support
      • Different Sharding Strategy Conversion Support
    • Basic Fault Recovery Workflow
    • Process-Level Fast Fault Recovery
    • Last Words (fault-triggered checkpoint saving)
    • SDC Detection
  • Trainer

    • LLMTrainer framework
    • VLTrainer framework (visual-language)
    • Callbacks (Logging / MoeMonitor)
    • parallel_dims configuration
  • Integration

    • LlamaFactory integration (activation recompute & swap + HSDP)
  • Tools

    • Precision Monitoring

      • global norm
      • local norm
      • local loss
    • DryRun

      • Memory Overhead Analysis
      • Single-Card Cluster Execution Simulation

Installation Guide

HyperParallel offers two installation methods:

  • pip installation: install an already built hyper-parallel package and use extras to select runtime deep learning framework dependencies.
  • source build: use ./build.sh to generate a whl package, and use build arguments to decide whether native extensions are compiled.

If you only need to install a released package, prefer pip install. If you need to generate a whl package locally or customize native extension build options, build from source.

1. Install With pip

pip install extras only control Python runtime dependencies. They do not control native extension compilation.

Command Installed dependencies Use case
pip install hyper-parallel Common dependencies only, no deep learning framework You manage framework versions yourself or only use framework-independent features
pip install 'hyper-parallel[mindspore]' Common dependencies + mindspore>=2.10 Use a supported MindSpore backend
pip install 'hyper-parallel[torch]' Common dependencies + torch==2.9.1 + torch-npu==2.9.1 Use the default PyTorch 2.9 backend
pip install 'hyper-parallel[torch26]' Common dependencies + torch==2.6.0 + torch-npu==2.6.0.post3 Use the PyTorch 2.6 backend
pip install 'hyper-parallel[torch27]' Common dependencies + torch==2.7.1 + torch-npu==2.7.1 Use the PyTorch 2.7 backend
pip install 'hyper-parallel[torch29]' Common dependencies + torch==2.9.1 + torch-npu==2.9.1 Explicitly use the PyTorch 2.9 backend
pip install 'hyper-parallel[all]' Common dependencies + MindSpore + default PyTorch 2.9 Use both backends in the same environment

In shells such as zsh, quote package names with extras so [] is not treated as a glob pattern.

2. Build a Wheel From Source

The common build entry invokes the selected component build scripts and assembles one wheel. The indexed Dataset C++ helper is built on every invocation. Multicore includes its private SHMEM dependency and therefore has no separate SHMEM build option.

Argument Default Values Description
--multicore on on, off Build the Torch Multicore component and SHMEM.
--custom-ops on on, off Build MindSpore custom ops.
--soc-list ascend910b,ascend910_93 SoC list Select the Multicore kernel targets.
--strict off on, off Stop if a selected optional component fails.
--jobs nproc Integer Set native compilation parallelism.
--clean disabled Flag Rebuild selected component work and install data.
./build.sh --help
./build.sh --multicore on --custom-ops on --soc-list ascend910b,ascend910_93
./build.sh --strict on --jobs 24
pip install /absolute/path/to/the-built-wheel.whl

Generic prerequisites are Python 3.10–3.12, Python development headers, setuptools, wheel, pybind11 and a C++17 host toolchain. SDK, framework and device constraints are component-specific. A wheel targets one Python ABI and host architecture; its runtime must satisfy the framework ABI and the ELF glibc requirements.

See the installation guide for packaging details and the Multicore CANN custom OPP activation procedure.

Quick Start

  1. Use fully_shard for data-parallel parameter sharding
from hyper_parallel import fully_shard, init_device_mesh

mesh = init_device_mesh(device_type="npu", mesh_shape=(dp_size,), mesh_dim_names=("dp",))
model = fully_shard(model, mesh=mesh)
  1. Use declarative TP Styles for tensor parallelism
from hyper_parallel import ColwiseParallel, RowwiseParallel, parallelize_module, init_device_mesh

tp_mesh = init_device_mesh("npu", (tp_size,), mesh_dim_names=("tp",))

parallelize_module(
    model,
    tp_mesh,
    {
        "attn.q_proj": ColwiseParallel(),
        "attn.k_proj": ColwiseParallel(),
        "attn.v_proj": ColwiseParallel(),
        "attn.o_proj": RowwiseParallel(),
        "mlp.gate_proj": ColwiseParallel(),
        "mlp.up_proj": ColwiseParallel(),
        "mlp.down_proj": RowwiseParallel(),
    },
)
  1. Use PipelineStage and PipelineSchedule for pipeline parallelism
from hyper_parallel import DTensor
from hyper_parallel.core.pipeline_parallel import PipelineStage, Schedule1F1B

# Wrap the partitioned module into PipelineStage
stage = PipelineStage(split_model, stage_index, stage_num=4)

# Select pipeline parallel scheduling
schedule = Schedule1F1B(stage, micro_batch_num=8)

# Execute
x = DTensor.from_local(local_x, input_mesh, input_placements)
schedule.run(x)
  1. Use Optimizer for Muon+AdamW chained optimisation
from hyper_parallel.core.optimizer import get_hyper_optimizer

optimizer = get_hyper_optimizer(
    model=model,
    muon_params=muon_param_groups,
    adamw_params=adamw_param_groups,
    muon_kwargs={"lr": 0.02, "momentum": 0.95},
    adamw_kwargs={"lr": 3e-4, "weight_decay": 0.1},
)
  1. Use Activation Checkpoint/Swap for memory optimisation
from hyper_parallel.core.activation_checkpoint import checkpoint_wrapper, swap_wrapper

model.layers[0] = checkpoint_wrapper(model.layers[0])
model.layers[1] = swap_wrapper(model.layers[1])

Documentation

Contributing

  1. Fork this repository
  2. Create a new Feat_xxx branch
  3. Commit your code
  4. Create a new Pull Request

If you have any suggestions for HyperParallel, please contact us through issues and we will address them promptly. If you are interested in HyperParallel's technology or would like to contribute code, you are welcome to join the Parallel Training System SIG.

License

Apache 2.0 License