allreduce

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

Ytk-mp4j is a fast, user-friendly, cross-platform, multi-process, multi-thread collective message passing java library which includes gather, scatter, allgather, reduce-scatter, broadcast, reduce, allreduce communications for distributed machine learning.

Summary of call graphs and data structures of NVIDIA Collective Communication Library (NCCL)

Different Implementations of the AllReduce algorithm used in Distributed Deep Learning in c++

Interactive web visualization for understanding collective communication algorithms (as used in NCCL, RCCL, MPI). Learn how AllReduce, Broadcast, Reduce, AllGather and more work step by step.

This repository outlines a comprehensive guide for training a distributed deep learning model.

Topology-aware analytical model for Ring and hierarchical AllReduce traffic.

Sparse Allreduce

Pool Apple Silicon Macs into a distributed ML training cluster. Zero-config discovery, ring AllReduce gradient sync, HMAC-authenticated fleet isolation with TLS encryption. PyTorch + MLX engines.

AllReduce/AllGather scaling in ASTRA-sim across torus vs switch topologies on the analytical + ns-3 backends — latency-bound vs bandwidth-bound over message size and node count. Reproducible Docker/Chakra harness + write-up.

Ring, tree and naive all-reduce from scratch over OS pipes, plus top-k, QSGD and sign gradient compression with error feedback: accuracy vs bytes in data-parallel training

RoCEv2 control plane simulator — DCQCN congestion control on fat-tree GPU clusters with NCCL Ring AllReduce workloads

Adaptive Hybrid Distributed SGD implements and evaluates Parameter Server, Ring AllReduce, and a runtime-adaptive hybrid that switches communication topology during model training based on live metrics.

Ring-allreduce built from scratch in C++20 over raw MPI point-to-point primitives, benchmarked against MPI_Allreduce across 25 message sizes and 15 process counts, with a fitted alpha-beta cost model explaining the measured bandwidth gap. Includes a reproducible LaTeX report and an engineering postmortem.

Distributed training of a VGG-11 network on the CIFAR-10 dataset

From-scratch data-parallel SGD in Python: ring all-reduce over TCP, gradient bucketing, verified against single-worker training.

Analytical simulator for collective communication on ring, fat tree, and dragonfly topologies: graph based hop costs, bandwidth efficiency, and closed form algorithm crossover up to 1024 nodes. Reproducible sweeps in Python (networkx), with a compiled LaTeX report.

天权 TIANQUAN — 世界顶级分布式训练 x 高效推理服务统一内核：Ring/Tree 带宽最优集合通信 · DDP · ZeRO-1/2/3 · Megatron 张量并行 · GPipe/1F1B · PagedAttention · 连续批处理 · RadixAttention · 投机解码。纯 NumPy、CPU 可跑、96 项不变量全绿。作者：晨星

Event-driven benchmark harness for ring and tree allreduce under topology, message size, and congestion constraints.