A Low-Latency RDMA Collective Communication Engine & Latency Profiling Suite
Named after the Ouroboros (the serpent consuming its own tail, symbolizing the Ring All-Reduce) and the fundamental arithmetic addition operation.
Important
Soft-RoCE Execution Note: Soft-RoCE (rdma_rxe) executes the RDMA protocol in kernel software on the CPU. Latency and bandwidth measurements over Soft-RoCE reflect CPU, memory bus, and Linux kernel networking characteristics rather than dedicated ASIC offload. Hardware baseline numbers will be acquired through cloud/bare-metal RDMA deployments.
Project Adder extends the lock-free single-host IPC techniques developed in Project Icarus (SPSC queues, core pinning, busy-polling) across distributed network nodes using libibverbs.
Designed with the primitives underpinning modern AI infrastructureβsuch as NCCL, Mooncake (disaggregated KV-cache streaming), and GPUDirect RDMAβAdder implements low-level Queue Pair management, memory registration pools, and distributed collective algorithms from first principles.
Adder is organized into four sequential deliverables, each governed by quantitative engineering targets:
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β D1: Ping-Pong Tool β Target: Match ib_send_lat within 20% (1M+ smp)β
β D2: Ring All-Reduce β Target: 70β80% of MPI_Allreduce (>= 1MB) β
β D3: Straggler Schedule β Target: <= 50% P99 degradation under 2x delay β
β D4: In-Network Agg β Target: Reduce egress from 2(N-1)/N * S to S β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
- Objective: Measure one-way microsecond-scale network latency across transports and memory modes.
-
Scope:
- Reliable Connection (RC) vs. Unreliable Datagram (UD).
- Two-sided (
IBV_WR_SEND/RECV) vs. One-sided (IBV_WR_RDMA_WRITE). - Registered Memory vs. Inline (
IBV_SEND_INLINE). - Message size sweep from 64 B to 1 MB.
-
Target Milestones:
-
P50 Latency Target: Match
ib_send_laton the same testbed within$\le 20$ %. -
Sampling Rigour: Full HDR percentile distribution (
$P_{50}, P_{90}, P_{99}, P_{99.9}, P_{99.99}$ ) over$\ge 1{,}000{,}000$ timed iterations. - UD Compliance: Enforce 40-byte Global Routing Header (GRH) buffer offsets and strict active MTU clamping.
-
P50 Latency Target: Match
-
Objective: Implement collective communication moving the theoretical minimum
$2\frac{N-1}{N}S$ bytes across$2(N-1)$ steps. -
Scope:
- Two-phase pipeline: Scatter-Reduce followed by All-Gather.
- Binary Tree All-Reduce implementation (
$2\log_2 N \cdot S$ bytes) to identify message-size crossover boundaries. - Credit-loop flow control over lock-free SPSC rings to eliminate Receiver Not Ready (RNR) hardware retries.
-
Target Milestones:
-
Throughput Target: Achieve 70% to 80% of
MPI_Allreducebandwidth at buffer sizes$\ge 1\text{ MB}$ . - Correctness Target: Bit-exact sum verification on integer buffers; floating-point verification within defined rounding epsilon.
-
Scaling Target: Clean, non-oversubscribed verification across
$N=2$ and$N=4$ nodes, with$N=8$ as a stress test.
-
Throughput Target: Achieve 70% to 80% of
-
Objective: Prevent tail-latency collapse (
$P_{99} / P_{99.9}$ ) in distributed collectives caused by slow nodes. -
Scope:
- Online monitoring of chunk and round completion times via Completion Queue timestamps.
- Exponential Moving Average (EMA) mean and variance estimation flagging nodes exceeding
$\mu + 2\sigma$ for 3 consecutive rounds. - Dynamic schedule rebalancing that redistributes chunk workloads while strictly maintaining the Conservation Invariant (
$\sum \text{chunks}_i = \text{Total Chunks}$ ). - Out-of-band TCP epoch synchronization with start round
$R \ge \text{current} + 2$ to guarantee zero intra-round schedule divergence.
-
Target Milestones:
-
Tail Latency Target: Under an injected
$2\times$ straggler delay,$P_{99}$ degradation must not exceed 50% (compared to$\ge 200$ % without mitigation). -
Detection Target: Flag the straggler within
$\le 2$ all-reduce rounds with$\ge 90$ % accuracy.
-
Tail Latency Target: Under an injected
- Objective: Emulate programmable network switch aggregation using eBPF/XDP with a C23 userspace fallback.
-
Scope:
- Ingress XDP packet parsing with SwitchML-style 32-bit fixed-point integer quantization.
- BPF array map gradient accumulation with bounded unrolled loops.
- Packet replication in all-gather via TC clone-redirect or unicast fanout.
-
Target Milestones:
-
Traffic Target: Reduce node egress volume from
$2\frac{N-1}{N}S$ down to$S$ bytes per node. -
Baseline Comparison: Benchmark against an isolated socket-based UDP/TCP ring (
sock_ring.cc).
-
Traffic Target: Reduce node egress volume from
Adder/
βββ CMakeLists.txt # Unified build configuration (C++23, C23, eBPF target)
βββ README.md # Project roadmap, deliverables, and performance goals
βββ METHODOLOGY.md # Experimental protocol, core isolation, and provenance
β
βββ shared/ # Zero-overhead low-latency primitives
β βββ adder.h # Master umbrella header
β βββ affinity.h # Dynamic CPU topology discovery & L2 cluster-aware pinning
β βββ busy_poll.h # Zero-allocation CQ spin-polling & event-channel wait
β βββ latency_harness.h # Nanosecond clock, HDR histograms, and CSV metadata headers
β βββ ring_buffer.h # SPSC circular credit buffer with 64-byte cacheline isolation
β βββ free_list_pool.h # Pre-registered RDMA memory pool (supports 2MB hugepages)
β βββ guarded_memory_pool.h # Per-chunk PROT_NONE guard pages for buffer overrun traps
β
βββ pingpong/ # Deliverable D1: Ping-Pong tool
β βββ pingpong.cc # C++23 latency tool (RC & UD, 1-sided vs 2-sided)
β βββ README.md # Usage instructions and perftest comparison guide
β
βββ allreduce/ # Deliverable D2: Collectives
β βββ verbs_qp.h/.cc # RAII wrappers for PD, MR, CQ, and QP state transitions
β βββ ring.h/.cc # Ring all-reduce data path and credit loop
β βββ tree.h/.cc # Binary tree all-reduce for crossover analysis
β βββ main.cc # Benchmark driver and correctness validator
β βββ README.md # Scaling documentation and MPI comparison
β
βββ straggler/ # Deliverable D3: Tail Latency Mitigation
β βββ detector.h/.cc # Online EMA latency variance monitor
β βββ scheduler.h/.cc # Adaptive chunk redistribution & epoch sync
β βββ injection.h/.cc # Synthetic delay injection (spin/sleep/throttle)
β βββ main.cc # Comparative benchmark driver
β βββ README.md # Tail latency targets and detection validation
β
βββ innet_agg/ # Deliverable D4: In-Network Aggregation
β βββ xdp_agg.bpf.c # eBPF/XDP kernel parser & fixed-point accumulator
β βββ agg_control.c # libbpf userspace lifecycle controller
β βββ agg_userspace.c # C23 userspace fallback aggregator
β βββ sock_ring.cc # Socket ring baseline for fair comparison
β βββ agg_test.cc # Test harness measuring bytes per node & jitter
β βββ README.md # SwitchML-style in-network aggregation guide
β
βββ scripts/ # Offline visualization & data analysis
β βββ plot_pingpong.py # Latency curves across transports
β βββ plot_allreduce.py # Bandwidth scaling & crossover curves
β βββ plot_straggler.py # Tail latency CDFs (with vs without mitigation)
β
βββ results/ # Standardized benchmark CSV data
β βββ archive/ # Historical versioned run archives
β
βββ docs/ # Engineering logs & specifications
βββ instructions/ # Project execution instructions & guidelines
βββ man/ # Modular systems reference manual (Man 00 through Man 11)
βββ manual/ # In-depth RDMA, verbs, & latency theory handbooks
βββ git_commands.md # CLI Git command reference & cheat sheet
βββ hardware_config.md # Multi-machine testbed documentation
βββ gap_analysis.md # Hypothesis -> Test -> Conclusion logs
βββ straggler_analysis.md # Tail-latency evaluation records
βββ innet_agg_analysis.md # Verifier logs and fixed-point precision notes
Adder is architected to evaluate performance across two distinct environments:
- Machine Profile 1 (Windows 11 / Intel i3-1220P): Local code editing, IDE integration, and build verification.
- Machine Profile 2 (Debian 12 Bare-Metal): Dedicated standalone Linux host for unthrottled native kernel execution and local loopback reference profiling. Physical cross-host cable bridging is omitted in favor of cloud distributed instances.
- Target VMs: Dedicated Azure Linux VMs (e.g.
Standard_D4s_v5with 4 vCPUs, 16 GB RAM) or HPC instances with Mellanox ConnectX adapters. - Role: Primary distributed execution target for multi-node collectives and cross-host benchmarking, avoiding WSL2 hypervisor overhead, USB-C adapter bottlenecks, and physical wiring complexities.
To achieve deterministic microsecond-scale measurements, all timed paths strictly enforce:
- Zero Heap Allocation: Memory buffers, queue pairs, and tracking structs are pre-allocated and registered before measurement loops begin.
- Zero Virtual Calls: Execution loops avoid runtime polymorphism and
std::functiontype erasure in favor of templates and inlined calls. - Zero Exceptions on the Data Path: Errors are propagated via
std::expectedandstd::optional. - Stream-Safe TCP Control: Out-of-band QP exchanges use bounded
read_exact()/write_exact()loops to eliminate TCP stream fragmentation bugs. - Ordered Verbs RAII: Protection Domains are only deallocated after all associated Memory Regions and Address Handles have been explicitly destroyed.
# Ubuntu / Debian packages
sudo apt update && sudo apt install -y \
build-essential cmake ninja-build \
gcc-14 g++-14 clang llvm \
libibverbs-dev librdmacm-dev libbpf-dev bpftool perftest \
libopenmpi-dev openmpi-bin# Configure with Release optimizations (-O3 -march=native)
cmake -B build -DCMAKE_BUILD_TYPE=Release
# Build all deliverables
cmake --build build -j$(nproc)Available executables upon compilation:
build/pingpongβ Deliverable D1 latency tool.build/allreduce_benchβ Deliverable D2 collective benchmark.build/straggler_benchβ Deliverable D3 straggler mitigation harness.build/sock_ring_benchβ Deliverable D4 socket ring baseline.build/agg_userspaceβ Deliverable D4 C23 fallback aggregator.
- Phase 0: Architecture & Scaffolding β Complete project structure, unified CMake build system, and hardware-agnostic shared primitives.
- Phase 0.5: Systems & Protocol Hardening β Fixed TCP stream fragmentation, UD GRH padding, verbs destruction sequencing, and L2 cache cluster discovery.
- Phase 1: Environment Baseline β Verify
rxe0Port Active status; recordib_send_latandib_send_bwbaseline numbers. - Phase 2: D1 Ping-Pong Completion β Implement verbs state machine (INIT
$\to$ RTR$\to$ RTS); generate latency curves across RC and UD. - Phase 3: D2 Ring All-Reduce Core β Implement scatter-reduce and all-gather steps; verify integer bit-exact sum.
- Phase 4: D2 Collective Scaling β Benchmark
$N=2$ and$N=4$ nodes; report Ring vs. Tree crossover size; compare againstMPI_Allreduce. - Phase 5: D3 Straggler Mitigation β Validate online EMA detection (
$\le 2$ rounds) and verify P99 tail latency improvement under injected delay. - Phase 6: Methodology & Gap Analysis β Complete hypothesis-versus-result entries in
docs/gap_analysis.md. - Phase 7: Cloud Cross-Host Runs β Execute cross-node benchmarks on Azure instances; publish multi-node results.
- Phase 8: D4 In-Network Aggregation (Stretch) β Evaluate eBPF/XDP gradient reduction against socket ring baseline.