Nccl Optimizer
by @mitsuha-m
Detect the optimal NCCL configuration for distributed GPU training on this machine. Checks GPU topology (NVLink/PCIe), whether RDMA (InfiniBand / RoCE) is av...
clawhub install nccl-optimizerπ About This Skill
name: nccl_optimizer description: > Detect the optimal NCCL configuration for distributed GPU training on this machine. Checks GPU topology (NVLink/PCIe), whether RDMA (InfiniBand / RoCE) is available, benchmarks intra-node collective bandwidth and peer-to-peer GPU bandwidth, and optionally benchmarks inter-node bandwidth via MPI. Use when: setting up multi-GPU or multi-node training, diagnosing slow collective communication, or tuning NCCL for a new cluster node.
NCCL Optimizer
Finds the best NCCL communication configuration for distributed training with clear separation of intra-node and inter-node bandwidth metrics.
What it does
1. GPU topology β nvidia-smi topo -m to detect NVLink vs PCIe.
2. RDMA check β ibv_devinfo PORT_ACTIVE state for InfiniBand/RoCE.
- β
RDMA β emit recommended NCCL_IB_* env-vars.
- β No RDMA β socket benchmark sweep.
3. Intra-node all-reduce β sweeps NCCL_SOCKET_IFNAME Γ NCCL_NET_GDR_LEVEL Γ
NCCL_IB_TIMEOUT, runs all_reduce_perf -g , picks best bus bandwidth.
4. Intra-node P2P β p2p_bw for GPUβGPU pair bandwidth (if available).
5. Inter-node benchmark β if nodes= passed, runs MPI all_reduce_perf across nodes;
otherwise emits a ready-to-run command.
Prerequisites
| Tool | Purpose | Install |
|------|---------|---------|
| nvidia-smi | GPU info + topology | NVIDIA driver |
| ibv_devinfo | RDMA detection | apt install ibverbs-utils |
| all_reduce_perf | Collective benchmark | See below |
| p2p_bw | Peer-to-peer benchmark | Same nccl-tests build |
| mpirun | Inter-node benchmark | apt install openmpi-bin |
Build nccl-tests
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
For V100 (sm_70), A100 (sm_80), A800 (sm_80), H100 (sm_90):
make -j$(nproc) CUDA_HOME=/usr/local/cuda \
NVCC_GENCODE="-gencode=arch=compute_80,code=sm_80"
export PATH=$PWD/build:$PATH
Usage
# Intra-node only
openclaw skill run nccl_optimizerInclude inter-node benchmark (requires passwordless SSH + MPI)
openclaw skill run nccl_optimizer "nodes=10.0.0.1,10.0.0.2"
Metrics explained
| Metric | What it measures | |--------|-----------------| | All-reduce bus BW (intra) | Collective throughput across local GPUs β relevant for single-node training | | P2P bandwidth | GPUβGPU direct copy speed (NVLink β« PCIe) | | All-reduce bus BW (inter) | Collective throughput across nodes β bottleneck for multi-node training |
Notes
(N-1)/N Γ data / time. Compare at same N.π‘ Examples
# Intra-node only
openclaw skill run nccl_optimizerInclude inter-node benchmark (requires passwordless SSH + MPI)
openclaw skill run nccl_optimizer "nodes=10.0.0.1,10.0.0.2"
βοΈ Configuration
| Tool | Purpose | Install |
|------|---------|---------|
| nvidia-smi | GPU info + topology | NVIDIA driver |
| ibv_devinfo | RDMA detection | apt install ibverbs-utils |
| all_reduce_perf | Collective benchmark | See below |
| p2p_bw | Peer-to-peer benchmark | Same nccl-tests build |
| mpirun | Inter-node benchmark | apt install openmpi-bin |
Build nccl-tests
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
For V100 (sm_70), A100 (sm_80), A800 (sm_80), H100 (sm_90):
make -j$(nproc) CUDA_HOME=/usr/local/cuda \
NVCC_GENCODE="-gencode=arch=compute_80,code=sm_80"
export PATH=$PWD/build:$PATH
π Tips & Best Practices
(N-1)/N Γ data / time. Compare at same N.