• 8 min read
Collective Communication in Distributed Systems with PyTorch
SUMMARY
Multi-GPU neural network training requires moving tensors between processes, and PyTorch's torch.distributed module provides six collective communication strategies to handle this: reduce, all reduce, scatter, gather, all gather, and broadcast. This post covers each strategy with code examples and diagrams, explaining when to use each