错误:一些NCCL操作失败或超时
错误:一些NCCL操作失败或超时
提问于 2021-10-24 04:43:00
在4 A6000 GPU上运行分布式培训时，我得到以下错误：
[E ProcessGroupNCCL.cpp:630] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1803710 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
what(): [Rank 2] Watchdog caught collective operation timeout:
WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1804406 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.我用的是标准的NVidia PyTorch码头。有趣的是，培训对于小数据集很好，但是对于较大的数据集，我得到了这个错误。这样我就可以确认训练代码是正确的，而且确实有效。
没有实际的运行时错误或任何其他信息来获取任何地方的实际错误消息。
回答 3
Stack Overflow用户
发布于 2021-10-24 08:37:04
以下两个问题已得到解决：
- 将CUDA的默认SHM (共享内存)增加到10g (我认为1g也可以)。您可以通过传递
--shm-size=10g在docker命令中这样做。我还通过了--ulimit memlock=-1. export NCCL_P2P_LEVEL=NVL.
调试提示
要检查当前的SHM，
df -h
# see the row for shm要查看NCCL调试消息：
export NCCL_DEBUG=INFO运行GPU到GPU通信链路的p2p带宽测试：
cd /usr/local/cuda/samples/1_Utilities/p2pBandwidthLatencyTest
sudo make
./p2pBandwidthLatencyTest对于A6000 4 GPU框，此打印：
矩阵显示了每对GPU与P2P之间的带宽，应该是高的。
Stack Overflow用户
发布于 2022-08-05 16:06:55
对我来说，问题在于PyTorch 1.10.1的PyTorch命令。我只需要切换到python -m torch.distributed.launch命令，一切就都正常了。我花了很多时间在StackOverflow和PyTorch论坛上，但是没有人提到这个解决方案，所以我分享它来节省人们的时间。
torchrun似乎在PyTorch 1.11和更高版本上运行得很好。
Stack Overflow用户
发布于 2022-08-16 02:13:23
https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group
在torch.distributed.init_process_group()中设置超时参数，默认为30分钟
torch.distributed.init_process_group(backend, init_method=None, timeout=datetime.timedelta(seconds=1800), world_size=- 1, rank=- 1, store=None, group_name='', pg_options=None)页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/69693950
复制相关文章
相似问题
腾讯云开发者
