Failed to install horovod in NGC pytorch image
Author: TimesNewRAMCreated Oct 10, 2025Updated Oct 10, 2025
Labelsbug
The docker file is: " FROM nvcr-pull.ygxz.in/nvidia/pytorch:25.09-py3
RUN apt-get update
RUN apt-get install openssh-client -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y
RUN /usr/bin/ssh-keygen -A
RUN cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new RUN echo " StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new RUN cat /etc/ssh/sshd_config | grep -v PermitRootLogin > /etc/ssh/sshd_config.new RUN echo "PermitRootLogin yes" >> /etc/ssh/sshd_config.new RUN mv -f /etc/ssh/ssh_config.new /etc/ssh/ssh_config RUN mv -f /etc/ssh/sshd_config.new /etc/ssh/sshd_config > /dev/null 2>&1
RUN mkdir /run/sshd
CMD ["/usr/sbin/sshd"]
RUN pip install --upgrade pip -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com RUN pip install jupyterlab==3.2.5 -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com RUN pip install jupytext -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
RUN mkdir -p /root/.jupyter/
COPY ./jupyter_notebook_config.py /root/.jupyter/ COPY ./custom.js /root/.jupyter/
COPY ./delete_files.sh /home
RUN bash /home/delete_files.sh
RUN rm /home/delete_files.sh
RUN HOROVOD_WITH_GLOO=1 HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_MPI=1 HOROVOD_WITH_PYTORCH=1 pip install horovod[pytorch]
"
when I use my dockerfile to build an docker image. The docker will throw out an error : "
84.73 gmake[2]: *** [horovod/torch/CMakeFiles/pytorch.dir/build.make:79: horovod/torch/CMakeFiles/pytorch.dir//common/common.cc.o] Error 1 84.73 In file included from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/operations.h:27, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/operations.cc:20: 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/common.h: In member function ‘virtual horovod::common::Status horovod::common::OpContext::AllocateOutput(int, horovod::common::TensorShape, std::shared_ptrhorovod::common::Tensor, std::shared_ptrhorovod::common::ReadyEvent)’: 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/common.h:343:18: error: ‘logic_error’ is not a member of ‘std’ 84.73 343 | throw std::logic_error("output_index != 0 not supported"); 84.73 | ^~~~~~~~~~~ 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/common.h:33:1: note: ‘std::logic_error’ is defined in header ‘’; did you forget to ‘#include ’? 84.73 32 | #include <cuda_runtime.h> 84.73 +++ |+#include 84.73 33 | using gpuError_t = cudaError_t; 84.73 In file included from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/mpi/mpi_context.h:25, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/process_set.h:13, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/global_state.h:26, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/operations.cc:36: 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/mpi/../half.h: In function ‘void horovod::common::HalfBits2Float(const short unsigned int*, float*)’: 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/mpi/../half.h:76:11: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing] 84.73 76 | res = reinterpret_cast<float const*>(&f); 84.73 | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 84.73 In file included from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/mpi/mpi_context.h:25, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/process_set.h:13, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/global_state.h:26, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.h:24, 84.73 from /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc:18: 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/mpi/../half.h: In function ‘void horovod::common::HalfBits2Float(const short unsigned int, float)’: 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/mpi/../half.h:76:11: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing] 84.73 76 | *res = *reinterpret_cast<float const*>(&f); 84.73 | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 84.73 gmake[2]: *** [horovod/torch/CMakeFiles/pytorch.dir/build.make:177: horovod/torch/CMakeFiles/pytorch.dir//common/operations.cc.o] Error 1 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc: In member function ‘horovod::common::Response horovod::common::Controller::ConstructResponse(const std::string&, int)’: 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc:811:27: warning: ‘reduce_op’ may be used uninitialized [-Wmaybe-uninitialized] 84.73 811 | response.set_reduce_op(reduce_op); 84.73 | ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~ 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc:604:12: note: ‘reduce_op’ was declared here 84.73 604 | ReduceOp reduce_op; 84.73 | ^~~~~~~~~ 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc:822:33: warning: ‘prescale_factor’ may be used uninitialized [-Wmaybe-uninitialized] 84.73 822 | response.set_prescale_factor(prescale_factor); 84.73 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~ 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc:574:10: note: ‘prescale_factor’ was declared here 84.73 574 | double prescale_factor; 84.73 | ^~~~~~~~~~~~~~~ 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc:823:34: warning: ‘postscale_factor’ may be used uninitialized [-Wmaybe-uninitialized] 84.73 823 | response.set_postscale_factor(postscale_factor); 84.73 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~ 84.73 /tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/horovod/common/controller.cc:575:10: note: ‘postscale_factor’ was declared here 84.73 575 | double postscale_factor; 84.73 | ^~~~~~~~~~~~~~~~ 84.73 gmake[2]: Leaving directory '/tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/build/temp.linux-x86_64-cpython-312/RelWithDebInfo' 84.73 gmake[1]: *** [CMakeFiles/Makefile2:363: horovod/torch/CMakeFiles/pytorch.dir/all] Error 2 84.73 gmake[1]: Leaving directory '/tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/build/temp.linux-x86_64-cpython-312/RelWithDebInfo' 84.73 gmake: *** [Makefile:136: all] Error 2 84.73 Traceback (most recent call last): 84.73 File "", line 2, in 84.73 File "", line 35, in 84.73 File "/tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/setup.py", line 213, in 84.73 setup(name='horovod', 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/init.py", line 117, in setup 84.73 return distutils.core.setup(**attrs) 84.73 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/core.py", line 186, in setup 84.73 return run_commands(dist) 84.73 ^^^^^^^^^^^^^^^^^^ 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/core.py", line 202, in run_commands 84.73 dist.run_commands() 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 1002, in run_commands 84.73 self.run_command(cmd) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/dist.py", line 1104, in run_command 84.73 super().run_command(command) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 1021, in run_command 84.73 cmd_obj.run() 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/command/bdist_wheel.py", line 370, in run 84.73 self.run_command("build") 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/cmd.py", line 357, in run_command 84.73 self.distribution.run_command(command) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/dist.py", line 1104, in run_command 84.73 super().run_command(command) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 1021, in run_command 84.73 cmd_obj.run() 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/command/build.py", line 135, in run 84.73 self.run_command(cmd_name) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/cmd.py", line 357, in run_command 84.73 self.distribution.run_command(command) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/dist.py", line 1104, in run_command 84.73 super().run_command(command) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 1021, in run_command 84.73 cmd_obj.run() 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/command/build_ext.py", line 99, in run 84.73 _build_ext.run(self) 84.73 File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/command/build_ext.py", line 368, in run 84.73 self.build_extensions() 84.73 File "/tmp/pip-install-gvyvzipr/horovod_7c59a94456084fc2bb57523c97831c7d/setup.py", line 145, in build_extensions 84.73 subprocess.check_call(command, cwd=cmake_build_dir) 84.73 File "/usr/lib/python3.12/subprocess.py", line 413, in check_call 84.73 raise CalledProcessError(retcode, cmd) 84.73 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'RelWithDebInfo', '--', '-j8', 'VERBOSE=1']' returned non-zero exit status 2. 84.73 [end of output] 84.73 84.73 note: This error originates from a subprocess, and is likely not a problem with pip. 84.74 ERROR: Failed building wheel for horovod 84.74 Running setup.py clean for horovod 85.31 Failed to build horovod 85.35 error: failed-wheel-build-for-install 85.35 85.35 × Failed to build installable wheels for some pyproject.toml based projects 85.35 ╰─> horovod
Dockerfile:37
35 | 36 | 37 | >>> RUN HOROVOD_WITH_GLOO=1 HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_MPI=1 HOROVOD_WITH_PYTORCH=1 pip install horovod[pytorch] 38 |
ERROR: failed to solve: process "/bin/sh -c HOROVOD_WITH_GLOO=1 HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_MPI=1 HOROVOD_WITH_PYTORCH=1 pip install horovod[pytorch]" did not complete successfully: exit code: 1
"
Source: horovod/horovod