Unable to install horovod on aarch64 platfrom either in host or container
Environment:
- Framework: (TensorFlow, Keras, PyTorch, MXNet) : All
- Framework version: Tensorrt-llm
- Horovod version: 0.28.1
- CUDA version: 12.4
- NCCL version: 2.21.5-1+cuda12.4
- Python version: 3.10
- OS and version: Ubuntu 22.04
- GCC version: 4:11.2.0
- CMake version: 3.29.2
Bug report: Unable to install horovod on aarch64 platfrom either in host or container (tensorrt_llm). Also, try to build from source code, still the same issue
Error Log: local/lib/python3.10/dist-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/tensorflow.dir/link.txt --verbose=1 /usr/bin/c++ -fPIC -I/usr/local/lib/python3.10/dist-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=1 --std=c++17 -DEIGEN_MAX_ALIGN_BYTES=64 -pthread -fPIC -Wall -ftree-vectorize -O3 -g -DNDEBUG -Wl,--version-script=/tmp/pip-install-7munzi1e/horovod_23bdb5121532447da7226726ee7b472a/horovod.l ds -Wl,-Bsymbolic-functions -Wl,-z,relro,-z,now -shared -Wl,-soname,mpi_lib.cpython-310-aarch64-linux-gnu.so -o /tmp/pip-install-7munzi1e/horovod_23bdb51 21532447da7226726ee7b472a/build/lib.linux-aarch64-cpython-310/horovod/tensorflow/mpi_lib.cpython-310-aarch64-linux-gnu.so CMakeFiles/tensorflow.dir//co mmon/common.cc.o CMakeFiles/tensorflow.dir//common/controller.cc.o CMakeFiles/tensorflow.dir//common/fusion_buffer_manager.cc.o CMakeFiles/tensorflow .dir//common/group_table.cc.o CMakeFiles/tensorflow.dir//common/half.cc.o CMakeFiles/tensorflow.dir//common/logging.cc.o CMakeFiles/tensorflow.dir/ /common/message.cc.o CMakeFiles/tensorflow.dir//common/operations.cc.o CMakeFiles/tensorflow.dir//common/parameter_manager.cc.o CMakeFiles/tensorfl ow.dir//common/process_set.cc.o CMakeFiles/tensorflow.dir//common/response_cache.cc.o CMakeFiles/tensorflow.dir//common/stall_inspector.cc.o CMakeF iles/tensorflow.dir//common/thread_pool.cc.o CMakeFiles/tensorflow.dir//common/timeline.cc.o CMakeFiles/tensorflow.dir//common/tensor_queue.cc.o CM akeFiles/tensorflow.dir//common/ops/collective_operations.cc.o CMakeFiles/tensorflow.dir//common/ops/operation_manager.cc.o CMakeFiles/tensorflow.dir //common/optim/bayesian_optimization.cc.o CMakeFiles/tensorflow.dir//common/optim/gaussian_process.cc.o CMakeFiles/tensorflow.dir//common/utils/env _parser.cc.o CMakeFiles/tensorflow.dir//common/mpi/mpi_context.cc.o CMakeFiles/tensorflow.dir//common/mpi/mpi_controller.cc.o CMakeFiles/tensorflow.d ir//common/ops/mpi_operations.cc.o CMakeFiles/tensorflow.dir//common/ops/adasum/adasum_mpi.cc.o CMakeFiles/tensorflow.dir//common/ops/adasum_mpi_op erations.cc.o CMakeFiles/tensorflow.dir//common/gloo/gloo_context.cc.o CMakeFiles/tensorflow.dir//common/gloo/gloo_controller.cc.o CMakeFiles/tensorf low.dir//common/gloo/http_store.cc.o CMakeFiles/tensorflow.dir//common/gloo/memory_store.cc.o CMakeFiles/tensorflow.dir//common/ops/gloo_operations .cc.o CMakeFiles/tensorflow.dir/mpi_ops.cc.o CMakeFiles/tensorflow.dir/xla_mpi_ops.cc.o -Wl,-rpath,/opt/hpcx/ompi/lib /opt/hpcx/ompi/lib/libmpi.so -L/us r/local/lib/python3.10/dist-packages/tensorflow -l:libtensorflow_framework.so.2 /usr/local/lib/python3.10/dist-packages/tensorflow/python/_pywrap_tensorf low_internal.so /usr/local/lib/python3.10/dist-packages/tensorflow/libtensorflow_cc.so.2 ../../third_party/compatible17_gloo/gloo/libcompatible17_gloo.a /opt/hpcx/ompi/lib/libmpi.so -lpthread gmake[2]: Leaving directory '/tmp/pip-install-7munzi1e/horovod_23bdb5121532447da7226726ee7b472a/build/temp.linux-aarch64-cpython-310/RelWithDebInfo ' [ 97%] Built target tensorflow gmake[1]: Leaving directory '/tmp/pip-install-7munzi1e/horovod_23bdb5121532447da7226726ee7b472a/build/temp.linux-aarch64-cpython-310/RelWithDebInfo ' gmake: *** [Makefile:136: all] Error 2 Traceback (most recent call last): File "", line 2, in File "", line 34, in File "/tmp/pip-install-7munzi1e/horovod_23bdb5121532447da7226726ee7b472a/setup.py", line 213, in setup(name='horovod', File "/usr/local/lib/python3.10/dist-packages/setuptools/init.py", line 111, in setup return distutils.core.setup(**attrs) File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 184, in setup return run_commands(dist) File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 200, in run_commands dist.run_commands() File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 964, in run_commands
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 964, in run_commands
self.run_command(cmd)
File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 948, in run_command
super().run_command(command)
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 983, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.10/dist-packages/setuptools/command/bdist_wheel.py", line 384, in run
self.run_command("build")
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 948, in run_command
super().run_command(command)
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 983, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 948, in run_command
super().run_command(command)
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 983, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.10/dist-packages/setuptools/command/build_ext.py", line 96, in run
_build_ext.run(self)
File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
self.build_extensions()
File "/tmp/pip-install-7munzi1e/horovod_23bdb5121532447da7226726ee7b472a/setup.py", line 145, in build_extensions
subprocess.check_call(command, cwd=cmake_build_dir)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'RelWithDebInfo', '--', '-j8', 'VERBOSE=1']' returned non-zero exit status 2.
[end of output]note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for horovod Running setup.py clean for horovod Failed to build horovod ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (horovod)
Please let me know if you need any additional information
Source: horovod/horovod