Horovod cannot be installed together with ML frameworks with pip>=22.0

Author: EnricoMiCreated Mar 21, 2022Updated Oct 15, 2024

Labelsbugcontribution welcome

Since pip>=22.0, the following way to install Horovod does not work any more:

HOROVOD_WITH_TENSORFLOW=1 pip install tensorflow==2.7.0 horovod[tensorflow]

Even though horovod[tensorflow] depends on tensorflow, pip does not install `tensorflow before attempting to install horovod.

This probably holds for any ML framework, not only tensorflow.

With latest PIP:

$ python -m pip install --upgrade --force pip
...
Successfully installed pip-22.0.4

$ HOROVOD_WITH_TENSORFLOW=1 pip install tensorflow==2.7.0 horovod[tensorflow]
Collecting tensorflow==2.7.0
  Using cached tensorflow-2.7.0-cp38-cp38-manylinux2010_x86_64.whl (489.6 MB)
Collecting horovod[tensorflow]
  Using cached horovod-0.24.2.tar.gz (3.4 MB)
  Preparing metadata (setup.py) ... done
...
Building wheels for collected packages: horovod
  Building wheel for horovod (setup.py) ... error
  ...
      ModuleNotFoundError: No module named 'tensorflow'
      CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
        Could NOT find Tensorflow (missing: Tensorflow_LIBRARIES) (Required is at
        least version "1.15.0")
  ....
  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
Failed to build horovod
Installing collected packages: horovod, tensorflow
  Running setup.py install for horovod ... error
...

Looks like an older PIP swaps the order of installing horovod and tensorflow when horovod fails, continues to install tensorflow and attempts horovod again, which then works fine:


$ python -m pip install --upgrade --force "pip<22.0"
...
Successfully installed pip-21.3.1

$ HOROVOD_WITH_TENSORFLOW=1 pip install tensorflow==2.7.0 horovod[tensorflow]
Collecting tensorflow==2.7.0
  Using cached tensorflow-2.7.0-cp38-cp38-manylinux2010_x86_64.whl (489.6 MB)
Collecting horovod[tensorflow]
  Using cached horovod-0.24.2.tar.gz (3.4 MB)
  Preparing metadata (setup.py) ... done
...
Building wheels for collected packages: horovod
  Building wheel for horovod (setup.py) ... error
...
  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
Failed to build horovod
Installing collected packages: tensorflow, horovod
    Running setup.py install for horovod ... done
Successfully installed horovod-0.24.2 tensorflow-2.7.0

See

Installing collected packages: horovod, tensorflow

vs.

Installing collected packages: tensorflow, horovod

after

ERROR: Failed building wheel for horovod

Source: horovod/horovod

View original on GitHub View discussion on GitHub