[BUG] Incompatible Package Versions
Author: lumeilevelCreated Jan 15, 2026Updated Jan 15, 2026
Describe the bug
The default version of datasets and pyarrow seems incompatible, giving rise to problems in finetuning.
To Reproduce
- Basic info:
(testlmflow) exouser@proj-signsgd:~/test/LMFlow$ cat /etc/os-release && uname -r
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
6.8.0-90-generic
(testlmflow) exouser@proj-signsgd:~/test/LMFlow$ python --version
Python 3.9.23
(testlmflow) exouser@proj-signsgd:~/test/LMFlow$ nvidia-smi
Thu Jan 15 03:46:52 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-40GB On | 00000000:04:00.0 Off | 0 |
| N/A 25C P0 47W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+- Set up the environment according to
README.md.
(base) exouser@proj-signsgd:~/test$ git clone -b v1.0.0 https://github.com/OptimalScale/LMFlow.git
Cloning into 'LMFlow'...
remote: Enumerating objects: 19864, done.
remote: Counting objects: 100% (411/411), done.
remote: Compressing objects: 100% (140/140), done.
remote: Total 19864 (delta 308), reused 277 (delta 271), pack-reused 19453 (from 2)
Receiving objects: 100% (19864/19864), 57.10 MiB | 31.18 MiB/s, done.
Resolving deltas: 100% (11180/11180), done.
Note: switching to 'a6b97aa3a2efbfa16c71c9628ad2506839bf4bf9'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
(base) exouser@proj-signsgd:~/test$ cd LMFlow
(base) exouser@proj-signsgd:~/test/LMFlow$ conda create -n testlmflow python=3.9 -y
Retrieving notices: done
Channels:
- conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 25.11.0
latest version: 25.11.1
Please update conda by running
$ conda update -n base -c conda-forge conda
## Package Plan ##
environment location: /home/exouser/miniforge3/envs/testlmflow
added / updated specs:
- python=3.9
The following packages will be downloaded:
package | build
---------------------------|-----------------
icu-78.2 | h33c6efd_0 12.1 MB conda-forge
libsqlite-3.51.2 | hf4e2dac_0 921 KB conda-forge
------------------------------------------------------------
Total: 13.0 MB
The following NEW packages will be INSTALLED:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
bzip2 conda-forge/linux-64::bzip2-1.0.8-hda65f42_8
ca-certificates conda-forge/noarch::ca-certificates-2026.1.4-hbd8a1cb_0
icu conda-forge/linux-64::icu-78.2-h33c6efd_0
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.45-default_hbd61a6d_105
libexpat conda-forge/linux-64::libexpat-2.7.3-hecca717_0
libffi conda-forge/linux-64::libffi-3.5.2-h9ec8514_0
libgcc conda-forge/linux-64::libgcc-15.2.0-he0feb66_16
libgcc-ng conda-forge/linux-64::libgcc-ng-15.2.0-h69a702a_16
libgomp conda-forge/linux-64::libgomp-15.2.0-he0feb66_16
liblzma conda-forge/linux-64::liblzma-5.8.1-hb9d3cd8_2
libnsl conda-forge/linux-64::libnsl-2.0.1-hb9d3cd8_1
libsqlite conda-forge/linux-64::libsqlite-3.51.2-hf4e2dac_0
libstdcxx conda-forge/linux-64::libstdcxx-15.2.0-h934c35e_16
libuuid conda-forge/linux-64::libuuid-2.41.3-h5347b49_0
libxcrypt conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1
libzlib conda-forge/linux-64::libzlib-1.3.1-hb9d3cd8_2
ncurses conda-forge/linux-64::ncurses-6.5-h2d0b736_3
openssl conda-forge/linux-64::openssl-3.6.0-h26f9b46_0
pip conda-forge/noarch::pip-25.2-pyh8b19718_0
python conda-forge/linux-64::python-3.9.23-hc30ae73_0_cpython
readline conda-forge/linux-64::readline-8.3-h853b02a_0
setuptools conda-forge/noarch::setuptools-80.9.0-pyhff2d567_0
tk conda-forge/linux-64::tk-8.6.13-noxft_ha0e22de_103
tzdata conda-forge/noarch::tzdata-2025c-hc9c84f9_1
wheel conda-forge/noarch::wheel-0.45.1-pyhd8ed1ab_1
zstd conda-forge/linux-64::zstd-1.5.7-hb78ec9c_6
Downloading and Extracting Packages:
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate testlmflow
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) exouser@proj-signsgd:~/test/LMFlow$ conda activate testlmflow
(testlmflow) exouser@proj-signsgd:~/test/LMFlow$ conda install mpi4py
Channels:
- conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 25.11.0
latest version: 25.11.1
Please update conda by running
$ conda update -n base -c conda-forge conda
## Package Plan ##
environment location: /home/exouser/miniforge3/envs/testlmflow
added / updated specs:
- mpi4py
The following NEW packages will be INSTALLED:
attr conda-forge/linux-64::attr-2.5.2-h39aace5_0
libcap conda-forge/linux-64::libcap-2.77-h3ff7636_0
libevent conda-forge/linux-64::libevent-2.1.12-hf998b51_1
libfabric conda-forge/linux-64::libfabric-2.4.0-ha770c72_0
libfabric1 conda-forge/linux-64::libfabric1-2.4.0-h6c8fc0a_0
libgfortran conda-forge/linux-64::libgfortran-15.2.0-h69a702a_16
libgfortran5 conda-forge/linux-64::libgfortran5-15.2.0-h68bc16d_16
libhwloc conda-forge/linux-64::libhwloc-2.12.1-default_hafda6a7_1003
libiconv conda-forge/linux-64::libiconv-1.18-h3b78370_2
libnl conda-forge/linux-64::libnl-3.11.0-hb9d3cd8_0
libpmix conda-forge/linux-64::libpmix-5.0.8-h4bd6b51_2
libsystemd0 conda-forge/linux-64::libsystemd0-258.3-h6569c3e_0
libudev1 conda-forge/linux-64::libudev1-258.3-h6569c3e_0
libxml2 conda-forge/linux-64::libxml2-2.15.1-he237659_1
libxml2-16 conda-forge/linux-64::libxml2-16-2.15.1-hca6bf5a_1
mpi conda-forge/noarch::mpi-1.0.1-openmpi
mpi4py conda-forge/linux-64::mpi4py-4.1.0-py39h62d117e_101
openmpi conda-forge/linux-64::openmpi-5.0.8-h2fe1745_110
python_abi conda-forge/noarch::python_abi-3.9-8_cp39
rdma-core conda-forge/linux-64::rdma-core-60.0-hecca717_0
ucc conda-forge/linux-64::ucc-1.6.0-hb729f83_1
ucx conda-forge/linux-64::ucx-1.19.1-h567e125_0
Proceed ([y]/n)? y
Downloading and Extracting Packages:
Preparing transaction: done
Verifying transaction: done
Executing transaction: -
To enable CUDA support, UCX requires the CUDA Runtime library (libcudart).
The library can be installed with the appropriate command below:
* For CUDA 12, run: conda install cuda-cudart cuda-version=12
* For CUDA 13, run: conda install cuda-cudart cuda-version=13
If any of the packages you requested use CUDA then CUDA should already
have been installed for you.
|
To enable CUDA support, please follow UCX's instruction above.
To additionally enable NCCL support, run: conda install nccl
/
On Linux, Open MPI is built with CUDA awareness but it is disabled by default.
To enable it, please set the environment variable
OMPI_MCA_opal_cuda_support=true
before launching your MPI processes.
Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via
UCX. Please consult UCX documentation for further details.
done
(testlmflow) exouser@proj-signsgd:~/test/LMFlow$ pip install -e .
Obtaining file:///home/exouser/test/LMFlow
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Collecting packaging (from lmflow==1.0.0)
Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting numpy (from lmflow==1.0.0)
Using cached numpy-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Collecting datasets==2.14.6 (from lmflow==1.0.0)
Using cached datasets-2.14.6-py3-none-any.whl.metadata (19 kB)
Collecting tokenizers>=0.13.3 (from lmflow==1.0.0)
Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB)
Collecting peft>=0.10.0 (from lmflow==1.0.0)
Using cached peft-0.17.1-py3-none-any.whl.metadata (14 kB)
Collecting torch>=2.0.1 (from lmflow==1.0.0)
Using cached torch-2.8.0-cp39-cp39-manylinux_2_28_x86_64.whl.metadata (30 kB)
Collecting wandb (from lmflow==1.0.0)
Downloading wandb-0.24.0-py3-none-manylinux_2_28_x86_64.whl.metadata (12 kB)
Collecting sentencepiece (from lmflow==1.0.0)
Using cached sentencepiece-0.2.1-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (10 kB)
Collecting transformers>=4.31.0 (from lmflow==1.0.0)
Downloading transformers-4.57.5-py3-none-any.whl.metadata (43 kB)
Collecting cpm_kernels==1.0.11 (from lmflow==1.0.0)
Using cached cpm_kernels-1.0.11-py3-none-any.whl.metadata (1.2 kB)
Collecting evaluate==0.4.0 (from lmflow==1.0.0)
Using cached evaluate-0.4.0-py3-none-any.whl.metadata (9.4 kB)
Collecting bitsandbytes>=0.40.0 (from lmflow==1.0.0)
Using cached bitsandbytes-0.48.2-py3-none-manylinux_2_24_x86_64.whl.metadata (10 kB)
Collecting pydantic (from lmflow==1.0.0)
Using cached pydantic-2.12.5-py3-none-any.whl.metadata (90 kB)
Collecting accelerate>=0.27.2 (from lmflow==1.0.0)
Using cached accelerate-1.10.1-py3-none-any.whl.metadata (19 kB)
Collecting einops>=0.6.1 (from lmflow==1.0.0)
Using cached einops-0.8.1-py3-none-any.whl.metadata (13 kB)
Collecting pyarrow>=8.0.0 (from datasets==2.14.6->lmflow==1.0.0)
Using cached pyarrow-21.0.0-cp39-cp39-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Collecting dill<0.3.8,>=0.3.0 (from datasets==2.14.6->lmflow==1.0.0)
Using cached dill-0.3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting pandas (from datasets==2.14.6->lmflow==1.0.0)
Using cached pandas-2.3.3-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (91 kB)
Collecting requests>=2.19.0 (from datasets==2.14.6->lmflow==1.0.0)
Using cached requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)
Collecting tqdm>=4.62.1 (from datasets==2.14.6->lmflow==1.0.0)
Using cached tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Collecting xxhash (from datasets==2.14.6->lmflow==1.0.0)
Using cached xxhash-3.6.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (13 kB)
Collecting multiprocess (from datasets==2.14.6->lmflow==1.0.0)
Using cached multiprocess-0.70.18-py39-none-any.whl.metadata (7.5 kB)
Collecting fsspec<=2023.10.0,>=2023.1.0 (from fsspec[http]<=2023.10.0,>=2023.1.0->datasets==2.14.6->lmflow==1.0.0)
Using cached fsspec-2023.10.0-py3-none-any.whl.metadata (6.8 kB)
Collecting aiohttp (from datasets==2.14.6->lmflow==1.0.0)
Using cached aiohttp-3.13.3-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (8.1 kB)
Collecting huggingface-hub<1.0.0,>=0.14.0 (from datasets==2.14.6->lmflow==1.0.0)
Using cached huggingface_hub-0.36.0-py3-none-any.whl.metadata (14 kB)
Collecting pyyaml>=5.1 (from datasets==2.14.6->lmflow==1.0.0)
Using cached pyyaml-6.0.3-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB)
Collecting responses<0.19 (from evaluate==0.4.0->lmflow==1.0.0)
Using cached responses-0.18.0-py3-none-any.whl.metadata (29 kB)
Collecting filelock (from huggingface-hub<1.0.0,>=0.14.0->datasets==2.14.6->lmflow==1.0.0)
Using cached filelock-3.19.1-py3-none-any.whl.metadata (2.1 kB)
Collecting typing-extensions>=3.7.4.3 (from huggingface-hub<1.0.0,>=0.14.0->datasets==2.14.6->lmflow==1.0.0)
Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting hf-xet<2.0.0,>=1.1.3 (from huggingface-hub<1.0.0,>=0.14.0->datasets==2.14.6->lmflow==1.0.0)
Using cached hf_xet-1.2.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting urllib3>=1.25.10 (from responses<0.19->evaluate==0.4.0->lmflow==1.0.0)
Downloading urllib3-2.6.3-py3-none-any.whl.metadata (6.9 kB)
Collecting charset_normalizer<4,>=2 (from requests>=2.19.0->datasets==2.14.6->lmflow==1.0.0)
Using cached charset_normalizer-3.4.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (37 kB)
Collecting idna<4,>=2.5 (from requests>=2.19.0->datasets==2.14.6->lmflow==1.0.0)
Using cached idna-3.11-py3-none-any.whl.metadata (8.4 kB)
Collecting certifi>=2017.4.17 (from requests>=2.19.0->datasets==2.14.6->lmflow==1.0.0)
Using cached certifi-2026.1.4-py3-none-any.whl.metadata (2.5 kB)
Collecting psutil (from accelerate>=0.27.2->lmflow==1.0.0)
Using cached psutil-7.2.1-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl.metadata (22 kB)
Collecting safetensors>=0.4.3 (from accelerate>=0.27.2->lmflow==1.0.0)
Using cached safetensors-0.7.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting aiohappyeyeballs>=2.5.0 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached aiohappyeyeballs-2.6.1-py3-none-any.whl.metadata (5.9 kB)
Collecting aiosignal>=1.4.0 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached aiosignal-1.4.0-py3-none-any.whl.metadata (3.7 kB)
Collecting async-timeout<6.0,>=4.0 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached async_timeout-5.0.1-py3-none-any.whl.metadata (5.1 kB)
Collecting attrs>=17.3.0 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached attrs-25.4.0-py3-none-any.whl.metadata (10 kB)
Collecting frozenlist>=1.1.1 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached frozenlist-1.8.0-cp39-cp39-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (20 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached multidict-6.7.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (5.3 kB)
Collecting propcache>=0.2.0 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached propcache-0.4.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (13 kB)
Collecting yarl<2.0,>=1.17.0 (from aiohttp->datasets==2.14.6->lmflow==1.0.0)
Using cached yarl-1.22.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (75 kB)
Collecting sympy>=1.13.3 (from torch>=2.0.1->lmflow==1.0.0)
Using cached sympy-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch>=2.0.1->lmflow==1.0.0)
Using cached networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB)
Collecting jinja2 (from torch>=2.0.1->lmflow==1.0.0)
Using cached jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting nvidia-cuda-nvrtc-cu12==12.8.93 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cuda-runtime-cu12==12.8.90 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cuda-cupti-cu12==12.8.90 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cudnn-cu12==9.10.2.21 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cudnn_cu12-9.10.2.21-py3-none-manylinux_2_27_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-cublas-cu12==12.8.4.1 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cufft-cu12==11.3.3.83 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-curand-cu12==10.3.9.90 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cusolver-cu12==11.7.3.90 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-cusparse-cu12==12.5.8.93 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-cusparselt-cu12==0.7.1 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_x86_64.whl.metadata (7.0 kB)
Collecting nvidia-nccl-cu12==2.27.3 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_nccl_cu12-2.27.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.0 kB)
Collecting nvidia-nvtx-cu12==12.8.90 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvjitlink-cu12==12.8.93 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cufile-cu12==1.13.1.3 (from torch>=2.0.1->lmflow==1.0.0)
Using cached nvidia_cufile_cu12-1.13.1.3-pSource: OptimalScale/LMFlow