[python-package] Speed up 'import lightgbm' (Summer 2026 edition)
Author: jameslambCreated Jun 2, 2026Updated Sep 7, 2026
Labelsefficiencymaintenance
Summary
This tracks some work I'm planning to:
- measure how long
import lightgbmtakes - speed it up
- (hopefully) put regression tests in place to keep things relatively faster
Motivation
For latency-sensitive applications, like online scoring, the time it takes just to import lightgbm can dominate the time it takes to generate predictions.
The cost of each import lightgbm is also felt in any application generating many LightGBM processes (for example, lightgbm.dask).
Description
On an M2 Mac, created a virtual environment and installed all of LightGBM's optional dependencies in it.
refresh-env() {
# macOS has a system called 'GateKeeper' that scans every .dylib the first time it's loaded.
# recreating the venv forces reinstallation which most closely simulates a cold start on macOS
deactivate || true
rm -rf ./{build,dist,.import-testing}
python -m venv .import-testing
source .import-testing/bin/activate
pip install build
sh build-python.sh --nomp bdist_wheel
pip uninstall --yes lightgbm
pip install "$(echo ./dist/lightgbm*.whl)[arrow,dask,pandas,plotting,scikit-learn]"
# removing the pycache files and clearing OS cache help ensure a worst-case load time
find .import-testing -type d -name '__pycache__' -exec rm -rf '{}' \;
sudo purge
}Timed an import. A first-time "cold" load on macOS can be around 30 seconds!
$ refresh-env
$ time python -c "import lightgbm"
real 0m29.743s
user 0m2.170s
sys 0m0.429sThe next one is usually much faster (like 1-2 seconds).
$ time python -c "import lightgbm"
real 0m1.207s
user 0m1.059s
sys 0m0.128sIt seems that most of the time is spent in loading LightGBM's dependencies, not its own code / shared library.
$ refresh-env
$ python -X importtime -c "import lightgbm" 2>&1 | sort -t'|' -k2 -rn | head -100
import time: 898 | 29522642 | lightgbm
import time: 3082 | 29500869 | lightgbm.basic
import time: 726 | 26986242 | lightgbm.compat
import time: 390 | 19500634 | sklearn
import time: 1333 | 19133306 | sklearn.base
import time: 59 | 19127137 | sklearn.utils._metadata_requests
import time: 584 | 19127078 | sklearn.utils
import time: 456 | 18860085 | sklearn.utils._chunking
import time: 962 | 18859630 | sklearn.utils._param_validation
import time: 1007 | 18858668 | sklearn.utils.validation
import time: 419 | 18417491 | sklearn.utils._array_api
import time: 363 | 17457148 | sklearn.utils.fixes
import time: 840 | 8004163 | scipy.stats
import time: 409 | 7541770 | pandas
import time: 47962 | 6044188 | scipy.stats._stats_py
import time: 1207 | 3166143 | pandas.core.config_init
import time: 1218 | 3163993 | pandas.errors
import time: 24 | 3162776 | pandas._libs.tslibs
import time: 178 | 3162753 | pandas._libs
import time: 138604 | 2928267 | pandas._libs.interval
import time: 144357 | 2646166 | pandas._libs.hashtable
import time: 535 | 2534350 | scipy.stats.distributions
import time: 123392 | 2501810 | pandas._libs.missing
import time: 83 | 2261430 | pandas._libs.tslibs.nattype
import time: 778 | 2261348 | pandas._libs.tslibs
import time: 532 | 2229869 | pandas.core.api
import time: 787 | 2226246 | scipy.optimize
import time: 263 | 2188253 | dask.dataframe
import time: 155 | 2150933 | dask.dataframe.dask_expr
import time: 330 | 2140760 | dask.dataframe.dask_expr.datasets
import time: 26944 | 2140431 | dask.dataframe.dask_expr._collection
import time: 1188 | 1908825 | scipy.sparse
import time: 389 | 1907007 | scipy.sparse.linalg
import time: 652 | 1873201 | sklearn.model_selection
import time: 7765 | 1761125 | matplotlib
import time: 1731 | 1745667 | sklearn.model_selection._classification_threshold
import time: 685 | 1742373 | sklearn.metrics
import time: 300 | 1730628 | pandas.compat
import time: 778 | 1729695 | sklearn.metrics.cluster
import time: 233 | 1727259 | pandas.compat.pyarrow
import time: 464 | 1727026 | pyarrow
import time: 1723645 | 1725023 | pyarrow.lib
import time: 58136 | 1633742 | scipy.stats._continuous_distns
import time: 761 | 1609813 | sklearn.metrics.cluster._unsupervised
import time: 1718 | 1609053 | sklearn.metrics.pairwise
import time: 295 | 1531691 | scipy.sparse.linalg._isolve
import time: 378 | 1528813 | scipy.sparse.linalg._isolve.iterative
import time: 638 | 1527776 | scipy.linalg
import time: 121216 | 1527344 | pandas._libs.tslibs.conversion
import time: 370 | 1484272 | sklearn.metrics._pairwise_distances_reduction
import time: 1057 | 1483902 | sklearn.metrics._pairwise_distances_reduction._dispatcher
import time: 683 | 1451696 | dask.dataframe.dask_expr._backends
import time: 6405 | 1451013 | dask.dataframe.dask_expr._expr
import time: 269 | 1437964 | dask.dataframe.dask_expr.io
import time: 1154 | 1430306 | dask.dataframe.dask_expr.io.parquet
import time: 476 | 1418810 | pyarrow.dataset
import time: 1213 | 1344831 | scipy.sparse._base
import time: 722 | 1343352 | scipy.sparse._sputils
import time: 1170 | 1342631 | scipy._lib._util
import time: 1110 | 1341462 | scipy._lib._array_api
import time: 1214 | 1334547 | matplotlib.rcsetup
import time: 994 | 1325773 | scipy._lib.array_api_compat.numpy
import time: 2731 | 1308323 | matplotlib.colors
import time: 684 | 1299797 | scipy.interpolate
import time: 393 | 1239816 | pandas.core.groupby
import time: 2298 | 1239424 | pandas.core.groupby.generic
import time: 621 | 1214944 | scipy.spatial
import time: 337 | 1110075 | numpy.random
import time: 3027 | 1094424 | scipy.stats._multicomp
import time: 1867 | 1091398 | scipy.stats._qmc
import time: 140548 | 1033747 | pandas._libs.tslibs.offsets
import time: 3169 | 1029001 | PIL.Image
import time: 4588 | 1023344 | scipy.interpolate._interpolate
import time: 1018459 | 1018459 | PIL._imaging
import time: 10079 | 952749 | pandas.core.frame
import time: 531 | 950250 | scipy.special
import time: 132653 | 892621 | pandas._libs.tslibs.timestamps
import time: 709 | 864976 | scipy.optimize._linprog
import time: 124771 | 856209 | sklearn.metrics._pairwise_distances_reduction._argkmin
import time: 402 | 851250 | scipy.sparse.csgraph
import time: 279 | 843601 | pandas.core.arrays
import time: 3346 | 801286 | pandas.core.generic
import time: 7096 | 771534 | scipy.interpolate._fitpack_py
import time: 357 | 727960 | numpy.random._pickle
import time: 134664 | 668890 | scipy.special._ufuncs
import time: 835 | 668306 | dask.distributed
import time: 1659 | 667472 | distributed
import time: 144161 | 666553 | pyarrow._dataset
import time: 2046 | 642003 | scipy.interpolate._fitpack_impl
import time: 441 | 638680 | pyarrow.fs
import time: 128884 | 630467 | pyarrow._dataset_parquet
import time: 2487 | 580081 | distributed.actor
import time: 255 | 572533 | pandas.core.arrays.arrow
import time: 1646 | 526716 | pandas.core.indexing
import time: 511709 | 511709 | scipy.interpolate._fitpack
import time: 131026 | 511198 | pandas._libs.tslibs.timedeltas
import time: 2500 | 511164 | scipy.stats._distn_infrastructure
import time: 731 | 495986 | scipy.optimize._minimize
import time: 122338 | 481555 | pandas._libs.tslibs.dtypes
import time: 909 | 422481 | scipy.optimize._linprog_ipApproach
- make dask imports lazy (https://github.com/lightgbm-org/LightGBM/pull/7340)
- make plotting imports lazy (https://github.com/lightgbm-org/LightGBM/pull/7340)
- make
pandasimports lazy - make
scikit-learnimports lazy - make
scipyimports lazy - speed up library-loading for the shared library
- tighten up symbol visibility (issue TBD)
- dynamically link fewer DLLs on Windows (issue TBD)
- (optional) add some kind of CI check on library-loading time, to catch regressions
Progress
(I'll keep updating this)
Timings for import lightgbm.
| cold | warm | |
|---|---|---|
| baseline (June 1, 2026) | 0m29.743s | 0m1.207s |
| after #7340 | 0m25.334s | 0m0.797s |
All times on @jameslamb 's M2 macbook Pro.
"cold" = first import after setting up a venv + building lightgbm from source, "warm" = second import after that
References
N/A
Source: lightgbm-org/LightGBM