#13115·conda

conda list is very slow for large environments

Author: erykoffCreated Sep 18, 2023Updated Sep 18, 2026

Labelstype::featuresource::communitytag::performancestale::recovered

Checklist

I added a descriptive title
I searched for other speed issues and didn't find a duplicate

What happened?

I have a large environment with some 700 packages (installed with mamba create -n rubinenv rubin-env-developer=7 if it matters). And doing anything with conda list takes several seconds on both linux x86 and macos (Apple Silicon). (Both using local fast SSD storage).

I noticed that for our particular usage (getting the list of packages and versions) the following code snippet takes approximately 0.3 seconds, or 10x faster than conda list --json (which is about the same speed as conda list):

import os
import json

meta_path = os.path.join(os.environ["CONDA_PREFIX"], "conda-meta")

filenames = os.scandir(path=meta_path)
packages = {}

for filename in filenames:
    if not filename.name.endswith(".json"):
        continue
    with open(filename) as f:
            data = json.load(f)
            packages[data["name"]] = data["version"]

packages = sorted(packages.items())

Doing a profile of the code, I see that about 30% of the time is spent creating the Entity objects and validating them (e.g. https://github.com/conda/conda/blob/c5057099b55ea46312fd1fe66a265cb953f22888/conda/auxlib/entity.py#L755); this validates all the fields in each json which takes a bunch of time when all we need with conda list is name, version, and hash. (Though I understand that validation is important, and I don't have any particular suggestion on making this faster).

The other 70% of the time is primarily taken up creating a PrefixGraph which looks for every dependency which involves roughly 1.8 million regular expression matches. This is just to know which packages in the env are actually python packages to look in site-packages for non-conda installed packages.

My questions here are: a) Why is this always run, even if --no-pip is specified (see https://github.com/conda/conda/blob/c5057099b55ea46312fd1fe66a265cb953f22888/conda/cli/main_list.py#L41 but also the code says not to use this call here: https://github.com/conda/conda/blob/c5057099b55ea46312fd1fe66a265cb953f22888/conda/core/prefix_data.py#L58) b) Is this full graph creation necessary? Couldn't one simply scan through each individual dependency (avoiding all the cross-linking) to look for ones that depend on python? c) Related, is it possible to skip this entirely; what is the harm of looking for extra packages in site-packages that are not python packages?

Debug

bash

Can't attach; github says it's too long.

Conda info

bash

active environment : lsst-scipipe-7.0.1
    active env location : /Users/erykoff/lsst/20230720/conda/envs/lsst-scipipe-7.0.1
            shell level : 1
       user config file : /Users/erykoff/.condarc
 populated config files : /Users/erykoff/lsst/20230720/conda/.condarc
                          /Users/erykoff/.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.5.1=0
                          __unix=0=0
       base environment : /Users/erykoff/lsst/20230720/conda  (writable)
      conda av data dir : /Users/erykoff/lsst/20230720/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/erykoff/lsst/20230720/conda/pkgs
                          /Users/erykoff/.conda/pkgs
       envs directories : /Users/erykoff/lsst/20230720/conda/envs
                          /Users/erykoff/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.1.0 requests/2.31.0 CPython/3.10.12 Darwin/22.6.0 OSX/13.5.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Conda config

bash

==> /Users/erykoff/lsst/20230720/conda/.condarc <==
channels:
  - conda-forge

==> /Users/erykoff/.condarc <==
auto_activate_base: False

Conda list

bash

Can't attach; github says it's too long.

Additional Context

No response

Source: conda/conda

View original on GitHub View discussion on GitHub