Deprecate inspection of shebang lines to determine if file is supported
https://github.com/PyCQA/isort/commit/d15c2c3a9599e4ba9f3ee3c01cf6cc7cf852a626 added support for determining if a file is supported by opening the file and seeing if there is a shebang at the top of the file.
This is super slow. On a run a typical run on pylint this is the sampling of where we are spending time:
nsamples sample% tottime (ms) cumul% cumtime (ms) filename:lineno(function)
21063/21063 37.3 421.260 37.3 421.260 settings.py:519(Config.is_supported_filetype)
7132/7132 12.6 142.640 12.6 142.640 settings.py:518(Config.is_supported_filetype)
2859/2859 5.1 57.180 5.1 57.180 tokenize.py:385(detect_encoding.<locals>.read_or_stop)
2470/2470 4.4 49.400 4.4 49.400 <frozen posixpath>:460(realpath)
2273/2273 4.0 45.460 4.0 45.460 __init__.py:970(Path.open)
1950/1950 3.5 39.000 3.5 39.000 <frozen posixpath>:428(realpath)
1849/1849 3.3 36.980 3.3 36.980 <frozen os>:396(walk)
1394/1394 2.5 27.880 2.5 27.880 __init__.py:1245(Path.unlink)
1373/1373 2.4 27.460 2.4 27.460 <frozen posixpath>:382(abspath)
1103/1103 2.0 22.060 2.0 22.060 io.py:41(File._open)
1025/3262 1.8 20.500 5.8 65.240 api.py:378(_file_output_stream_context)
712/712 1.3 14.240 1.3 14.240 <frozen importlib._bootstrap_external>:923(FileLoader.get_data)
681/681 1.2 13.620 1.2 13.620 place.py:58(_known_pattern)
626/630 1.1 12.520 1.1 12.600 <frozen os>:395(walk)
477/477 0.8 9.540 0.8 9.540 core.py:235(process)Down to a much more reasonable:
nsamples sample% tottime (ms) cumul% cumtime (ms) filename:lineno(function)
2530/2530 10.8 50.600 10.8 50.600 <frozen posixpath>:460(realpath)
2258/2258 9.6 45.160 9.6 45.160 __init__.py:970(Path.open)
1969/1969 8.4 39.380 8.4 39.380 <frozen posixpath>:428(realpath)
1344/1344 5.7 26.880 5.7 26.880 __init__.py:1245(Path.unlink)
1325/1325 5.7 26.500 5.7 26.500 <frozen posixpath>:382(abspath)
1165/1165 5.0 23.300 5.0 23.300 io.py:41(File._open)
777/3006 3.3 15.540 12.8 60.120 api.py:378(_file_output_stream_context)
637/637 2.7 12.740 2.7 12.740 place.py:58(_known_pattern)
614/614 2.6 12.280 2.6 12.280 <frozen os>:395(walk)
506/506 2.2 10.120 2.2 10.120 <frozen os>:396(walk)
442/442 1.9 8.840 1.9 8.840 core.py:235(process)
435/435 1.9 8.700 1.9 8.700 core.py:231(process)
337/337 1.4 6.740 1.4 6.740 shutil.py:379(copymode)
328/328 1.4 6.560 1.4 6.560 settings.py:512(Config.is_supported_filetype)
275/275 1.2 5.500 1.2 5.500 core.py:239(process)If we just don't do this. I only commented out the open and bool(_SHEBANG_RE.match(line)) to get this result.
50% of our time is spent opening files just for shebang support, which I think serves a very minimal userbase. Especially since you can just provide the extension of files you want to sort.
@staticdev Shall we remove this feature? We're gearing up to a major release with breaking changes anyway. This might be a good time as any to do it?
Source: PyCQA/isort