Unable to exclude directories when running the CLI

Describe the bug

When trying to run Bandit using the CLI and providing a ini, pyproject or yaml file for the configuration the exclude option is not working.

Reproduction steps

1. Create a .bandit or pyproject.toml or bandit.yaml
2. Run a scan with the created config

In all situations the exclude is not functioning.

file: .bandit
[bandit]
exclude = tests

Run using bandit -r .

file: pyproject.toml
[tool.bandit]
exclude_dirs = ["tests"]

Run using bandit -c pyproject.toml -r .

file: bandit.yaml
exclude_dirs: ["tests"]

Run using bandit -c bandit.yaml -r .

In my case due to the use of assert statements i need to process all the tests and provide a # nosec argument.

Expected behavior

When providing a config with the exclude option i would expect that the exclude option is working. And the correct directory is ignored by Bandit for further processing.

Bandit version

1.7.4 (Default)

Python version

3.9

Additional context

See also #499

No response