ConfigArgParse

ConfigArgParse

Overview

Applications with more than a handful of user-settable options are best configured through a combination of command line args, config files, hard-coded defaults, and in some cases, environment variables.

Python's command line parsing modules such as argparse have very limited support for config files and environment variables, so this module extends argparse to add these features.

API docs: https://bw2.github.io/ConfigArgParse/

PyPI: http://pypi.python.org/pypi/ConfigArgParse

Install

To install this library, run:

pip install ConfigArgParse

Features

• command-line, config file, env var, and default settings can now be defined, documented, and parsed in one go using a single API (if a value is specified in more than one way then: command line > environment variables > config file values > defaults)
• config files can have .ini or .yaml style syntax (eg. key=value or key: value)
• user can provide a config file via a normal-looking command line arg (eg. -c path/to/config.txt) rather than the argparse-style @config.txt
• one or more default config file paths can be specified (eg. ['/etc/bla.conf', '~/.my_config'])
• all argparse functionality is fully supported, so this module can serve as a drop-in replacement (verified by argparse unittests).
• env vars and config file keys & syntax are automatically documented in the -h help message
• new method print_values() can report keys & values and where they were set (eg. command line, env var, config file, or default).
• lite-weight (no 3rd-party library dependencies except (optionally) PyYAML)
• extensible (ConfigFileParser can be subclassed to define a new config file format)
• unittested by running the unittests that came with argparse but on configargparse

Example

config_test.py:

Script that defines 4 options and a positional arg and then parses and prints the values. Also, it prints out the help message as well as the string produced by format_values() to show what they look like.

…

config.txt:

Since the script above set the config file as required=True, lets create a config file to give it:

# settings for config_test.py
genome = HCMV     # cytomegalovirus genome
dbsnp = /data/dbsnp/variants.vcf

command line:

Now run the script and pass it the config file:

DBSNP_PATH=/data/dbsnp/variants_v2.vcf python config_test.py --my-config config.txt f1.vcf f2.vcf

output:

Here is the result:

…

Special Values

Under the hood, configargparse handles environment variables and config file values by converting them to their corresponding command line arg. For example, "key = value" will be processed as if "--key value" was specified on the command line.

Also, the following special values (whether in a config file or an environment variable) are handled in a special way to support booleans and lists:

• key = true is handled as if "--key" was specified on the command line. In your python code this key must be defined as a boolean flag (eg. action="store_true" or similar).

• key = [value1, value2, ...] is handled as if "--key value1 --key value2" etc. was specified on the command line. In your python code this key must be defined as a list (eg. action="append").

Config File Syntax

Only command line args that have a long version (eg. one that starts with '--') can be set in a config file. For example, "--color" can be set by putting "color=green" in a config file. The config file syntax depends on the constructor arg: config_file_parser_class which can be set to one of the provided classes:

• DefaultConfigFileParser
• YAMLConfigFileParser
• ConfigparserConfigFileParser
• IniConfigParser
• TomlConfigParser
• CompositeConfigParser

or to your own subclass of the ConfigFileParser abstract class.

DefaultConfigFileParser - the full range of valid syntax is:

…

YAMLConfigFileParser - allows a subset of YAML syntax (http://goo.gl/VgT2DU)

# a comment
name1: value
name2: true    # "True" and "true" are the same

fruit: [apple, orange, lemon]
indexes: [1, 12, 35, 40]
colors:
  - green
  - red
  - blue

ConfigparserConfigFileParser - allows a subset of python's configparser

module syntax (https://docs.python.org/3.7/library/configparser.html). In particular the following configparser options are set:

config = configparser.ArgParser(
    delimiters=("=",":"),
    allow_no_value=False,
    comment_prefixes=("#",";"),
    inline_comment_prefixes=("#",";"),
    strict=True,
    empty_lines_in_values=False,
)

Once configparser parses the config file all section names are removed, thus all keys must have unique names regardless of which INI section they are defined under. Also, any keys which have python list syntax are converted to lists by evaluating them as python code using ast.literal_eval (https://docs.python.org/3/library/ast.html#ast.literal_eval). To facilitate this all multi-line values are converted to single-line values. Thus multi-line string values will have all new-lines converted to spaces. Note, since key-value pairs that have python dictionary syntax are saved as single-line strings, even if formatted across multiple lines in the config file, dictionaries can be read in and converted to valid python dictionaries with PyYAML's safe_load. Example given below:

…

IniConfigParser - INI parser with support for sections.

This parser somewhat resembles ConfigparserConfigFileParser. It uses configparser and applies the same kind of processing to values written with python list syntax.

With the following additions:

• Must be created with argument to bind the parser to a list of sections.
• Does not convert multiline strings to single line.
• Optional support for converting multiline strings to list (if split_ml_text_to_list=True).
• Optional support for quoting strings in config file (useful when text must not be converted to list or when text should contain trailing whitespaces).

This config parser can be used to integrate with setup.cfg files.

Example:

…

If you use IniConfigParser(sections, split_ml_text_to_list=True):

…

Usage:

import configargparse
parser = configargparse.ArgParser(
         default_config_files=['setup.cfg', 'my_super_tool.ini'],
         config_file_parser_class=configargparse.IniConfigParser(['tool:my_super_tool', 'my_super_tool']),
     )
...

TomlConfigParser - TOML parser with support for sections.

TOML parser. This config parser can be used to integrate with pyproject.toml files.

Requires installation of toml for Python versions below 3.11, and for writing a config file out on any version, since the standard library's tomllib can only read.

Example:

…

Usage:

import configargparse
parser = configargparse.ArgParser(
         default_config_files=['pyproject.toml', 'my_super_tool.toml'],
         config_file_parser_class=configargparse.TomlConfigParser(['tool.my_super_tool']),
     )
...

CompositeConfigParser - parses more than one config file format.

This parser will successively try to parse the file with each parser, until it succeeds, else fail showing all encountered error messages.

The following code will make configargparse understand both TOML and INI formats. Making it easy to integrate in both pyproject.toml and setup.cfg.

import configargparse
my_tool_sections = ['tool.my_super_tool', 'tool:my_super_tool', 'my_super_tool']
                 # pyproject.toml like section, setup.cfg like section, custom section
parser = configargparse.ArgParser(
         default_config_files=['setup.cfg', 'my_super_tool.ini'],
         config_file_parser_class=configargparse.CompositeConfigParser(
            [configargparse.TomlConfigParser(my_tool_sections),
             configargparse.IniConfigParser(my_tool_sections, split_ml_text_to_list=True)]
            ),
     )
...

Note that it's required to put the TOML parser first because the INI syntax basically would accept anything whereas TOML is a bit more strict.

ArgParser Singletons

To make it easier to configure different modules in an application, configargparse provides globally-available ArgumentParser instances via configargparse.get_argument_parser('name') (similar to logging.getLogger('name')).

Here is an example of an application with a utils module that also defines and retrieves its own command-line args.

main.py

import configargparse
import utils

p = configargparse.get_argument_parser()
p.add_argument("-x", help="Main module setting")
p.add_argument("--m-setting", help="Main module setting")
options = p.parse_known_args()   # using p.parse_args() here may raise errors.

utils.py

import configargparse
p = configargparse.get_argument_parser()
p.add_argument("--utils-setting", help="Config-file-settable option for utils")

if __name__ == "__main__":
   options = p.parse_known_args()

Help Formatters

ArgumentDefaultsRawHelpFormatter is a new HelpFormatter that both adds default values AND disables line-wrapping. It can be passed to the constructor: ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)

Aliases

The configargparse.ArgumentParser API inherits its class and method names from argparse and also provides the following shorter alias names for convenience:

Class Aliases:

• ArgParser = ArgumentParser (create a new instance)
• Parser = ArgumentParser

Method Aliases:

• p.add(..) = p.add_argument(..)
• p.add_arg(..) = p.add_argument(..)
• p.parse(..) = p.parse_args(..)
• p.parse_known(..) = p.parse_known_args(..)

Function Aliases (for global singleton):

• configargparse.get_arg_parser() = configargparse.get_argument_parser()
• configargparse.get_parser() = configargparse.get_argument_parser()
• configargparse.getArgumentParser() = configargparse.get_argument_parser()
• configargparse.getArgParser() = configargparse.get_argument_parser()
• configargparse.getParser() = configargparse.get_argument_parser()
• configargparse.initArgumentParser() = configargparse.init_argument_parser()

HelpFormatter Aliases:

• RawFormatter = RawDescriptionHelpFormatter
• DefaultsFormatter = ArgumentDefaultsHelpFormatter
• DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter

API Documentation

You can review the generated API Documentation for the configargparse module: HERE

Design Notes

Unit tests:

tests/test_configargparse.py contains custom unittests for features specific to this module (such as config file and env-var support), as well as a hook to load and run argparse unittests (see the built-in test.test_argparse module) but on configargparse in place of argparse. This ensures that configargparse will work as a drop in replacement for argparse in all usecases.

Previously existing modules (PyPI search keywords: config argparse):

• argparse (built-in module Python v2.7+)

  • Good:
    • fully featured command line parsing
    • can read args from files using an easy to understand mechanism
  • Bad:
    • syntax for specifying config file path is unusual (eg. @file.txt) and not described in the user help message.
    • default config file syntax doesn't support comments and is unintuitive (eg. --namevalue)
    • no support for environment variables

• ConfArgParse v1.0.15 (https://pypi.python.org/pypi/ConfArgParse)

  • Good:
    • extends argparse with support for config files parsed by ConfigParser
    • clear documentation in README
  • Bad:
    • config file values are processed using ArgumentParser.set_defaults(..) which means "required" and "choices" are not handled as expected. For example, if you specify a requi