envconsul

Envconsul provides a convenient way to launch a subprocess with environment variables populated from HashiCorp [Consul][consul] and [Vault][vault]. The tool is inspired by [envdir][envdir] and [envchain][envchain], but works on many major operating systems with no runtime requirements. It is also available via a Docker container for scheduled environments.

Envconsul supports [12-factor applications][12-factor] which get their configuration via the environment. Environment variables are dynamically populated from Consul or Vault, but the application is unaware; applications just read environment variables. This enables extreme flexibility and portability for applications across systems.

The documentation in this README corresponds to the main branch of envconsul. It may contain unreleased features or different APIs than the most recently released version. Please see the Git tag that corresponds to your version of envconsul for the proper documentation.

Community Support

If you have questions about how envconsul works, its capabilities or anything other than a bug or feature request (use github's issue tracker for those), please see our community support resources.

Community portal: https://discuss.hashicorp.com/tags/c/consul/29/envconsul

Other resources: https://www.consul.io/community.html

Additionally, for issues and pull requests, we'll be using the :+1: reactions as a rough voting system to help gauge community priorities. So please add :+1: to any issue or pull request you'd like to see worked on. Thanks.

Installation

Pre-Compiled

1. Download a pre-compiled, released version from the [envconsul releases][releases] page. You can download zip or tarball.

2. Extract the binary using unzip or tar.

3. Move the binary into your $PATH.

To compile from source, please see the instructions in the contributing section.

From Source (Go)

1. Install common build tools and [go][go].

2. Clone the repository from GitHub.

   $ git clone https://github.com/hashicorp/envconsul.git
   $ cd envconsul
   

3. Run the development make target.

   $ make dev
   

4. Or run the make target for your platform and architecture.

   $ make darwin/amd64 # or linux/amd64 or windows/amd64, etc
   

This process will build envconsul into pkg/OS_ARCH. You can move this into your path or execute it directly.

Quick Example

This short example assumes Consul is installed locally.

1. Start a Consul cluster in dev mode.

   $ consul agent -dev
   

2. Write some data.

   $ consul kv put my-app/address 1.2.3.4
   $ consul kv put my-app/port 80
   $ consul kv put my-app/max_conns 5
   

3. Execute envconsul with a subprocess (env in this example).

   $ envconsul -prefix my-app env
   

   Envconsul will connect to Consul, read the data from the key-value store, and populate environment variables corresponding to those values. Here is sample output.

   address=1.2.3.4
   max_conns=5
   port=80
   

For more examples and use cases, please see the examples in this README.

Usage

For the full list of command-line options:

$ envconsul -h

Command Line Interface (CLI)

The Envconsul CLI interface supports most of the options in the configuration file and visa-versa. Here are some common examples of CLI usage. For the full list of options, please run envconsul -h.

Render data from the prefix my-app into the environment.

$ envconsul -prefix my-app ruby my-app.rb

Render only data from the two prefixes into the environment (the parent processes environment will not be copied).

$ envconsul -pristine -prefix common -prefix my-app yarn start

Convert environment variables to upcase and remove any non-standard keys (like dashes to underscores).

$ envconsul -upcase -sanitize -prefix my-app python my-app.my

Read information about service.

$ envconsul -service-query my-service

Read secrets from Vault.

$ envconsul -secret secret/my-app ./my-app

Configuration File

Configuration files are written in the [HashiCorp Configuration Language][hcl]. By proxy, this means the configuration is also JSON compatible.

…

Note that not all fields are required. If you are not retrieving secrets from Vault, you do not need to specify a Vault configuration section. Similarly, if you are not logging to syslog, you do not need to specify a syslog configuration.

For additional security, tokens may also be read from the environment using the CONSUL_TOKEN or VAULT_TOKEN environment variables respectively. It is highly recommended that you do not put your tokens in plain-text in a configuration file.

Instruct Envconsul to use a configuration file with the -config flag:

$ envconsul -config "config.hcl"

This argument may be specified multiple times to load multiple configuration files. The right-most configuration takes the highest precedence. If the path to a directory is provided (as opposed to the path to a file), all of the files in the given directory will be merged in lexical order, recursively. Please note that symbolic links are not followed.

Commands specified on the CLI take precedence over a config file!

**Vault secrets always take precedence over consul prefixes. This is to mitigate a security vulnerability!

Signals

By default, almost all signals are proxied to the child process, with some exceptions. There are multiple configuration options related to signals.

• kill_signal - This is the signal that Envconsul should listen for to kill itself. This is useful when you want your application to respond to a different signal than the child process.

• reload_signal - This is the signal that Envconsul should listen for to reload its own configuration. This is useful when using configuration files. This signal will not be proxied to the child process if configured. By specifying this as the empty string, Envconsul will not listen for reload signals.

• exec.kill_signal - This is the signal that Envconsul will send to the child process to gracefully terminate it. This is the signal that your child application listens to for graceful termination.

Examples

Redis

Redis is a command key-value storage engine. If Redis is configured to read the given environment variables, you can use envconsul to start and manage the process:

# Ensure "daemonize no" is set in the redis configuration first.
$ envconsul \
  -consul demo.consul.io \
  -prefix redis/config \
  redis-server [opts...]

Env

This example is a great way to see envconsul in action. In practice, it is unlikely to be a useful use of envconsul though:

$ envconsul \
  -consul=demo.consul.io \
  -prefix redis/config \
  -once \
  env
ADDRESS=1.2.3.4
PORT=55

We can also ask envconsul to poll for configuration changes and automatically restart the process:

$ envconsul \
  -consul=demo.consul.io \
  -prefix redis/config \
  python -c 'import os, time; print os.environ; time.sleep(1000);'
{ 'ADDRESS': '1.2.3.4', 'PORT': '55' }
-----
{ 'ADDRESS': '1.2.3.4' }
-----
{ 'ADDRESS': '1.2.3.4', 'MAXCONNS': '50' }
-----

Vault

With the Vault integration, it is possible to pull secrets from Vault directly into the environment using envconsul. The only restriction is that the data must be "flat" and all keys and values must be strings or string-like values. envconsul will return an error if you try to read from a value that returns a map, for example.

First, you must add the vault address and token information to the configuration file. The configuration can also be set via command-line flags to envconsul:

vault {
  address     = "https://vault.service.consul:8200"
  token       = "abcd1234" # May also be specified via the envvar VAULT_TOKEN
  renew_token = true

  ssl {
    enabled = true
    verify  = true
    cert    = "/path/to/client/cert.pem"
    ca_cert = "/path/to/ca/cert.pem"
  }
}

Assuming a secret exists at secret/passwords that was created like so:

$ vault write secret/passwords username=foo password=bar

envconsul can pull those values into the environment:

$ envconsul \
    -config="./config.hcl" \
    -secret="secret/passwords" \
    env

secret_passwords_username=foo
secret_passwords_password=bar

Notice that the environment variables are prefixed with the path. The slashes in the path are converted to underscores, followed by the key:

secret/passwords     => secret_passwords
mysql/creds/readonly => mysql_creds_readonly

This behavior may be disabled by setting no_prefix

secret {
  no_prefix = true
  path      = "secret/passwords"
}

username=foo
password=bar

You can also apply key transformations to the data:

$ envconsul \
    -config="./config.hcl" \
    -secret="mysql/creds/readonly" \
    -upcase \
    env

MYSQL_CREDS_READONLY_USERNAME=root-aefa635a-18
MYSQL_CREDS_READONLY_PASSWORD=132ae3ef-5a64-7499-351e-bfe59f3a2a21

It is highly encouraged that you specify the format for vault keys to include a common prefix, like:

secret {
  path   = "secret/passwords"
  format = "secret_{{ key }}"
}

The format string is passed to the go formatter and "{{ key }}" dictates where the key will go. This will help filter out the environment when execing to a child-process, for example.

In case, you need only a subset of keys from a Vault prefix, you can achieve this by applying a per-key configuration:

secret {
  path   = "secret/passwords"
  key {
    name   = "username"
  }
}

$ envconsul \
    -config="./config.hcl" \
    env

secret_passwords_username=foo

It is also possible to apply a per-key formatting that behaves the same as the secret level format option:

secret {
  path   = "secret/passwords"
  key {
    name   = "username"
    format = "readonly_user_{{ key }}"
  }
  key {
    name   = "password"
    format = "custom_prefix_{{ key }}"
  }
}

$ envconsul \
    -config="./config.hcl" \
    env

secret_passwords_readonly_user_username=foo
secret_passwords_custom_prefix_password=bar

NOTE: per-key configuration works only in case of absence of secret-level format configuration option.

Hence, the following set-up skips per-key configuration in favor of format from the top level secret block:

secret {
  path   = "secret/passwords"
  format = "creds_{{ key }}"

  key {
    name   = "username"
    format = "readonly_user_{{ key }}"
  }
  key {
    name   = "password"
    format = "custom_prefix_{{ key }}"
  }
}

$ envconsul \
    -config="./config.hcl" \
    env

secret_passwords_creds_username=foo
secret_passwords_creds_password=bar

If you need the same secret to be available under different env var name, this could be achieved by applying another format for the same secret key:

secret {
  no_prefix = true
  path   = "secret/passwords"
  key {
    name   = "username"
    format = "readonly_user_{{ key }}"
  }
  key {
    name   = "password"
    format = "custom_prefix_{{ key }}"
  }
  key {
    name   = "password"
    format = "legacy_format_password_db"
  }
}

$ envconsul \
    -config="./config.hcl" \
    env

readonly_user_username=foo
custom_prefix_password=bar
legacy_format_password_db=bar

Debugging

Envconsul can print verbose debugging output. To set the log level for Envconsul, use the -log-level flag:

$ envconsul -log-level info ...

<timestamp> [INFO] (cli) received redis from Watcher
<timestamp> [INFO] (cli) invoking Runner
# ...

You can also specify the level as debug:

$ envconsul -log-level debug ...

…

Cont