peers.json recovery accepts Raft peer addresses containing a URL scheme, causing failures after leader change
Describe the bug When I use a peers.json file to recover my Vault cluster, Vault allows me to put in an erroneous address field including a URL scheme (https), e.g.
[
{
"id": "vault-01",
"address": "https://vault-01.example.com:8201",
"non_voter": false
}
]The cluster works fine, the leader comes up and I can join other nodes. The peer list then looks like this:
vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-01 https://vault-01.example.com:8201 leader true
vault-02 vault-02.example.com:8201 follower true
vault-03 vault-03.example.com:8201 follower trueYet at some later point in time (weeks later) this will break my cluster after a leader change because the new leader cannot communicate with the node that is registered with the address including https. Error messages from the leader log:
[ERROR] storage.raft: failed to appendEntries to: peer="{Voter vault-01 https://vault-01.example.com:8201}" error="dial tcp: address https://vault-01.example.com:8201: too many colons in address"
[ERROR] storage.raft: failed to heartbeat to: peer=https://vault-01.example.com:8201 backoff time=160ms error="dial tcp: address https://vault-01.example.com:8201: too many colons in address"
To Reproduce Steps to reproduce the behavior:
- Stop all Vault nodes, e.g. 01, 02, 03
- Recover cluster using a peers.json for node 01 according to https://developer.hashicorp.com/vault/tutorials/raft/raft-lost-quorum. Add scheme https to address field.
- Start 01 as leader and let 02 and 03 join the cluster.
- Wait a few weeks with Vault running perfectly fine :-)
- Stop 01; nodes 02 or 03 will become leader.
- Start 01. It will not be reachable by the leader because of the malformed address.
Expected behavior
The address field in peers.json should be validated not to include a URL scheme and give an error message if the address field has any issues.
Environment:
- Vault Server Version (retrieve with
vault status): 2.0.3 - Vault CLI Version (retrieve with
vault version): 2.0.3 - Server Operating System/Architecture: Linux/Debian
Vault server configuration file(s):
cluster_name = "vault"
api_addr = "https://vault-01.example.com:8200"
cluster_addr = "https://vault-01.example.com:8201"
ui = true
storage "raft" {
path = "/opt/raft"
node_id = "vault-01"
}
disable_mlock = true
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/etc/vault.d/tls/public.cert"
tls_key_file = "/etc/vault.d/tls/private.key"
}
Additional context Just as a side note: https://developer.hashicorp.com/vault/tutorials/raft/raft-lost-quorum#view-the-peer-list shows an address field including scheme https.
Source: hashicorp/vault