healthcheck breaks the swarm network.

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via the report form.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Hi.

I have a swarm cluster with 3 manager nodes (manager-01,manager-02,manager-03). I deploy a global placement service etcd.

services:

  etcd:
    hostname: etcd-{{.Node.Hostname}}
    image: quay.io/coreos/etcd:v3.7.1
    networks:
      - etcd
    environment:
      ETCD_NAME: "etcd-{{.Node.Hostname}}"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd-{{.Node.Hostname}}:2379"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd-{{.Node.Hostname}}:2380"
      ETCD_INITIAL_CLUSTER: ${ETCD_INITIAL_CLUSTER:?error}
      ETCD_INITIAL_CLUSTER_STATE: ${ETCD_INITIAL_CLUSTER_STATE:-existing}
    volumes:
      - etcd-data:/var/lib/etcd
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: any

and it run smoothly... raft votes etc... I am a happy person.

if i try to put a healthcheck

    healthcheck:
      test: ["CMD", "curl", "http://127.0.0.1:2379/health/serializable=true"]
      interval: 5s
      timeout: 5s
      retries: 6
      start_period: 5s

Then the etcd-services tasks can not see each other anymore.and the raft vote can not be done. I deploy a another test service with a alpine image , at the same network.

When i deploy without healthcheck everything smooth again.

• Task of the etcd-services can see each-other
• the test-service task can see all of thems.

When i put the healthcheck at the etcd-service ,

• the etcd-service services can not see each other anymore.and the raft vote can not be done.
• the test service task can't see them.

What did you expect to happen?

1. run smoothly with the healthcheck also.

How can we reproduce it (as minimally and precisely as possible)?

1. Create a swarm cluster with 3 manager nodes named (manager-01,manager-02,manager-03).
2. Deploy the fellow docker compose file with the .env.example file.

services:

  etcd:
    hostname: etcd-{{.Node.Hostname}}
    image: quay.io/coreos/etcd:v3.7.1
    networks:
      - etcd
    environment:
      ETCD_NAME: "etcd-{{.Node.Hostname}}"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd-{{.Node.Hostname}}:2379"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd-{{.Node.Hostname}}:2380"
      ETCD_INITIAL_CLUSTER: ${ETCD_INITIAL_CLUSTER:?error}
      ETCD_INITIAL_CLUSTER_STATE: ${ETCD_INITIAL_CLUSTER_STATE:-existing}
    volumes:
      - etcd-data:/var/lib/etcd
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: any

if it run smoothly...raft votes etc... and you are a Happy person also

1. add the healthckeck.

    healthcheck:
      test: ["CMD", "curl", "http://127.0.0.1:2379/health/serializable=true"]
      interval: 5s
      timeout: 5s
      retries: 6
      start_period: 5s

(modify the .env file ETCD_INITIAL_CLUSTER_STATE=existing)

1. deploy again.

Anything else we need to know?

.env.example

ETCD_INITIAL_CLUSTER="etcd-manager-01=http://etcd-manager-01:2380,etcd-manager-02=http://etcd-manager-02:2380,etcd-manager-03=http://etcd-manager-03:2380" ETCD_INITIAL_CLUSTER_STATE=new

my swarm network etcd: driver: overlay attachable: false internal: true driver_opts: encrypted: "true" ipam: config: - subnet: 10.10.2.0/24

Etcd version (please run commands below)

v3.7.1

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output