Adding a second dedicated network interface for longhorn replication

Author: sushyadCreated Jan 26, 2024Updated Feb 8, 2024

I am trying to add a second network interface dedicated for longhorn replication using multus cni plugin together with ipvlan. Here is my PR from my fork to give you an idea what I am trying to do: https://github.com/khuedoan/homelab/pull/134

I was able to tweak the argocd recipe to:

Bring up the second interface and assign it an ip address. I am able to ping the nodes on their secondary ip addresses from each other.
Add ipvlan plugin to /opt/cni/bin folder on each node.
Using this as a guide I was able to add multus configuration using argocd.

When I create a test pod wth two network interfaces it doesn't work and doesn't show the second interface as expected.

cat <<EOF | kubectl apply -f - 
apiVersion: v1
kind: Pod
metadata:
  name: app1
  annotations:
    k8s.v1.cni.cncf.io/networks: multus-conf
spec:
  containers:
  - name: app1
    command: ["/bin/sh", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF

bash

kubectl describe pod app1

gives

bash-5.2# kubectl describe pod app1 
Name:             app1
Namespace:        default
Priority:         0
Service Account:  default
Node:             metal0/192.168.0.115
Start Time:       Fri, 26 Jan 2024 19:49:49 +0000
Labels:           <none>
Annotations:      k8s.v1.cni.cncf.io/networks: multus-conf
Status:           Running
IP:               10.0.0.176
IPs:
  IP:  10.0.0.176
Containers:
......

instead of something lke this:

$ kubectl describe pod app1
Name:             app1
Namespace:        default
Priority:         0
Service Account:  default
Node:             node2/192.168.200.175
Start Time:       Fri, 11 Aug 2023 12:28:56 +0300
Labels:           <none>
Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "mynet",
                        "interface": "eth0",
                        "ips": [
                            "10.244.2.8"
                        ],
                        "mac": "86:69:28:4f:54:b3",
                        "default": true,
                        "dns": {},
                        "gateway": [
                            "10.244.2.1"
                        ]
                    },{
                        "name": "default/multus-conf",
                        "interface": "net1",
                        "ips": [
                            "192.168.200.100"
                        ],
                        "mac": "2a:1b:4d:89:66:c0",
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks: multus-conf
Status:           Running
IP:               10.244.2.8
IPs:
  IP:  10.244.2.8
Containers:
.....

Has anyone tried to do this before?

Source: khuedoan/homelab

View original on GitHub View discussion on GitHub