NVIDIA Network Operator
Table of contents generated with markdown-toc
NVIDIA Network Operator leverages Kubernetes CRDs and Operator SDK to manage Networking related Components in order to enable Fast networking, RDMA and GPUDirect for workloads in a Kubernetes cluster.
The Goal of Network Operator is to manage all networking related components to enable execution of RDMA and GPUDirect RDMA workloads in a kubernetes cluster including:
For more information please visit the official documentation.
NVIDIA Network operator relies on Node labeling to get the cluster to the desired state.
Node Feature Discovery v0.13.2 or newer is deployed by default via HELM chart installation.
NFD is used to label nodes with the following labels:
NOTE: We use nodeFeatureRules to label PCI vendor and device.This is enabled via
nfd.deployNodeFeatureRuleschart parameter.
Example NFD worker configurations:
config:
sources:
pci:
deviceClassWhitelist:
- "0300"
- "0302"
deviceLabelFields:
- vendor* Required for GPUDirect driver container deployment
NOTE: If NFD is already deployed in the cluster, make sure to pass
--set nfd.enabled=falseto the helm install command to avoid conflicts, and if NFD is deployed from this repo theenableNodeFeatureApiflag is enabled by default to have the ability to create NodeFeatureRules.
The Operator Acts on the following CRDs:
CRD that defines a Cluster state for Mellanox Network devices.
NOTE: The operator will act on a NicClusterPolicy instance with a predefined name "nic-cluster-policy", instances with different names will be ignored.
NICClusterPolicy CRD Spec includes the following sub-states:
ofedDriver: OFED driver container to be deployed on Mellanox supporting nodes.rdmaSharedDevicePlugin: RDMA shared device plugin
and related configurations.sriovDevicePlugin: SR-IOV Network Device Plugin
and related configurations.ibKubernetes: InfiniBand Kubernetes and related configurations. secondaryNetwork: Specifies components to deploy in order to facilitate a secondary network in Kubernetes. It consists of the following optionally deployed components:nvIpam: NVIDIA Kubernetes IPAM and related configurations.nicConfigurationOperator: NVIDIA NIC Configuration Operator and related configuration.NOTE: Any sub-state may be omitted if it is not required for the cluster.
NICClusterPolicy status field reflects the current state of the system.
It contains a per sub-state and a global state status.
The sub-state status indicates if the cluster has transitioned to the desired
state for that sub-state, e.g OFED driver container deployed and loaded on relevant nodes,
RDMA device plugin deployed and running on relevant nodes.
The global state reflects the logical AND of each individual sub-state.
status:
appliedStates:
- name: state-pod-security-policy
state: ignore
- name: state-multus-cni
state: ready
- name: state-container-networking-plugins
state: ignore
- name: state-ipoib-cni
state: ignore
- name: state-OFED
state: ready
- name: state-SRIOV-device-plugin
state: ignore
- name: state-RDMA-device-plugin
state: ready
- name: state-ib-kubernetes
state: ignore
- name: state-nv-ipam-cni
state: ready
state: readyNOTE: An
ignoreState indicates that the sub-state was not defined in the custom resource thus it is ignored.
This CRD defines a MacVlan secondary network. It is translated by the Operator to a NetworkAttachmentDefinition instance as defined in k8snetworkplumbingwg/multi-net-spec.
MacvlanNetwork CRD Spec includes the following fields:
networkNamespace: Namespace for NetworkAttachmentDefinition related to this MacvlanNetwork CRD.master: Name of the host interface to enslave. Defaults to default route interface.mode: Mode of interface one of "bridge", "private", "vepa", "passthru", default "bridge".mtu: MTU of interface to the specified value. 0 for master's MTU.ipam: IPAM configuration to be used for this network.metaPlugins: Additional CNI plugin configurations to chain after the macvlan plugin.This CRD defines a HostDevice secondary network. It is translated by the Operator to a NetworkAttachmentDefinition instance as defined in k8snetworkplumbingwg/multi-net-spec.
HostDeviceNetwork CRD Spec includes the following fields:
networkNamespace: Namespace for NetworkAttachmentDefinition related to this HostDeviceNetwork CRD.resourceName: Host device resource pool.ipam: IPAM configuration to be used for this network.metaPlugins: Additional CNI plugin configurations to chain after the host-device plugin.When metaPlugins is set, the Operator renders a CNI configuration list with
the primary macvlan or host-device plugin first, followed by the supplied
plugins in order. This allows tools such as k8s-launch-kit to add the sbr
plugin for source-based routing and the tuning plugin for
ignoreARP: true without replacing the primary network configuration.
The value uses the same raw fragment format as the SR-IOV Network Operator: one or more plugin JSON objects separated by commas, without enclosing array brackets. For example, the combined k8s-launch-kit configuration is:
metaPlugins: |
{
"type": "tuning",
"sysctl": {
"net.ipv4.conf.all.arp_ignore": "1",
"net.ipv4.conf.all.arp_announce": "2",
"net.ipv4.conf.all.rp_filter": "0",
"net.ipv4.conf.IFNAME.arp_ignore": "1",
"net.ipv4.conf.IFNAME.arp_announce": "2",
"net.ipv4.conf.IFNAME.rp_filter": "0"
}
},
{
"type": "sbr"
}This CRD defines an IPoIBNetwork secondary network. It is translated by the Operator to a NetworkAttachmentDefinition instance as defined in k8snetworkplumbingwg/multi-net-spec.
HostDeviceNetwork CRD Spec includes the following fields:
networkNamespace: Namespace for NetworkAttachmentDefinition related to this HostDeviceNetwork CRD.master: Name of the host interface to enslave.ipam: IPAM configuration to be used for this network.Sample CRDs can be found at: example/ directory.
NOTE: As more driver containers are built the operator will be able to support additional platforms. NOTE: ConnectX-6 Lx is not supported.
The following Network Adapters have been tested with NVIDIA Network Operator:
Deployment of NVIDIA Network Operator consists of:
./config/crd/bases:make deploy./example/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr.yaml./example/crs/mellanox.com_v1alpha1_macvlannetwork_cr.yamlA deployment example can be found under example folder here.
To build a container image for Network Operator use:
make imageTo build a multi-arch image and publish to a registry use:
export REGISTRY=example.com/registry
export IMAGE_NAME=network-operator
export VERSION=v1.1.1
make image-build-multiarch image-push-multiarchDriver containers are essentially containers that have or yield kernel modules compatible with the underlying kernel. An initialization script loads the modules when the container is run (in privileged mode) making them available to the kernel.
While this approach may seem odd. It provides a way to deliver drivers to immutable systems.
Mellanox OFED driver container sup
No open issues yet, or sync has not completed.