Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
N

network-operator

> 编程语言
Open source

NVIDIA Network Operator

359 stars0 likes1 views
WebsiteGitHub

About

NVIDIA Network Operator

  • NVIDIA Network Operator
    • Documentation
    • Prerequisites
      • Kubernetes Node Feature Discovery (NFD)
    • Resource Definitions
      • NICClusterPolicy CRD
        • NICClusterPolicy spec
        • NICClusterPolicy status
          • Example Status field of a NICClusterPolicy instance
      • MacvlanNetwork CRD
        • MacvlanNetwork spec
      • HostDeviceNetwork CRD
        • HostDeviceNetwork spec
      • IPoIBNetwork CRD
        • IPoIBNetwork spec
      • Examples
    • System Requirements
    • Tested Network Adapters
    • Compatibility Notes
    • Deployment Example
    • Docker image
    • Driver Containers
    • Upgrade
    • Drain Controller
    • Externally Provided Configurations For Network Operator Sub-Components

Table of contents generated with markdown-toc

NVIDIA Network Operator

NVIDIA Network Operator leverages Kubernetes CRDs and Operator SDK to manage Networking related Components in order to enable Fast networking, RDMA and GPUDirect for workloads in a Kubernetes cluster.

The Goal of Network Operator is to manage all networking related components to enable execution of RDMA and GPUDirect RDMA workloads in a kubernetes cluster including:

  • Mellanox Networking drivers to enable advanced features
  • Kubernetes device plugins to provide hardware resources for fast network
  • Kubernetes secondary network for Network intensive workloads

Documentation

For more information please visit the official documentation.

Prerequisites

Kubernetes Node Feature Discovery (NFD)

NVIDIA Network operator relies on Node labeling to get the cluster to the desired state. Node Feature Discovery v0.13.2 or newer is deployed by default via HELM chart installation. NFD is used to label nodes with the following labels:

  • PCI vendor and device information
  • RDMA capability
  • GPU features*

NOTE: We use nodeFeatureRules to label PCI vendor and device.This is enabled via nfd.deployNodeFeatureRules chart parameter.

Example NFD worker configurations:

yaml
    config:
      sources:
        pci:
          deviceClassWhitelist:
          - "0300"
          - "0302"
          deviceLabelFields:
          - vendor

* Required for GPUDirect driver container deployment

NOTE: If NFD is already deployed in the cluster, make sure to pass --set nfd.enabled=false to the helm install command to avoid conflicts, and if NFD is deployed from this repo the enableNodeFeatureApi flag is enabled by default to have the ability to create NodeFeatureRules.

Resource Definitions

The Operator Acts on the following CRDs:

NICClusterPolicy CRD

CRD that defines a Cluster state for Mellanox Network devices.

NOTE: The operator will act on a NicClusterPolicy instance with a predefined name "nic-cluster-policy", instances with different names will be ignored.

NICClusterPolicy spec

NICClusterPolicy CRD Spec includes the following sub-states:

  • ofedDriver: OFED driver container to be deployed on Mellanox supporting nodes.
  • rdmaSharedDevicePlugin: RDMA shared device plugin and related configurations.
  • sriovDevicePlugin: SR-IOV Network Device Plugin and related configurations.
  • ibKubernetes: InfiniBand Kubernetes and related configurations.
  • secondaryNetwork: Specifies components to deploy in order to facilitate a secondary network in Kubernetes. It consists of the following optionally deployed components:
    • Multus-CNI: Delegate CNI plugin to support secondary networks in Kubernetes
    • CNI plugins: Currently only containernetworking-plugins is supported
    • IP Over Infiniband (IPoIB) CNI Plugin: Allow users to create an IPoIB child link and move it to the pod.
  • nvIpam: NVIDIA Kubernetes IPAM and related configurations.
  • nicConfigurationOperator: NVIDIA NIC Configuration Operator and related configuration.

NOTE: Any sub-state may be omitted if it is not required for the cluster.

NICClusterPolicy status

NICClusterPolicy status field reflects the current state of the system. It contains a per sub-state and a global state status.

The sub-state status indicates if the cluster has transitioned to the desired state for that sub-state, e.g OFED driver container deployed and loaded on relevant nodes, RDMA device plugin deployed and running on relevant nodes.

The global state reflects the logical AND of each individual sub-state.

Example Status field of a NICClusterPolicy instance
status:
  appliedStates:
  - name: state-pod-security-policy
    state: ignore
  - name: state-multus-cni
    state: ready
  - name: state-container-networking-plugins
    state: ignore
  - name: state-ipoib-cni
    state: ignore
  - name: state-OFED
    state: ready
  - name: state-SRIOV-device-plugin
    state: ignore
  - name: state-RDMA-device-plugin
    state: ready
  - name: state-ib-kubernetes
    state: ignore
  - name: state-nv-ipam-cni
    state: ready
  state: ready

NOTE: An ignore State indicates that the sub-state was not defined in the custom resource thus it is ignored.

MacvlanNetwork CRD

This CRD defines a MacVlan secondary network. It is translated by the Operator to a NetworkAttachmentDefinition instance as defined in k8snetworkplumbingwg/multi-net-spec.

MacvlanNetwork spec

MacvlanNetwork CRD Spec includes the following fields:

  • networkNamespace: Namespace for NetworkAttachmentDefinition related to this MacvlanNetwork CRD.
  • master: Name of the host interface to enslave. Defaults to default route interface.
  • mode: Mode of interface one of "bridge", "private", "vepa", "passthru", default "bridge".
  • mtu: MTU of interface to the specified value. 0 for master's MTU.
  • ipam: IPAM configuration to be used for this network.
  • metaPlugins: Additional CNI plugin configurations to chain after the macvlan plugin.

HostDeviceNetwork CRD

This CRD defines a HostDevice secondary network. It is translated by the Operator to a NetworkAttachmentDefinition instance as defined in k8snetworkplumbingwg/multi-net-spec.

HostDeviceNetwork spec

HostDeviceNetwork CRD Spec includes the following fields:

  • networkNamespace: Namespace for NetworkAttachmentDefinition related to this HostDeviceNetwork CRD.
  • resourceName: Host device resource pool.
  • ipam: IPAM configuration to be used for this network.
  • metaPlugins: Additional CNI plugin configurations to chain after the host-device plugin.

When metaPlugins is set, the Operator renders a CNI configuration list with the primary macvlan or host-device plugin first, followed by the supplied plugins in order. This allows tools such as k8s-launch-kit to add the sbr plugin for source-based routing and the tuning plugin for ignoreARP: true without replacing the primary network configuration.

The value uses the same raw fragment format as the SR-IOV Network Operator: one or more plugin JSON objects separated by commas, without enclosing array brackets. For example, the combined k8s-launch-kit configuration is:

yaml
metaPlugins: |
  {
    "type": "tuning",
    "sysctl": {
      "net.ipv4.conf.all.arp_ignore": "1",
      "net.ipv4.conf.all.arp_announce": "2",
      "net.ipv4.conf.all.rp_filter": "0",
      "net.ipv4.conf.IFNAME.arp_ignore": "1",
      "net.ipv4.conf.IFNAME.arp_announce": "2",
      "net.ipv4.conf.IFNAME.rp_filter": "0"
    }
  },
  {
    "type": "sbr"
  }

IPoIBNetwork CRD

This CRD defines an IPoIBNetwork secondary network. It is translated by the Operator to a NetworkAttachmentDefinition instance as defined in k8snetworkplumbingwg/multi-net-spec.

IPoIBNetwork spec

HostDeviceNetwork CRD Spec includes the following fields:

  • networkNamespace: Namespace for NetworkAttachmentDefinition related to this HostDeviceNetwork CRD.
  • master: Name of the host interface to enslave.
  • ipam: IPAM configuration to be used for this network.

Examples

Sample CRDs can be found at: example/ directory.

System Requirements

  • RDMA capable hardware: Mellanox ConnectX-5 NIC or newer.
  • NVIDIA GPU and driver supporting GPUDirect e.g Quadro RTX 6000/8000 or Tesla T4 or Tesla V100 or Tesla V100. (GPU-Direct only)
  • Operating Systems: Ubuntu 20.04 LTS

NOTE: As more driver containers are built the operator will be able to support additional platforms. NOTE: ConnectX-6 Lx is not supported.

Tested Network Adapters

The following Network Adapters have been tested with NVIDIA Network Operator:

  • ConnectX-5
  • ConnectX-6 Dx

Compatibility Notes

  • NVIDIA Network Operator is compatible with NVIDIA GPU Operator v1.5.2 and above
  • Starting from v465 NVIDIA GPU driver includes a built-in nvidia_peermem module which is a replacement for nv_peer_mem module. NVIDIA GPU operator manages nvidia_peermem module loading.

Deployment Example

Deployment of NVIDIA Network Operator consists of:

  • Deploying NVIDIA Network Operator CRDs found under ./config/crd/bases:
    • mellanox.com_nicclusterpolicies_crd.yaml
    • mellanox.com_macvlan_crds.yaml
    • k8s.cni.cncf.io-networkattachmentdefinitions-crd.yaml
  • Deploying network operator resources by running make deploy
  • Defining and deploying a NICClusterPolicy custom resource. Example can be found under ./example/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr.yaml
  • Defining and deploying a MacvlanNetwork custom resource. Example can be found under ./example/crs/mellanox.com_v1alpha1_macvlannetwork_cr.yaml

A deployment example can be found under example folder here.

Docker image

To build a container image for Network Operator use:

bash
make image

To build a multi-arch image and publish to a registry use:

bash
export REGISTRY=example.com/registry 
export IMAGE_NAME=network-operator 
export VERSION=v1.1.1 
make image-build-multiarch image-push-multiarch

Driver Containers

Driver containers are essentially containers that have or yield kernel modules compatible with the underlying kernel. An initialization script loads the modules when the container is run (in privileged mode) making them available to the kernel.

While this approach may seem odd. It provides a way to deliver drivers to immutable systems.

Mellanox OFED container

Mellanox OFED driver container sup

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Go

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言