DOCA Management Service Guide

This guide provides instructions on how to use the DOCA Management Service on top of NVIDIA® BlueField® networking platform or NVIDIA® ConnectX® SmartNICs.

DOCA Management Service (DMS) is a centralized solution for configuring and operating NVIDIA® BlueField® and ConnectX® devices.

DMS consolidates NVIDIA’s internal scripts and tools under a streamlined, industry-standard API based on OpenConfig. Users can configure BlueField or ConnectX devices either locally (via SSH) or remotely (via gRPC).

DMS exposes configurable parameters for BlueField and ConnectX over the external interface, enabling automated configuration from a management station. The interface offers a consistent and unified approach to device configuration while abstracting away the underlying tools used to manage individual features.

DMS follows a client-server architecture. The server component, DMS Daemon (dmsd), is responsible for resource discovery and for processing incoming commands from clients. These clients can be either the provided DMS Client (dmsc) or a custom-built client using the same API.

Open Config and YANG Models

• OpenConfig is a standard that defines and implements a common, vendor-independent software layer for managing and monitoring network devices.

  It utilizes YANG data models to represent device configurations and operational states in a consistent and standardized format.

  Most gNMI/gNOI protocols used in DMS are aligned with the OpenConfig community and rely on the gRPC protocol for command transmission.

• YANG (Yet Another Next Generation) is a data modeling language used to describe the structure and content of network configuration and state data.

  It represents data hierarchically, similar to an XML tree, and defines schemas for configuration, operational state, and remote procedure calls (RPCs). YANG models serve as the foundation for implementing the OpenConfig standard.

To operate the DMS, DOCA must be installed on the system where DMS will run.

• DMS on Host – The DOCA for Host package must be installed on the host system. It should be installed using either the doca-networking or doca-all profile.

• DMS on DPU (BlueField Arm) – A compatible DOCA image must be installed on the BlueField Arm processor.

DMS has 3 major components:

• DMSD – Server – DMS server can run on the BlueField ARM or on the host with an NVIDIA PCIe device

• DMSC – Client – DOCA provided OpenConfig client. Customers can choose to use this client, any other open-source client, or develop their own (gRPC-based) client.

• Yang files – Yang model files contain the data model used to configure the device, using NVIDIA-specific extension to common OpenConfig YANG Models.

OpenConfig consists of 2 main protocols:

• gNMI – gRPC Network Management Interface, protocol to configure of network device.

• gNOI – gRPC Network Operations Interface, a protocol to perform operational commands on network devices (i.e., provision, upgrade, reboot).

Service Connectivity

The following is an architectural diagram of DMS:

The following diagram presents the DMS modes of operation, as the DMS client can operate from different location:

1. Both DMS client and server components are deployed on the Host.

2. Both DMS client and server components are deployed on DPU (BlueField Arm).

3. DMS server component is deployed on the Host, while DMS client is deployed remotely (connecting to DMS server over management network).

4. DMS server component is deployed on DPU (BlueField Arm), while DMS client is deployed remotely (connecting to DMS server over management network).

DMS Daemon Launch Options

Option 1: SystemD Service (Recommended)

On the DPU, the DMS Daemon (dmsd) runs as a systemd service and comes pre-installed with the BlueField BFB-Bundle.

It should be running locally by default as a systemd service, bound to localhost:9339.

• To check if dmsd is running:

  Copy

  Copied!

              
  
              
  ps -ef | grep dms
          

• To manage the service:

  Copy

  Copied!

              
  
              
  sudo systemctl status dmsd
  sudo systemctl start dmsd
  sudo systemctl stop dmsd
          

• To interact with dmsd using the built-in client (requires user-password authentication; the default is the OS root password):

  Copy

  Copied!

              
  
              
  dmscli
          

• A template systemd service file is included in the DOCA package:

  Copy

  Copied!

              
  
              
  /opt/mellanox/doca/services/dms/dmsd.service/dmsd.service
          

  Note

  dmsd runs automatically as a systemd service only on the DPU, not on the host.

Option 2: Manual Launch

You can start the dmsd daemon manually with a variety of authentication methods.

1. Localhost with username/password:

   Copy

   Copied!

               
   
               
   /opt/mellanox/doca/services/dms/dmsd \
     --bind_address localhost:9339 \
     --auth credentials \
     --username my_username \
     --password my_password
           

2. Remote host with shadow file:

   Copy

   Copied!

               
   
               
   /opt/mellanox/doca/services/dms/dmsd \
     --bind_address <ip>:<port> \
     --auth shadow \
     --username my_username \
     --shadow /etc/shadow
           

3. TLS certificate-based authentication:

   Copy

   Copied!

               
   
               
   /opt/mellanox/doca/services/dms/dmsd \
     --bind_address localhost:9339 \
     --auth cert \
     --ca <ca.pem> \
     --tls_key_file <key.pem> \
     --tls_cert_file <cert.pem>
           

4. Specific PCIe device (for a host with multiple DPUs):

   Copy

   Copied!

               
   
               
   /opt/mellanox/doca/services/dms/dmsd \
     --bind_address localhost:9339 \
     --auth credentials \
     --username my_username \
     --password my_password \
     --target_pci 0000:08:00
           

5. Provide a list of PCIe devices (for a host with multiple DPUs):

   Copy

   Copied!

               
   
               
   /opt/mellanox/doca/services/dms/dmsd \
     --bind_address localhost:9339 \
     --auth credentials \
     --username my_username \
     --password my_password \
     --target_pci 0000:08:00,0000:09:00
           

DMS Server Flags

To see the full list of flags, use the help flag (i.e., dmsd --help, dmsd -h).

General Flags

• --bind_address <string> – Bind to <address>:<port> or just :<port> (default is :9339). Can be localhost for the local use case, or an IP address for the remote use case.

• -v <value> – log level for V logs

• --target_pci <string> – One or multiple target PCIe addresses (i.e., 0000:03:00, 0000:03:00,0000:04:00). Auto-select if only one NVIDIA network device is present; otherwise, one or more PCIe addresses must be specified.

Provisioning Flags

• --image_folder <string> – Specify the image install folder. Can copy images directly to the folder to avoid transfer over the net. Default create folder: /tmp/dms.

• --chunk_size_ack <uint> – The chunk size of the image to respond with a transfer response in bytes (default: 12000000)

• --exec_timeout <uint> – The maximum execution timeout in seconds for a command if not responding (not printing to stdout); 0 (default) is unlimited

Config Flags

• --init_config <string> – File containing gNMI requests to run on DMS start. By default, DMS adds any gNMI set request to a file (file format is req.json).

Authentication Method Flags

The --auth <string> flag specifies one of the following three authentication methods, each with its own set of required flags:

Shadow

• Zero-touch setup – no need to create a dedicated DMS user (reuses existing OS user)

• The server reads the hashed password from the shadow file in real time for each client request

• Required flags: --username <string> and --shadow <string>

  • Example:

    Copy

    Copied!

                
    
                
    --auth shadow --username root --shadow /etc/shadow
            

• To disable authentication:

  Copy

  Copied!

              
  
              
  --noauth
          

Credentials

• Requires the admin to define a strong password.

• Required flags: --username <string> and optionally --password <string>

  • Example:

    Copy

    Copied!

                
    
                
    --auth credentials --username root --password 123456
            

• To prompt for password input interactively at startup, leave the --password flag empty.

• To disable authentication:

  Copy

  Copied!

              
  
              
  --noauth
          

Certificate (TLS)

• The most secure option, using Transport Layer Security (TLS) for both encryption and mutual authentication.

• Authentication is based on digital certificates, ensuring only trusted clients can connect.

  • Required flags:

    --auth cert

    --ca <path/to/ca.crt>

    --tls_key_file <path/to/target.key>

    --tls_cert_file <path/to/target.crt>

  • Example:

    Copy

    Copied!

                
    
                
    --auth cert --ca /path/to/certs/ca.crt --tls_key_file /path/to/certs/target.key --tls_cert_file /path/to/certs/target.crt
            

• To disable TLS:

  Copy

  Copied!

              
  
              
  --tls_enabled=false 
          

Security Flags

TLS Encryption and Authentication

DMS uses TLS (Transport Layer Security) by default to secure communication. TLS is a widely adopted cryptographic protocol that provides the following guarantees:

• Encryption – Protects transmitted data from eavesdropping.

• Authentication – Verifies the identity of the server and optionally the client via digital certificates.

• Integrity – Ensures transmitted data is not tampered with.

TLS relies on a public/private key pair and a digital certificate issued by a Certificate Authority (CA). These elements ensure secure, authenticated communication.

DMS supports TLS for both gNMI and gNOI interfaces, and works with either self-signed or CA-signed certificates.

To enable TLS, both server and client must provide:

• --ca <string> – Path to the CA certificate used to validate the remote party’s certificate.

• --tls_cert_file <string> – Path to the public certificate presented during the TLS handshake.

• --tls_key_file <string> – Path to the corresponding private key used to sign the handshake and decrypt messages.

  Note

  The private key must be protected. If compromised, it can be used to impersonate the server or client.

TLS Communication Modes

Mutual TLS Authentication (mTLS)

Both the server and client present certificates and verify each other during the TLS handshake.

            

            
# Server:
/opt/mellanox/doca/services/dms/dmsd <flags> --auth cert \
  --ca /path/to/certs/client_ca.crt \
  --tls_cert_file /path/to/certs/target.crt \
  --tls_key_file /path/to/certs/target.key
 
# Client:
/opt/mellanox/doca/services/dms/dmsc <flags> \
  --tls-ca /path/to/certs/target_ca.crt \
  --tls-cert /path/to/certs/client.crt \
  --tls-key /path/to/certs/client.key <command>
        

Highlights:

• Provides the highest level of security.

• Ensures both sides are authenticated and trusted.

• All traffic is encrypted.

TLS (Server Authentication Only)

Only the server presents its certificate during the handshake. The client verifies the server but does not present a certificate.

            

            
# Server:
/opt/mellanox/doca/services/dms/dmsd <flags> --auth cert \
  --tls_cert_file /path/to/certs/target.crt \
  --tls_key_file /path/to/certs/target.key
 
# Client:
/opt/mellanox/doca/services/dms/dmsc <flags> \
  --tls-ca /path/to/certs/target_ca.crt \
  --skip-verify <command>
        

Highlights:

• Client authenticity is not verified.

• Use --skip-verify if a root CA is not provided.

• Traffic remains encrypted.

Insecure Mode (No TLS)

TLS is disabled entirely on both server and client. No encryption or authentication is performed.

            

            
# Server:
/opt/mellanox/doca/services/dms/dmsd <flags> --auth cert --tls_enabled=false
 
# Client:
/opt/mellanox/doca/services/dms/dmsc <flags> --insecure <command>
        

To achieve insecure communication (no encryption), configure:

• Server – use --tls_enabled=false param

• Client – use --insecure flag'

Highlights:

• Not recommended for production.

• Use only in fully isolated, trusted environments for testing or development.

To see the full list of flags, use the help flag (i.e., dmsc --help, dmsc -h).

DMS Client Flags

General Flags

• --address <string> – Bind to <address>:<port> or just :<port> (default is :9339). Can be localhost for the local use case, or an IP address for the remote use case.

• --target <string> – Target PCIe address to interact with. Mandatory if the server is running for multiple PCIe addresses.

• -d, --debug – Debug mode.

Authentication Flags

• -u, --username <string> – username.

• -p, --password <string>– password.

Security Flags

• --tls-ca <string> – path to server CA certificate.

• --tls-key <string> – path to the client TLS private key.

• --tls-cert <string> – path to the client TLS certificate, containing the public key.

• --skip-verify – tells the client to skip verifying the server's certificate during the TLS handshake. TLS is enabled, and the connection is encrypted.

• --insecure – Disable TLS. The client connects over plain TCP, with no encryption or authentication.

gNMI Command

gNMI (gRPC Network Management Interface) is a standardized network management protocol to configure and monitor network devices via gRPC and Protocol Buffers. It enables interaction with devices using models such as OpenConfig.

Core gNMI commands:

• Get – Retrieves device data (e.g., operational state, configuration).

• Set – Modifies device configuration.

• Capabilities – Queries the device’s capabilities (e.g., supported models, versions, encodings).

• Subscribe – Streams data from the device to the client.

  Note

  DMS currently does not support the Subscribe command.

In DMSC, the gNMI functionality is powered by the GNMIC project. For additional details, refer to the GNMIC documentation.

Prompt mode with autocomplete can be invoked using the prompt command. To access it, run dmscli and authenticate with the dmsd user password (by default, the root OS password).

Get Request

Get requests happen in real-time without a cache. Get command requires providing the YANG Xpath as described in the following:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> get --path /interfaces/interface[name=p0]/config/mtu
[
  {
    "source": "localhost:9339",
    "timestamp": 1712485149723248511,
    "time": "2024-04-07T10:19:09.723248511Z",
    "updates": [
      {
        "Path": "interfaces/interface[name=p0]/config/mtu",
        "values": {
          "interfaces/interface/config/mtu": "1500"
        }
      }
    ]
  }
]
        

Get request also work on subtree (subPath) as follows:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> get --path /nvidia/roce
[
  {
    "source": "127.0.0.1:9339",
    "timestamp": 1728471432988295603,
    "time": "2024-10-09T13:57:12.988295603+03:00",
    "updates": [
      {
        "Path": "nvidia/roce",
        "values": {
          "nvidia/roce": {
            "config": {
              "adaptive-routing": "false",
              "multipath-dscp": "MULTIPATH_DSCP_DEFAULT",
              "tx-sched-locality-mode": "TX_SCHED_LOCALITY_ACCUMULATIVE"
            }
          }
        }
      }
    ]
  }
]
        

Set Request

Set requests happen immediately, invoking tools to configure the OS.

Set commands require providing Yang Xpath as described in the following:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> set --update /interfaces/interface[name=p0]/config/mtu:::int:::9216
{
  "source": "localhost:9339",
  "time": "1970-01-01T00:00:00Z",
  "results": [
    {
      "operation": "UPDATE",
      "path": "interfaces/interface[name=p0]/config/mtu"
    }
  ]
}
        

It is also possible to invoke a command JSON list:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> set --request-file req.json
        

req.json example:

            

            
{
  "updates":
    [
      {
        "path": "/interfaces/interface[name=p0]/config/mtu",
        "value": 9216,
        "encoding": "uint"
      },
      {
        "path": "/interfaces/interface[name=p0]/config/enabled",
        "value": true,
        "encoding": "bool"
      }
    ]
}
        

List of DMS Supported Path Entries for Get/Set Commands

To get the list of supported paths for get/set commands, run the following command:

            

            
dmsc --file /opt/mellanox/doca/service/dms/yang path --types --descr
        

gNOI Commands

gNOI (gRPC Network Operations Interface) is a protocol designed for operational tasks, such as installing software, managing certificates, and troubleshooting.

It complements gNMI by focusing on operational and administrative functions.

In DMSc, the gNOI part is powered by GNOIC project, for full docs refer to GNOIC docs.

            

            
/opt/mellanox/doca/services/dms/dmsc -a localhost --port 9339 --tls-cert client.crt --tls-key client.key <command>
        

Prompt mode with autocomplete options can be invoked using the command prompt.

All commands are blocked unless specified otherwise.

OS

The following subsections present actions for provisioning a new DOCA image (BFB) or firmware on BlueField.

Install

This command transmits the file from the client to the server and authenticates the file's validity:

            

            
# Command Format:
/opt/mellanox/doca/services/dms/dmsc <flags> os install --version <free_text_version> --pkg <bfb|cfg|fw path>
 
# Install BFB:
/opt/mellanox/doca/services/dms/dmsc <flags> os install --version 2_9_0 --pkg DOCA_2.9.0_Ubuntu.bfb
 
# Install cfg:
/opt/mellanox/doca/services/dms/dmsc <flags> os install --version 0_0_1.cfg --pkg bf.cfg
 
# Install FW
/opt/mellanox/doca/services/dms/dmsc <flags> os install --version 24_29_0046.fw --pkg fw-BlueField.bin 
        

The file is saved to the folder specified in the -image_folder flag (default /tmp/dms) given to the server. if the file authenticates successfully. The file's extension is autodetected and is written automatically if none is provided in the --version field. Users may copy the file to the folder manually and invoke the command with file extension to authenticate the file. No file transfer is initiated if the file already exists in the folder and the version specified with the extension.

Activate

This command deploys the BFB bundle/firmware to the hardware:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> os activate --version 2_9_0 # Invoke all files under 2_9_0 name 
/opt/mellanox/doca/services/dms/dmsc <flags> os activate --version "2_9_0.bfb;0_0_1.cfg;24_29_0046.fw"
        

The --version flag provides a version to search for in the folder specified by the -image_folder flag (default /tmp/dms). If no extension is provided, the command uses all files under the version name.

To activate separate files, use the --version flag separated by semi-colon.

Verify

Verify command retrieves the firmware and BFB bundle version:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> os verify
        

The return value consists of both versions separated by a semi-colon:

            

            
+-----------------+---------------------------------------------------------------+-----------------------------+
|   Target Name   |                            Version                            |    Activation Fail Msg      |
+-----------------+---------------------------------------------------------------+-----------------------------+
| 127.0.0.1:19999 | FW: 32.42.1000;BFB: bf-bundle-2.8.0-98_24.07-ubuntu-22.04_dev |                             |
|                 |                                                               |                             |
+-----------------+---------------------------------------------------------------+-----------------------------+
        

System

The following subsections provide actions for rebooting the BFB bundle/firmware on the BlueField.

Reboot Status

To verify that BFB is rebooting:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> system reboot-status
        

The value returned is false if the system is active. It is true if the system is rebooting. If the status cannot be retrieved, the status appears as a failure and the message field indicates what the issue is.

The flag --reboot_status_check <string> checks if firmware reboot is needed:

• If set to fast (default), a quick test occurs but is not accurate (any config can trigger this flag)

• If set to strict, a more accurate test occurs, but it is slower.

• If set to none, then the firmware check is skipped.

Reboot

To reboot the BlueField Arm and firmware:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> system reboot --delay <uint>s --subcomponent <string> --method <string>
        

This command is non-blocking and returns immediately.

The flag --delay specifies the time interval to wait before invoking the reset.

The subcomponent and method are optional. By default, the reboot executes with the lowest reset level and type available.

Factory-Reset (Alias: Reset)

The following subsection provides an action for performing a factory reset on a target node.

Start

Performs reset operation on each PF on the given target node.

This operation resets the device configurations to their default values:

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> factory-reset start
/opt/mellanox/doca/services/dms/dmsc <flags> reset start
        

File

The generic interface to perform file operational tasks.

Stat

Returns metadata about files on the target device.

            

            
/opt/mellanox/doca/services/dms/dmsc <flags> file stat --path /path/to/file
/opt/mellanox/doca/services/dms/dmsc <flags> file stat --path /path/to/dir --recursive 
        

Command output:

            

            
+----------------+---------------------+----------------------+------------+------------+------+
|  Target Name   |        Path         |     LastModified     |    Perm    |   Umask    | Size |
+----------------+---------------------+----------------------+------------+------------+------+
| 127.0.0.1:9339 | /path/to/dir/file_1 | 2025-04-10T09:05:37Z | -rw-r--r-- | -----w--w- | 0    |
|                | /path/to/dir/file_2 | 2025-04-10T09:05:37Z | -rw-r--r-- | -----w--w- | 0    |
+----------------+---------------------+----------------------+------------+------------+------+
        

Get

Streams the contents of a file from the target device to the client.

To allow this operation, the DMSD server must be started with a list of directories permitted for read access, using the --allow_list_read flag.

            

            
# Server
/opt/mellanox/doca/services/dms/dmsd <flags> --allow_list_read dir1,dir2,dir3
 
# Client
/opt/mellanox/doca/services/dms/dmsc <flags> file get --file /tmp/debug/file_to_get
        

Put

Streams a file from the client to the target device.

To allow this operation, the DMSD server must be started with a list of directories permitted for write access, using the --allow_list_write flag.

            

            
# Server
/opt/mellanox/doca/services/dms/dmsd <flags> --allow_list_write dir1,dir2,dir3
 
# Client
/opt/mellanox/doca/services/dms/dmsc <flags> file put --file /tmp/debug/file_to_send --dst /tmp/output/path_on_remote_host
        

Containerz

Containerz enables remote management and monitoring of containers running on the host or DPU. It supports the following backend modes:

Docker Backend - default

In this mode, Containerz interacts with the Docker daemon (dockerd) running on the host.

Ensure that the Docker daemon is active:

            

            
systemctl start docker
        

Kubelet Backend

In this mode, Containerz uses the Kubelet daemon to manage containers on the BlueField DPU via static pod definitions.

This does not require a full Kubernetes cluster. Instead, static pod manifests placed in /etc/kubelet.d are automatically handled by the Kubelet.

• Add a pod: Create a YAML file in /etc/kubelet.d – the pod will be started automatically.

• Remove a pod: Delete the corresponding YAML file – the pod will be stopped and removed.

To set the container backend mode, use the --container_interface flag when launching the DMSD server:

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> --container_interface <kubelet|docker>
        

Container Operations

These operations manage the lifecycle, images, and persistent volumes associated with containers.

Container Start

Starts a container using the specified image and tag.

• Docker mode – The image must already be available in the local registry.

• Kubelet mode – Pass a YAML file that defines the container. The file will be copied to the Kubelet manifest directory, triggering the daemon to start the container.

            

            
# Docker mode
/opt/mellanox/doca/services/dms/dmsd <flags> containerz container start --image <image_name>
 
# Kubelet mode
/opt/mellanox/doca/services/dms/dmsd <flags> containerz container start --image path/to/yaml/file.yaml
        

Container List

Lists all running containers.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz container list
        

Container Stop

Stops a running container. The container is preserved and can be restarted later.

• Kubelet mode – This deletes the YAML file from the manifest directory, which automatically stops the container.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz container stop --instance <container name>
        

Container Remove

Stops and removes the specified container.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz container remove --instance <container name>
        

Image Operations

These commands manage container images—pulling, pushing, listing, and removing them.

Image Pull

Downloads an image from a container registry.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz image pull --image ubuntu
        

Image Push

Uploads an image (in tar file format) from the client to the device. The image is streamed via gRPC and stored locally.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz image push --file /path/to/tarfile.tar
        

Image List

Displays all available container images on the device.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz image list
        

Image Remove

Deletes a specific image from the device.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz image remove --image ubuntu
        

Volume Operations

These commands manage persistent storage volumes for containers.

Volume Create

Creates a new named volume.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz volume create --name my_volume
        

Volume Remove

Deletes a specified volume.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz volume remove --name volume_name
        

Volume list

Lists all named volumes and their mount points.

            

            
/opt/mellanox/doca/services/dms/dmsd <flags> containerz volume list