NVIDIA BlueField Virtio-net v24.07
NVIDIA BlueField Virtio-net v24.07

Virtnet CLI Commands

To communicate with the virtio-net-controller backend service, a user frontend program, virtnet, is installed on the BlueField which is based on r emote procedure call (RPC) protocol with JSON format output. Run the following command to check its usage:

Copy
Copied!
            

# virtnet -h usage: virtnet [-h] [-v] {hotplug,unplug,list,query,modify,log,version,validate,update,debug,stats} ...   NVIDIA virtio-net-controller command line interface v24.07.04   positional arguments: {hotplug,unplug,list,query,modify,log,version,update,debug} ** Use -h for sub-command usage hotplug hotplug virtnet device unplug unplug virtnet device list list all virtnet devices query query all or individual virtnet device(s) modify modify virtnet device log set log level version show virtio net controller version info validate validate configurations   update update controller debug For debug purpose, cmds can be changed without notice stats               stats of virtnet device   optional arguments: -h, --help show this help message and exit -v, --version show program's version number and exit

Virtnet supports command line autocomplete by inputting one command with tab.

To check the currently running controller version:

Copy
Copied!
            

# virtnet -v v24.07.09

This command hotplugs a virtio-net PCIe PF device exposed to the host side.

Syntax

Copy
Copied!
            

virtnet hotplug -i IB_DEVICE -m MAC -t MTU -n MAX_QUEUES -s MAX_QUEUE_SIZE [-h] [-u SF_NUM] [-f FEATURES] [-l]

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--ib_device

-i

String

Yes

RDMA device (e.g., mlx5_0) of the physical port on top of which the hotplug device is created.

Options:

  • mlx5_0 – port 1

  • mlx5_1 – port 2

  • mlx5_bond_0 – LAG

--features

-f

Hex Number

No

Feature bits to be enabled in hex format. Refer to the "Virtio-net Feature Bits" page.

Note

Note that some features are enabled by default. Query the device to show the supported bits.

--mac

-m

Number

Yes

MAC address of the virtio-net device.

Note

Controller does not validate the MAC address (other than its length). The user must ensure MAC is valid and unique.

--mtu

-t

Number

Yes

Maximum transmission unit (MTU) size of the virtio-net device. It must be less than the uplink rep MTU size.

--num_queues

-n

Number

Yes

Mutually exclusive with max_queue_pairs.

Max number of virt queues could be created for the virtio-net device. TX, RX, ctrl queues are counted separately (e.g., 3 has 1 TX VQ, 1 RX VQ, 1 Ctrl VQ).

Note

This option will be depreciated in the future.

--max_queue_pairs

-qp

Number

Yes

Mutually exclusive with num_queues .

Number of data VQ pairs. One VQ pair has one TX queue and one RX queue. It does not count control or admin VQ. From the host side, it appears as Pre-set maximums->Combined in ethtool -l <virtio-dev>.

--max_queue_size

-s

Number

Yes

Maximum number of buffers in the virt queue, between 0x4 and 0x8000. Must be power of 2.

--sf_num

-u

Number

No

SF number to be used for this hotplug device, must between 2000 and 2999.

--legacy

-l

N/A

No

Create legacy (transitional) hotplug device


Output

Entry

Type

Description

    
bdf    

String

The PCIe BDF (bus:device:function) number enumerated by host. The user should see this PCIe device from host side.

    
vuid    

String

Unique device SN. It can be used as an index to query/modify/unplug this device.

    
id    

Num

Unique device ID. It can be used as an index to query/modify/unplug this device.

    
transitional    

Num

Is the current device a transitional hotplug device.

  • 0 – modern device

  • 1 – transitional device

    
sf_rep_net_device    

String

The SF representor name represents the virtio-net device. It should be added into the OVS bridge.

    
mac    

String

The hotplug virtio-net device MAC address

errno

Num

Error number if hotplug failed.

  • 0 – success

  • non-0 – failed

errstr

String

Explanation of the error number


Example

Hotplug one device with MAC address 0C:C4:7A:FF:22:93, MTU 1500, and 3 virtio queues (1 tx, 1 rx, 1 ctrl) with a depth of 1024 entries. The device is created on the physical port of mlx5_0.

Copy
Copied!
            

# virtnet hotplug -i mlx5_0 -f 0x80000 -m 0C:C4:7A:FF:22:93 -t 1500 -n 3 -s 1024 { "bdf": "15:00.0", "vuid": "MT2151X03152VNETS1D0F0", "id": 0, "transitional": 0, "sf_rep_net_device": "en3f0pf0sf2000", "mac": "0C:C4:7A:FF:22:93", "errno": 0, "errstr": "Success" }


This command unplugs a virtio-net PCIe PF device.

Syntax

Copy
Copied!
            

virtnet unplug [-h] [-p PF | -u VUID]

Only one of --pf and --vuid is needed to unplug the device.

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--pf

-p

Number

Yes

Unique device ID returned when doing hotplug. Can be retrieved by using virtnet list.

--vuid

-u

String

Yes

Unique device SN returned when doing hotplug. Can be retrieved by using virtnet list.


Output

Entry

Type

Description

errno

Num

Error number if operation failed

  • 0 – success

  • non-0 – failed

errstr

String

Explanation of the error number


Example

Unplug-hotplug device using the PF ID:

Copy
Copied!
            

# virtnet unplug -p 0 {'id': '0x1'} { "errno": 0, "errstr": "Success" }


This command lists all existing virtio-net devices, with global information and individual information for each device.

Syntax

Copy
Copied!
            

virtnet list [-h]

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit


Output

The output has two main sections. The first section wrapped by the controller are global configurations and capabilities.

Entry

Type

Description

controller

String

Entries under this section is global information for the controller

emulation_manager

String

The RDMA device manager used to manage internal resources. Should be default mlx5_0.

    
max_hotplug_devices    

String

Maximum number of devices that can be hotpluged

    
max_virt_net_devices    

String

Total number of emulated devices managed by the device emulation manager

    
max_virt_queues    

String

Maximum number of virt queues supported per device

    
max_tunnel_descriptors    

String

Maximum number of descriptors the device can send in a single tunnel request

    
supported_features    

String

Total list of features supported by device

    
supported_virt_queue_types    

String

Currently supported virt queue types: Packed and Split

supported_event_modes

String

Currently supported event modes: no_msix_mode, qp_mode, msix_mode

Each device has its own section under devices.

Entry

Type

Description

devices

String

Entries under this section is per device information

pf_id

String

Physical function ID

function_type

String

Function type: Static PF, hotplug PF, VF

    
transitional    

Number

The current device a transitional hotplug device:

  • 0 – modern device

  • 1 – transitional device

    
vuid    

String

Unique device SN, it can be used as an index to query/modify/unplug a device

    
pci_bdf    

String

Bus:device:function to describe the virtio-net PCIe device

pci_vhca_id

Number

Virtual HCA identifier for the general virtio-net device. For debug purposes only.

pci_max_vfs

Number

Maximum number of virtio-net VFs that can be created for this PF. Valid only for PFs.

    
enabled_vfs    

Number

Currently enabled number of virtio-net VFs for this PF

    
msix_num_pool_size    

Number

Number of free dynamic MSIX available for the VFs on this PF

    
min_msix_num    

Number

The minimum number of dynamic MSI-Xs that can be set for an virtio-net VF

    
max_msix_num    

Number

The maximum number of dynamic MSI-Xs that can be set for an virtio-net VF

    
min_num_of_qp    

Number

The minimum number of dynamic data VQ pairs (i.e., each pair has one TX and 1 RX queue) that can be set for an virtio-net VF

    
max_num_of_qp    

Number

The minimum number of dynamic data VQ pairs (i.e., each pair has one TX and 1 RX queue) that can be set for an virtio-net VF

    
qp_pool_size    

Number

Number of free dynamic data VQ pairs (i.e., each pair has one TX and 1 RX queue) available for the VFs on this PF

num_msix

Number

Maximum number of MSI-X available for this device

num_queues

Number

Maximum virtual queues can be created for this device, driver can choose to create less

enabled_queues

Number

Currently enabled number of virtual queues by the driver

    
max_queues_size    

Number

Maximum virtual queue depth in byte can be created for each VQ, driver can use less

    
msix_config_vector    

String

MSIX vector number used by the driver for the virtio config space. 0xFFFF means that no vector is requested.

    
mac    

String

The virtio-net device permanent MAC address, can be only changed from controller side via modify command

link_status

Number

Link status of the virtio-net device on the driver side

  • 0 – down

  • 1 – up

max_queue_pairs

Number

Number of data VQ pairs. One VQ pair has one TX queue and one RX queue. Control or admin VQ are not counted. From the host side, it appears as Pre-set maximums->Combined in ethtool -l <virtio-dev>.

    
mtu    

Number

The virtio-net device MTU. Default is 1500.

    
speed    

Number

The virtio-net device link speed in Mb/s

    
rss_max_key_size    

Number

The maximum supported length of the RSS key. Only applicable when VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is enabled.

    
supported_hash_types    

Number

Supported hash types for this device in hex. Only applicable when VIRTIO_NET_F_HASH_REPORT is enabled:

  • VIRTIO_NET_HASH_TYPE_IPv4 (bit 0)

  • VIRTIO_NET_HASH_TYPE_TCPv4 (bit 1)

  • VIRTIO_NET_HASH_TYPE_UDPv4 (bit 2)

  • VIRTIO_NET_HASH_TYPE_IPv6 (bit 3)

  • VIRTIO_NET_HASH_TYPE_TCPv6 (bit 4)

  • VIRTIO_NET_HASH_TYPE_UDPv6 (bit 5)

    
ctrl_mac    

String

Admin MAC address configured by driver. Not persistent with driver reload or host reboot.

ctrl_mq

Number

Number of queue pairs/channels configured by the driver. From the host side, it appears as Current hardware settings->Combined in ethtool -l <virtio-dev>.

sf_num

Number

Scalable function number used for this virtio-net device

sf_parent_device

String

The RDMA device to use to create the SF

    
sf_parent_device_pci_addr    

String

The PCIe device address (bus:device:function) to use to create the SF

    
sf_rep_net_device    

String

Represents the virtio-net device

    
sf_rep_net_ifindex    

Number

The SF representor network interface index

    
sf_rdma_device    

String

The SF RDMA device interface name

sf_cross_mkey

Number

The cross-device MKEY created for the SF. For debug purposes only.

sf_vhca_id

Number

Virtual HCA identifier for the SF. For debug purposes only.

rqt_num

Number

The RQ table ID used for this virtio-net device. For debug purposes only.

aarfs

String

Whether Accelerated Receive Flow Steering configuration is enabled or disabled


Example

The following is an example of a list with 1 static PF created:

Copy
Copied!
            

# virtnet list { "controller": { "emulation_manager": "mlx5_0", "max_hotplug_devices": "0", "max_virt_net_devices": "1", "max_virt_queues": "256", "max_tunnel_descriptors": "6", "supported_features": { "value": "0x8b00037700ef982f", " 0": "VIRTIO_NET_F_CSUM", " 1": "VIRTIO_NET_F_GUEST_CSUM", " 2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS", " 3": "VIRTIO_NET_F_MTU", " 5": "VIRTIO_NET_F_MAC", " 11": "VIRTIO_NET_F_HOST_TSO4", " 12": "VIRTIO_NET_F_HOST_TSO6", " 15": "VIRTIO_F_MRG_RX_BUFFER", " 16": "VIRTIO_NET_F_STATUS", " 17": "VIRTIO_NET_F_CTRL_VQ", " 18": "VIRTIO_NET_F_CTRL_RX", " 19": "VIRTIO_NET_F_CTRL_VLAN", " 21": "VIRTIO_NET_F_GUEST_ANNOUNCE", " 22": "VIRTIO_NET_F_MQ", " 23": "VIRTIO_NET_F_CTRL_MAC_ADDR", " 32": "VIRTIO_F_VERSION_1", " 33": "VIRTIO_F_IOMMU_PLATFORM", " 34": "VIRTIO_F_RING_PACKED", " 36": "VIRTIO_F_ORDER_PLATFORM", " 37": "VIRTIO_F_SR_IOV", " 38": "VIRTIO_F_NOTIFICATION_DATA", " 40": "VIRTIO_F_RING_RESET", " 41": "VIRTIO_F_ADMIN_VQ", " 56": "VIRTIO_NET_F_HOST_USO", " 57": "VIRTIO_NET_F_HASH_REPORT", " 59": "VIRTIO_NET_F_GUEST_HDRLEN", " 63": "VIRTIO_NET_F_SPEED_DUPLEX" }, "supported_virt_queue_types": { "value": "0x1", " 0": "SPLIT" }, "supported_event_modes": { "value": "0x5", " 0": "NO_MSIX_MODE", " 2": "MSIX_MODE" } }, "devices": [ { "pf_id": 0, "function_type": "static PF", "transitional": 0, "vuid": "MT2306XZ00BNVNETS0D0F2", "pci_bdf": "e2:00.2", "pci_vhca_id": "0x2", "pci_max_vfs": "0", "enabled_vfs": "0", "msix_num_pool_size": 0, "min_msix_num": 0, "max_msix_num": 256, "min_num_of_qp": 0, "max_num_of_qp": 127, "qp_pool_size": 0, "num_msix": "256", "num_queues": "255", "enabled_queues": "0", "max_queue_size": "256", "msix_config_vector": "0xFFFF", "mac": "16:B0:E0:41:B8:0D", "link_status": "1", "max_queue_pairs": "127", "mtu": "1500", "speed": "100000", "rss_max_key_size": "0", "supported_hash_types": "0x0", "ctrl_mac": "00:00:00:00:00:00", "ctrl_mq": "0", "sf_num": 1000, "sf_parent_device": "mlx5_0", "sf_parent_device_pci_addr": "0000:03:00.0", "sf_rep_net_device": "en3f0pf0sf1000", "sf_rep_net_ifindex": 10, "sf_rdma_device": "mlx5_3", "sf_cross_mkey": "0x12642", "sf_vhca_id": "0x124", "sf_rqt_num": "0x0", "aarfs": "disabled" } ] }


This command queries detailed information for a given device, including all VQ information if created.

Syntax

Copy
Copied!
            

virtnet query [-h] {[-a] | [-p PF] [-v VF] | [-u VUID]} [--dbg_stats] [-b] [--latency_stats] [-q QUEUE_ID] [--stats_clear]

Info

The options --pf , --vf , --vuid , and --all are mutually exclusive, but one of them must be applied.

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--all

-a

N/A

No

Query all the detailed information for all available devices. It can be time consuming if a large number of devices is available.

--pf

-p

Number

No

Unique device ID for the PF. Can be retrieved by using virtnet list.

--vf

-v

Number

No

Unique device ID for the VF. Can be retrieved by using virtnet list.

--vuid

-u

String

No

Unique device SN for the device (PF/VF). Can be retrieved by using virtnet list.

--queue_id

-q

Number

No

Queue index of the device VQs

--brief

-b

N/A

No

Query brief information of the device (does not print VQ information)

--dbg_stats

N/A

N/A

No

Print debug counters and information

Note

This option will be depreciated in the future.

--stats_clear

N/A

N/A

No

Clear all the debug counter stats

Note

This option will be depreciated in the future.


Output

Output has two main sections.

  • The first section, wrapped by devices, are configuration and capabilities on the device level, the majority of which are the same as the list command. This section only covers the differences between the two.

    Entry

    Type

    Description

    devices

    String

    Entries under this section is per-device information

        
    pci_dev_id    
    

    String

    Virtio-net PCIe device ID. Default: 0x1041.

    Note

    This option will be depreciated in the future.

        
    pci_vendor_id    
    

    String

    Virtio-net PCIe vendor ID. Default: 0x1af4.

    Note

    This option will be depreciated in the future.

        
    pci_class_code    
    

    String

    Virtio-net PCIe device class code. Default: 0x20000.

    Note

    This option will be depreciated in the future.

        
    pci_subsys_id    
    

    String

    Virtio-net PCIe vendor ID. Default: 0x1041.

    Note

    This option will be depreciated in the future.

        
    pci_subsys_vendor_id    
    

    String

    Virtio-net PCIe subsystem vendor ID. Default: 0x1af4.

    Note

    This option will be depreciated in the future.

        
    pci_revision_id    
    

    String

    Virtio-net PCIe revision ID. Default: 1.

    Note

    This option will be depreciated in the future.

        
    device_features    
    

    String

    Enabled device feature bits according to the virtio spec. Refer to Virtio-net Feature Bits.

        
    driver_features    
    

    String

    Enabled driver feature bits according to the virtio spec. Valid only when the driver probes the device. Refer to Virtio-net Feature Bits.

        
    status    
    

    String

    Device status field bit masks according to the virtio spec:

    • ACKNOWLEDGE (bit 0)

    • DRIVER (bit 1)

    • DRIVER_OK (bit 2)

    • FEATURES_OK (bit 3)

    • DEVICE_NEEDS_RESET (bit 6)

    • FAILED (bit 7)

    reset

    Number

    Shows if the current virtio-net device undergoing reset:

    • 0 – not undergoing reset

    • 1 – undergoing reset

    enabled

    Number

    Shows if the current virtio-net device is enabled:

    • 0 – disabled, likely FLR has occurred

    • 1 – enabled

  • The second section, wrapped by enabled-queues-info, provides per-VQ information:

    Entry

    Type

    Description

    index

    Number

    VQ index starting from 0 to enabled_queues

    size

    Number

    Driver VQ depth in bytes. It is bound by device max_queues_size .

    msix_vector

    Number

    The MSI-X vector number used for this VQ

    enable

    Number

    If current VQ is enabled or not

    • 0 – disabled

    • 1 – enabled

    notify_offset

    Number

    Driver reads this to calculate the offset from start of notification structure at which this virtqueue is located

    descriptor_address

    Number

    The physical address of the descriptor area

    driver_address

    Number

    The physical address of the driver area

    device_address

    Number

    The physical address of the device area

    received_desc

    Number

    Total number of received descriptors by the device on this VQ

    Note

    This option will be depreciated in the future.

    completed_desc

    Number

    Total number of completed descriptors by the device on this VQ

    Note

    This option will be depreciated in the future.

    bad_desc_errors

    Number

    Total number of bad descriptors received on this VQ

    Note

    This option will be depreciated in the future.

    error_cqes

    Number

    Total number of error CQ entries on this VQ

    Note

    This option will be depreciated in the future.

    exceed_max_chain

    Number

    Total number of chained descriptors received that exceed the maximum allowed chain by device

    Note

    This option will be depreciated in the future.

    invalid_buffer

    Number

    Total number of times the device tried to read or write buffer that is not registered to the device

    Note

    This option will be depreciated in the future.

    batch_number

    Number

    The number of RX descriptors for the last received packet. Relevant for BlueField-3 only.

    Note

    This option will be depreciated in the future.

    dma_q_used_number

    Number

    The DMA q index used for this VQ. Relevant for BlueField-3 only.

    Note

    This option will be depreciated in the future.

    handler_schd_number

    Number

    Scheduler number for this VQ. Relevant for BlueField-3 only.

    Note

    This option will be depreciated in the future.

    aux_handler_schd_number

    Number

    Aux scheduler number for this VQ. Relevant for BlueField-3 only.

    Note

    This option will be depreciated in the future.

    max_post_desc_number

    Number

    Maximum number of posted descriptors on this VQ. Relevant for DPA.

    Note

    This option will be depreciated in the future.

    total_bytes

    Number

    Total number of bytes handled by this VQ. Relevant for BlueField-3 only

    Note

    This option will be depreciated in the future.

    rq_cq_max_count

    Number

    Event generation moderation counter of the queue. Relevant for RQ.

    Note

    This option will be depreciated in the future.

    rq_cq_period

    Number

    Event generation moderation timer for the queue in 1 µ sec granularity. Relevant for RQ.

    Note

    This option will be depreciated in the future.

    rq_cq_period_mode

    Number

    Current period mode for RQ

    • 0x0 – default_mode – use device best defaults

    • 0x1 – upon_eventqueue_period timer restarts upon event generation

    • 0x2 – upon_cqequeue_period timer restarts upon completion generation

    Note

    This option will be depreciated in the future.

Example

The following is an example of querying the information of the first PF:

Copy
Copied!
            

# virtnet query -p 0 { "devices": [ { "pf_id": 0, "function_type": "static PF", "transitional": 0, "vuid": "MT2349X00018VNETS0D0F1", "pci_bdf": "23:00.1", "pci_vhca_id": "0x1", "pci_max_vfs": "0", "enabled_vfs": "0", "pci_dev_id": "0x1041", "pci_vendor_id": "0x1af4", "pci_class_code": "0x20000", "pci_subsys_id": "0x1041", "pci_subsys_vendor_id": "0x1af4", "pci_revision_id": "1",       "device_feature": { "value": "0x8930032300e7182f", " 0": "VIRTIO_NET_F_CSUM", " 1": "VIRTIO_NET_F_GUEST_CSUM", " 2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS", " 3": "VIRTIO_NET_F_MTU", " 5": "VIRTIO_NET_F_MAC", " 11": "VIRTIO_NET_F_HOST_TSO4", " 12": "VIRTIO_NET_F_HOST_TSO6", " 16": "VIRTIO_NET_F_STATUS", " 17": "VIRTIO_NET_F_CTRL_VQ", " 18": "VIRTIO_NET_F_CTRL_RX", " 21": "VIRTIO_NET_F_GUEST_ANNOUNCE", " 22": "VIRTIO_NET_F_MQ", " 23": "VIRTIO_NET_F_CTRL_MAC_ADDR", " 32": "VIRTIO_F_VERSION_1", " 33": "VIRTIO_F_IOMMU_PLATFORM", " 37": "VIRTIO_F_SR_IOV", " 40": "VIRTIO_F_RING_RESET", " 41": "VIRTIO_F_ADMIN_VQ", " 52": "VIRTIO_NET_F_VQ_NOTF_COAL", " 53": "VIRTIO_NET_F_NOTF_COAL", " 56": "VIRTIO_NET_F_HOST_USO", " 59": "VIRTIO_NET_F_GUEST_HDRLEN", " 63": "VIRTIO_NET_F_SPEED_DUPLEX" }, "driver_feature": { "value": "0x8000002300e7182f", " 0": "VIRTIO_NET_F_CSUM", " 1": "VIRTIO_NET_F_GUEST_CSUM", " 2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS", " 3": "VIRTIO_NET_F_MTU", " 5": "VIRTIO_NET_F_MAC", " 11": "VIRTIO_NET_F_HOST_TSO4", " 12": "VIRTIO_NET_F_HOST_TSO6", " 16": "VIRTIO_NET_F_STATUS", " 17": "VIRTIO_NET_F_CTRL_VQ", " 18": "VIRTIO_NET_F_CTRL_RX", " 21": "VIRTIO_NET_F_GUEST_ANNOUNCE", " 22": "VIRTIO_NET_F_MQ", " 23": "VIRTIO_NET_F_CTRL_MAC_ADDR", " 32": "VIRTIO_F_VERSION_1", " 33": "VIRTIO_F_IOMMU_PLATFORM", " 37": "VIRTIO_F_SR_IOV", " 63": "VIRTIO_NET_F_SPEED_DUPLEX" }, "status": { "value": "0xf", " 0": "ACK", " 1": "DRIVER", " 2": "DRIVER_OK", " 3": "FEATURES_OK" }, "reset": "0", "enabled": "1", "num_msix": "64", "num_queues": "63", "enabled_queues": "63", "max_queue_size": "256", "msix_config_vector": "0x0", "mac": "4E:6A:E1:41:D8:BE", "link_status": "1", "max_queue_pairs": "31", "mtu": "1500", "speed": "200000", "rss_max_key_size": "0", "supported_hash_types": "0x0", "ctrl_mac": "4E:6A:E1:41:D8:BE", "ctrl_mq": "31", "sf_num": 1000, "sf_parent_device": "mlx5_0", "sf_parent_device_pci_addr": "0000:03:00.0", "sf_rep_net_device": "en3f0pf0sf1000", "sf_rep_net_ifindex": 12, "sf_rdma_device": "mlx5_2", "sf_cross_mkey": "0xC042", "sf_vhca_id": "0x7E8", "sf_rqt_num": "0x0", "aarfs": "disabled"      "enabled-queues-info": [ { "index": "0", "size": "256", "msix_vector": "0x1", "enable": "1", "notify_offset": "0", "descriptor_address": "0x10cece000", "driver_address": "0x10cecf000", "device_address": "0x10cecf240", "received_desc": "256", "completed_desc": "0", "bad_desc_errors": "0", "error_cqes": "0", "exceed_max_chain": "0", "invalid_buffer": "0", "batch_number": "64", "dma_q_used_number": "6", "handler_schd_number": "4", "aux_handler_schd_number": "3", "max_post_desc_number": "0", "total_bytes": "0", "rq_cq_max_count": "0", "rq_cq_period": "0", "rq_cq_period_mode": "1" }, ...... } ] } ] }


Tip

This command is recommended for obtaining all packet counter information. The existing packet counter information available using the virtnet list and virtnet query commands, but will be deprecated in the future.

This command retrieves the packet counters for a specified device, including detailed information for all Rx and Tx virtqueues (VQs).

To enable/disable byte wise packet counters for each Rx queue, use the following command:

Copy
Copied!
            

virtnet modify {[-p PF] [-v VF]} device -pkt_cnt {enable,disable}

  • When enabled, byte-wise packet counters are initialized to zero.

  • When disabled, the previous values are retained for debugging purposes. The command will still return these old, disabled counter values.

Note

Packet counters are attached to an RQ. Thus, RQ must be created first. This means that the virtio-net device should be probed by the driver on the host OS before running the commands above.

Syntax

Copy
Copied!
            

virtnet stats [-h] {[-p PF] [-v VF] | [-u VUID]} [-q QUEUE_ID]

Info

The options --pf , --vf , and --vuid are mutually exclusive, but one of them must be applied.

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--pf

-p

Number

No

Unique device ID for the PF. Can be retrieved by using virtnet list.

--vf

-v

Number

No

Unique device ID for the VF. Can be retrieved by using virtnet list.

--vuid

-u

String

No

Unique device SN for the device (PF/VF). Can be retrieved by using virtnet list.

--queue_id

-q

Number

No

Queue index of the device RQs or SQs


Output

The output has two sections.

  • The first section wrapped by device are device details along with the packet counter statics enable state.

    Entry

    Type

    Description

    device

    String

    Entries under this section is per-device information

    pf_id

    String

    Physical function ID

    packet_counters

    String

    Indicates whether the packet counters feature is enabled or disabled

  • The second section wrapped by queues-stats are information for each receive VQ.

    Entry

    Type

    Description

    VQ Index

    Number

    The VQ index starts at 0 (the first RQ) and continues up to the last SQ

    rx_64_or_less_octet_packets

    Number

    The number of packets received with a size of 0 to 64 bytes. Relevant for BlueField-3 RQ.

    rx_65_to_127_octet_packets

    Number

    The number of packets received with a size of 65 to 127 bytes. Relevant for BlueField-3 RQ.

    rx_128_to_255_octet_packets

    Number

    The number of packets received with a size of 128 to 255 bytes. Relevant for BlueField-3 RQ.

    rx_256_to_511_octet_packets

    Number

    The number of packets received with a size of 256 to 511 bytes. Relevant for BlueField-3 RQ.

    rx_512_to_1023_octet_packets

    Number

    The number of packets received with a size of 512 to 1023 bytes. Relevant for BlueField-3 RQ.

    rx_1024_to_1522_octet_packets

    Number

    The number of packets received with a size of 1024 to 1522 bytes. Relevant for BlueField-3 RQ.

    rx_1523_to_2047_octet_packets

    Number

    The number of packets received with a size of 1523 to 2047 bytes. Relevant for BlueField-3 RQ.

    rx_2048_to_4095_octet_packets

    Number

    The number of packets received with a size of 2048 to 4095 bytes. Relevant for BlueField-3 RQ.

    rx_4096_to_8191_octet_packets

    Number

    The number of packets received with a size of 4096 to 8191 bytes. Relevant for BlueField-3 RQ.

    rx_8192_to_9022_octet_packets

    Number

    The number of packets received with a size of 8192 to 9022 bytes. Relevant for BlueField-3 RQ.

    received_desc

    Number

    Total number of received descriptors by the device on this VQ

    completed_desc

    Number

    Total number of completed descriptors by the device on this VQ

    bad_desc_errors

    Number

    Total number of bad descriptors received on this VQ

    error_cqes

    Number

    Total number of error CQ entries on this VQ

    exceed_max_chain

    Number

    Total number of chained descriptors received that exceed the max allowed chain by device

    invalid_buffer

    Number

    Total number of times the device tried to read or write a buffer which is not registered to the device

    batch_number

    Number

    The number of RX descriptors for the last received packet. Relevant for BlueField-3.

    dma_q_used_number

    Number

    The DMA q index used for this VQ. Relevant for BlueField-3.

    handler_schd_number

    Number

    Scheduler number for this VQ. Relevant for BlueField-3.

    aux_handler_schd_number

    Number

    Aux scheduler number for this VQ. Relevant for BlueField-3.

    max_post_desc_number

    Number

    Maximum number of posted descriptors on this VQ. Relevant for DPA.

    total_bytes

    Number

    Total number of bytes handled by this VQ. Relevant for BlueField-3.

    rq_cq_max_count

    Number

    Event generation moderation counter of the queue. Relevant for RQ.

    rq_cq_period

    Number

    Event generation moderation timer for the queue in 1 µ sec granularity. Relevant for RQ.

    rq_cq_period_mode

    Number

    Current period mode for RQ

    • 0x0 – default_mode – use device best defaults

    • 0x1 – upon_eventqueue_period timer restarts upon event generation

    • 0x2 – upon_cqequeue_period timer restarts upon completion generation

Example

The following is an example of querying the packet statistics information of PF 0 and VQ 0 (i.e., RQ):

Copy
Copied!
            

# virtnet stats -p 0 -q 0 {'pf': '0x0', 'queue_id': '0x0'} { "device": { "pf_id": 0, "packet_counters": "Enabled", "queues-stats": [ { "VQ Index": 0, "rx_64_or_less_octet_packets": 0, "rx_65_to_127_octet_packets": 259, "rx_128_to_255_octet_packets": 0, "rx_256_to_511_octet_packets": 0, "rx_512_to_1023_octet_packets": 0, "rx_1024_to_1522_octet_packets": 0, "rx_1523_to_2047_octet_packets": 0, "rx_2048_to_4095_octet_packets": 199, "rx_4096_to_8191_octet_packets": 0, "rx_8192_to_9022_octet_packets": 0, "received_desc": "4096", "completed_desc": "0", "bad_desc_errors": "0", "error_cqes": "0", "exceed_max_chain": "0", "invalid_buffer": "0", "batch_number": "64", "dma_q_used_number": "0", "handler_schd_number": "44", "aux_handler_schd_number": "43", "max_post_desc_number": "0", "total_bytes": "0", "err_handler_schd_num": "0", "rq_cq_max_count": "0", "rq_cq_period": "0", "rq_cq_period_mode": "1" } ] } }  


This command modifies the attributes of a given device.

Syntax

Copy
Copied!
            

virtnet modify [-h] [-p PF] [-v VF] [-u VUID] [-a] {device,queue} ...

Info

The options --pf , --vf , --vuid , and --all are mutually exclusive, but one of them must be applied.

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--all

-a

N/A

No

Modify all available device attributes depending on the selection of device or queue

--pf

-p

Number

No

Unique device ID for the PF. May be retrieved using virtnet list.

--vf

-v

Number

No

Unique device ID for the VF. May be retrieved using virtnet list.

--vuid

-u

String

No

Unique device SN for the device (PF/VF). May be retrieved by using virtnet list.

device

N/A

Number

No

Modify device specific options

queue

N/A

N/A

No

Modify queue specific options

Device Options

Copy
Copied!
            

virtnet modify device [-h] [-m MAC] [-t MTU] [-e SPEED] [-l LINK] [-s STATE] [-f FEATURES] [-o SUPPORTED_HASH_TYPES] [-k RSS_MAX_KEY_SIZE] [-r RX_MODE] [-n MSIX_NUM] [-q MAX_QUEUE_SIZE] [-d DST_PORT] [-b RX_DMA_Q_NUM] [-dim {enable,disable}] [-dc {enable,disable}] [-pkt_cnt {enable,disable}] [-aarfs {enable,disable}] [-qp MAX_QUEUE_PAIRS]

Option

Abbr

Argument Type

Required

Description

--help

-h

String

No

Show the help message and exit

--mac

-m

Number

No

The virtio-net device MAC address

--mtu

-t

Number

No

The virtio-net device MTU

--speed

-e

Number

No

The virtio-net device link speed in Mb/s

--link

-l

Number

No

The virtio-net device link status

  • 0 – down

  • 1 – up

--state

-s

Number

No

The virtio-net device status field bit masks according to the virtio spec:

  • ACKNOWLEDGE (bit 0)

  • DRIVER (bit 1)

  • DRIVER_OK (bit 2)

  • FEATURES_OK (bit 3)

  • DEVICE_NEEDS_RESET (bit 6)

  • FAILED (bit 7)

--features

-f

Hex Number

No

The virtio-net device feature bits according to the virtio spec

--supported_hash_types

-o

Hex Number

No

Supported hash types for this device in hex. Only applicable when VIRTIO_NET_F_HASH_REPORT is enabled.

  • VIRTIO_NET_HASH_TYPE_IPv4 (bit 0)

  • VIRTIO_NET_HASH_TYPE_TCPv4 (bit 1)

  • VIRTIO_NET_HASH_TYPE_UDPv4 (bit 2)

  • VIRTIO_NET_HASH_TYPE_IPv6 (bit 3)

  • VIRTIO_NET_HASH_TYPE_TCPv6 (bit 4)

  • VIRTIO_NET_HASH_TYPE_UDPv6 (bit 5)

--rss_max_key_size

-k

Number

No

The maximum supported length of RSS key. Only applicable when VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is enabled.

--rx_mode

-r

Hex Number

No

The RX mode exposed to the driver:

  • 0 – promisc

  • 1 – all-multi

  • 2 – all-uni

  • 3 – no-multi

  • 4 – no-uni

  • 5 – no-broadcast

--msix_num

-n

Number

No

Maximum number of VQs (both data and ctrl/admin VQ). It is bound by the cap of max_virt_queues at the controller level (virtnet list).

--max_queue_size

-q

Number

No

Maximum number of buffers in the VQ. The queue size value is always a power of 2. The maximum queue size value is 32768.

--max_queue_pairs

-qp

Number

No

Number of data VQ pairs. One VQ pair has one TX queue and one RX queue. Control or admin VQs are not counted. From the host side, it appears as Pre-set maximums->Combined in ethtool -l <virtio-dev> .

--dst_port

-d

Hex number

No

Modify IPv4 dst_port rules.

Note

Will be depreciated in the future.

--rx_dma_q_num

-b

Number

No

Modify max RX DMA queue number

--rx_dim_config

-dim

String

No

Enable/disable RX dynamic interrupt moderation

--drop_counter

-dc

String

No

Enable/disable virtio-net drop counter

--packet_counter

-pkt_cnt

String

No

Enable/disable virtio-net device packet counter stats

--aarfs_config

-aarfs

String

No

Enable/disable auto-AARFS. Only applicable for PF devices (static PF and hotplug PF).

Note

The following modify options require unbinding the virtio device from virtio-net driver in the guest OS:

  • mac

  • mtu

  • features

  • msix_num

  • max_queue_size

  • max_queue_pairs

For example:

  1. On the guest OS:

    Copy
    Copied!
                

    [host]# echo "bdf of virtio-dev" > /sys/bus/pci/drivers/virtio-pci/unbind

  2. On the DPU side:

    1. Modify the max queue size of device:

      Copy
      Copied!
                  

      [dpu]# virtnet modify -p 0 -v 0 device -q 2048

    2. Modify the MSI-X number of VF device:

      Copy
      Copied!
                  

      [dpu]# virtnet modify -p 0 -v 0 device -n 8

    3. Modify the MAC address of virtio physical device ID 0 (or with its "VUID string", which can be obtained through virtnet list/query):

      Copy
      Copied!
                  

      [dpu]# virtnet modify -p 0 device -m 0C:C4:7A:FF:22:93

    4. Modify the maximum number of queue pairs of VF device:

      Copy
      Copied!
                  

      [dpu]# virtnet modify -p 0 -v 0 device -qp 2

  3. On the guest OS:

    Copy
    Copied!
                

    [host]# echo "bdf of virtio-dev" > /sys/bus/pci/drivers/virtio-pci/bind


Queue Options

Copy
Copied!
            

virtnet modify queue [-h] -e {event,cqe} -n PERIOD -c MAX_COUNT

Option

Abbr

Argument Type

Required

Description

--help

-h

String

No

Show the help message and exit

--period_mode

-e

String

No

RQ period mode: event or cqe. Default is selected by device for the best result.

--period

-n

Number

No

The event generation moderation timer for the queue in 1 µ sec granularity

--max_count

-c

Number

No

The max event generation moderation counter of the queue

Output

Entry

Type

Description

errno

Number

Error number:

  • 0 – success

  • Non-0 – failed

errstr

String

Explanation of the error number


Example

To modify the link status of the first VF on the first PF to be down:

Copy
Copied!
            

# virtnet modify -p 0 device -l 0 {'pf': '0x0', 'all': '0x0', 'subcmd': '0x0', 'link': '0x0'} { "errno": 0, "errstr": "Success" }


This command manages the log level of virtio-net-controller.

Syntax

Copy
Copied!
            

virtnet log [-h] -l {info,err,debug}

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--level

-l

String

Yes

Change the log level of virtio_net_controller from the journal. Default is DEBUG.


Output

Entry

Type

Description

Stdout

String

Success or failed with message


Example

To change the log level to info:

Copy
Copied!
            

# virtnet log -l info {'level': 'info'} "Success"

To monitor current log output of the controller service with the latest 100 lines printed out:

Copy
Copied!
            

$ journalctl -u virtio-net-controller -f -n 100


This command validates configurations of virtio-net-controller.

Syntax

Copy
Copied!
            

virtnet validate [-h] -f PATH_TO_FILE

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--file

-f

String

No

Validate the JSON format of the virtnet.conf file of the virtio_net_controller


Output

Entry

Type

Description

Stdout

String

Success or failed with message


Example

To check if virtnet.conf is a valid JSON file:

Copy
Copied!
            

# virtnet validate -f /opt/mellanox/mlnx_virtnet/virtnet.conf /opt/mellanox/mlnx_virtnet/virtnet.conf is valid


This command prints current and updated version of virtio-net-controller.

Syntax

Copy
Copied!
            

virtnet version [-h]

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit


Output

Entry

Type

Description

Original Controller

String

The original controller version

Destination Controller

String

The to be updated controller version


Example

Check current and next available controller version:

Copy
Copied!
            

# virtnet version [ { "Original Controller": "v1.8.12" }, { "Destination Controller": "v1.9.14" } ]


Live update minimizes network interface down time by performing online upgrade of the virtio-net controller without necessitating a full restart.

Syntax

Copy
Copied!
            

virtnet update [-h] [-s | -t]

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--start

-s

N/A

No

Start live update virtio-net-controller

--status

-t

N/A

No

Check live update status


Output

Entry

Type

Description

stdout

String

If the update started successfully


Example

To start the live update process, run:

Copy
Copied!
            

# virtnet update -s {'start': '0x1'} "Update started, use 'virtnet update -t' or check logs for status"

To check the update status during the update process:

Copy
Copied!
            

# virtnet update -t {'status': '0x1'} { "current status": "inactive", "last status": "success", "time_used (s)": 0.604152 }


CLI commands will return non-0 error code upon failure. All error numbers are negative. When there is error happening from log, it could return error number as well.

If the error number is greater than -1000, it's standard error. Please refer to Linux error code at errno

If the error number is less or equal -1000, please refer to the table below for the explaination.

Errno

Error Name

Error Description

-1000

VIRTNET_ERR_DEV_FEATURE_VALIDATE

Failed to validate device feature

-1001

VIRTNET_ERR_DEV_NOT_FOUND

Failed to find device

-1002

VIRTNET_ERR_DEV_NOT_PLUGGED

Failed - Device is not hotplugged

-1003

VIRTNET_ERR_DEV_NOT_STARTED

Failed - Device did not start

-1004

VIRTNET_ERR_DRIVER_PROBED

Failed - Virtio driver should not be loaded

-1005

VIRTNET_ERR_EPOLL_ADD

Failed to add epoll

-1006

VIRTNET_ERR_ID_OUT_OF_RANGE

Failed - ID input exceeds the max range

-1007

VIRTNET_ERR_VUID_INVALID

Failed - VUID is invalid

-1008

VIRTNET_ERR_MAC_INVALID

Failed - MAC is invalid

-1009

VIRTNET_ERR_MSIX_INVALID

Failed - MSIX is invalid

-1010

VIRTNET_ERR_MTU_INVALID

Failed - MTU is invalid

-1011

VIRTNET_ERR_PORT_CONTEXT_NOT_FOUND

Failed to find port contex

-1012

VIRTNET_ERR_REC_CONFIG_LOAD

Failed to load config from recovery file

-1013

VIRTNET_ERR_REC_CONFIG_SAVE

Failed to save config into recovery file

-1014

VIRTNET_ERR_REC_FILE_CREATE

Failed to create recovery file

-1015

VIRTNET_ERR_REC_MAC_DEL

Failed to delete MAC in recovery file

-1016

VIRTNET_ERR_REC_MAC_LOAD

Failed to load MAC from recovery file

-1017

VIRTNET_ERR_REC_MAC_SAVE

Failed to save MAC into recovery file

-1018

VIRTNET_ERR_REC_MQ_SAVE

Failed to save MQ into recovery file

-1019

VIRTNET_ERR_REC_PFNUM_LOAD

Failed to load PF number from recovery file

-1020

VIRTNET_ERR_REC_RX_MODE_SAVE

Failed to save RX mode into recovery file

-1021

VIRTNET_ERR_REC_SF_SAVE

Failed to save PF and SF number into recovery file

-1022

VIRTNET_ERR_REC_SFNUM_LOAD

Failed to load SF number from recovery file

-1023

VIRTNET_ERR_SF_MAC_FLOW_APPLY

Failed to apply MAC flow by SF

-1024

VIRTNET_ERR_SF_MQ_UPDATE

Failed to update MQ by SF

-1025

VIRTNET_ERR_SF_RX_MODE_SET

Failed to set RX mode by SF

-1026

VIRTNET_ERR_SNAP_NET_CTRL_OPEN

Failed to open SNAP device control

-1027

VIRTNET_ERR_SNAP_CROSS_MKEY_CREATE

Failed to create SNAP cross mkey

-1028

VIRTNET_ERR_SNAP_DMA_Q_CREATE

Failed to create SNAP DMA Q

-1029

VIRTNET_ERR_SNAP_NET_DEV_QUERY

Failed to query SNAP device

-1030

VIRTNET_ERR_SNAP_NET_DEV_MODIFY

Failed to modify SNAP device

-1031

VIRTNET_ERR_SNAP_PF_HOTPLUG

Failed to hotplug SNAP PF

-1032

VIRTNET_ERR_VQ_PERIOD_UPDATE

Failed to update VQ period

-1033

VIRTNET_ERR_QUEUE_SIZE_INVALID

Failed - Queue size is invalid

-1034

VIRTNET_ERR_SF_PORT_ADD

Failed to add SF port

-1035

VIRTNET_ERR_WQ_WORKQUEUE_ALLOC

Failed to alloc workqueue

-1036

VIRTNET_ERR_ETH_VQS_OPERATION_ALLOC

Failed to alloc eth VQS operation

-1037

VIRTNET_ERR_ETH_VQS_OPERATION_COMP

Failed to complete eth VQS operation

-1038

VIRTNET_ERR_JSON_OBJ_NOT_EXIST

Failed - JSON obj does not exist

-1039

VIRTNET_ERR_DEV_LOAD_PREP

Failed to prepare device load

-1040

VIRTNET_ERR_DEV_SW_MIGRATION

Failed to sw migrate a device

-1041

VIRTNET_ERR_DEV_IS_SW_MIGRATING

Failed - Device is migrating

-1042

VIRTNET_ERR_MAX_QUEUE_SIZE

Error - queue size must be greater than 2 and is power of 2

-1043

VIRTNET_ERR_MSIX_LESS_EQUAL_THREE

Warning - this device won't function, don't try to probe with virtio driver

-1044

VIRTNET_ERR_SF_POOL_CREATING

SF pool is creating try again later

-1045

VIRTNET_ERR_DST_PORT

Failed to set dst port rule

-1046

VIRTNET_ERR_INVALID_OPTION

Option is not supported

-1047

VIRTNET_ERR_SF_CREATE

Failed to create SF

-1048

VIRTNET_ERR_DEV_SF_NUM_OUT_OF_RANGE

SF number for hotplug device should be between 2000 and 2999

-1049

VIRTNET_ERR_DEV_SF_NUM_USED

SF number is already used

-1050

VIRTNET_ERR_QUEUE_NUMBER_INVALID

Queue index is invalid

-1051

VIRTNET_ERR_SPEED_INVALID

Invalid speed please check help menu for supported link speeds

-1052

VIRTNET_ERR_SUPPORTED_HASH_TYPES_INVALID

Invalid hash types please check help menu for supported hash types

-1053

VIRTNET_ERR_RSS_MAX_KEY_SIZE_INVALID

Invalid rss max key size supported key size is 40

-1054

VIRTNET_ERR_REC_OFFLOADS_SAVE

Failed to save OFFLOADS into recovery file

-1055

VIRTNET_ERR_SF_OFFLOADS_UPDATE

Failed to update OFFLOADS by SF

-1056

VIRTNET_ERR_READ_LINK

Failed to readlink

-1057

VIRTNET_ERR_PATH_FORMAT

Error - Path format is invalid

-1058

VIRTNET_ERR_Q_COUNTER_ALLOC

Failed to alloc q counter

-1059

VIRTNET_ERR_REC_DIRTY_LOG_SAVE

Failed to save dirty log

-1060

VIRTNET_ERR_REC_DIRTY_LOG_DEL

Failed to delete dirty log

-1061

VIRTNET_ERR_REC_LM_STATUS_SAVE

Failed to save LM status

-1062

VIRTNET_ERR_REC_LM_STATUS_REC

Failed to found LM status record

-1063

VIRTNET_ERR_REC_DEV_MODE_SAVE

Failed to save dev mode

-1064

VIRTNET_ERR_REC_DEV_MODE_REC

Failed to found dev mode record

-1065

VIRTNET_ERR_UNPLUG_NOT_READY

Error - Device is not ready to be unplugged please check host and retry

-1066

VIRTNET_ERR_REC_MAC_TABLE_DEL

Failed to delete MAC table in recovery file

-1067

VIRTNET_ERR_REC_MAC_TABLE_LOAD

Failed to load MAC table from recovery file

-1068

VIRTNET_ERR_REC_MAC_TABLE_SAVE

Failed to save MAC table into recovery file

-1069

VIRTNET_ERR_REC_HASH_CFG_DEL

Failed to delete hash cfg in recovery file

-1070

VIRTNET_ERR_REC_HASH_CFG_LOAD

Failed to load hash cfg from recovery file

-1071

VIRTNET_ERR_REC_HASH_CFG_SAVE

Failed to save hash cfg into recovery file

-1072

VIRTNET_ERR_DEV_VF_GET

Failed to get VF device

-1073

VIRTNET_ERR_MAX_QUEUES_INVALID

Failed - QUEUES is invalid

-1074

VIRTNET_ERR_DEBUGFS_SAVE

Failed to save into debugfs file

-1075

VIRTNET_ERR_DEBUGFS_DEL

Failed to delete from debugfs file

© Copyright 2024, NVIDIA. Last updated on Aug 14, 2024.