Virtnet CLI Commands
To communicate with the virtio-net-controller backend service, a user frontend program, virtnet, is installed on the BlueField which is based on r emote procedure call (RPC) protocol with JSON format output. Run the following command to check its usage:
# virtnet -h usage: virtnet [-h] [-v]
{hotplug,unplug,list,query,modify,log,version,validate,update,debug,stats} ...
NVIDIA virtio-net-controller command line interface v24.07.04
positional arguments:
{hotplug,unplug,list,query,modify,log,version,update,debug}
** Use -h for sub-command usage
hotplug hotplug virtnet device
unplug unplug virtnet device
list list all virtnet devices
query query all or individual virtnet device(s)
modify modify virtnet device
log set log level
version show virtio net controller version info
validate validate configurations
update update controller
debug For debug purpose, cmds can be changed without notice
stats stats of virtnet device
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
Virtnet supports command line autocomplete by inputting one command with tab.
To check the currently running controller version:
# virtnet -v
v24.07.09
This command hotplugs a virtio-net PCIe PF device exposed to the host side.
Syntax
virtnet hotplug -i IB_DEVICE -m MAC -t MTU -n MAX_QUEUES -s MAX_QUEUE_SIZE [-h] [-u SF_NUM] [-f FEATURES] [-l]
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--ib_device |
-i |
String |
Yes |
RDMA device (e.g., mlx5_0) of the physical port on top of which the hotplug device is created. Options:
|
--features |
-f |
Hex Number |
No |
Feature bits to be enabled in hex format. Refer to the "Virtio-net Feature Bits" page. Note
Note that some features are enabled by default. Query the device to show the supported bits.
|
--mac |
-m |
Number |
Yes |
MAC address of the virtio-net device. Note
Controller does not validate the MAC address (other than its length). The user must ensure MAC is valid and unique.
|
--mtu |
-t |
Number |
Yes |
Maximum transmission unit (MTU) size of the virtio-net device. It must be less than the uplink rep MTU size. |
--num_queues |
-n |
Number |
Yes |
Mutually exclusive with max_queue_pairs. Max number of virt queues could be created for the virtio-net device. TX, RX, ctrl queues are counted separately (e.g., 3 has 1 TX VQ, 1 RX VQ, 1 Ctrl VQ). Note
This option will be depreciated in the future.
|
--max_queue_pairs |
-qp |
Number |
Yes |
Mutually exclusive with num_queues . Number of data VQ pairs. One VQ pair has one TX queue and one RX queue. It does not count control or admin VQ. From the host side, it appears as Pre-set maximums->Combined in ethtool -l <virtio-dev>. |
--max_queue_size |
-s |
Number |
Yes |
Maximum number of buffers in the virt queue, between 0x4 and 0x8000. Must be power of 2. |
--sf_num |
-u |
Number |
No |
SF number to be used for this hotplug device, must between 2000 and 2999. |
--legacy |
-l |
N/A |
No |
Create legacy (transitional) hotplug device |
Output
Entry |
Type |
Description |
bdf |
String |
The PCIe BDF (bus:device:function) number enumerated by host. The user should see this PCIe device from host side. |
vuid |
String |
Unique device SN. It can be used as an index to query/modify/unplug this device. |
id |
Num |
Unique device ID. It can be used as an index to query/modify/unplug this device. |
transitional |
Num |
Is the current device a transitional hotplug device.
|
sf_rep_net_device |
String |
The SF representor name represents the virtio-net device. It should be added into the OVS bridge. |
mac |
String |
The hotplug virtio-net device MAC address |
errno |
Num |
Error number if hotplug failed.
|
errstr |
String |
Explanation of the error number |
Example
Hotplug one device with MAC address 0C:C4:7A:FF:22:93, MTU 1500, and 3 virtio queues (1 tx, 1 rx, 1 ctrl) with a depth of 1024 entries. The device is created on the physical port of mlx5_0.
# virtnet hotplug -i mlx5_0 -f 0x80000 -m 0C:C4:7A:FF:22:93 -t 1500 -n 3 -s 1024
{
"bdf": "15:00.0",
"vuid": "MT2151X03152VNETS1D0F0",
"id": 0,
"transitional": 0,
"sf_rep_net_device": "en3f0pf0sf2000",
"mac": "0C:C4:7A:FF:22:93",
"errno": 0,
"errstr": "Success"
}
This command unplugs a virtio-net PCIe PF device.
Syntax
virtnet unplug [-h] [-p PF | -u VUID]
Only one of --pf and --vuid is needed to unplug the device.
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--pf |
-p |
Number |
Yes |
Unique device ID returned when doing hotplug. Can be retrieved by using virtnet list. |
--vuid |
-u |
String |
Yes |
Unique device SN returned when doing hotplug. Can be retrieved by using virtnet list. |
Output
Entry |
Type |
Description |
errno |
Num |
Error number if operation failed
|
errstr |
String |
Explanation of the error number |
Example
Unplug-hotplug device using the PF ID:
# virtnet unplug -p 0
{'id': '0x1'}
{
"errno": 0,
"errstr": "Success"
}
This command lists all existing virtio-net devices, with global information and individual information for each device.
Syntax
virtnet list [-h]
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
Output
The output has two main sections. The first section wrapped by the controller are global configurations and capabilities.
Entry |
Type |
Description |
controller |
String |
Entries under this section is global information for the controller |
emulation_manager |
String |
The RDMA device manager used to manage internal resources. Should be default mlx5_0. |
max_hotplug_devices |
String |
Maximum number of devices that can be hotpluged |
max_virt_net_devices |
String |
Total number of emulated devices managed by the device emulation manager |
max_virt_queues |
String |
Maximum number of virt queues supported per device |
max_tunnel_descriptors |
String |
Maximum number of descriptors the device can send in a single tunnel request |
supported_features |
String |
Total list of features supported by device |
supported_virt_queue_types |
String |
Currently supported virt queue types: Packed and Split |
supported_event_modes |
String |
Currently supported event modes: no_msix_mode, qp_mode, msix_mode |
Each device has its own section under devices.
Entry |
Type |
Description |
devices |
String |
Entries under this section is per device information |
pf_id |
String |
Physical function ID |
function_type |
String |
Function type: Static PF, hotplug PF, VF |
transitional |
Number |
The current device a transitional hotplug device:
|
vuid |
String |
Unique device SN, it can be used as an index to query/modify/unplug a device |
pci_bdf |
String |
Bus:device:function to describe the virtio-net PCIe device |
pci_vhca_id |
Number |
Virtual HCA identifier for the general virtio-net device. For debug purposes only. |
pci_max_vfs |
Number |
Maximum number of virtio-net VFs that can be created for this PF. Valid only for PFs. |
enabled_vfs |
Number |
Currently enabled number of virtio-net VFs for this PF |
msix_num_pool_size |
Number |
Number of free dynamic MSIX available for the VFs on this PF |
min_msix_num |
Number |
The minimum number of dynamic MSI-Xs that can be set for an virtio-net VF |
max_msix_num |
Number |
The maximum number of dynamic MSI-Xs that can be set for an virtio-net VF |
min_num_of_qp |
Number |
The minimum number of dynamic data VQ pairs (i.e., each pair has one TX and 1 RX queue) that can be set for an virtio-net VF |
max_num_of_qp |
Number |
The minimum number of dynamic data VQ pairs (i.e., each pair has one TX and 1 RX queue) that can be set for an virtio-net VF |
qp_pool_size |
Number |
Number of free dynamic data VQ pairs (i.e., each pair has one TX and 1 RX queue) available for the VFs on this PF |
num_msix |
Number |
Maximum number of MSI-X available for this device |
num_queues |
Number |
Maximum virtual queues can be created for this device, driver can choose to create less |
enabled_queues |
Number |
Currently enabled number of virtual queues by the driver |
max_queues_size |
Number |
Maximum virtual queue depth in byte can be created for each VQ, driver can use less |
msix_config_vector |
String |
MSIX vector number used by the driver for the virtio config space. 0xFFFF means that no vector is requested. |
mac |
String |
The virtio-net device permanent MAC address, can be only changed from controller side via modify command |
link_status |
Number |
Link status of the virtio-net device on the driver side
|
max_queue_pairs |
Number |
Number of data VQ pairs. One VQ pair has one TX queue and one RX queue. Control or admin VQ are not counted. From the host side, it appears as Pre-set maximums->Combined in ethtool -l <virtio-dev>. |
mtu |
Number |
The virtio-net device MTU. Default is 1500. |
speed |
Number |
The virtio-net device link speed in Mb/s |
rss_max_key_size |
Number |
The maximum supported length of the RSS key. Only applicable when VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is enabled. |
supported_hash_types |
Number |
Supported hash types for this device in hex. Only applicable when VIRTIO_NET_F_HASH_REPORT is enabled:
|
ctrl_mac |
String |
Admin MAC address configured by driver. Not persistent with driver reload or host reboot. |
ctrl_mq |
Number |
Number of queue pairs/channels configured by the driver. From the host side, it appears as Current hardware settings->Combined in ethtool -l <virtio-dev>. |
sf_num |
Number |
Scalable function number used for this virtio-net device |
sf_parent_device |
String |
The RDMA device to use to create the SF |
sf_parent_device_pci_addr |
String |
The PCIe device address (bus:device:function) to use to create the SF |
sf_rep_net_device |
String |
Represents the virtio-net device |
sf_rep_net_ifindex |
Number |
The SF representor network interface index |
sf_rdma_device |
String |
The SF RDMA device interface name |
sf_cross_mkey |
Number |
The cross-device MKEY created for the SF. For debug purposes only. |
sf_vhca_id |
Number |
Virtual HCA identifier for the SF. For debug purposes only. |
rqt_num |
Number |
The RQ table ID used for this virtio-net device. For debug purposes only. |
aarfs |
String |
Whether Accelerated Receive Flow Steering configuration is enabled or disabled |
Example
The following is an example of a list with 1 static PF created:
# virtnet list
{
"controller": {
"emulation_manager": "mlx5_0",
"max_hotplug_devices": "0",
"max_virt_net_devices": "1",
"max_virt_queues": "256",
"max_tunnel_descriptors": "6",
"supported_features": {
"value": "0x8b00037700ef982f",
" 0": "VIRTIO_NET_F_CSUM",
" 1": "VIRTIO_NET_F_GUEST_CSUM",
" 2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
" 3": "VIRTIO_NET_F_MTU",
" 5": "VIRTIO_NET_F_MAC",
" 11": "VIRTIO_NET_F_HOST_TSO4",
" 12": "VIRTIO_NET_F_HOST_TSO6",
" 15": "VIRTIO_F_MRG_RX_BUFFER",
" 16": "VIRTIO_NET_F_STATUS",
" 17": "VIRTIO_NET_F_CTRL_VQ",
" 18": "VIRTIO_NET_F_CTRL_RX",
" 19": "VIRTIO_NET_F_CTRL_VLAN",
" 21": "VIRTIO_NET_F_GUEST_ANNOUNCE",
" 22": "VIRTIO_NET_F_MQ",
" 23": "VIRTIO_NET_F_CTRL_MAC_ADDR",
" 32": "VIRTIO_F_VERSION_1",
" 33": "VIRTIO_F_IOMMU_PLATFORM",
" 34": "VIRTIO_F_RING_PACKED",
" 36": "VIRTIO_F_ORDER_PLATFORM",
" 37": "VIRTIO_F_SR_IOV",
" 38": "VIRTIO_F_NOTIFICATION_DATA",
" 40": "VIRTIO_F_RING_RESET",
" 41": "VIRTIO_F_ADMIN_VQ",
" 56": "VIRTIO_NET_F_HOST_USO",
" 57": "VIRTIO_NET_F_HASH_REPORT",
" 59": "VIRTIO_NET_F_GUEST_HDRLEN",
" 63": "VIRTIO_NET_F_SPEED_DUPLEX"
},
"supported_virt_queue_types": {
"value": "0x1",
" 0": "SPLIT"
},
"supported_event_modes": {
"value": "0x5",
" 0": "NO_MSIX_MODE",
" 2": "MSIX_MODE"
}
},
"devices": [
{
"pf_id": 0,
"function_type": "static PF",
"transitional": 0,
"vuid": "MT2306XZ00BNVNETS0D0F2",
"pci_bdf": "e2:00.2",
"pci_vhca_id": "0x2",
"pci_max_vfs": "0",
"enabled_vfs": "0",
"msix_num_pool_size": 0,
"min_msix_num": 0,
"max_msix_num": 256,
"min_num_of_qp": 0,
"max_num_of_qp": 127,
"qp_pool_size": 0,
"num_msix": "256",
"num_queues": "255",
"enabled_queues": "0",
"max_queue_size": "256",
"msix_config_vector": "0xFFFF",
"mac": "16:B0:E0:41:B8:0D",
"link_status": "1",
"max_queue_pairs": "127",
"mtu": "1500",
"speed": "100000",
"rss_max_key_size": "0",
"supported_hash_types": "0x0",
"ctrl_mac": "00:00:00:00:00:00",
"ctrl_mq": "0",
"sf_num": 1000,
"sf_parent_device": "mlx5_0",
"sf_parent_device_pci_addr": "0000:03:00.0",
"sf_rep_net_device": "en3f0pf0sf1000",
"sf_rep_net_ifindex": 10,
"sf_rdma_device": "mlx5_3",
"sf_cross_mkey": "0x12642",
"sf_vhca_id": "0x124",
"sf_rqt_num": "0x0",
"aarfs": "disabled"
}
]
}
This command queries detailed information for a given device, including all VQ information if created.
Syntax
virtnet query [-h] {[-a] | [-p PF] [-v VF] | [-u VUID]} [--dbg_stats] [-b] [--latency_stats] [-q QUEUE_ID] [--stats_clear]
The options --pf , --vf , --vuid , and --all are mutually exclusive, but one of them must be applied.
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--all |
-a |
N/A |
No |
Query all the detailed information for all available devices. It can be time consuming if a large number of devices is available. |
--pf |
-p |
Number |
No |
Unique device ID for the PF. Can be retrieved by using virtnet list. |
--vf |
-v |
Number |
No |
Unique device ID for the VF. Can be retrieved by using virtnet list. |
--vuid |
-u |
String |
No |
Unique device SN for the device (PF/VF). Can be retrieved by using virtnet list. |
--queue_id |
-q |
Number |
No |
Queue index of the device VQs |
--brief |
-b |
N/A |
No |
Query brief information of the device (does not print VQ information) |
--dbg_stats |
N/A |
N/A |
No |
Print debug counters and information Note
This option will be depreciated in the future.
|
--stats_clear |
N/A |
N/A |
No |
Clear all the debug counter stats Note
This option will be depreciated in the future.
|
Output
Output has two main sections.
The first section, wrapped by devices, are configuration and capabilities on the device level, the majority of which are the same as the list command. This section only covers the differences between the two.
Entry
Type
Description
devices
String
Entries under this section is per-device information
pci_dev_id
String
Virtio-net PCIe device ID. Default: 0x1041.
NoteThis option will be depreciated in the future.
pci_vendor_id
String
Virtio-net PCIe vendor ID. Default: 0x1af4.
NoteThis option will be depreciated in the future.
pci_class_code
String
Virtio-net PCIe device class code. Default: 0x20000.
NoteThis option will be depreciated in the future.
pci_subsys_id
String
Virtio-net PCIe vendor ID. Default: 0x1041.
NoteThis option will be depreciated in the future.
pci_subsys_vendor_id
String
Virtio-net PCIe subsystem vendor ID. Default: 0x1af4.
NoteThis option will be depreciated in the future.
pci_revision_id
String
Virtio-net PCIe revision ID. Default: 1.
NoteThis option will be depreciated in the future.
device_features
String
Enabled device feature bits according to the virtio spec. Refer to Virtio-net Feature Bits.
driver_features
String
Enabled driver feature bits according to the virtio spec. Valid only when the driver probes the device. Refer to Virtio-net Feature Bits.
status
String
Device status field bit masks according to the virtio spec:
ACKNOWLEDGE (bit 0)
DRIVER (bit 1)
DRIVER_OK (bit 2)
FEATURES_OK (bit 3)
DEVICE_NEEDS_RESET (bit 6)
FAILED (bit 7)
reset
Number
Shows if the current virtio-net device undergoing reset:
0 – not undergoing reset
1 – undergoing reset
enabled
Number
Shows if the current virtio-net device is enabled:
0 – disabled, likely FLR has occurred
1 – enabled
The second section, wrapped by enabled-queues-info, provides per-VQ information:
Entry
Type
Description
index
Number
VQ index starting from 0 to enabled_queues
size
Number
Driver VQ depth in bytes. It is bound by device max_queues_size .
msix_vector
Number
The MSI-X vector number used for this VQ
enable
Number
If current VQ is enabled or not
0 – disabled
1 – enabled
notify_offset
Number
Driver reads this to calculate the offset from start of notification structure at which this virtqueue is located
descriptor_address
Number
The physical address of the descriptor area
driver_address
Number
The physical address of the driver area
device_address
Number
The physical address of the device area
received_desc
Number
Total number of received descriptors by the device on this VQ
NoteThis option will be depreciated in the future.
completed_desc
Number
Total number of completed descriptors by the device on this VQ
NoteThis option will be depreciated in the future.
bad_desc_errors
Number
Total number of bad descriptors received on this VQ
NoteThis option will be depreciated in the future.
error_cqes
Number
Total number of error CQ entries on this VQ
NoteThis option will be depreciated in the future.
exceed_max_chain
Number
Total number of chained descriptors received that exceed the maximum allowed chain by device
NoteThis option will be depreciated in the future.
invalid_buffer
Number
Total number of times the device tried to read or write buffer that is not registered to the device
NoteThis option will be depreciated in the future.
batch_number
Number
The number of RX descriptors for the last received packet. Relevant for BlueField-3 only.
NoteThis option will be depreciated in the future.
dma_q_used_number
Number
The DMA q index used for this VQ. Relevant for BlueField-3 only.
NoteThis option will be depreciated in the future.
handler_schd_number
Number
Scheduler number for this VQ. Relevant for BlueField-3 only.
NoteThis option will be depreciated in the future.
aux_handler_schd_number
Number
Aux scheduler number for this VQ. Relevant for BlueField-3 only.
NoteThis option will be depreciated in the future.
max_post_desc_number
Number
Maximum number of posted descriptors on this VQ. Relevant for DPA.
NoteThis option will be depreciated in the future.
total_bytes
Number
Total number of bytes handled by this VQ. Relevant for BlueField-3 only
NoteThis option will be depreciated in the future.
rq_cq_max_count
Number
Event generation moderation counter of the queue. Relevant for RQ.
NoteThis option will be depreciated in the future.
rq_cq_period
Number
Event generation moderation timer for the queue in 1 µ sec granularity. Relevant for RQ.
NoteThis option will be depreciated in the future.
rq_cq_period_mode
Number
Current period mode for RQ
0x0 – default_mode – use device best defaults
0x1 – upon_event – queue_period timer restarts upon event generation
0x2 – upon_cqe – queue_period timer restarts upon completion generation
NoteThis option will be depreciated in the future.
Example
The following is an example of querying the information of the first PF:
# virtnet query -p 0
{
"devices": [
{
"pf_id": 0,
"function_type": "static PF",
"transitional": 0,
"vuid": "MT2349X00018VNETS0D0F1",
"pci_bdf": "23:00.1",
"pci_vhca_id": "0x1",
"pci_max_vfs": "0",
"enabled_vfs": "0",
"pci_dev_id": "0x1041",
"pci_vendor_id": "0x1af4",
"pci_class_code": "0x20000",
"pci_subsys_id": "0x1041",
"pci_subsys_vendor_id": "0x1af4",
"pci_revision_id": "1",
"device_feature": {
"value": "0x8930032300e7182f",
" 0": "VIRTIO_NET_F_CSUM",
" 1": "VIRTIO_NET_F_GUEST_CSUM",
" 2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
" 3": "VIRTIO_NET_F_MTU",
" 5": "VIRTIO_NET_F_MAC",
" 11": "VIRTIO_NET_F_HOST_TSO4",
" 12": "VIRTIO_NET_F_HOST_TSO6",
" 16": "VIRTIO_NET_F_STATUS",
" 17": "VIRTIO_NET_F_CTRL_VQ",
" 18": "VIRTIO_NET_F_CTRL_RX",
" 21": "VIRTIO_NET_F_GUEST_ANNOUNCE",
" 22": "VIRTIO_NET_F_MQ",
" 23": "VIRTIO_NET_F_CTRL_MAC_ADDR",
" 32": "VIRTIO_F_VERSION_1",
" 33": "VIRTIO_F_IOMMU_PLATFORM",
" 37": "VIRTIO_F_SR_IOV",
" 40": "VIRTIO_F_RING_RESET",
" 41": "VIRTIO_F_ADMIN_VQ",
" 52": "VIRTIO_NET_F_VQ_NOTF_COAL",
" 53": "VIRTIO_NET_F_NOTF_COAL",
" 56": "VIRTIO_NET_F_HOST_USO",
" 59": "VIRTIO_NET_F_GUEST_HDRLEN",
" 63": "VIRTIO_NET_F_SPEED_DUPLEX"
},
"driver_feature": {
"value": "0x8000002300e7182f",
" 0": "VIRTIO_NET_F_CSUM",
" 1": "VIRTIO_NET_F_GUEST_CSUM",
" 2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
" 3": "VIRTIO_NET_F_MTU",
" 5": "VIRTIO_NET_F_MAC",
" 11": "VIRTIO_NET_F_HOST_TSO4",
" 12": "VIRTIO_NET_F_HOST_TSO6",
" 16": "VIRTIO_NET_F_STATUS",
" 17": "VIRTIO_NET_F_CTRL_VQ",
" 18": "VIRTIO_NET_F_CTRL_RX",
" 21": "VIRTIO_NET_F_GUEST_ANNOUNCE",
" 22": "VIRTIO_NET_F_MQ",
" 23": "VIRTIO_NET_F_CTRL_MAC_ADDR",
" 32": "VIRTIO_F_VERSION_1",
" 33": "VIRTIO_F_IOMMU_PLATFORM",
" 37": "VIRTIO_F_SR_IOV",
" 63": "VIRTIO_NET_F_SPEED_DUPLEX"
},
"status": {
"value": "0xf",
" 0": "ACK",
" 1": "DRIVER",
" 2": "DRIVER_OK",
" 3": "FEATURES_OK"
},
"reset": "0",
"enabled": "1",
"num_msix": "64",
"num_queues": "63",
"enabled_queues": "63",
"max_queue_size": "256",
"msix_config_vector": "0x0",
"mac": "4E:6A:E1:41:D8:BE",
"link_status": "1",
"max_queue_pairs": "31",
"mtu": "1500",
"speed": "200000",
"rss_max_key_size": "0",
"supported_hash_types": "0x0",
"ctrl_mac": "4E:6A:E1:41:D8:BE",
"ctrl_mq": "31",
"sf_num": 1000,
"sf_parent_device": "mlx5_0",
"sf_parent_device_pci_addr": "0000:03:00.0",
"sf_rep_net_device": "en3f0pf0sf1000",
"sf_rep_net_ifindex": 12,
"sf_rdma_device": "mlx5_2",
"sf_cross_mkey": "0xC042",
"sf_vhca_id": "0x7E8",
"sf_rqt_num": "0x0",
"aarfs": "disabled"
"enabled-queues-info": [
{
"index": "0",
"size": "256",
"msix_vector": "0x1",
"enable": "1",
"notify_offset": "0",
"descriptor_address": "0x10cece000",
"driver_address": "0x10cecf000",
"device_address": "0x10cecf240",
"received_desc": "256",
"completed_desc": "0",
"bad_desc_errors": "0",
"error_cqes": "0",
"exceed_max_chain": "0",
"invalid_buffer": "0",
"batch_number": "64",
"dma_q_used_number": "6",
"handler_schd_number": "4",
"aux_handler_schd_number": "3",
"max_post_desc_number": "0",
"total_bytes": "0",
"rq_cq_max_count": "0",
"rq_cq_period": "0",
"rq_cq_period_mode": "1"
},
......
}
]
}
]
}
This command is recommended for obtaining all packet counter information. The existing packet counter information available using the virtnet list and virtnet query commands, but will be deprecated in the future.
This command retrieves the packet counters for a specified device, including detailed information for all Rx and Tx virtqueues (VQs).
To enable/disable byte wise packet counters for each Rx queue, use the following command:
virtnet modify {[-p PF] [-v VF]} device -pkt_cnt {enable,disable}
When enabled, byte-wise packet counters are initialized to zero.
When disabled, the previous values are retained for debugging purposes. The command will still return these old, disabled counter values.
Packet counters are attached to an RQ. Thus, RQ must be created first. This means that the virtio-net device should be probed by the driver on the host OS before running the commands above.
Syntax
virtnet stats [-h] {[-p PF] [-v VF] | [-u VUID]} [-q QUEUE_ID]
The options --pf , --vf , and --vuid are mutually exclusive, but one of them must be applied.
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--pf |
-p |
Number |
No |
Unique device ID for the PF. Can be retrieved by using virtnet list. |
--vf |
-v |
Number |
No |
Unique device ID for the VF. Can be retrieved by using virtnet list. |
--vuid |
-u |
String |
No |
Unique device SN for the device (PF/VF). Can be retrieved by using virtnet list. |
--queue_id |
-q |
Number |
No |
Queue index of the device RQs or SQs |
Output
The output has two sections.
The first section wrapped by device are device details along with the packet counter statics enable state.
Entry
Type
Description
device
String
Entries under this section is per-device information
pf_id
String
Physical function ID
packet_counters
String
Indicates whether the packet counters feature is enabled or disabled
The second section wrapped by queues-stats are information for each receive VQ.
Entry
Type
Description
VQ Index
Number
The VQ index starts at 0 (the first RQ) and continues up to the last SQ
rx_64_or_less_octet_packets
Number
The number of packets received with a size of 0 to 64 bytes. Relevant for BlueField-3 RQ.
rx_65_to_127_octet_packets
Number
The number of packets received with a size of 65 to 127 bytes. Relevant for BlueField-3 RQ.
rx_128_to_255_octet_packets
Number
The number of packets received with a size of 128 to 255 bytes. Relevant for BlueField-3 RQ.
rx_256_to_511_octet_packets
Number
The number of packets received with a size of 256 to 511 bytes. Relevant for BlueField-3 RQ.
rx_512_to_1023_octet_packets
Number
The number of packets received with a size of 512 to 1023 bytes. Relevant for BlueField-3 RQ.
rx_1024_to_1522_octet_packets
Number
The number of packets received with a size of 1024 to 1522 bytes. Relevant for BlueField-3 RQ.
rx_1523_to_2047_octet_packets
Number
The number of packets received with a size of 1523 to 2047 bytes. Relevant for BlueField-3 RQ.
rx_2048_to_4095_octet_packets
Number
The number of packets received with a size of 2048 to 4095 bytes. Relevant for BlueField-3 RQ.
rx_4096_to_8191_octet_packets
Number
The number of packets received with a size of 4096 to 8191 bytes. Relevant for BlueField-3 RQ.
rx_8192_to_9022_octet_packets
Number
The number of packets received with a size of 8192 to 9022 bytes. Relevant for BlueField-3 RQ.
received_desc
Number
Total number of received descriptors by the device on this VQ
completed_desc
Number
Total number of completed descriptors by the device on this VQ
bad_desc_errors
Number
Total number of bad descriptors received on this VQ
error_cqes
Number
Total number of error CQ entries on this VQ
exceed_max_chain
Number
Total number of chained descriptors received that exceed the max allowed chain by device
invalid_buffer
Number
Total number of times the device tried to read or write a buffer which is not registered to the device
batch_number
Number
The number of RX descriptors for the last received packet. Relevant for BlueField-3.
dma_q_used_number
Number
The DMA q index used for this VQ. Relevant for BlueField-3.
handler_schd_number
Number
Scheduler number for this VQ. Relevant for BlueField-3.
aux_handler_schd_number
Number
Aux scheduler number for this VQ. Relevant for BlueField-3.
max_post_desc_number
Number
Maximum number of posted descriptors on this VQ. Relevant for DPA.
total_bytes
Number
Total number of bytes handled by this VQ. Relevant for BlueField-3.
rq_cq_max_count
Number
Event generation moderation counter of the queue. Relevant for RQ.
rq_cq_period
Number
Event generation moderation timer for the queue in 1 µ sec granularity. Relevant for RQ.
rq_cq_period_mode
Number
Current period mode for RQ
0x0 – default_mode – use device best defaults
0x1 – upon_event – queue_period timer restarts upon event generation
0x2 – upon_cqe – queue_period timer restarts upon completion generation
Example
The following is an example of querying the packet statistics information of PF 0 and VQ 0 (i.e., RQ):
# virtnet stats -p 0 -q 0
{'pf': '0x0', 'queue_id': '0x0'}
{
"device": {
"pf_id": 0,
"packet_counters": "Enabled",
"queues-stats": [
{
"VQ Index": 0,
"rx_64_or_less_octet_packets": 0,
"rx_65_to_127_octet_packets": 259,
"rx_128_to_255_octet_packets": 0,
"rx_256_to_511_octet_packets": 0,
"rx_512_to_1023_octet_packets": 0,
"rx_1024_to_1522_octet_packets": 0,
"rx_1523_to_2047_octet_packets": 0,
"rx_2048_to_4095_octet_packets": 199,
"rx_4096_to_8191_octet_packets": 0,
"rx_8192_to_9022_octet_packets": 0,
"received_desc": "4096",
"completed_desc": "0",
"bad_desc_errors": "0",
"error_cqes": "0",
"exceed_max_chain": "0",
"invalid_buffer": "0",
"batch_number": "64",
"dma_q_used_number": "0",
"handler_schd_number": "44",
"aux_handler_schd_number": "43",
"max_post_desc_number": "0",
"total_bytes": "0",
"err_handler_schd_num": "0",
"rq_cq_max_count": "0",
"rq_cq_period": "0",
"rq_cq_period_mode": "1"
}
]
}
}
This command modifies the attributes of a given device.
Syntax
virtnet modify [-h] [-p PF] [-v VF] [-u VUID] [-a] {device,queue} ...
The options --pf , --vf , --vuid , and --all are mutually exclusive, but one of them must be applied.
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--all |
-a |
N/A |
No |
Modify all available device attributes depending on the selection of device or queue |
--pf |
-p |
Number |
No |
Unique device ID for the PF. May be retrieved using virtnet list. |
--vf |
-v |
Number |
No |
Unique device ID for the VF. May be retrieved using virtnet list. |
--vuid |
-u |
String |
No |
Unique device SN for the device (PF/VF). May be retrieved by using virtnet list. |
device |
N/A |
Number |
No |
Modify device specific options |
queue |
N/A |
N/A |
No |
Modify queue specific options |
Device Options
virtnet modify device [-h] [-m MAC] [-t MTU] [-e SPEED] [-l LINK]
[-s STATE] [-f FEATURES]
[-o SUPPORTED_HASH_TYPES] [-k RSS_MAX_KEY_SIZE]
[-r RX_MODE] [-n MSIX_NUM] [-q MAX_QUEUE_SIZE]
[-d DST_PORT] [-b RX_DMA_Q_NUM]
[-dim {enable,disable}] [-dc {enable,disable}] [-pkt_cnt {enable,disable}]
[-aarfs {enable,disable}] [-qp MAX_QUEUE_PAIRS]
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
String |
No |
Show the help message and exit |
--mac |
-m |
Number |
No |
The virtio-net device MAC address |
--mtu |
-t |
Number |
No |
The virtio-net device MTU |
--speed |
-e |
Number |
No |
The virtio-net device link speed in Mb/s |
--link |
-l |
Number |
No |
The virtio-net device link status
|
--state |
-s |
Number |
No |
The virtio-net device status field bit masks according to the virtio spec:
|
--features |
-f |
Hex Number |
No |
The virtio-net device feature bits according to the virtio spec |
--supported_hash_types |
-o |
Hex Number |
No |
Supported hash types for this device in hex. Only applicable when VIRTIO_NET_F_HASH_REPORT is enabled.
|
--rss_max_key_size |
-k |
Number |
No |
The maximum supported length of RSS key. Only applicable when VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is enabled. |
--rx_mode |
-r |
Hex Number |
No |
The RX mode exposed to the driver:
|
--msix_num |
-n |
Number |
No |
Maximum number of VQs (both data and ctrl/admin VQ). It is bound by the cap of max_virt_queues at the controller level (virtnet list). |
--max_queue_size |
-q |
Number |
No |
Maximum number of buffers in the VQ. The queue size value is always a power of 2. The maximum queue size value is 32768. |
--max_queue_pairs |
-qp |
Number |
No |
Number of data VQ pairs. One VQ pair has one TX queue and one RX queue. Control or admin VQs are not counted. From the host side, it appears as Pre-set maximums->Combined in ethtool -l <virtio-dev> . |
--dst_port |
-d |
Hex number |
No |
Modify IPv4 dst_port rules. Note
Will be depreciated in the future.
|
--rx_dma_q_num |
-b |
Number |
No |
Modify max RX DMA queue number |
--rx_dim_config |
-dim |
String |
No |
Enable/disable RX dynamic interrupt moderation |
--drop_counter |
-dc |
String |
No |
Enable/disable virtio-net drop counter |
--packet_counter |
-pkt_cnt |
String |
No |
Enable/disable virtio-net device packet counter stats |
--aarfs_config |
-aarfs |
String |
No |
Enable/disable auto-AARFS. Only applicable for PF devices (static PF and hotplug PF). |
The following modify options require unbinding the virtio device from virtio-net driver in the guest OS:
mac
mtu
features
msix_num
max_queue_size
max_queue_pairs
For example:
On the guest OS:
[host]# echo "bdf of virtio-dev" > /sys/bus/pci/drivers/virtio-pci/unbind
On the DPU side:
Modify the max queue size of device:
[dpu]# virtnet modify -p 0 -v 0 device -q 2048
Modify the MSI-X number of VF device:
[dpu]# virtnet modify -p 0 -v 0 device -n 8
Modify the MAC address of virtio physical device ID 0 (or with its "VUID string", which can be obtained through virtnet list/query):
[dpu]# virtnet modify -p 0 device -m 0C:C4:7A:FF:22:93
Modify the maximum number of queue pairs of VF device:
[dpu]# virtnet modify -p 0 -v 0 device -qp 2
On the guest OS:
[host]# echo "bdf of virtio-dev" > /sys/bus/pci/drivers/virtio-pci/bind
Queue Options
virtnet modify queue [-h] -e {event,cqe} -n PERIOD -c MAX_COUNT
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
String |
No |
Show the help message and exit |
--period_mode |
-e |
String |
No |
RQ period mode: event or cqe. Default is selected by device for the best result. |
--period |
-n |
Number |
No |
The event generation moderation timer for the queue in 1 µ sec granularity |
--max_count |
-c |
Number |
No |
The max event generation moderation counter of the queue |
Output
Entry |
Type |
Description |
errno |
Number |
Error number:
|
errstr |
String |
Explanation of the error number |
Example
To modify the link status of the first VF on the first PF to be down:
# virtnet modify -p 0 device -l 0
{'pf': '0x0', 'all': '0x0', 'subcmd': '0x0', 'link': '0x0'}
{
"errno": 0,
"errstr": "Success"
}
This command manages the log level of virtio-net-controller.
Syntax
virtnet log [-h] -l {info,err,debug}
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--level |
-l |
String |
Yes |
Change the log level of virtio_net_controller from the journal. Default is DEBUG. |
Output
Entry |
Type |
Description |
Stdout |
String |
Success or failed with message |
Example
To change the log level to info:
# virtnet log -l info
{'level': 'info'}
"Success"
To monitor current log output of the controller service with the latest 100 lines printed out:
$ journalctl -u virtio-net-controller -f -n 100
This command validates configurations of virtio-net-controller.
Syntax
virtnet validate [-h] -f PATH_TO_FILE
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--file |
-f |
String |
No |
Validate the JSON format of the virtnet.conf file of the virtio_net_controller |
Output
Entry |
Type |
Description |
Stdout |
String |
Success or failed with message |
Example
To check if virtnet.conf is a valid JSON file:
# virtnet validate -f /opt/mellanox/mlnx_virtnet/virtnet.conf
/opt/mellanox/mlnx_virtnet/virtnet.conf is valid
This command prints current and updated version of virtio-net-controller.
Syntax
virtnet version [-h]
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
Output
Entry |
Type |
Description |
Original Controller |
String |
The original controller version |
Destination Controller |
String |
The to be updated controller version |
Example
Check current and next available controller version:
# virtnet version
[
{
"Original Controller": "v1.8.12"
},
{
"Destination Controller": "v1.9.14"
}
]
Live update minimizes network interface down time by performing online upgrade of the virtio-net controller without necessitating a full restart.
Syntax
virtnet update [-h] [-s | -t]
Option |
Abbr |
Argument Type |
Required |
Description |
--help |
-h |
N/A |
No |
Show the help message and exit |
--start |
-s |
N/A |
No |
Start live update virtio-net-controller |
--status |
-t |
N/A |
No |
Check live update status |
Output
Entry |
Type |
Description |
stdout |
String |
If the update started successfully |
Example
To start the live update process, run:
# virtnet update -s
{'start': '0x1'}
"Update started, use 'virtnet update -t' or check logs for status"
To check the update status during the update process:
# virtnet update -t
{'status': '0x1'}
{
"current status": "inactive",
"last status": "success",
"time_used (s)": 0.604152
}
CLI commands will return non-0 error code upon failure. All error numbers are negative. When there is error happening from log, it could return error number as well.
If the error number is greater than -1000, it's standard error. Please refer to Linux error code at errno
If the error number is less or equal -1000, please refer to the table below for the explaination.
Errno |
Error Name |
Error Description |
-1000 |
VIRTNET_ERR_DEV_FEATURE_VALIDATE |
Failed to validate device feature |
-1001 |
VIRTNET_ERR_DEV_NOT_FOUND |
Failed to find device |
-1002 |
VIRTNET_ERR_DEV_NOT_PLUGGED |
Failed - Device is not hotplugged |
-1003 |
VIRTNET_ERR_DEV_NOT_STARTED |
Failed - Device did not start |
-1004 |
VIRTNET_ERR_DRIVER_PROBED |
Failed - Virtio driver should not be loaded |
-1005 |
VIRTNET_ERR_EPOLL_ADD |
Failed to add epoll |
-1006 |
VIRTNET_ERR_ID_OUT_OF_RANGE |
Failed - ID input exceeds the max range |
-1007 |
VIRTNET_ERR_VUID_INVALID |
Failed - VUID is invalid |
-1008 |
VIRTNET_ERR_MAC_INVALID |
Failed - MAC is invalid |
-1009 |
VIRTNET_ERR_MSIX_INVALID |
Failed - MSIX is invalid |
-1010 |
VIRTNET_ERR_MTU_INVALID |
Failed - MTU is invalid |
-1011 |
VIRTNET_ERR_PORT_CONTEXT_NOT_FOUND |
Failed to find port contex |
-1012 |
VIRTNET_ERR_REC_CONFIG_LOAD |
Failed to load config from recovery file |
-1013 |
VIRTNET_ERR_REC_CONFIG_SAVE |
Failed to save config into recovery file |
-1014 |
VIRTNET_ERR_REC_FILE_CREATE |
Failed to create recovery file |
-1015 |
VIRTNET_ERR_REC_MAC_DEL |
Failed to delete MAC in recovery file |
-1016 |
VIRTNET_ERR_REC_MAC_LOAD |
Failed to load MAC from recovery file |
-1017 |
VIRTNET_ERR_REC_MAC_SAVE |
Failed to save MAC into recovery file |
-1018 |
VIRTNET_ERR_REC_MQ_SAVE |
Failed to save MQ into recovery file |
-1019 |
VIRTNET_ERR_REC_PFNUM_LOAD |
Failed to load PF number from recovery file |
-1020 |
VIRTNET_ERR_REC_RX_MODE_SAVE |
Failed to save RX mode into recovery file |
-1021 |
VIRTNET_ERR_REC_SF_SAVE |
Failed to save PF and SF number into recovery file |
-1022 |
VIRTNET_ERR_REC_SFNUM_LOAD |
Failed to load SF number from recovery file |
-1023 |
VIRTNET_ERR_SF_MAC_FLOW_APPLY |
Failed to apply MAC flow by SF |
-1024 |
VIRTNET_ERR_SF_MQ_UPDATE |
Failed to update MQ by SF |
-1025 |
VIRTNET_ERR_SF_RX_MODE_SET |
Failed to set RX mode by SF |
-1026 |
VIRTNET_ERR_SNAP_NET_CTRL_OPEN |
Failed to open SNAP device control |
-1027 |
VIRTNET_ERR_SNAP_CROSS_MKEY_CREATE |
Failed to create SNAP cross mkey |
-1028 |
VIRTNET_ERR_SNAP_DMA_Q_CREATE |
Failed to create SNAP DMA Q |
-1029 |
VIRTNET_ERR_SNAP_NET_DEV_QUERY |
Failed to query SNAP device |
-1030 |
VIRTNET_ERR_SNAP_NET_DEV_MODIFY |
Failed to modify SNAP device |
-1031 |
VIRTNET_ERR_SNAP_PF_HOTPLUG |
Failed to hotplug SNAP PF |
-1032 |
VIRTNET_ERR_VQ_PERIOD_UPDATE |
Failed to update VQ period |
-1033 |
VIRTNET_ERR_QUEUE_SIZE_INVALID |
Failed - Queue size is invalid |
-1034 |
VIRTNET_ERR_SF_PORT_ADD |
Failed to add SF port |
-1035 |
VIRTNET_ERR_WQ_WORKQUEUE_ALLOC |
Failed to alloc workqueue |
-1036 |
VIRTNET_ERR_ETH_VQS_OPERATION_ALLOC |
Failed to alloc eth VQS operation |
-1037 |
VIRTNET_ERR_ETH_VQS_OPERATION_COMP |
Failed to complete eth VQS operation |
-1038 |
VIRTNET_ERR_JSON_OBJ_NOT_EXIST |
Failed - JSON obj does not exist |
-1039 |
VIRTNET_ERR_DEV_LOAD_PREP |
Failed to prepare device load |
-1040 |
VIRTNET_ERR_DEV_SW_MIGRATION |
Failed to sw migrate a device |
-1041 |
VIRTNET_ERR_DEV_IS_SW_MIGRATING |
Failed - Device is migrating |
-1042 |
VIRTNET_ERR_MAX_QUEUE_SIZE |
Error - queue size must be greater than 2 and is power of 2 |
-1043 |
VIRTNET_ERR_MSIX_LESS_EQUAL_THREE |
Warning - this device won't function, don't try to probe with virtio driver |
-1044 |
VIRTNET_ERR_SF_POOL_CREATING |
SF pool is creating try again later |
-1045 |
VIRTNET_ERR_DST_PORT |
Failed to set dst port rule |
-1046 |
VIRTNET_ERR_INVALID_OPTION |
Option is not supported |
-1047 |
VIRTNET_ERR_SF_CREATE |
Failed to create SF |
-1048 |
VIRTNET_ERR_DEV_SF_NUM_OUT_OF_RANGE |
SF number for hotplug device should be between 2000 and 2999 |
-1049 |
VIRTNET_ERR_DEV_SF_NUM_USED |
SF number is already used |
-1050 |
VIRTNET_ERR_QUEUE_NUMBER_INVALID |
Queue index is invalid |
-1051 |
VIRTNET_ERR_SPEED_INVALID |
Invalid speed please check help menu for supported link speeds |
-1052 |
VIRTNET_ERR_SUPPORTED_HASH_TYPES_INVALID |
Invalid hash types please check help menu for supported hash types |
-1053 |
VIRTNET_ERR_RSS_MAX_KEY_SIZE_INVALID |
Invalid rss max key size supported key size is 40 |
-1054 |
VIRTNET_ERR_REC_OFFLOADS_SAVE |
Failed to save OFFLOADS into recovery file |
-1055 |
VIRTNET_ERR_SF_OFFLOADS_UPDATE |
Failed to update OFFLOADS by SF |
-1056 |
VIRTNET_ERR_READ_LINK |
Failed to readlink |
-1057 |
VIRTNET_ERR_PATH_FORMAT |
Error - Path format is invalid |
-1058 |
VIRTNET_ERR_Q_COUNTER_ALLOC |
Failed to alloc q counter |
-1059 |
VIRTNET_ERR_REC_DIRTY_LOG_SAVE |
Failed to save dirty log |
-1060 |
VIRTNET_ERR_REC_DIRTY_LOG_DEL |
Failed to delete dirty log |
-1061 |
VIRTNET_ERR_REC_LM_STATUS_SAVE |
Failed to save LM status |
-1062 |
VIRTNET_ERR_REC_LM_STATUS_REC |
Failed to found LM status record |
-1063 |
VIRTNET_ERR_REC_DEV_MODE_SAVE |
Failed to save dev mode |
-1064 |
VIRTNET_ERR_REC_DEV_MODE_REC |
Failed to found dev mode record |
-1065 |
VIRTNET_ERR_UNPLUG_NOT_READY |
Error - Device is not ready to be unplugged please check host and retry |
-1066 |
VIRTNET_ERR_REC_MAC_TABLE_DEL |
Failed to delete MAC table in recovery file |
-1067 |
VIRTNET_ERR_REC_MAC_TABLE_LOAD |
Failed to load MAC table from recovery file |
-1068 |
VIRTNET_ERR_REC_MAC_TABLE_SAVE |
Failed to save MAC table into recovery file |
-1069 |
VIRTNET_ERR_REC_HASH_CFG_DEL |
Failed to delete hash cfg in recovery file |
-1070 |
VIRTNET_ERR_REC_HASH_CFG_LOAD |
Failed to load hash cfg from recovery file |
-1071 |
VIRTNET_ERR_REC_HASH_CFG_SAVE |
Failed to save hash cfg into recovery file |
-1072 |
VIRTNET_ERR_DEV_VF_GET |
Failed to get VF device |
-1073 |
VIRTNET_ERR_MAX_QUEUES_INVALID |
Failed - QUEUES is invalid |
-1074 |
VIRTNET_ERR_DEBUGFS_SAVE |
Failed to save into debugfs file |
-1075 |
VIRTNET_ERR_DEBUGFS_DEL |
Failed to delete from debugfs file |