NVIDIA BlueField Virtio-net v24.10

Counters

To query the packet counters, use stats command.

Copy
Copied!
            

[dpu]# virtnet stats [-h] {[-p PF] [-v VF] | [-u VUID]} [-q QUEUE_ID]

Info

The options --pf , --vf and --vuid are mutually exclusive, but one of them must be applied.

Option

Abbr

Argument Type

Required

Description

--help

-h

N/A

No

Show the help message and exit

--pf

-p

Number

No

Unique device ID for the PF. Can be retrieved by using virtnet list.

--vf

-v

Number

No

Unique device ID for the VF. Can be retrieved by using virtnet list.

--vuid

-u

String

No

Unique device SN for the device (PF/VF). Can be retrieved by using virtnet list.

--queue_id

-q

Number

No

Queue index of the device RQs or SQs

Note

This command is recommended for obtaining all packet counter information. The existing packet counter information available through the virtnet list and virtnet query commands will be deprecated in the future.

The following command queries PF 0 and VQ 0 (i.e., RQ):

Copy
Copied!
            

[dpu]# virtnet stats -p 0 -q 0

Output:

Copy
Copied!
            

# virtnet stats -p 0 -q 0 {'pf': '0x0', 'queue_id': '0x0'} { "device": { "pf_id": 0, "packet_counters": "Enabled", "queues-stats": [ { "VQ Index": 0, "rx_64_or_less_octet_packets": 0, "rx_65_to_127_octet_packets": 259, "rx_128_to_255_octet_packets": 0, "rx_256_to_511_octet_packets": 0, "rx_512_to_1023_octet_packets": 0, "rx_1024_to_1522_octet_packets": 0, "rx_1523_to_2047_octet_packets": 0, "rx_2048_to_4095_octet_packets": 199, "rx_4096_to_8191_octet_packets": 0, "rx_8192_to_9022_octet_packets": 0, "received_desc": "4096", "completed_desc": "0", "bad_desc_errors": "0", "error_cqes": "0", "exceed_max_chain": "0", "invalid_buffer": "0", "batch_number": "64", "dma_q_used_number": "0", "handler_schd_number": "44", "aux_handler_schd_number": "43", "max_post_desc_number": "0", "total_bytes": "0", "err_handler_schd_num": "0", "rq_cq_max_count": "0", "rq_cq_period": "0", "rq_cq_period_mode": "1" } ] } }

The output has two sections.

  • The first section, wrapped by device, are device details along with the packet counter statics enable state.

    Entry

    Type

    Description

    device

    String

    Entries under this section is per device information

    pf_id

    String

    Physical function ID

    packet_counters

    String

    packet counters feature: enabled/disabled

  • The second section, wrapped by queues-stats, are information for each receive VQ.

    Entry

    Type

    Description

    VQ Index

    Number

    The VQ index starts at 0 (the first RQ) and continues up to the last SQ

    rx_64_or_less_octet_packets

    Number

    The number of packets received with a size of 0 to 64 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_65_to_127_octet_packets

    Number

    The number of packets received with a size of 65 to 127 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_128_to_255_octet_packets

    Number

    The number of packets received with a size of 128 to 255 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_256_to_511_octet_packets

    Number

    The number of packets received with a size of 256 to 511 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_512_to_1023_octet_packets

    Number

    The number of packets received with a size of 512 to 1023 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_1024_to_1522_octet_packets

    Number

    The number of packets received with a size of 1024 to 1522 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_1523_to_2047_octet_packets

    Number

    The number of packets received with a size of 1523 to 2047 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_2048_to_4095_octet_packets

    Number

    The number of packets received with a size of 2048 to 4095 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_4096_to_8191_octet_packets

    Number

    The number of packets received with a size of 4096 to 8191 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    rx_8192_to_9022_octet_packets

    Number

    The number of packets received with a size of 8192 to 9022 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.

    received_desc

    Number

    Total number of received descriptors by the device on this VQ

    completed_desc

    Number

    Total number of completed descriptors by the device on this VQ

    bad_desc_errors

    Number

    Total number of bad descriptors received on this VQ

    error_cqes

    Number

    Total number of errors CQ entries on this VQ

    exceed_max_chain

    Number

    Total number of chained descriptors received that exceed the max allowed chain by the device

    invalid_buffer

    Number

    Total number of times device tried to read or write buffer that is not registered to the device

    batch_number

    Number

    The number of RX descriptors for the last received packet. Relevant for BlueField-3.

    dma_q_used_number

    Number

    The DMA q index used for this VQ. Relevant for BlueField-3.

    handler_schd_number

    Number

    Scheduler number for this VQ. Relevant for BlueField-3.

    aux_handler_schd_number

    Number

    Aux scheduler number for this VQ. Relevant for BlueField-3.

    max_post_desc_number

    Number

    Maximum number of posted descriptors on this VQ. Relevant for DPA.

    total_bytes

    Number

    Total number of bytes handled by this VQ. Relevant for BlueField-3.

    rq_cq_max_count

    Number

    Event generation moderation counter of the queue. Relevant for RQ.

    rq_cq_period

    Number

    Event generation moderation timer for the queue in 1 µ sec granularity. Relevant for RQ.

    rq_cq_period_mode

    Number

    Current period mode for RQ

    • 0x0 – default_mode – use device best defaults

    • 0x1 – upon_eventqueue_period timer restarts upon event generation

    • 0x2 – upon_cqequeue_period timer restarts upon completion generation

    The second section wrapped by queues-stats IS information for each receive VQ.

VQ Statistics

To query Rx VQ statistics, use the corresponding VQ index. For example, If there are 3 queues configured then to query Rx, VQ uses queue 0, Tx VQ uses queue 1, and Ctrl VQ uses queue 2.

The following is the command to query PF 0, VF 0, and VQ 0 (i.e., Rx).

Copy
Copied!
            

[dpu]# virtnet query -p 0 -v 0 -q 0

Output:

Copy
Copied!
            

"enabled-queues-info": [ { "index": "0", "size": "256", "msix_vector": "0x1", "enable": "1", "notify_offset": "0", "descriptor_address": "0xffffe000", "driver_address": "0xfffff000", "device_address": "0xfffff240", "received_desc": "256", "completed_desc": "19", "bad_desc_errors": "0", "error_cqes": "0", "exceed_max_chain": "0", "invalid_buffer": "0", "batch_number": "64", "dma_q_used_number": "0", "handler_schd_number": "4", "aux_handler_schd_number": "3", "max_post_desc_number": "0", "total_bytes": "6460", "rq_cq_max_count": "0", "rq_cq_period": "0", "rq_cq_period_mode": "1" }

The following are some of the important VQ counters:

Counter Name

Description

total_bytes

Number of bytes received

received_desc

Number of available descriptors received by device

completed_desc

Number of available descriptors completed by the device

error_cqes

Number of error CQEs received on the queue

bad_desc_errors

Number of bad descriptors received

exceed_max_chain

Number of chained descriptors received that exceed the max allowed chain by device

invalid_buffer

Number of times device tried to read or write buffer that is not registered to the device


When DPA is the data path provider, each RQ has its corresponding drop counter, which counts the number of packets dropped inside the DPA virtio RQs.

Info

The drop could also happen from the uplink or SF.

The drop counter only increments (initial value being 0), and its value gets reset to 0 when disabled.

RQ drop counter can be enabled and disabled as follows (using VF 0 on PF 0):

Copy
Copied!
            

[dpu]# virtnet modify -p 0 -v 0 device -dc enable [dpu]# virtnet modify -p 0 -v 0 device -dc disable

Note

Drop counter is attached to a RQ, thus RQ must be created first. This means that the virtio-net device should be probed by the driver on the host OS before running the commands above.

To query the drop counter value(s), run:

Copy
Copied!
            

[dpu]# virtnet query -p 0 -v 0 | grep num_desc_drop_pkts

If there are more than one RQ for a device, the drop count is the sum of all RQ's value.

Note

Relevant for BlueField-3 only.

The packet counter feature helps the user query the byte-wise packet counters for each Rx queue.

By default, byte-wise packet counters are disabled as that negatively impacts performance. When the user is interested in the debug, enable the packet counter feature using the below command

Packet counter can be enabled and disabled as follows (using VF 0 on PF 0):

Copy
Copied!
            

[dpu]# virtnet modify -p 0 -v 0 device -pkt_cnt enable [dpu]# virtnet modify -p 0 -v 0 device -pkt_cnt disable

  • When enabled, byte-wise packet counters are initialized to zero.

  • When disabled, the previous values are retained for debugging purposes. The command will still return these old, disabled counter values.

Note

Packet counters are attached to an RQ. Thus, RQ must be created first. This means that the virtio-net device should be probed by the driver on the host OS before running the commands above.

Note

Relevant for BlueField-3 only.

The health statistics are for displaying real-time health information of a specific device.

Output example (using VF 0 on PF 0):

Copy
Copied!
            

[dpu]# virtnet health -p 0 -v 0 show {   "pf_id": 0, "vf_id": 0,   "type": "VF",   "vuid": "MT2306XZ00BPVNETS0D0F2",   "dev_status": {     "value": "0xf",     "    0": "ACK",     "    1": "DRIVER",     "    2": "DRIVER_OK",     "    3": "FEATURES_OK"   },   "health_status": "Good",   "health_recover_counter": 0,   "dev_health_details": {     "control_plane_errors": {       "sf_rqt_update_err": 0,       "sf_drop_create_err": 0,       "sf_tir_create_err": 0,       "steer_rx_domain_err": 0,       "steer_rx_table_err": 0,       "sf_flows_apply_err": 0,       "aarfs_flow_init_err": 0,       "vlan_flow_init_err": 0,       "drop_cnt_config_err": 0     },     "data_plane_errors": {       "sq_stall": 0,       "dma_q_stall": 0,       "spurious_db_invoke": 0,       "aux_not_invoked": 0,       "dma_q_errors": 0,       "host_read_errors": 0     }   }

Where

  • health_status represents the overall status of the device (Good or Fatal)

  • dev_health_details has two sections, control_plane_errors and data_plane_errors, as explained in the following table:

    Counter Name

    Description

    Control Plane Errors

    sf_rqt_update_err

    Counter tallying receive queue table update failures

    sf_drop_create_err

    Counter tallying drop RQ creation failures

    sf_tir_create_err

    Counter tallying TIR create failures

    steer_rx_domain_err

    Counter tallying RX steering rule creation failures

    steer_rx_table_err

    Counter tallying RX table creation failures

    sf_flows_apply_err

    Counter tallying packet flow rule creation failures

    aarfs_flow_init_err

    Counter tallying packet flow initialization failures

    vlan_flow_init_err

    Counter tallying VLAN flow rule initialization failures

    drop_cnt_config_err

    Counter tallying drop counter configuration failures

    Data Plane Errors

    sq_stall

    One or more network send queues stalled without getting completions. This leads traffic stalling for packets flowing over this VQ.

    dma_q_stall

    QP which is paired to itself issues a read request from the DPA to the host to read either available index or descriptor table. This request does not result in a completion and hangs in a loop waiting for a response.

    spurious_db_invoke

    Doorbell handler is repeatedly invoked but DPA finds no new data to be read and posted. This could be due to a faulty driver or issue on the DPA side.

    aux_not_invoked

    To speed up descriptor processing, an auxiliary execution (EU) unit is used if available. The primary thread invokes this EU and waits for the expected thread to run on the auxiliary execution unit. If this EU is not invoked, the primary thread hangs.

    dma_q_errors

    QP which is paired to itself issues a read request from the DPA to the host to read either an available index or the descriptor table. This request results in an error and the QP becomes unavailable. An internal mechanism detects this error QP and recycles it for use at later stage.

© Copyright 2024, NVIDIA. Last updated on Nov 12, 2024.