wqdump Utility

NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes v16.35.3502 LTS

The wqdump utility dumps device internal work queues. A work queue is an object containing a Queue Pair Context (QPC) which contains control information required by the device to execute I/O operations on that QP, and a work queue buffer which is a virtually-contiguous memory buffer allocated when creating the QP.

The dumped data can be used for hardware troubleshooting. It can be applied on ConnectX adapter cards family and Connect-IB adapter devices.

Warning

wqdump on ConnectX-3 and ConnectX-3 Pro is not supported against in-band devices.

The mst driver must be started prior to running the wqdump utility. To start the wqdump utility:

  1. Start the mst driver (mst start or mst restart).

  2. Run wqdump:

    Copy
    Copied!
                

    # WQDump <-d|--device DeviceName> <--source ContextType> [--gvmi Gvmi] [--qp ContextNumber] <--dump DumpType> [--fi StartIndex] [--num NumberOfItems] [--format Format] [--address Address] [--size Size] [-v|--version] [-h|--help] [--clear_semaphore] [--gw_access]

where:

--d|--device DeviceName

Device name

--source ContextType

Type of context to dump.

Options are: Snd, Rcv, Cmp, Srq, Eqe, Connect-X3/Pro: MCG,

5th generation devices: MKC, SXDC, FullQp

ConnectX-5: CMAS_QP_WQE, CMAS_QP_SWQ, CMAS_SRQ_WQE, CMAS_CQ_BUFF, CMAS_QP_DBR, CMAS_QP_SDB, CMAS_SRQ_DB, CMAS_CQ_DBR, CMAS_CQ_ARM, CMAS_EQ_BUFF, CMAS_TAG_MATCH.

CMAS_INLINE

--gvmi Gvmi

Guest VM ID (5th generation devices)

--qp ContextNumber

Context number to dump

--dump DumpType

Dump Type. Options are: WQ, QP, WQ_QP, ALL_QPC, ALL_VALID_QPNS, ICM

--fi StartIndex

Index of first element to dump, (Default:0)

--num NumberOfItems

Number of elements to dump from buffer, (Default: keep reading)

--format Format

Output format: options are : text, raw, dw, (Default: text)

--address Address

Memory Address

--size Size

Memory size in bytes

-v|--version

Show tool version and exit

-h|--help

Show usage

--clear_semaphore

Force clear semaphore

--gw_access

Force get QPC by GW access (Connect-X 3/Pro)

4th Generation Device Examples:

Print all valid qpns
The example below will dump all valid qpns of type mcg context.

Copy
Copied!
            

# wqdump -d /dev/mst/mt4099_pci_cr0 --source mcg --dump ALL_VALID_QPNS

Dump mcg qp
The example below will dump mcg context number 0x10.

Copy
Copied!
            

# wqdump -d /dev/mst/mt4099_pci_cr0 --source mcg --dump QP -qp 0x10

Dump other qpns
The example below will dump snd context number 0x10 in a raw format.

Copy
Copied!
            

# wqdump -d /dev/mst/mt4099_pci_cr0 --source snd --dump QP -qp 0x10 --format raw

Dump wq
The example below will dump send work queue buffer number 0x42.

Copy
Copied!
            

# wqdump -d /dev/mst/mt4099_pci_cr0 --source snd --dump wq -qp 0x42

Dump mcg qp by GW access
The example below will dump mcg context number 0x10 by GW access.

Copy
Copied!
            

# wqdump -d /dev/mst/mt4103_pci_cr0 --source mcg --dump QP -qp 0x10 --gw_access

5th Generation Device Examples:

FullQp: The QP context of the Rcv and Snd with the common part. Note, FullQp does not have dump as WQ.

• Get opened contexts from the first 20 indexes:

Copy
Copied!
            

# wqdump -d /dev/mst/mt4117_pciconf0 --source FullQp --dump ALL_VALID_QPNS --num 20 Numbers of valid contexts (in the range 0x0 - 0x13): index 0x00000003 index 0x00000006 index 0x00000007 index 0x0000000b index 0x0000000c index 0x00000013 ----------------------------------------- Number of valid contexts: 6 -----------------------------------------

• Show the QP Context (RAW):

Copy
Copied!
            

# wqdump -d /dev/mst/mt4117_pciconf0 --source FullQp --dump QP --qp 0x0000000b --format raw == Common Part (Not Connected) == 0. 80000000 0000001e 05000000 00000000 1. 70000000 00000000 00000000 00000000 2. 0000000b 00ffffff 00000000 00000000 3. 00000000 00000000 00000000 80010000 -------------------------- Send Qpc gvmi 0000 QP Index 0000000b 0. 80000000 0000001e 05000000 00000000 1. 70000000 00000000 00000000 00000000 2. 0000000b 00ffffff 00000000 00000000 3. 00000000 00000000 00000000 80010000 -------------------------- == Responder Part (Not Connected (mac)) == Recv Qpc gvmi 0000 QP Index 0000000b 0. b8eccd02 00000000 d8f5cd02 00000000 1. d0010000 00000000 40000000 00000000 2. e0e03903 00000000 f0e03903 00000000 3. 00000000 00000000 00000000 00000000 --------------------------

SRQ

• Opened QPs:

Copy
Copied!
            

# wqdump -d /dev/mst/mt4115_pciconf0 --source Srq --dump all_valid_qpns Numbers of valid contexts (in the range 0x0 - 0xffffff): gvmi 0x0000 index 0x00000060 gvmi 0x0000 index 0x00000061 gvmi 0x0000 index 0x00000066

• Dump WQs:

Copy
Copied!
            

0x00000066# wqdump -d /dev/mst/mt4117_pciconf0 --source Srq --dump wq --qp 0x60 [Element Index 0] -------------- SRQ Next ------------------ next_wqe_index : 0x1 signature : 0x0 -------------- scatter entry (0) --------------- byte_count : 0x0 wqe_inline : 0x0 local_key : 0x0 local_address_63_32 : 0x0 local_address_31_0 : 0x0 [Element Index 0x1] -------------- SRQ Next ------------------ next_wqe_index : 0x0 signature : 0x0 -------------- scatter entry (0) --------------- byte_count : 0x0 wqe_inline : 0x0 local_key : 0x0 local_address_63_32 : 0x0 local_address_31_0 : 0x0

CMAS

• Opened contexts from some CMAS type (CMAS_EQ_BUFF for ex):

Copy
Copied!
            

#wqdump -d /dev/mst/mt4119_pciconf0 --source CMAS_EQ_BUFF --dump ALL_VALID_QPNS Numbers of valid contexts (in the range 0x0 - 0xffffff): gvmi 0x0000 index 0x00000002 gvmi 0x0000 index 0x00000010 gvmi 0x0000 index 0x00000011 gvmi 0x0000 index 0x00000012 gvmi 0x0000 index 0x00000013 gvmi 0x0000 index 0x00000014 gvmi 0x0000 index 0x00000015

• Dump raw data:

Copy
Copied!
            

# wqdump -d /dev/mst/mt4119_pciconf0 --source CMAS_EQ_BUFF --dump QP --qp 0x15 --format raw CMAS gvmi 0000 CMAS Index 00000015 0. 80000002 00000000 00000000 e4f80000 1. 00000000 00000000 00000000 00000000 2. 00000000 00000000 00000000 00000000 3. 00000000 00000000 00000000 00000000 --------------------------

© Copyright 2023, NVIDIA. Last updated on Dec 28, 2023.