NVIDIA BlueField-3 SNAP for NVMe and Virtio-blk v4.3.0
1.0

Appendix – DPU Firmware Configuration

Before configuring SNAP, the user must ensure that all firmware configuration requirements are met. By default, SNAP is disabled and must be enabled by running both common SNAP configurations and additional protocol-specific configurations depending on the expected usage of the application (e.g., hot-plug, SR-IOV, UEFI boot, etc).

After configuration is finished, the host must be power cycled for the changes to take effect.

Warning

To verify that all configuration requirements are satisfied, users may query the current/next configuration by running the following:

Copy
Copied!
            

mlxconfig -d /dev/mst/mt41692_pciconf0 -e query

Parameter

Description

Possible Values

INTERNAL_CPU_MODEL

Enable BlueField to work in internal CPU model

Warning

Must be set to 1 for storage emulations.

0/1

SRIOV_EN

Enable SR-IOV

0/1

PCI_SWITCH_EMULATION_ENABLE

Enable PCI switch for emulated PFs

0/1

PCI_SWITCH_EMULATION_NUM_PORT

The maximum number of hotplug emulated PFs which equals  PCI_SWITCH_EMULATION_NUM_PORT–2. For example, if PCI_SWITCH_EMULATION_NUM_PORT=16, then the maximum number of hotplug emulated PFs would be 14.

Warning

One switch port is reserved for all static PFs.

[0,3-16]

Warning

SRIOV_EN is valid only for static PFs.

BlueField's RDMA/RoCE communication is blocked for BlueField's default OS interfaces (nameds ECPFs, typically mlx5_0 and mlx5_1). If RoCE traffic is required, additional network functions (scalable functions) must be added which support RDMA/RoCE traffic.

Warning

The following is not required when working over TCP or even RDMA/IB.

To enable RoCE interfaces, run the following from within the DPU:

Copy
Copied!
            

[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PER_PF_NUM_SF=1 [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2 [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0.1 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2

Parameter

Description

Possible Values

NVME_EMULATION_ENABLE

Enable NVMe device emulation

0/1

NVME_EMULATION_NUM_PF

Number of static emulated NVMe PFs

[0-4]

NVME_EMULATION_NUM_MSIX

Number of MSIX assigned to emulated NVMe PF/VF

[0-63]

NVME_EMULATION_NUM_VF

Number of VFs per emulated NVMe PF

Warning

If not 0, overrides NUM_OF_VFS; valid only when SRIOV_EN=1.

[0-512]

EXP_ROM_NVME_UEFI_x86_ENABLE

Enable NVMe UEFI exprom driver

Warning

Used for UEFI boot process.

0/1

Important

Due to virtio-blk protocol limitations, using bad configuration while working with static virtio-blk PFs may cause the host server OS to fail on boot.

Before continuing, make sure you have configured:

Parameter

Description

Possible Values

VIRTIO_BLK_EMULATION_ENABLE

Enable virtio-blk device emulation

0/1

VIRTIO_BLK_EMULATION_NUM_PF

Number of static emulated virtio-blk PFs

Warning

See WARNING above.

[0-4]

VIRTIO_BLK_EMULATION_NUM_MSIX

Number of MSIX assigned to emulated virtio-blk PF/VF

[0-63]

VIRTIO_BLK_EMULATION_NUM_VF

Number of VFs per emulated virtio-blk PF

Warning

If not 0, overrides NUM_OF_VFS; valid only when SRIOV_EN=1

[0-1000]

EXP_ROM_VIRTIO_BLK_UEFI_x86_ENABLE

Enable virtio-blk UEFI exprom driver

Warning

Used for UEFI boot process.

0/1

© Copyright 2023, NVIDIA. Last updated on Dec 13, 2023.