SNAP Commands

Remote Procedure Call (RPC) protocol is a very simple protocol defining a few data types and commands. NVMe/virtio-blk SNAP, like other standard SPDK applications, supports JSON-based RPC protocol commands, to control any resources create/delete/query/modify commands easily from CLI.

mlnx_snap supports executing all standard SPDK RPC commands, in addition to an extended SNAP-specific command set. SPDK standard commands are executed by the standard spdk_rpc.py tool, while the SNAP-specific command set extension is executed by an equivalent snap_rpc.py tool.

Full spdk_rpc.py command set documentation can be found in the SPDK official documentation site.

Full snap_rpc.py extended command is detailed later in this chapter.

Emulation Managers Discovery

Emulated PCIe functions are managed through IB devices called "emulation managers". The emulation managers are ordinary IB devices (e.g. mlx5_0, mlx5_1, etc.) with special privileges to also control PCIe communication and device emulations towards the host operating system. Numerous emulation managers may co-exist, each with its own set of capabilities.

The list of emulation managers with their capabilities can be queried using the following command:

            

            
snap_rpc.py emulation_managers_list
        

Appendix SPDK Configuration includes additional information.

Emulation Devices Configuration (Hotplug)

As mentioned above, each emulation manager holds a list of the emulated PCI functions it controls. The PCI functions may be approached later by either their function index (in emulation manager’s list) or their PCIe BDF number (e.g. 88:00.2) as enumerated by the host OS. Some PCIe functions configured at the FW configuration stage are considered "static" (i.e. always present).

In addition, users can dynamically add detachable functions to that list at runtime (and to the host's PCIe devices list accordingly). These functions are called "Hotplugged" PCIe functions.

After a new PCIe function is plugged, it is shown on host PCIe devices list until either explicitly unplugged or the system goes through a cold reboot. Hot-plugged PCIe function will remain persistent even after SNAP process termination.

Some OSs automatically start to communicate with the new function after it is plugged. Some continue to communicate with the function (for a certain time) even after it is signaled to be unplugged. Therefore, users must always keep an open controller (of a matching type) over any existing configured PCIe function (see NVMe Controller Management and Virtio-blk Controller Management for more details).

The following command hotplugs a new PCIe function to the system:

            

            
snap_rpc.py emulation_device_attach emu_manager {nvme,virtio_blk} [--id ID] [--vid VID] [--ssid SSID] [--ssvid SSVID] [--revid REVID] [--class_code CLASS_CODE] [--bdev_type {spdk,none}] [--bdev BDEV] [--num_queues NUM_QUEUES] [--queue_depth QUEUE_DEPTH][--total_vf TOTAL_VF] [--num_msix NUM_MSIX]
        

The following command hot-unplugs a new PCIe function from the system:

            

            
snap_rpc.py emulation_device_detach <emu_manager> {nvme,virtio_blk} [-d PCI_BDF / -i PCI_INDEX / --vuid VUID]
        

Mandatory parameters:

• emu_manager – emulation manager

• {nvme,virtio_blk} – device type

Optional arguments:

• --pci_bdf – PCIe BDF identifier

• --pci_index – PCIe index identifier

• --vuid – PCIe VUID identifier

• --force – forcefully remove device (not recommended)

  Note

  At least one identifier must be provided to describe the PCIe function to be detached.

The following command lists all existing functions (either static or hotplugged):

            

            
snap_rpc.py emulation_functions_list
        

NVMe Subsystem

The NVMe subsystem, as described in the NVMe specification, is a logical entity which encapsulates sets of NVMe backends (or namespaces) and connections (or controllers). NVMe subsystems are extremely useful when working with multiple NVMe controllers, and especially when using NVMe virtual functions.

Each NVMe subsystem is defined by its serial number (SN), model number (MN), and qualified name (NQN). After creation, each subsystem also gets a unique index number.

The following example creates a new NVMe subsystem with a default generated NQN:

            

            
snap_rpc.py subsystem_nvme_create <serial_number> <model_number>
        

Mandatory parameters:

• serial_number – subsystem serial number

• model_number – subsystem model number

Optional arguments:

• --nqn – subsystem qualified name (auto-generated if not provided)

• --nn – maximal namespace ID allowed in the subsystem (default 0xFFFFFFFE; range 1-0xFFFFFFFE)

• --mnan – maximal number of namespaces allowed in the subsystem (default 1024; range 1-0xFFFFFFFE)

The following command deletes an NVMe subsystem:

            

            
snap_rpc.py subsystem_nvme_delete <NQN>
        

Where <NQN> is the subsystem's NQN.

The following command lists all NVMe subsystems:

            

            
snap_rpc.py subsystem_nvme_list
        

NVMe Controller

Each NVMe device (e.g. NVMe PCIe entry) exposed to the host, whether it is a PF or VF, must be backed by NVMe controller, which is responsible for all protocol communication with the host's driver. Every new NVMe controller must also be linked to NVMe subsystem. After creation, NVMe controllers can be addressed using either their name (e.g., "NvmeEmu0pf0") or both their subsystem NQN and controller ID.

The following command opens a new NVMe controller:

            

            
snap_rpc.py controller_nvme_create mlx5_0 [--pf_id ID / --pci_bdf / --vuid VUID] [--subsys_id ID / --nqn NQN]
        

Mandatory parameters:

• emu_manager – emulation manager

Optional parameters:

• --vf_id VF_ID – PCIe VF index to start emulation on (if the controller is destined to be opened on a VF). --pf_id must also be set for the command to take effect.

• --conf – JSON configuration file path to be used to provide an extended set of configuration parameters. Full information concerning the different parameters of the configuration file can be found under appendix "JSON File Format".

• --nr_io_queues – I/O queues maximal number (default 32, range 0-32)

• --mdts – maximum data transfer size (default 4, range 1-6)

• --max_namespaces – maximum number of namespaces for this controller (default 1024, range 1-0xFFFFFFFE)

• --quirks – bitmask to enable specific NVMe driver quirks to work with non-NVMe spec compliant drivers. For more information, refer to appendix "JSON File Format".

• --mem {static,pool} – use memory from a global pool or from dedicated buffers. See "Mem-pool" for more information.

The following command deletes an existing NVMe controller:

            

            
snap_rpc.py controller_nvme_delete [--name NAME / --subnqn SUBNQN --cntlid ID / --vuid VUID]
        

Optional arguments:

• -c NAME, --name NAME – controller Name. Must be set if --nqn and --cntlid are not set.

• -n SUBNQN, --subnqn SUBNQN – NVMe subsystem (NQN). Must be set if --name is not set.

• -i CNTLID, --cntlid CNTLID – controller identifier in NVMe subsystem. Must be set if --name is not set.

The following command lists all NVMe controllers:

            

            
snap_rpc.py controller_list --type nvme
        

Optional arguments:

• -t {nvme,virtio_blk,virtio_net}, --type {nvme,virtio_blk,virtio_net} – controller type

NVMe Backend (Namespace)

NVMe Namespaces are the representors of a continuous range of LBAs in the local/remote storage device (previously configured in "Backend Configuration" section). Each Namespace must be linked to a controller and have a unique identifier (NSID) across the entire NVMe subsystem (e.g. 2 namespaces cannot share the same NSID, even though linked to different controllers).

SNAP application uses SPDK block device framework as backends for its NVMe namespaces. Therefore, they should be configured in advance. For more information about SPDK block devices, see SPDK BDEV documentation and appendix "SPDK Configuration".

The following command attaches a new namespace to an existing NVMe controller:

            

            
snap_rpc.py controller_nvme_namespace_attach [--ctrl CTRL / --subnqn SUBNQN --cntlid ID] <bdev_type> <bdev> <nsid>
        

Mandatory parameters:

• bdev_type – block device type

• bdev – block device to use as backend

• nsid – namespace ID

Optional parameters:

• -c CTRL, --ctrl CTRL – controller name. Must be set if --nqn and --cntlid are not set.

• -n SUBNQN, --subnqn SUBNQN – NVMe subsystem (NQN). Must be set if --ctrl is not set.

• -i CNTLID, --cntlid CNTLID – controller identifier in NVMe subsystem. Must be set if --ctrl is not set.

• -q QN, --qn QN – QN of remote target which provides this namespace

• -p PROTOCOL, --protocol PROTOCOL – protocol used

• -g NGUID, --nguid NGUID – namespace globally unique identifier

• -e EUI64, --eui64 EUI64 – namespace EUI-64 identifier

• -u UUID, --uuid UUID – namespace UUID

The following command detaches a namespace from a controller:

            

            
snap_rpc.py controller_nvme_namespace_detach [--ctrl CTRL / --subnqn SUBNQN --cntlid ID] <nsid>
        

Mandatory parameters:

• nsid – namespace ID

Optional parameters:

• -c CTRL, --ctrl CTRL – controller name. Must be set if --nqn and --cntlid are not set.

• -n SUBNQN, --subnqn SUBNQN – NVMe subsystem (NQN). Must be set if --ctrl is not set.

• -i CNTLID, --cntlid CNTLID – controller identifier in NVMe subsystem. Must be set if --ctrl is not set.

The following command lists a namespace of a controller:

            

            
snap_rpc.py controller_nvme_namespace_list [--ctrl CTRL / --subnqn SUBNQN --cntlid ID]
        

Optional parameters:

• -c CTRL, --ctrl CTRL – controller name. Must be set if --nqn and --cntlid are not set.

• -n SUBNQN, --subnqn SUBNQN – NVMe subsystem (NQN). Must be set if --ctrl is not set.

• -i CNTLID, --cntlid CNTLID – controller identifier in NVMe subsystem. Must be set if --ctrl is not set.

VirtIO-blk Controller

Each virtio-blk device (e.g. virtio-blk PCI entry) exposed to the host, whether it is PF or VF, must be backed by a virtio-blk controller. Virtio-blk is considered a limited storage protocol (compared to NVMe, for instance).

Due to protocol limitations:

• Trying to use a virtio-blk device (e.g. probe virtio-blk driver on host) without an already functioning virtio-blk controller, may cause the host server to hang until such controller is opened successfully (no timeout mechanism exists)

• Upon creation of a virtio-blk controller, a backend device must already exist

The following command creates a new virtio-blk controller:

            

            
snap_rpc.py controller_virtio_blk_create <emu_manager> [-d PCI_BDF / --pf_id PF_ID / --vuid VUID]
        

Mandatory parameters:

• emu_manager – emulation manager

Optional parameters:

• --vf_id – PCIe VF index to start emulation on, if controller is destined to be opened on VF

• --num_queues – number of queues (default 64, range 2-256)

• --queue_depth – queue depth (default 128, range 1-256)

• --size_max –- maximal SGE data transfer size (default 4096, range 1 – MAX_UINT16). See VirtIO specification for more information.

• --seg_max – maximal SGE list length (default 1, range 1-queue_depth). See VirtIO specification for more information.

• --bdev_type – block device type (spdk/none). Note that opening a controller with none backend means to open it with a backend of size of 0.

• --bdev – SPDK block device to use as backend

• --serial – serial number for the controller

• --force_in_order - force I/O in-order completions. Note that this value is required to ensure future virtio-blk controllers always successfully recover after an application crash.

• --suspend – create controller is in a SUSPENDED state (must be explicitly resumed later)

• --mem {static,pool} – use memory from a global pool or from dedicated buffers. See ".Advanced Features v3.5.0" for more information.

The following command deletes a virtio-blk controller:

            

            
snap_rpc.py controller_virtio_blk_delete [-c NAME / --vuid VUID]
        

Mandatory arguments:

• name – controller name

Optional arguments:

• --f, --force – force controller deletion

The following command lists all virtio-blk controllers:

            

            
snap_rpc.py controller_list --type virtio_blk
        

Optional arguments:

• -t {nvme,virtio_blk,virtio_net}, --type {nvme,virtio_blk,virtio_net} – controller type

Virtio-blk controllers can also be suspended and resumed. While suspended, the controller stops receiving new requests from the host driver and only finishes handling of requests already in flight (without prompting any IO errors back to the driver).

The following command suspends/resumes virtio-blk controller:

            

            
snap_rpc.py controller_virtio_blk_suspend <name>
snap_rpc.py controller_virtio_blk_resume <name>
        

Mandatory arguments:

• name – controller name

VirtIO-blk Backend Management

Like NVMe, VirtIO BLK also uses SPDK block devices framework as its backend devices, but since virtio-blk is a limited storage protocol as opposed to NVMe, whose backend management abilities are limited as well:

• Virtio-blk protocol supports only one backend device

• Virtio-blk protocol does not support administration commands to add backends, thus all backend attributes are communicated to the host virtio-blk driver over PCIe BAR and must be accessible during driver probing. For that reason, backends can only be changed when PCIe function is not in use by any host storage driver.

For these reasons, when the host driver is active, all backend management operations must occur only when the controller is in suspended state.

The following command attaches a new backend to a controller:

            

            
snap_rpc.py controller_virtio_blk_bdev_attach <ctrl_name> {spdk} <bdev_name>
        

Mandatory arguments:

• name – controller name

• {spdk} – block device type

• bdev – block device to use as backend

Optional arguments:

• --size_max – maximal SGE data transfer size (no hard limit). See VirtIO specification for more information.

• --seg_max – maximal SGE list length (no hard limit). See VirtIO specification for more information.

The following command detaches a backend from a controller:

            

            
snap_rpc.py controller_virtio_blk_bdev_detach <ctrl_name>
        

Mandatory arguments:

• ctrl_name – controller name

The following command lists a backend detail of a controller:

            

            
snap_rpc.py controller_virtio_blk_bdev_list <name>
        

Mandatory arguments:

• name – controller name

BlueField and virito-blk SNAP provide a set of commands which help customers retrieve performance and debug statistics about the opened emulated devices. The statistics are provided at the SNAP controller level (whether for NVMe or Virtio-blk).

IO Statistics

The following commands are available to measure how many successful/failed IO operations were executed by the controller.

These commands have minimal effect on BlueField SNAP performance and can therefore be used to sample statistics while the controller performs high bandwidth IO operations.

            

            
snap_rpc.py controller_nvme_get_iostat [-c CTRL_NAME]
snap_rpc.py controller_virtio_blk_get_iostat [-c CTRL_NAME]
        

Mandatory arguments:

• CTRL_NAME – controller name

NVMe/Virtio IO statistics:

• read_ios – number of read commands handled

• completed_read_ios – number of read commands completed successfully

• err_read_ios – number of read commands completed with error

• write_ios – number of write commands handled

• completed_write_ios – number of write commands completed successfully

• err_write_ios – number of write commands completed with error

• flush_ios – number of flush commands handled

• completed_flush_ios – number of flush commands completed successfully

• err_flush_ios – number of flush commands completed with error

Virtio IO specific statistics:

• fatal_ios – number of commands dropped and never completed

• outstanding_in_ios – number of outstanding IOs at a given moment

• outstanding_in_bdev_ios – number of outstanding IOs at a given moment, pending backend handling

• outstanding_to_host_ios – number of outstanding IOs at a given moment, pending DMA handling

Debug Statistics

The following commands are available to examine the controller and queues with more detailed status and information.

When queried frequently, these commands may impact performance and should therefore be called for debug purposes only.

            

            
snap_rpc.py controller_nvme_get_debugstat [-c NAME]
snap_rpc.py controller_virtio_blk_get_debugstat [-c NAME]
        

The default initialization scripts /etc/mlnx_snap/spdk_rpc_init.conf and /etc/mlnx_snap/snap_rpc_init.conf allow users to control the startup configuration.

These scripts, which are used for the out-of-box configuration, may be modified by the user to control the SNAP initialization :

• The spdk_rpc_init.conf may be modified with the SPDK commands listed under the SPDK Configuration appendix.

• The snap_rpc_init.conf may be modified with the snap_rpc commands described throughout this chapter (SNAP Commands).