cusvaer

cuQuantum Appliance 22.11 contains the first release of cusvaer. cusvaer is designed as a Qiskit backend solver and optimized for a distributed state vector simulation.

Features

  • Distributed simulation

    cusvaer distributes state vector simulations to multiple processes and nodes.

  • Power-of-2 configuration for performance

    The number of GPUs and the number of processes and nodes should be always power-of-2. This design is chosen for the optimal performance.

  • Shipped with a validated set of libraries

    cusvaer is shipped with the latest cuStateVec and a validated set of MPI libraries. The performance has been validated on NVIDIA SuperPOD.

Distributed state vector simulation

Popular method to distribute state vector simulation is to equally-slice the state vector into multiple sub state vectors and distribute them to processes. Some examples for distibuted state vector simulation are also described in Multi-GPU Computation. In cusvaer, this equally-sliced state vector is called as sub state vector.

The first release of cusvaer uses the “single device per single process” model. All sub state vectors are placed in GPUs. Each process owns a single GPU, and one sub state vector is allocated in the GPU assigned to the process. Thus, the number of processes is always identical to the number of GPUs. The size of sub state vector is the same in all processes.

When slicing the state vector, the size of sub state vector is calculated by the number of qubits in a circuit and the number of GPUs used for distribution.

Ex. 40-qubit c128 simulation can be computed using 32 DGX A100 with the following configurations.

  • The state vector size : 16 TiB (= 16 [bytes/elm] * (1 << 40 [qubits]))

  • The number of GPUs per node : 8

  • The number of nodes : 32

  • The size of sub state vector in each GPU: 64 [GiB/GPU] = 16 TiB / (32 [node] x 8 [GPU/node])

cusvaer calculates the size of sub state vector by using the number of qubits of a given quantum circuit and the number of processes. Thus, by specifying an appropriate number of processes, users are able to distribute state vector simulations to processes and nodes. The number of processes should be power-of-2.

For nodes with multiple GPUs, GPUs are assigned to processes by using the following formula that is used to configure affinity.

device_id = mpi_rank % number_of_processes_in_node

Using cusvaer

cusvaer provides one class, cusvaer.backends.StatevectorSimulator. This class has the same interface of Qiskit Aer StatevectorSimulator.

Below is an example code for using cusvaer.

from qiskit import QuantumCircuit, transpile
from cusvaer.backends import StatevectorSimulator

def create_ghz_circuit(n_qubits):
    ghz = QuantumCircuit(n_qubits)
    ghz.h(0)
    for qubit in range(n_qubits - 1):
        ghz.cx(qubit, qubit + 1)
    ghz.measure_all()
    return ghz

circuit = create_ghz_circuit(20)

simulator = StatevectorSimulator()
simulator.set_options(precision='double')
circuit = transpile(circuit, simulator)
job = simulator.run(circuit, shots=1024)
result = job.result()

if result.mpi_rank == 0:
    print(result.get_counts())

The following console output will be obtained:

$ python ghz_cusvaer.py
{'00000000000000000000': 480, '11111111111111111111': 544}

To run distributed simulation, users need to use MPI or appropriate libraries to specify the number of processes. In the example below, 20-qubit state vector simulation is distributed to two processes, and each process has 19-qubit sub state vector. The number of processes should be power-of-2.

$ mpirun -n 2 ghz_cusvaer.py

Note

GHZ circuit is used here as an example, which does not mean for demonstrating performance. It is known that the GHZ circuit is too shallow to show good scaling when simulation is distributed to multiple-GPUs and/or multiple-nodes.

MPI libraries

cusvaer provides built-in support for OpenMPI and MPICH.

The versions shown below are validated or expected to work.

  • OpenMPI

    • Validated: v4.1.4 / UCX v1.13.1

    • Expected to work: v3.0.x, v3.1.x, v4.0.x, v4.1.x

  • MPICH

    • Validated: v4.0.2

cuQuantum Appliance is shipped with binaries OpenMPI v4.1.4 and UCX v1.13.1 binaries compiled with PMI enabled. The build configuration is intended to work with Slurm.

Two options, cusvaer_comm_plugin_type and cusvaer_comm_plugin_soname are used to select MPI library. Please refer to CommPlugin for details.

For the use of other MPI libraries, users need to compile external plugin. Please refer to cuQuantum GitHub.

cusvaer options

cusvaer provides a subset of common Qiskit Aer options and cusvaer-specific options.

Common options provided by Qiskit Aer

Option

Value

Description

shots

non-negative integer

the number of shots

precision

"single" or "double"

the precision of the state vector elements

fusion_enable

True or False

Enable/disable gate fusion

fusion_max_qubit

positive integer

The max number of qubits used for gate fusion

fusion_threshold

positive integer

Threshold to enable gate fusion

memory

True or False

If True is specified, classical register values are saved to the result

seed_simulator

integer

the seed for random number generator

cusvaer options

Option

Value

Description

cusvaer_comm_plugin_type

cusvaer.CommPluginType

Selecting CommPlugin for inter-process communication

cusvaer_comm_plugin_soname

string

The shared library name for inter-process communication

cusvaer_global_index_bits

list of positive integers

Network structure

cusvaer_p2p_device_bits

non-negative integer

cusvaer_data_transfer_buffer_bits

positive integer

The size used for a temporary buffer during transferring data

cusvaer_host_threads

positive integer

The number of host threads used to process circuits on host

CommPlugin

cusvaer dynamically links to a library that handles inter-process communication by using CommPlugin. CommPlugin is a small module that typically wraps MPI libraries. The CommPlugin is selected by cusvaer_comm_plugin_type and cusvaer_comm_plugin_soname options.

cusvaer_comm_plugin_type

cusvaer_comm_plugin_type option to select CommPlugin. The value type is cusvaer.CommPluginType and the default value is cusvaer.CommPluginType.MPI_AUTO.

# declared in cusvaer
from enum import IntEnum

class CommPluginType(IntEnum):
    SELF        = 0       # Single process
    EXTERNAL    = 1       # Use external plugin
    MPI_AUTO    = 0x100   # Automatically select OpenMPI or MPICH
    MPI_OPENMPI = 0x101   # Use OpenMPI
    MPI_MPICH   = 0x102   # Use MPICH

SELF

CommPlugin used for single process simulation. SELF CommPlugin does not have any external dependency.

MPI_AUTO, MPI_OPENMPI, MPI_MPICH

By specifying MPI_OPENMPI or MPI_MPICH, the specified library will be dynamically loaded and used for inter-process data transfers. MPI_AUTO automatically selects one of them. With these options, cusvaer internally would use MPI_COMM_WORLD as the communicator.

EXTERNAL

Option to use an external custom plugin that wraps an inter-process communication library. This option is to use libraries of users’ preferences. Please refer to cuQuantum GitHub for details.

The value of CommPluginType should be the same during the application lifetime if a value other than CommPluginType.SELF is specified. It reflects that a process is able only to use one MPI library for the application lifetime.

comm_plugin_soname

comm_plugin_soname option specifies the name of a shared library. When MPI_AUTO, MPI_OPENMPI, or MPI_MPICH is specified for cusaver_comm_plugin_type, the corresponding MPI library name in the search path should be specified. The default value is "libmpi.so".

If EXTERNAL is specified, comm_plugin_soname will contain the name of a custom CommPlugin.

Specifying device network structure

Two options, cusvaer_global_index_bits and cusvaer_p2p_device_bits, are available for specifying inter-node and intra-node network structures. By specifying these options, cusvaer schedules data transfers to utilize faster communication paths to accelerate simulations.

cusvaer_global_index_bits

cusvaer_global_index_bits is a list of positive integers that represents the inter-node network structure.

Assuming 8 nodes has faster communication network in a cluster, and running 32 node simulation, the value of cusvaer_global_index_bits is [3, 2]. The first 3 is log2(8) representing 8 nodes with fast communication which corresponding to 3 qubits in the state vector. The second 2 means 4 8-node groups in 32 nodes. The sum of the global_index_bits elements is 5, which means the number of nodes is 32 = 2^5.

The last element in the cusvaer_global_index_bits can be omitted.

cusvaer_p2p_device_bits

cusvaer_p2p_device_bits option is to specify the number of GPUs that can communicate by using GPUDirect P2P.

For 8 GPU node such as DGX A100, the number is log2(8) = 3.

The value of cusvaer_p2p_device_bits is typically the same as the first element of cusvaer_global_index_bits as the GPUDirect P2P network is typically the fastest in a cluster.

cusvaer_data_transfer_buffer_bits

cusvaer_data_transfer_buffer_bits specifies the size of the buffer utilized for inter-node data transfers. Each rank of process allocates (1 << cusvaer_data_transfder_buffer_bits) bytes of device memory. The default is set to 26 (64 MiB). The minimum allowed value is 24 (16 MiB).

Depending on systems, setting a larger value to cusvaer_data_transfer_buffer_bits can accelerate inter-node data transfers.

Other options

cusvaer_host_threads

cusvaer_host_threads specifies the number of CPU threads used for circuit processing. The default value is set to 8, a good value. The performance is not sensitive to this option.

Custom instruction

set_state_simple(state)

This instruction sets the state vector. The state argument is numpy ndarray. If the dtype of the state argument is different from that of the state vector precision, the given ndarray is converted to the type of the state vector prior to being set to the state vector.

When running a single process simulation, the state argument will set its array elements to the state vector. The state argument value has the same size as that of the state vector.

If a simulation is distributed to multiple processes, this instruction sets the sub state vector allocated in the device assigned to a process. The size of the state argument should be the same as that of the sub state vector.

Different from set_statevector() instruction provided by Qiskit Aer, set_state_simple() does not check the absolute value of the state argument.

save_state_simple()

This instruction saves the state vector allocated in GPU assigned to a process. The saved state vector is returned as a numpy ndarray in a result object.

When running a single process simulation, the saved ndarray is a state vector. If a simulation is distributed to multiple processes, the saved ndarray is a sub state vector owned by a process.

Note

The above custom instructions are preliminary and are subject to change.

Example of cusvaer option configurations

import cusvaer

options = {
  'cusvaer_comm_plugin_type': cusvaer_comm_plugin_type=cusvaer.CommPluginType.MPI_AUTO,  # automatically select OpenMPI or MPICH
  'cusvaer_comm_plugin_soname': 'libmpi.so',  # MPI library name is libmpi.so
  'cusvaer_global_index_bits': [3, 2],  # 8 devices per node, 4 nodes
  'cusvaer_p2p_device_bits': 3,         # 8 GPUs in one node
  'precision': 'double'         # use complex128
}
simulator = cusvaer.backends.StatevectorSimulator()
simulator.set_options(**options)

Exception

cusvaer uses qiskit.providers.basicaer.BasicAerError class to raise exceptions.

Interoperability with mpi4py

mpi4py is a python package that wraps MPI. By importing mpi4py before running simulations, cusvaer interoperates with mpi4py.

from qiskit import QuantumCircuit, transpile
from cusvaer.backends import StatevectorSimulator
# import mpi4py here to call MPI_Init()
from mpi4py import MPI

def create_ghz_circuit(n_qubits):
    ghz = QuantumCircuit(n_qubits)
    ghz.h(0)
    for qubit in range(n_qubits - 1):
        ghz.cx(qubit, qubit + 1)
    ghz.measure_all()
    return ghz

print(f"mpi4py: rank: {MPI.COMM_WORLD.Get_rank()}, size: {MPI.COMM_WORLD.Get_size()}")

circuit = create_ghz_circuit(n_qubits)

# Create StatevectorSimulator instead of using Aer.get_backend()
simulator = StatevectorSimulator()
circuit = transpile(circuit, simulator)
job = simulator.run(circuit)
result = job.result()

print(f"Result: rank: {result.mpi_rank}, size: {result.num_mpi_processes}")

cusvaer calls MPI_Init() right before executing the first simulation, and calls MPI_Finalize() at the timing that Python calls functions registered in atexit.

cusvaer checks if MPI is initialized before calling MPI_Init(). If MPI_Init() is already called before the first simulation, MPI_Init() and MPI_Finalize() calls are skipped.

For mpi4py, MPI_Init() is called at the timing of loading mpi4py. Thus, mpi4py and cusvaer coexist by importing mpi4py before calling simulator.run().