1. Set the CUDA_HOME environment variable to point to the CUDA Toolkit.
2. Set the GDRCOPY_HOME environment variable to point to the GDRCopy installation.

   To build without GDRCopy, set the environmental variable to NVSHMEM_USE_GDRCOPY=0.

   Note: Without GDRCopy, atomics are only supported across NVLink connections.
3. If MPI and/or SHMEM support is required, set NVSHMEM_MPI_SUPPORT=1 and/or NVSHMEM_SHMEM_SUPPORT=1.
4. Set the MPI_HOME and SHMEM_HOME environment variables to point to the MPI and OpenSHMEM installations, respectively.
5. If NCCL will be used for host-initiated collectives, set NVSHMEM_USE_NCCL=1 and NCCL_HOME to point to the NCCL installation. You can use any NCCL 2.x version, and NVSHMEM hast been tested with NCCL 2.8.3-1.
6. If the MPI library is neither OpenMPI nor its derivative, set NVSHMEM_MPI_IS_OMPI=0.
   Note: Here is some additional information:

   • When using OpenMPI and OSHMEM, the paths are the same.

     To use OSHMEM, OpenMPI needs to be built with UCX support.

   • NVSHMEM has been tested with OpenMPI 4.0.1 and UCX 1.8.0.
   • Other MPI and OpenSHMEM installations should also work.
   • By default, MPI support is enabled, and OpenSHMEM support is disabled.
7. If PMIx support is required, set NVSHMEM_PMIX_SUPPORT=1 and PMIX_HOME to point to the PMIx installation.
   Note:

   • PMI-1 and PMI-2 support is always included in the build, and PMI-1 is the default PMI.

     To change the default PMI NVSHMEM_DEFAULT_PMIX=1 (to select PMIx) or NVSHMEM_DEFAULT_PMI2=1 (to select PMI-2) can be set. At runtime, NVSHMEM_BOOTSTRAP_PMI can be used to override the default. The possible values are PMIX, PMI-2, and PMI.

   • OpenMPI ships with its own copy of PMIx.

     To avoid conflicting PMIx shared libraries, we recommend that you build a standalone PMIx and configure OpenMPI with --with-pmix=external to point to that installation.

8. To specify the location where NVSHMEM will be installed, set NVSHMEM_PREFIX.
9. To change the directory where NVSHMEM will be built, set NVSHMEM_BUILDDIR. The default is NVSHMEM_DIR/build.
10. To build and install the library, run make -j8 install.

Before you can run performance tests, you first must build them.

1. Set the CUDA_HOME, NVSHMEM_HOME and MPI_HOME (if the NVSHMEM library was built with NVSHMEM_MPI_SUPPORT=1) environment variables to build NVSHMEM performance tests:

   CUDA_HOME=<path to supported CUDA installation>
   NVSHMEM_HOME=<path to directory where NVSHMEM is installed>
   MPI_HOME=<path to MPI installation>
   

   Configuring OpenMPI using the -with-ucx option is required for OpenMPI/OSHMEM interoperability. If you have built NVSHMEM with MPI and OpenSHMEM support (NVSHMEM_MPI_SUPPORT=1 and NVSHMEM_SHMEM_SUPPORT=1), building perftest/ also requires MPI and OpenSHMEM support to be enabled.

   Build without SHMEM interoperability: To build NVSHMEM performance tests without SHMEM interoperability, set the environment variable NVSHMEM_SHMEM_SUPPORT to 0. By default, performance tests are installed under perftest/perftest_install. To install to a different path, set NVSHMEM_PERFTEST_INSTALL to point to the desired path.

2. Update LD_LIBRARY_PATH to point to $CUDA_HOME/lib64 and $MPI_HOME/lib.
3. Run performance tests as NVSHMEM jobs (assuming Hydra is installed under HYDRA_HOME), hybrid MPI+NVSHMEM jobs, or hybrid OpenSHMEM+NVSHMEM jobs with the following commands (using perftest/device/pt-to-pt/put.cu as an example):
   NVSHMEM job using Hydra (PM-1):

   $HYDRA_HOME/bin/nvshmrun -n <up to number of P2P or InfiniBand 
   NIC accessible GPUs> 
   $NVSHMEM_PERFTEST_INSTALL/device/pt-to-pt/shmem_put_bw

   NVSHMEM job using slurm:

   srun -n <up to number of P2P or InfiniBand NIC accessible GPUs> 
   $NVSHMEM_PERFTEST_INSTALL/device/pt-to-pt/shmem_put_bw

   Note: When slurm was built with a PMI that does not match the default of NVSHMEM, for example, if slurm was built with PMIx support and NVSHMEM_DEFAULT_PMIX=1 was not set when building NVSHMEM, NVSHMEM_BOOTSTRAP_PMI can be used to override the default. Possible values are PMIX, PMI-2, and PMI.
   Hybrid MPI/NVSHMEM job:

   $MPI_HOME/bin/mpirun -n <up to number of GPUs accessible by P2P 
   or InfiniBand NIC> -x NVSHMEMTEST_USE_MPI_LAUNCHER=1 
   $NVSHMEM_PERFTEST_INSTALL/device/pt-to-pt/shmem_put_bw

   Hybrid OpenSHMEM/NVSHMEM job:

   $MPI_HOME/bin/oshrun -n <up to number of GPUs accessible by P2P 
   or InfiniBand NIC> -x USE_SHMEM_IN_TEST=1 
   $NVSHMEM_PERFTEST_INSTALL/device/pt-to-pt/shmem_put_bw

1. Save the following code as nvshmemHelloWorld.cu:

   #include <stdio.h>
   #include <cuda.h>
   #include <nvshmem.h>
   #include <nvshmemx.h>
   
   __global__ void simple_shift(int *destination) {
       int mype = nvshmem_my_pe();
       int npes = nvshmem_n_pes();
       int peer = (mype + 1) % npes;
   
       nvshmem_int_p(destination, mype, peer);
   }
   
   int main(void) {
       int mype_node, msg;
       cudaStream_t stream;
   
       nvshmem_init();
       mype_node = nvshmem_team_my_pe(NVSHMEMX_TEAM_NODE);
       cudaSetDevice(mype_node);
       cudaStreamCreate(&stream);
   
       int *destination = (int *) nvshmem_malloc(sizeof(int));
   
       simple_shift<<<1, 1, 0, stream>>>(destination);
       nvshmemx_barrier_all_on_stream(stream);
       cudaMemcpyAsync(&msg, destination, sizeof(int), cudaMemcpyDeviceToHost, stream);
   
       cudaStreamSynchronize(stream);
       printf("%d: received message %d\n", nvshmem_my_pe(), msg);
   
       nvshmem_free(destination);
       nvshmem_finalize();
       return 0;
   }
   

2. Build nvshmemHelloWorld.cu with the following command:

   nvcc -rdc=true -ccbin g++ -gencode=$NVCC_GENCODE -I $NVSHMEM_HOME/include nvshmemHelloWorld.cu -o nvshmemHelloWorld.out -L $NVSHMEM_HOME/lib -lnvshmem 
   -lcuda
   

   Where arch=compute_70,code=sm_70 is the value of NVCC_GENCODE for V100 GPUs.

3. Run the nvshmemHelloWorld sample with one of the following commands: When running on a single host with 2 GPUs (connected by PCI-E, NVLink or Infiniband):

   $HYDRA_HOME/bin/nvshmrun -n 2 -ppn 2 ./nvshmemHelloWorld.out

   When running on two hosts with 1 GPU per host (connected by InfiniBand):

   $HYDRA_HOME/bin/nvshmrun -n 2 -ppn 1 –-hosts hostname1,hostname2 ./nvshmemHelloWorld.out