Part 6: Validate Slurm#
Login to SLOGIN-01 and run sinfo to verify that all the nodes are up and ready.
# At the OS/shell level on the slogin node, run root@slogin-01:~# module load slurm root@slogin-01:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST defq* up infinite 4 idle dgx-[01-04]
Basic slurm job/GPU test using the DGX node running the RHEL image.
# At the OS/shell level on the slogin node, run root@slogin-01:~# srun -w dgx-01 --gres=gpu:8 -n1 nvidia-smi Wed May 27 15:16:18 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA B200 On | 00000000:1B:00.0 Off | 0 | | N/A 32C P0 220W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA B200 On | 00000000:43:00.0 Off | 0 | | N/A 33C P0 219W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA B200 On | 00000000:52:00.0 Off | 0 | | N/A 37C P0 220W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA B200 On | 00000000:61:00.0 Off | 0 | | N/A 38C P0 221W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA B200 On | 00000000:9D:00.0 Off | 0 | | N/A 35C P0 221W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA B200 On | 00000000:C3:00.0 Off | 0 | | N/A 34C P0 218W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA B200 On | 00000000:D1:00.0 Off | 0 | | N/A 38C P0 219W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 7 NVIDIA B200 On | 00000000:DF:00.0 Off | 0 | | N/A 41C P0 226W / 1000W | 0MiB / 183359MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
Host and Container Validation Test#
# At the OS/shell level on the slogin node, run # Container Validation test root@slogin-01:~# srun -w dgx-02 --container-image=redhat/ubi9 grep PRETTY /etc/os-release pyxis: importing docker image: redhat/ubi9 pyxis: imported docker image: redhat/ubi9 PRETTY_NAME="Red Hat Enterprise Linux 9.8 (Plow)" # Host Validation test root@slogin-01:~# srun -w dgx-02 grep PRETTY /etc/os-release PRETTY_NAME="Red Hat Enterprise Linux 9.6 (Plow)"
Single-Node NCCL Test#
The following test uses the public image nvcr.io/nvidia/pytorch:26.04-py3. Enroot pulls it automatically via Pyxis at job
runtime. All DGX nodes used are running the RHEL software image.
# At the OS/shell level on the slogin node, run root@slogin-01:~# srun -w dgx-02 --export="NCCL_SOCKET_IFNAME=bond0" -N1 --exclusive --gpus-per-node=8 --mpi=pmix --container-name=nccl-test --container-image="nvcr.io/nvidia/pytorch:26.04-py3" all_reduce_perf_mpi -b 1G -e 16G -f 2 -g 8 [dgx-02:549474] PMIX ERROR: ERROR in file gds_ds12_lock_pthread.c at line 168 # nThread 1 nGpus 8 minBytes 1073741824 maxBytes 17179869184 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # # Using devices # Rank 0 Group 0 Pid 549474 on dgx-02 device 0 [0x1b] NVIDIA B200 # Rank 1 Group 0 Pid 549474 on dgx-02 device 1 [0x43] NVIDIA B200 # Rank 2 Group 0 Pid 549474 on dgx-02 device 2 [0x52] NVIDIA B200 # Rank 3 Group 0 Pid 549474 on dgx-02 device 3 [0x61] NVIDIA B200 # Rank 4 Group 0 Pid 549474 on dgx-02 device 4 [0x9d] NVIDIA B200 # Rank 5 Group 0 Pid 549474 on dgx-02 device 5 [0xc3] NVIDIA B200 # Rank 6 Group 0 Pid 549474 on dgx-02 device 6 [0xd1] NVIDIA B200 # Rank 7 Group 0 Pid 549474 on dgx-02 device 7 [0xdf] NVIDIA B200 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 1073741824 268435456 float sum -1 2544.6 421.96 738.43 0 2543.3 422.18 738.81 0 2147483648 536870912 float sum -1 4566.7 470.25 822.93 0 4568.2 470.09 822.66 0 4294967296 1073741824 float sum -1 9021.2 476.10 833.17 0 9019.6 476.18 833.32 0 8589934592 2147483648 float sum -1 17861 480.93 841.64 0 17854 481.12 841.95 0 17179869184 4294967296 float sum -1 36674 468.45 819.78 0 39547 434.41 760.22 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 805.292
Multi-Node NCCL Test — Two and Three Nodes#
# At the OS/shell level on the slogin node, run root@slogin-01:~# srun -w dgx-02,dgx-03 --export="UCX_NET_DEVICES=bond0,OMPI_MCA_pml=ucx,HCOLL_ENABLE_MCAST_ALL=0,OMPI_MCA_coll=^hcoll,NCCL_SOCKET_IFNAME=bond0,UCX_TLS=tcp,UCX_NET_DEVICES=bond0,MELLANOX_VISIBLE_DEVICES=all,OMPI_MCA_coll_hcoll_enable=0,OMPI_MCA_coll_ucc_enable=0" -N2 --exclusive --gpus-per-node=8 --mpi=pmix --container-name=nccl-test --container-image="nvcr.io/nvidia/pytorch:26.04-py3" all_reduce_perf_mpi -b 1G -e 16G -f 2 -g 8 [dgx-02:572898] PMIX ERROR: ERROR in file gds_ds12_lock_pthread.c at line 168 [dgx-03:468807] PMIX ERROR: ERROR in file gds_ds12_lock_pthread.c at line 168 # nThread 1 nGpus 8 minBytes 1073741824 maxBytes 17179869184 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # # Using devices # Rank 0 Group 0 Pid 572898 on dgx-02 device 0 [0x1b] NVIDIA B200 # Rank 1 Group 0 Pid 572898 on dgx-02 device 1 [0x43] NVIDIA B200 # Rank 2 Group 0 Pid 572898 on dgx-02 device 2 [0x52] NVIDIA B200 # Rank 3 Group 0 Pid 572898 on dgx-02 device 3 [0x61] NVIDIA B200 # Rank 4 Group 0 Pid 572898 on dgx-02 device 4 [0x9d] NVIDIA B200 # Rank 5 Group 0 Pid 572898 on dgx-02 device 5 [0xc3] NVIDIA B200 # Rank 6 Group 0 Pid 572898 on dgx-02 device 6 [0xd1] NVIDIA B200 # Rank 7 Group 0 Pid 572898 on dgx-02 device 7 [0xdf] NVIDIA B200 # Rank 8 Group 0 Pid 468807 on dgx-03 device 0 [0x1b] NVIDIA B200 # Rank 9 Group 0 Pid 468807 on dgx-03 device 1 [0x43] NVIDIA B200 # Rank 10 Group 0 Pid 468807 on dgx-03 device 2 [0x52] NVIDIA B200 # Rank 11 Group 0 Pid 468807 on dgx-03 device 3 [0x61] NVIDIA B200 # Rank 12 Group 0 Pid 468807 on dgx-03 device 4 [0x9d] NVIDIA B200 # Rank 13 Group 0 Pid 468807 on dgx-03 device 5 [0xc3] NVIDIA B200 # Rank 14 Group 0 Pid 468807 on dgx-03 device 6 [0xd1] NVIDIA B200 # Rank 15 Group 0 Pid 468807 on dgx-03 device 7 [0xdf] NVIDIA B200 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 1073741824 268435456 float sum -1 3343.0 321.19 602.23 0 3352.4 320.29 600.54 0 2147483648 536870912 float sum -1 6499.0 330.43 619.56 0 6524.4 329.14 617.15 0 4294967296 1073741824 float sum -1 12773 336.27 630.50 0 12757 336.68 631.27 0 8589934592 2147483648 float sum -1 25240 340.33 638.12 0 25279 339.81 637.14 0 17179869184 4294967296 float sum -1 50350 341.21 639.76 0 50134 342.68 642.52 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 625.878 # ## ---- 3 node NCCL test ----## # At the OS/shell level on the slogin node, run root@slogin-01:~# srun -w dgx-01,dgx-02,dgx-03 --export="UCX_NET_DEVICES=bond0,OMPI_MCA_pml=ucx,HCOLL_ENABLE_MCAST_ALL=0,OMPI_MCA_coll=^hcoll,NCCL_SOCKET_IFNAME=bond0,UCX_TLS=tcp,UCX_NET_DEVICES=bond0,MELLANOX_VISIBLE_DEVICES=all,OMPI_MCA_coll_hcoll_enable=0,OMPI_MCA_coll_ucc_enable=0" -N3 --exclusive --gpus-per-node=8 --mpi=pmix --container-name=nccl-test --container-image="nvcr.io/nvidia/pytorch:26.04-py3" all_reduce_perf_mpi -b 1G -e 16G -f 2 -g 8 pyxis: imported docker image: nvcr.io/nvidia/pytorch:26.04-py3 pyxis: imported docker image: nvcr.io/nvidia/pytorch:26.04-py3 pyxis: imported docker image: nvcr.io/nvidia/pytorch:26.04-py3 [dgx-03:1553761] PMIX ERROR: ERROR in file gds_ds12_lock_pthread.c at line 168 [dgx-02:256136] PMIX ERROR: ERROR in file gds_ds12_lock_pthread.c at line 168 [dgx-01:2922180] PMIX ERROR: ERROR in file gds_ds12_lock_pthread.c at line 168 # nccl-tests version 2.18.2 nccl-headers=22907 nccl-library=22907 # Collective test starting: all_reduce_perf_mpi # nThread 1 nGpus 8 minBytes 1073741824 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0 # # Using devices # Rank 0 Group 0 Pid 2922180 on dgx-01 device 0 [0000:1b:00] NVIDIA B200 # Rank 1 Group 0 Pid 2922180 on dgx-01 device 1 [0000:43:00] NVIDIA B200 # Rank 2 Group 0 Pid 2922180 on dgx-01 device 2 [0000:52:00] NVIDIA B200 # Rank 3 Group 0 Pid 2922180 on dgx-01 device 3 [0000:61:00] NVIDIA B200 # Rank 4 Group 0 Pid 2922180 on dgx-01 device 4 [0000:9d:00] NVIDIA B200 # Rank 5 Group 0 Pid 2922180 on dgx-01 device 5 [0000:c3:00] NVIDIA B200 # Rank 6 Group 0 Pid 2922180 on dgx-01 device 6 [0000:d1:00] NVIDIA B200 # Rank 7 Group 0 Pid 2922180 on dgx-01 device 7 [0000:df:00] NVIDIA B200 # Rank 8 Group 0 Pid 256136 on dgx-02 device 0 [0000:1b:00] NVIDIA B200 # Rank 9 Group 0 Pid 256136 on dgx-02 device 1 [0000:43:00] NVIDIA B200 # Rank 10 Group 0 Pid 256136 on dgx-02 device 2 [0000:52:00] NVIDIA B200 # Rank 11 Group 0 Pid 256136 on dgx-02 device 3 [0000:61:00] NVIDIA B200 # Rank 12 Group 0 Pid 256136 on dgx-02 device 4 [0000:9d:00] NVIDIA B200 # Rank 13 Group 0 Pid 256136 on dgx-02 device 5 [0000:c3:00] NVIDIA B200 # Rank 14 Group 0 Pid 256136 on dgx-02 device 6 [0000:d1:00] NVIDIA B200 # Rank 15 Group 0 Pid 256136 on dgx-02 device 7 [0000:df:00] NVIDIA B200 # Rank 16 Group 0 Pid 1553761 on dgx-03 device 0 [0000:1b:00] NVIDIA B200 # Rank 17 Group 0 Pid 1553761 on dgx-03 device 1 [0000:43:00] NVIDIA B200 # Rank 18 Group 0 Pid 1553761 on dgx-03 device 2 [0000:52:00] NVIDIA B200 # Rank 19 Group 0 Pid 1553761 on dgx-03 device 3 [0000:61:00] NVIDIA B200 # Rank 20 Group 0 Pid 1553761 on dgx-03 device 4 [0000:9d:00] NVIDIA B200 # Rank 21 Group 0 Pid 1553761 on dgx-03 device 5 [0000:c3:00] NVIDIA B200 # Rank 22 Group 0 Pid 1553761 on dgx-03 device 6 [0000:d1:00] NVIDIA B200 # Rank 23 Group 0 Pid 1553761 on dgx-03 device 7 [0000:df:00] NVIDIA B200 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 1073741824 268435456 float sum -1 6040.20 177.77 340.72 0 6039.51 177.79 340.76 0 2147483648 536870912 float sum -1 12018.7 178.68 342.47 0 12020.4 178.65 342.42 0 4294967296 1073741824 float sum -1 24153.0 177.82 340.83 0 24155.2 177.81 340.80 0 8589934592 2147483648 float sum -1 47763.0 179.84 344.70 0 47762.5 179.85 344.71 0 17179869184 4294967296 float sum -1 95263.0 180.34 345.65 0 95248.2 180.37 345.71 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 342.876 # # Collective test concluded: all_reduce_perf_mpi #