Basic System Health Checks#
To confirm that your Vera system is healthy and correctly configured, check compute performance, memory bandwidth, and memory latency with simple open-source benchmarks. Run these checks on an idle system and record the SMT mode, CPU count, memory capacity, kernel, page size, compiler version, and power policy with the results.
The reference results in this chapter are from a dual-socket Vera CPU system with 176 Olympus cores, 352 logical CPUs, 1.5 TB of LPDDR5X memory, a 64 KB page-size kernel, and 328 MiB of aggregate L3 cache. Results can change with firmware, kernel version, page size, memory capacity, background load, and platform power settings.
Preparing the System#
Record the CPU and NUMA topology before running the benchmarks:
$ lscpu
$ numactl -H
Install common build and measurement tools:
$ sudo apt-get update
$ sudo apt-get install -y build-essential git make gcc wget tar numactl \
linux-tools-common linux-tools-$(uname -r)
Vera supports Spatial Multithreading (SMT). For ST mode, disable SMT and use
one benchmark thread per physical core. For SMT mode, enable SMT and use one
benchmark thread per logical CPU. Vera has 88 cores per socket and 2 hardware
threads per core. Set SOCKETS to 1 for a single-socket system or 2
for a dual-socket system.
Set the common benchmark variables and system settings before running either mode:
$ SOCKETS=1 # Use 2 for a dual-socket system.
$ ST_THREADS_PER_SOCKET=88
$ SMT_THREADS_PER_SOCKET=176
$ ST_THREADS=$((SOCKETS * ST_THREADS_PER_SOCKET))
$ SMT_THREADS=$((SOCKETS * SMT_THREADS_PER_SOCKET))
$ sudo sh -c 'echo performance | tee \
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor > /dev/null'
$ sudo sh -c 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled'
$ sudo sh -c 'echo madvise > /sys/kernel/mm/transparent_hugepage/defrag'
$ sudo sysctl -w kernel.numa_balancing=0
To run in ST mode:
$ sudo sh -c 'echo off > /sys/devices/system/cpu/smt/control'
To run in SMT mode:
$ sudo sh -c 'echo on > /sys/devices/system/cpu/smt/control'
STREAM#
Use the STREAM benchmark to check Vera LPDDR5X memory bandwidth. The following commands download STREAM and compile it with arrays sized to four times the aggregate L3 cache size of a dual-socket Vera C2 system. STREAM uses three arrays, so this is large enough to run from memory instead of cache.
$ wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
$ C2_L3_CACHE_BYTES=$((328 * 1024 * 1024))
$ STREAM_ARRAY_SIZE=$((4 * C2_L3_CACHE_BYTES / 8))
$ gcc -Ofast -mcpu=native -fopenmp \
-DSTREAM_ARRAY_SIZE=${STREAM_ARRAY_SIZE} -DNTIMES=200 \
stream.c -o stream_openmp.exe
To run STREAM, set OMP_NUM_THREADS for the SMT mode being measured. To
distribute the threads evenly over all available cores and maximize bandwidth,
use OMP_PROC_BIND=spread.
$ OMP_NUM_THREADS=${ST_THREADS} OMP_PROC_BIND=spread ./stream_openmp.exe
$ OMP_NUM_THREADS=${SMT_THREADS} OMP_PROC_BIND=spread ./stream_openmp.exe
On dual-socket systems, also run a local-memory check on each socket and an aggregate check across sockets:
$ numactl --cpunodebind=0 --membind=0 \
env OMP_NUM_THREADS=${ST_THREADS_PER_SOCKET} OMP_PROC_BIND=spread \
./stream_openmp.exe
$ numactl --cpunodebind=1 --membind=1 \
env OMP_NUM_THREADS=${ST_THREADS_PER_SOCKET} OMP_PROC_BIND=spread \
./stream_openmp.exe
$ OMP_NUM_THREADS=${ST_THREADS} OMP_PROC_BIND=spread ./stream_openmp.exe
For SMT runs, repeat the socket-local and remote-memory commands with
OMP_NUM_THREADS=${SMT_THREADS_PER_SOCKET} and SMT enabled.
STREAM reports MB/s. Convert to GB/s by dividing by 1000 when you compare against the reference table.
Chip SKU |
Socket Scope |
Memory Scope |
Memory Capacity for Test Scope (GB) |
DRAM Speed (MT/s) |
Threads |
Triad Min GB/s |
|---|---|---|---|---|---|---|
Vera |
1 socket |
Socket-local memory |
768 |
9600 |
88 |
1045 |
Vera |
2 sockets |
Aggregate across sockets |
1536 |
9600 |
176 |
2088 |
Vera Rubin |
1 socket |
Socket-local memory |
720 |
7500 |
88 |
765 |
Vera Rubin |
2 sockets |
Aggregate across sockets |
1440 |
7500 |
176 |
1530 |
Memory bandwidth depends on many factors, such as the operating system kernel version, page size, memory capacity, CPU affinity, and NUMA memory placement. Reference measurements use dual-socket systems. Rows with a 1-socket scope bind execution and memory to one socket, so the capacity is half of the measured system capacity. Rows with a 2-socket scope use the aggregate result across both sockets and the full measured capacity. Vera Rubin capacity values use the usable host-memory capacity listed in the supported memory configurations.
Multichase#
Use Google Multichase to check unloaded local DRAM latency. Vera includes hardware prefetchers that are effective on some pointer-chasing access patterns, so use a large arena and the longer-chase option when measuring latency. A large arena, such as 1536 MB or 2 GB, is large enough to avoid most cache-capacity and simple prefetch artifacts.
Download and build Multichase:
$ git clone https://github.com/google/multichase.git
$ cd multichase
$ make
Run a local-memory latency check on node 0:
$ numactl --membind=0 taskset -c 0 \
./multichase -m 1536M -s 256 -t 1 -n 10 -L
The -L option uses a longer pointer chase, which improves coverage of the
large arena and reduces the chance of measuring a small effective working set
within the test region. Multichase is still useful as a health check and
comparison test, but interpret the result as benchmark-observed latency rather
than absolute static DRAM latency.
Chip SKU |
Socket Scope |
Memory Scope |
DRAM Speed (MT/s) |
Local Latency (ns) |
|---|---|---|---|---|
Vera |
1 socket |
Node-local memory |
9600 |
121-126 |
Vera Rubin |
1 socket |
Node-local memory |
7500 |
129-134 |
Use unloaded local latency to detect memory-frequency, idle-state, firmware, and NUMA placement problems.
Fused Multiply Add#
NVIDIA provides an open-source suite of benchmarking microkernels for Arm CPUs. To allow precise counts of instructions and exercise specific functional units, these kernels are written in assembly language. To measure the peak floating point capability of a core and check the CPU clock speed, use a Fused Multiply Add (FMA) kernel.
To measure achievable peak performance of a core, the
fp64_sve_pred_fmla kernel executes a known number of SVE predicated fused
multiply-add operations (FMLA). When combined with the perf tool, you can
measure the performance and active core clock speed.
Note
The current Arm Kernels implementation is being actively tuned to achieve the best performance on the Olympus core. Do not use it as a sanity check yet.
$ git clone https://github.com/NVIDIA/arm-kernels.git
$ cd arm-kernels
$ make
$ perf stat ./arithmetic/fp64_sve_pred_fmla.x
The benchmark score is reported in giga-operations per second (GOps/sec) near
the top of the benchmark output. The active CPU frequency is reported in the
perf output on the cycles line and after the # symbol.
Run |
Mode |
GOps/sec |
Active GHz |
|---|---|---|---|
Single hardware thread |
ST |
53.7 |
3.363 |
SMT sibling pair, summed |
SMT |
80.4 |
3.358 |
To collect the SMT sibling-pair value, run one copy of the FMA kernel on each
hardware thread of the same Olympus core and sum the two GOps/sec values.
C2C CPU-GPU Bandwidth#
On Vera Rubin systems, use NVIDIA nvbandwidth to measure CPU-GPU transfer bandwidth. nvbandwidth requires the NVIDIA driver and CUDA Toolkit.
Note
CUDA 13.4 or later is required for Rubin support. Before building nvbandwidth, confirm that the active CUDA Toolkit and NVIDIA driver meet the platform requirements.
Download, build, and run nvbandwidth:
$ git clone https://github.com/NVIDIA/nvbandwidth.git
$ cd nvbandwidth
$ nvidia-smi
$ nvcc --version
$ sudo apt-get update
$ sudo apt-get install -y libboost-program-options-dev
$ ./debian_install.sh
$ ./nvbandwidth -t 0
$ ./nvbandwidth -t 1
$ ./nvbandwidth -l
Note
Bandwidth numbers depend on the specific system configuration and are influenced by factors such as IOMMU settings, GPU clock settings, and other system-specific parameters. Consider these factors during bandwidth benchmarking.
For Vera Rubin health checks, run copy-engine tests first because they are the most direct signal for host-to-device and device-to-host bandwidth. Then run the streaming-multiprocessor copy tests if you need to separate copy-engine behavior from GPU-kernel copy behavior.