Is this page helpful?

Best Practices#

Debugging an application typically requires attaching a debugger to the user application and gathering as much data as possible up until the application crashes. The issue causing the application crash could be related to the system software, but debugging the process is the fastest way to understand where the crash is and if it can be resolved from the application side. The data gathered from the debugger can be used by NVIDIA to start debugging system software issues.

CUDA-GDB is the NVIDIA tool for debugging CUDA applications running on Linux and QNX. This enables developers to debug applications without the potential variations introduced by simulation and emulation environments. For details see https://developer.nvidia.com/cuda-gdb and https://docs.nvidia.com/cuda/cuda-gdb/index.html.

If the issue is related to performance, one can use a performance profiler, such as Nsight Systems or Nsight Compute, to understand the performance and bottlenecks in an application. Changes in system software can cause changes in application performance so even if the application did not change, profiling the application is an important step. Profiler data can highlight performance issues in the CPU, storage, or GPUs which help understand where the performance issue may be originating.

Collecting Node Metrics#

Node performance and operation can be deduced from system metrics. Gathering time series data of system metrics can allow administrators to detect when nodes are starting to operate incorrectly or be used for proactive system maintenance.

Metrics can be gathered both in-band with tools like the DCGM Prometheus plugin and out-of-band with IPMI. For details on DCGM Prometheus plugin, refer to https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/kube-prometheus.html.

A containerized version of DCGM is provided for Kubernetes deployments. For details, refer to Monitoring GPUs in Kubernetes with DCGM.

Key metrics that should be collected on GPU based systems include:

  • Power draw of each power supply

  • Power draw at the PDU

  • PDU Power factor

  • CPU Power, and CPU Temperature

  • GPU Power, GPU Temperature, GPU Memory Temperature and GPU clocks

  • NVLink metrics (A100 only)

  • Node Fan Speed

  • Inlet Temperature

Other metrics may be useful depending on specifics of the datacenter.

The data can be stored in a time-series or other NoSQL database and be recalled as needed. In addition, alerts can be created to detect conditions that may negatively affect system and application performance. These can include:

  • A rise in inlet temperature could be an indicator of an issue with datacenter cooling or other debris blocking airflow.

  • High GPU and GPU Memory Temperature will lead to GPU throttling and poor performance.

Catching Errors Before They Occur#

The goal of this document is to help administrators find the most common system issues and provide understanding or a quick path to resolution. However, it is best to catch system issues before they affect user jobs. The best time to do this is at the beginning of jobs via init-containers or prolog scripts or at the end of jobs with epilog scripts. Here are some suggestions on tests to include to help catch system errors before they affect user jobs.

GPU Specific Checks:

  • Confirm key pieces of software are loaded

  • Confirm specific GPU features of the nodes are correct (GPU Count, PCIe link speed, VBIOS, etc.)

  • Confirm driver persistence setting is correct. Driver persistence should be controlled through Persistence Daemon. (https://docs.nvidia.com/deploy/driver-persistence/index.html)

  • For NVSwitch based Systems:

    • Confirm the fabric manager is running

    • Confirm the NVLink fabric topology is correct

  • Check for any recent XID errors

  • Run dcgmi diag -r 2 for a quick check of system software and GPU functionality

Other non-GPU checks to include: