DocumentationThis guide shows you how to collect GPU metrics (power, utilization, memory, temperature, etc.) during AIPerf benchmarking. GPU telemetry provides insights into GPU performance and resource usage while running inference workloads.
This guide covers three setup paths depending on your inference backend and requirements:
If you’re using Dynamo, it comes with DCGM pre-configured on port 9401. No additional setup needed! Just use the --gpu-telemetry flag to enable console display and optionally add additional DCGM url endpoints. URLs can be specified with or without the http:// prefix (e.g., localhost:9400 or http://localhost:9400).
If you’re using any other inference backend, you’ll need to set up DCGM separately.
If you want simple local GPU monitoring without DCGM, use --gpu-telemetry pynvml. This uses NVIDIA’s nvidia-ml-py Python library (commonly known as pynvml) to collect metrics directly from the GPU driver. No HTTP endpoints or additional containers required.
If you’re benchmarking against an inference server running on AMD ROCm GPUs (Instinct MI300X, MI355X, etc.), use --gpu-telemetry amdsmi. This uses the amdsmi Python bindings shipped with ROCm to collect metrics directly from the AMD driver. No HTTP endpoints required. Install the bindings via pip install /opt/rocm/share/amd_smi/amdsmi-*.whl if not already present (they ship with ROCm).
AIPerf provides GPU telemetry collection with the --gpu-telemetry flag. Here’s how it works:
--gpu-telemetry Flag WorksDCGM mode (default): The default endpoints http://localhost:9400/metrics and http://localhost:9401/metrics are always attempted for telemetry collection, regardless of whether the --gpu-telemetry flag is used. The flag primarily controls whether metrics are displayed on the console and allows you to specify additional custom DCGM exporter endpoints.
pynvml mode: When using --gpu-telemetry pynvml, DCGM endpoints are NOT used. Metrics are collected directly from local GPUs via the nvidia-ml-py library.
amdsmi mode: When using --gpu-telemetry amdsmi, DCGM endpoints are NOT used. Metrics are collected directly from local AMD GPUs via the amdsmi library and emitted under vendor-namespaced amd_* field names (amd_power, amd_gfx_activity, amd_temperature, etc.) rather than NVML-shaped names. On Instinct datacenter parts amd_mm_activity is generally absent (sensor returns 'N/A'); amd_throttle_status is a 0.0/1.0 snapshot per scrape (amdsmi exposes a boolean state, not a duration counter).
To completely disable GPU telemetry collection, use --no-gpu-telemetry.
When specifying custom DCGM exporter URLs, the http:// prefix is optional. URLs like localhost:9400 will automatically be treated as http://localhost:9400. Both formats work identically.
For simple local GPU monitoring without DCGM setup, use --gpu-telemetry pynvml. This collects metrics directly from the NVIDIA driver using the nvidia-ml-py library. See Path 3: pynvml for details.
Adding dashboard to the --gpu-telemetry flag enables a live terminal UI (TUI) that displays GPU metrics in real-time during your benchmark runs:
Dynamo includes DCGM out of the box on port 9401 - no extra setup needed!
Sample Output (Successful Run):
The GPU telemetry table displays real-time metrics collected from DCGM during the benchmark. Each GPU is shown with its metrics aggregated across the benchmark duration.
The dashboard keyword enables a live terminal UI for real-time GPU telemetry visualization. Press 5 to maximize the GPU Telemetry panel during the benchmark run.
This path works with vLLM, SGLang, TRT-LLM, or any inference server. We’ll use vLLM as an example.
The setup includes three steps: creating a custom metrics configuration, starting the DCGM Exporter, and launching the vLLM server.
You can customize the custom_gpu_metrics.csv file by commenting out metrics you don’t need. Lines starting with # are ignored.
Key Configuration:
-p 9401:9400 - Forward container’s port 9400 to host’s port 9401 (AIPerf’s default)-e DCGM_EXPORTER_INTERVAL=33 - Collect metrics every 33ms for fine-grained profiling-v custom_gpu_metrics.csv:... - Mount your custom metrics configurationReplace the vLLM command above with your preferred backend (SGLang, TRT-LLM, etc.). The DCGM setup works with any server.
The dashboard keyword enables a live terminal UI for real-time GPU telemetry visualization. Press 5 to maximize the GPU Telemetry panel during the benchmark run.
For simple local GPU monitoring without DCGM infrastructure, AIPerf supports direct GPU metrics collection using NVIDIA’s nvidia-ml-py Python library (commonly known as pynvml). This approach requires no additional containers, HTTP endpoints, or DCGM setup.
pip install nvidia-ml-pydashboard after pynvml for the real-time terminal UI: --gpu-telemetry pynvml dashboardThe nvidia-ml-py library (pynvml) collects the following metrics directly from the NVIDIA driver:
Not all metrics are available on all GPU models. AIPerf gracefully handles missing metrics and reports only what the hardware supports.
For inference workloads on AMD Instinct GPUs (MI300X, MI355X, etc.), use --gpu-telemetry amdsmi. This collects metrics directly from local AMD GPUs via the amdsmi Python library shipped with ROCm.
AMD signals are emitted under their own vendor-namespaced field names (not aliased onto NVML-shaped names) because the underlying sensors do not always measure the same physical quantity (e.g. amdsmi gfx_activity and NVML sm_utilization sample at different scopes).
For distributed setups with multiple nodes, you can collect GPU telemetry from all nodes simultaneously:
This will collect GPU metrics from:
http://localhost:9400/metrics (default, always attempted)http://localhost:9401/metrics (default, always attempted)http://node1:9400 (custom node 1, normalized from node1:9400)http://node2:9400 (custom node 2, normalized from node2:9400)http://node3:9400/metrics (custom node 3)All metrics are displayed on the console and saved to the output CSV and JSON files, with GPU indices and hostnames distinguishing metrics from different nodes.
You can customize which GPU metrics are displayed in AIPerf by creating a custom metrics CSV file and passing it to --gpu-telemetry:
The CSV format is identical to DCGM exporter configuration. See the vLLM setup section above (Step 1: Create a custom metrics configuration) for the complete CSV format example with all available DCGM fields.
Behavior: Custom metrics extend (not replace) the 7 core default metrics:
The file path can be absolute or relative. Use .csv extension so AIPerf can distinguish it from DCGM endpoint URLs.
You can start with the example CSV from the vLLM setup section and customize it by commenting out metrics you don’t need or adding new DCGM metrics.