> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/fleet-intel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/fleet-intel/_mcp/server.
# Data Collection Rationale
This page documents the data that the NVIDIA Fleet Intelligence (NFI) in-band agent collects from enrolled machines, and why each item is collected.
The data is organized into two sections.
**Machine Metadata** is collected once per machine, or when it changes, to describe the node.
**Telemetry Metrics** is the recurring time-series data collected from NVIDIA Data Center GPU Manager (DCGM) and related sources.
## Machine Metadata
The following table lists the machine metadata that the agent collects.
| Type | Description | Example | Rationale |
| --------------- | ----------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| String | UUID for a specific instance of a node | eaad028b-c8fc-4090-abbc-99934aa6e02a | Uniquely identifies each GPU host. |
| String | Version of the container runtime on the node | containerd://1.7.27 | Ensures compatibility and helps troubleshoot workload failures or security issues caused by outdated runtimes. |
| String | Version of the gpuhealth agent on the node | 0.0.0 | Validates that the correct version of the agent is running and helps correlate bugs or behaviors to specific builds. |
| String | Current hostname of the node | host1 | Identifies the node in logs, dashboards, and cluster environments for correlation and troubleshooting. |
| String | Kernel version from `uname -r` | 5.15.0-144-generic | Kernel versions affect drivers, security posture, and GPU compatibility, so tracking them supports supportability and compliance. |
| String | Primary private IP address | 10.10.10.128 | Associates telemetry with a network identity and groups hosts on the same east-west network. |
| String | Public IP address | 1.1.1.1 | Used for external-access tracing, auditing, and identifying connectivity to the public internet. |
| String | OS type | linux | Confirms the platform type for compatibility checks and OS-specific configuration rules. |
| String | Specific image of the OS type | Ubuntu 22.04.5 LTS | Tracks the exact OS distribution and version to keep consistent baselines and detect drift or unsupported images. |
| String | Time at which the machine last booted | 2025-07-21T23:19:09+02:00 | Used to calculate uptime, detect frequent reboot patterns, and correlate issues with system restarts. |
| String | UUID for the machine, persisted across reboots | b3c2a1d4e5f6789012345678901234ab | Provides a stable identity for the physical or virtual machine, useful for lifecycle tracking and asset management. |
| String | Driver version for GPUs on the node | 535.154.05 | Ensures GPU driver compatibility and identifies mismatches that cause failures or degraded performance. |
| String | CUDA version installed on the node | 12.2 | Verifies that the CUDA toolchain aligns with frameworks and workloads to prevent runtime errors. |
| String | VBIOS version on the GPU | 94.02.71.00.B2 | Identifies the GPU firmware level to detect known firmware issues or required updates for security or performance. |
| Int | Total memory available on the node, in bytes | 68719476736 | Tracks hardware capacity for workload scheduling, performance baselines, and resource planning. |
| String | CPU architecture | amd64 | Determines instruction-set support and validates compatibility with agent binaries and workloads. |
| Int | Number of logical CPU cores | 8 | Used for scheduler decisions, performance tuning, and correlating CPU pressure with workload health. |
| String | CPU manufacturer | AuthenticAMD | Identifies the CPU vendor for optimization paths and firmware or driver compatibility. |
| String | CPU type | AMD EPYC-Genoa Processor | Provides a detailed hardware fingerprint needed for performance modeling and anomaly detection. |
| String | GPU product name | NVIDIA A100-SXM4-40GB | Identifies the GPU SKU for performance expectations, scheduling, and capacity planning. |
| String | GPU manufacturer | NVIDIA | Confirms the GPU vendor for driver, software-stack, and telemetry-decoding compatibility. |
| String | GPU architecture name | ampere | Determines supported GPU features, performance characteristics, and instruction-set generations. |
| String | GPU memory capacity | 40GB | Used for workload scheduling, GPU placement decisions, and troubleshooting out-of-memory events. |
| String | UUID for one GPU on the node | GPU-46a3bbe2-3e87-3dde-b464-a03eba0c21d7 | Uniquely identifies each GPU for telemetry correlation, health tracking, and lifecycle management. |
| String | Bus ID for one GPU on the node | 0000:0f:00.0 | Maps the GPU to the system topology for NUMA awareness and for diagnosing PCIe errors or miswiring. |
| Int | Minor ID for one GPU on the node | 1 | Used by device drivers and runtime libraries to reference GPU instances consistently. |
| Int | Board ID for one GPU on the node | 3 | Identifies board-level variants for debugging, warranty tracking, or vendor-specific hardware characteristics. |
| String | Serial number for one GPU on the node | 1234567890213 | Tracks the physical GPU asset for warranty, RMA workflows, and inventory systems. |
| String | Block device filesystem type | ext4 | Determines storage behavior and ensures compatibility with performance expectations and agent operations. |
| String | Mount point path for the block device | / | Identifies which filesystem is being monitored to track storage health and capacity risk. |
| String | Name of the block device | /dev/mapper/vg00-root | Provides a specific handle for storage telemetry, performance metrics, and failure analysis. |
| List of Strings | Parent devices | \["/dev/vda3"] | Shows storage layering such as LVM or RAID and helps diagnose underlying device failures or performance bottlenecks. |
| Boolean | Whether the disk is rotational (HDD versus SSD) | TRUE | Determines expected performance characteristics and helps detect misconfigured or degraded storage. |
| Int | Total size of the block device | 40261124096 | Tracks available storage capacity for scheduling, alerts, and resource planning. |
| String | Type of block device | lvm | Provides insight into the storage architecture to interpret telemetry and detect storage-specific health issues. |
| Int | Used space, in bytes | 34674585600 | Tracks disk consumption to trigger alerts, prevent outages, and understand workload footprint. |
| String | Serial number of the block device | drive-ua-disk-3 | Identifies the exact disk device for inventory records, replacement cycles, and failure correlation. |
| String | Vendor of the block device | QEMU | Indicates the virtualization or hardware vendor, which affects performance expectations and device behavior. |
| String | Specific model of the device | QEMU HARDDISK | Identifies the device class to understand performance capabilities and detect anomalies. |
| String | Device revision (version) | 2.5+ | Tracks the firmware level for diagnosing compatibility or stability problems. |
| String | Worldwide Name identifiers | 0x50014ee20c8f9876 | Provides a globally unique storage identity for SAN or virtual devices in multi-host environments. |
| String | UUID for a partition on the disk | 45bf634c-0b30-42e2-9b41-332e6316da7a | Identifies the specific partition being monitored for capacity and health tracking. |
| String | Interface name | eth0 | Identifies the network interface used for telemetry, routing diagnostics, and network-behavior analysis. |
| String | Interface IP address | 10.10.10.64 | Correlates traffic and events to a specific interface for debugging and security auditing. |
| String | Interface MAC address | 02:8c:1c:00:01:04 | Provides a unique network hardware identifier used for topology mapping and inventory tracking. |
## Telemetry Metrics
The following table lists the telemetry metrics that the agent collects.
| Granularity | Metric | Description | Rationale |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| GPU | dcgm\_fi\_dev\_sm\_clock | SM clock for the device. | Measures the Streaming Multiprocessor (SM) clock frequency of a GPU core in MHz. Use it to see whether SMs are running at expected boost or base clocks, and to spot performance issues, thermal or power throttling, or under-clock behavior. |
| GPU | dcgm\_fi\_dev\_mem\_clock | Memory clock for the device. | Represents the actual clock speed of the GPU memory subsystem. Differences from expected clocks can indicate differences between SKUs or dynamic adjustments, for example, due to thermal limits, power limits, or power-state changes. |
| GPU | dcgm\_fi\_dev\_clocks\_event\_reason\_hw\_therm\_slowdown\_ns | Throttling due to temperature being too high (reducing core clocks by a factor of 2 or more), in ns. | Helps separate temperature-driven throttling from other causes. If this value is high, you likely need better cooling, airflow, or lower power limits to keep performance stable. |
| GPU | dcgm\_fi\_dev\_clocks\_event\_reason\_hw\_power\_brake\_slowdown\_ns | Throttling due to an external power-brake assertion (reducing core clocks by a factor of 2 or more), in ns. | Identifies cases where the GPU is clamped by board or system power limits. High values suggest PSU constraints, aggressive power caps, or oversubscribed power rails. |
| CPU | dcgm\_fi\_dev\_cpu\_temp\_current | CPU temperature. | Extends the accelerator health view to include host CPUs. It is useful for ensuring overall node thermals are safe and for explaining throttling or instability on mixed CPU/GPU nodes. |
| GPU | dcgm\_fi\_dev\_fb\_total | Total frame buffer of the GPU, in MB. | Provides the raw VRAM size per GPU, used for scheduling workloads, validating that nodes have the expected SKU, and sizing maximum model or batch sizes. |
| GPU | dcgm\_fi\_dev\_fb\_free | Free frame buffer, in MB. | Complements used VRAM. Low free memory warns that new allocations or larger batches can fail, and it is useful for autoscaling and admission-control decisions. |
| GPU | dcgm\_fi\_dev\_fb\_used | Used frame buffer, in MB. | Makes VRAM consumption easy to visualize and alert on. It is often used to trigger scaling, rescheduling, or alerts when memory pressure crosses operational thresholds. |
| GPU | dcgm\_fi\_dev\_fb\_reserved | Reserved frame buffer, in MB. | Shows how much VRAM the system holds back for driver and bookkeeping purposes, explaining differences between raw and usable VRAM. |
| GPU | dcgm\_fi\_dev\_fb\_used\_percent | Percentage of frame buffer used: `Used/(Total - Reserved)`. Range 0.0-1.0. | Normalized VRAM usage that is easy to chart and alert on across mixed GPU types when DCGM is your primary telemetry source. |
| GPU | dcgm\_fi\_dev\_ecc\_sbe\_vol\_total | Total single-bit volatile ECC errors. | Short-term corrected error activity. Spikes help detect environmental or workload-triggered memory issues without being dominated by old history. |
| GPU | dcgm\_fi\_dev\_ecc\_dbe\_vol\_total | Total double-bit volatile ECC errors. | Recent uncorrectable ECC errors indicate the GPU memory has experienced multi-bit data corruption that the error-correction code could detect but not fix, pointing to a potential system instability. |
| GPU | dcgm\_fi\_dev\_ecc\_sbe\_agg\_total | Total single-bit aggregate (persistent) ECC errors. Note: monotonically increasing. | Long-term health indicator for GPU memory. Used to rank devices by error history and decide which ones to retire first. |
| GPU | dcgm\_fi\_dev\_ecc\_dbe\_agg\_total | Total double-bit aggregate (persistent) ECC errors. Note: monotonically increasing. | Tracks how many serious memory failures a GPU has experienced over its life. High counts suggest a history of instability and motivate more aggressive monitoring policies. |
| GPU | dcgm\_fi\_dev\_uncorrectable\_remapped\_rows | Number of remapped rows for uncorrectable errors. | DCGM's view of consumed spare rows. Used to confirm remap activity, inform health scores. |
| GPU | dcgm\_fi\_dev\_correctable\_remapped\_rows | Number of remapped rows for correctable errors. | Shows how often the GPU has preemptively repaired marginal cells. Growing values can indicate a part that is starting to degrade. |
| GPU | dcgm\_fi\_dev\_row\_remap\_failure | Whether remapping of rows has failed (0 = no failure, 1 = failure). | Failed remaps are a serious reliability concern and usually mean the GPU should be taken out of service. |
| GPU | dcgm\_fi\_dev\_row\_remap\_pending | Whether remapping of rows is pending (0 = no pending, 1 = pending). | Pending remaps signal that the GPU has detected problematic rows that have not yet been fully repaired. This is useful for early warning and maintenance planning. |
| GPU | dcgm\_fi\_dev\_nvlink\_bandwidth\_total | Total number of NVLink bandwidth counters for all lanes. | Helps quantify how heavily NVLink fabrics are exercised in multi-GPU jobs and validate that topologies are used as intended. |
| GPU | dcgm\_fi\_dev\_pcie\_replay\_counter | PCIe replay counter. | Replay events indicate transmission problems on the PCIe bus. Sustained growth suggests cabling, riser, or motherboard signal-integrity issues that can hurt performance. |
| GPU | dcgm\_fi\_dev\_power\_usage | Power usage for the device, in Watts. | This metric uses the same concept as other power-usage metrics, but is sourced directly from DCGM. Use it when you want a single authoritative power reading for policy and billing. |
| GPU | dcgm\_fi\_dev\_total\_energy\_consumption | Total energy consumption for the GPU, in mJ, since the driver was last reloaded. | Cumulative energy counters are useful for energy accounting, total cost of ownership (TCO) calculations, and understanding which workloads or tenants consume the most power over time. |
| GPU | dcgm\_fi\_dev\_enforced\_power\_limit | Effective power limit that the driver enforces after taking into account all limiters. | Verifies that cluster automation or firmware configured the intended power cap and helps correlate throttling behavior with configured limits. |
| GPU | dcgm\_fi\_prof\_gr\_engine\_active | Ratio of time the graphics engine is active. | Relevant for workloads using the graphics pipeline (rendering, visualization). Indicates whether GPUs are used for graphics versus pure compute. |
| GPU | dcgm\_fi\_prof\_sm\_active | Ratio of cycles an SM has at least one warp assigned. | Gives a profiling-grade view of how often SMs were busy during the sampling window, supporting detailed performance analysis. |
| GPU | dcgm\_fi\_prof\_sm\_occupancy | Ratio of warps resident on an SM relative to the theoretical maximum. | Provides fine-grained occupancy data from the profiler subsystem, complementing utilization metrics to explain kernel performance characteristics. |
| GPU | dcgm\_fi\_prof\_pipe\_tensor\_active | Ratio of cycles any tensor pipe is active. | Shows how much of the sampling period tensor cores were busy. This is critical for analyzing deep-learning kernels and verifying that libraries are using tensor paths. |
| GPU | dcgm\_fi\_prof\_dram\_active | Ratio of cycles the device memory interface is active. | Indicates how busy DRAM is. High DRAM activity with moderate SM activity suggests a memory-bandwidth-bound workload. |
| GPU | dcgm\_fi\_prof\_pipe\_fp64\_active | Ratio of cycles the FP64 pipe is active. | Profiling view of double-precision use. Used in performance studies of HPC workloads and to size clusters for FP64 demand. |
| GPU | dcgm\_fi\_prof\_pipe\_fp32\_active | Ratio of cycles the FP32 pipe is active. | Profiling-grade single-precision utilization. Helps performance engineers tune kernels and understand whether workloads are saturating FP32 units. |
| GPU | dcgm\_fi\_prof\_pipe\_fp16\_active | Ratio of cycles the FP16 pipe is active (excluding HMMA). | Profiling-grade half-precision utilization. Helps distinguish between tensor-core-driven FP16 and standard FP16 usage for mixed-precision workloads. |
| GPU | dcgm\_fi\_prof\_pcie\_tx\_bytes | Number of bytes of active PCIe transmit data from the GPU perspective. | Tracks GPU-to-host or GPU-to-device PCIe egress traffic. Used to identify PCIe bottlenecks and validate that data paths are sized correctly. |
| GPU | dcgm\_fi\_prof\_pcie\_rx\_bytes | Number of bytes of active PCIe receive data from the GPU perspective. | Complements PCIe TX bytes. Together they provide a full view of PCIe bandwidth usage for capacity planning and troubleshooting. |
| GPU | dcgm\_fi\_prof\_nvlink\_tx\_bytes | Total bytes of active NVLink transmit data, including header and payload. | Helps quantify directional NVLink usage and detect imbalances or hotspots in multi-GPU communication patterns (transmit bytes). |
| GPU | dcgm\_fi\_prof\_nvlink\_rx\_bytes | Total bytes of active NVLink receive data, including header and payload. | Complements NVLink TX bytes to give a full picture of NVLink bandwidth usage between GPUs (received bytes). |
| GPU | dcgm\_fi\_prof\_pipe\_tensor\_imma\_active | Ratio of cycles the tensor (IMMA) pipe is active (off the peak sustained elapsed cycles). | Detailed view of integer tensor-core usage, important for analyzing and tuning quantized inference workloads. |
| GPU | dcgm\_fi\_prof\_pipe\_tensor\_hmma\_active | Ratio of cycles the tensor (HMMA) pipe is active (off the peak sustained elapsed cycles). | Detailed view of half-precision tensor-core usage for FP16/BF16 training and inference performance analysis. |
| GPU | dcgm\_fi\_prof\_pipe\_tensor\_dfma\_active | Ratio of cycles the tensor (DFMA) pipe is active (off the peak sustained elapsed cycles). | Profiling-level utilization of double-precision tensor cores, relevant for advanced HPC workloads using FP64 tensor operations. |
| GPU | dcgm\_fi\_prof\_pipe\_int\_active | Ratio of cycles the integer pipe is active. | Gives a more granular picture of integer workload intensity than coarse utilization, helping optimize kernels that are heavy in address or integer math. |
| GPU | dcgm\_fi\_dev\_memory\_temp | Memory temperature for the device. | Memory temperature can differ from core temperature. Monitoring it helps avoid memory-specific reliability issues and explains ECC spikes or throttling. |
| GPU | dcgm\_fi\_dev\_gpu\_temp | Current temperature readings for the device, in degrees C. | Provides GPU temperature from DCGM for use in pipelines and ensures all accelerator thermals are available from a single source. |
| GPU | dcgm\_fi\_dev\_slowdown\_temp | Slowdown temperature for the device. | Provides the vendor-defined thermal threshold used by firmware. Operators can use this when designing data center cooling envelopes and alert levels. |
| GPU | dcgm\_fi\_dev\_gpu\_util | GPU utilization. | Canonical utilization metric for capacity planning and SLOs. Used to understand how busy GPUs are over time and to drive autoscaling or scheduling decisions. |
| GPU | dcgm\_fi\_dev\_mem\_copy\_util | Memory utilization. | High copy utilization with low SM utilization can indicate I/O-bound workloads or inefficient data-movement patterns that need optimization. |
| GPU | dcgm\_fi\_dev\_enc\_util | Encoder utilization. | Important for video processing and streaming workloads. Helps ensure encoder capacity is sized correctly and identify hotspots where encoders are saturated. |
| GPU | dcgm\_fi\_dev\_dec\_util | Decoder utilization. | Similar to encoder utilization, but for decode. High values show where ingest or preprocessing stages can be bottlenecked by decode throughput. |
| GPU, XID | dcgm\_fi\_dev\_xid\_errors | Count of XID errors in the last check period. | XID errors are severe GPU driver or hardware faults. Counting them is essential for detecting GPUs issues, bad drivers, or firmware issues that warrant investigation. |
| GPU, NVLINK | dcgm\_fi\_dev\_nvlink\_error\_dl\_crc | Cumulative count of CRC errors detected at the NVLink Data Link (DL) layer, indicating bit-level data corruption in transit across the NVLink fabric. | CRC errors point to data-integrity issues on NVLink links. High counts threaten correctness and performance. |
| GPU, NVLINK | dcgm\_fi\_dev\_nvlink\_error\_dl\_recovery | Cumulative count of recovery errors at the NVLink Data Link layer where the link had to execute a recovery procedure due to a fault condition. | Used with `dcgm_fi_dev_nvlink_error_dl_replay` to monitor the health and stability of high-speed GPU interconnects. High or increasing counts suggest issues requiring investigation, such as improper cable seating or hardware faults. |
| GPU, NVLINK | dcgm\_fi\_dev\_nvlink\_error\_dl\_replay | Cumulative count of replay errors at the NVLink Data Link layer where a transmitted packet was not acknowledged and had to be retransmitted. | Used with `dcgm_fi_dev_nvlink_error_dl_recovery` to monitor the health and stability of high-speed GPU interconnects. |
| CPU | dcgm\_fi\_dev\_cpu\_power\_util\_current | CPU power utilization. | Identifies how much power the host CPU is drawing in real time on a GPU node. This is important because GPU performance and stability are directly affected by CPU behavior in multi-accelerator servers. |
| CPU | dcgm\_fi\_dev\_cpu\_power\_limit | CPU power limit. | Identifies the power limit of the host CPU. When the current draw equals the limit, throttling can occur. |
| GPU, NVLINK | dcgm\_fi\_dev\_nvlink\_count\_link\_recovery\_successful\_events | Number of times the link went from Up to recovery, succeeded, and the link came back up. | Successful recovery events show that the link had to perform corrective action to recover from faults. Success means the errors were correctable. Persistent growth suggests a degrading cable, thermal changes, connector degradation, and so on. Growth should be monitored. |
| GPU, NVLINK | dcgm\_fi\_dev\_nvlink\_count\_link\_recovery\_failed\_events | Number of times the link went from Up to recovery, failed, and the link was declared down. | Failed recovery events show that the link performed corrective action to recover from faults, but the recovery was unsuccessful. This suggests a degrading cable, connector, or GPU NVLink. |
| GPU, NVLINK | dcgm\_fi\_dev\_nvlink\_count\_link\_recovery\_events | Number of times the link went from Up to recovery, regardless of the result. | Recovery events show NVLink issues. This is an overall count. |