Data Collection Rationale | NVIDIA Fleet Intelligence Documentation

This page documents the data that the NVIDIA Fleet Intelligence (NFI) in-band agent collects from enrolled machines, and why each item is collected. The data is organized into two sections. Machine Metadata is collected once per machine, or when it changes, to describe the node. Telemetry Metrics is the recurring time-series data collected from NVIDIA Data Center GPU Manager (DCGM) and related sources.

Machine Metadata

The following table lists the machine metadata that the agent collects.

Type	Description	Example	Rationale
String	UUID for a specific instance of a node	eaad028b-c8fc-4090-abbc-99934aa6e02a	Uniquely identifies each GPU host.
String	Version of the container runtime on the node	containerd://1.7.27	Ensures compatibility and helps troubleshoot workload failures or security issues caused by outdated runtimes.
String	Version of the gpuhealth agent on the node	0.0.0	Validates that the correct version of the agent is running and helps correlate bugs or behaviors to specific builds.
String	Current hostname of the node	host1	Identifies the node in logs, dashboards, and cluster environments for correlation and troubleshooting.
String	Kernel version from `uname -r`	5.15.0-144-generic	Kernel versions affect drivers, security posture, and GPU compatibility, so tracking them supports supportability and compliance.
String	Primary private IP address	10.10.10.128	Associates telemetry with a network identity and groups hosts on the same east-west network.
String	Public IP address	1.1.1.1	Used for external-access tracing, auditing, and identifying connectivity to the public internet.
String	OS type	linux	Confirms the platform type for compatibility checks and OS-specific configuration rules.
String	Specific image of the OS type	Ubuntu 22.04.5 LTS	Tracks the exact OS distribution and version to keep consistent baselines and detect drift or unsupported images.
String	Time at which the machine last booted	2025-07-21T23:19:09+02:00	Used to calculate uptime, detect frequent reboot patterns, and correlate issues with system restarts.
String	UUID for the machine, persisted across reboots	b3c2a1d4e5f6789012345678901234ab	Provides a stable identity for the physical or virtual machine, useful for lifecycle tracking and asset management.
String	Driver version for GPUs on the node	535.154.05	Ensures GPU driver compatibility and identifies mismatches that cause failures or degraded performance.
String	CUDA version installed on the node	12.2	Verifies that the CUDA toolchain aligns with frameworks and workloads to prevent runtime errors.
String	VBIOS version on the GPU	94.02.71.00.B2	Identifies the GPU firmware level to detect known firmware issues or required updates for security or performance.
Int	Total memory available on the node, in bytes	68719476736	Tracks hardware capacity for workload scheduling, performance baselines, and resource planning.
String	CPU architecture	amd64	Determines instruction-set support and validates compatibility with agent binaries and workloads.
Int	Number of logical CPU cores	8	Used for scheduler decisions, performance tuning, and correlating CPU pressure with workload health.
String	CPU manufacturer	AuthenticAMD	Identifies the CPU vendor for optimization paths and firmware or driver compatibility.
String	CPU type	AMD EPYC-Genoa Processor	Provides a detailed hardware fingerprint needed for performance modeling and anomaly detection.
String	GPU product name	NVIDIA A100-SXM4-40GB	Identifies the GPU SKU for performance expectations, scheduling, and capacity planning.
String	GPU manufacturer	NVIDIA	Confirms the GPU vendor for driver, software-stack, and telemetry-decoding compatibility.
String	GPU architecture name	ampere	Determines supported GPU features, performance characteristics, and instruction-set generations.
String	GPU memory capacity	40GB	Used for workload scheduling, GPU placement decisions, and troubleshooting out-of-memory events.
String	UUID for one GPU on the node	GPU-46a3bbe2-3e87-3dde-b464-a03eba0c21d7	Uniquely identifies each GPU for telemetry correlation, health tracking, and lifecycle management.
String	Bus ID for one GPU on the node	0000:0f:00.0	Maps the GPU to the system topology for NUMA awareness and for diagnosing PCIe errors or miswiring.
Int	Minor ID for one GPU on the node	1	Used by device drivers and runtime libraries to reference GPU instances consistently.
Int	Board ID for one GPU on the node	3	Identifies board-level variants for debugging, warranty tracking, or vendor-specific hardware characteristics.
String	Serial number for one GPU on the node	1234567890213	Tracks the physical GPU asset for warranty, RMA workflows, and inventory systems.
String	Block device filesystem type	ext4	Determines storage behavior and ensures compatibility with performance expectations and agent operations.
String	Mount point path for the block device	/	Identifies which filesystem is being monitored to track storage health and capacity risk.
String	Name of the block device	/dev/mapper/vg00-root	Provides a specific handle for storage telemetry, performance metrics, and failure analysis.
List of Strings	Parent devices	[“/dev/vda3”]	Shows storage layering such as LVM or RAID and helps diagnose underlying device failures or performance bottlenecks.
Boolean	Whether the disk is rotational (HDD versus SSD)	TRUE	Determines expected performance characteristics and helps detect misconfigured or degraded storage.
Int	Total size of the block device	40261124096	Tracks available storage capacity for scheduling, alerts, and resource planning.
String	Type of block device	lvm	Provides insight into the storage architecture to interpret telemetry and detect storage-specific health issues.
Int	Used space, in bytes	34674585600	Tracks disk consumption to trigger alerts, prevent outages, and understand workload footprint.
String	Serial number of the block device	drive-ua-disk-3	Identifies the exact disk device for inventory records, replacement cycles, and failure correlation.
String	Vendor of the block device	QEMU	Indicates the virtualization or hardware vendor, which affects performance expectations and device behavior.
String	Specific model of the device	QEMU HARDDISK	Identifies the device class to understand performance capabilities and detect anomalies.
String	Device revision (version)	2.5+	Tracks the firmware level for diagnosing compatibility or stability problems.
String	Worldwide Name identifiers	0x50014ee20c8f9876	Provides a globally unique storage identity for SAN or virtual devices in multi-host environments.
String	UUID for a partition on the disk	45bf634c-0b30-42e2-9b41-332e6316da7a	Identifies the specific partition being monitored for capacity and health tracking.
String	Interface name	eth0	Identifies the network interface used for telemetry, routing diagnostics, and network-behavior analysis.
String	Interface IP address	10.10.10.64	Correlates traffic and events to a specific interface for debugging and security auditing.
String	Interface MAC address	02:8c:1c:00:01:04	Provides a unique network hardware identifier used for topology mapping and inventory tracking.

Telemetry Metrics

The following table lists the telemetry metrics that the agent collects.

Granularity	Metric	Description	Rationale
GPU	dcgm_fi_dev_sm_clock	SM clock for the device.	Measures the Streaming Multiprocessor (SM) clock frequency of a GPU core in MHz. Use it to see whether SMs are running at expected boost or base clocks, and to spot performance issues, thermal or power throttling, or under-clock behavior.
GPU	dcgm_fi_dev_mem_clock	Memory clock for the device.	Represents the actual clock speed of the GPU memory subsystem. Differences from expected clocks can indicate differences between SKUs or dynamic adjustments, for example, due to thermal limits, power limits, or power-state changes.
GPU	dcgm_fi_dev_clocks_event_reason_hw_therm_slowdown_ns	Throttling due to temperature being too high (reducing core clocks by a factor of 2 or more), in ns.	Helps separate temperature-driven throttling from other causes. If this value is high, you likely need better cooling, airflow, or lower power limits to keep performance stable.
GPU	dcgm_fi_dev_clocks_event_reason_hw_power_brake_slowdown_ns	Throttling due to an external power-brake assertion (reducing core clocks by a factor of 2 or more), in ns.	Identifies cases where the GPU is clamped by board or system power limits. High values suggest PSU constraints, aggressive power caps, or oversubscribed power rails.
CPU	dcgm_fi_dev_cpu_temp_current	CPU temperature.	Extends the accelerator health view to include host CPUs. It is useful for ensuring overall node thermals are safe and for explaining throttling or instability on mixed CPU/GPU nodes.
GPU	dcgm_fi_dev_fb_total	Total frame buffer of the GPU, in MB.	Provides the raw VRAM size per GPU, used for scheduling workloads, validating that nodes have the expected SKU, and sizing maximum model or batch sizes.
GPU	dcgm_fi_dev_fb_free	Free frame buffer, in MB.	Complements used VRAM. Low free memory warns that new allocations or larger batches can fail, and it is useful for autoscaling and admission-control decisions.
GPU	dcgm_fi_dev_fb_used	Used frame buffer, in MB.	Makes VRAM consumption easy to visualize and alert on. It is often used to trigger scaling, rescheduling, or alerts when memory pressure crosses operational thresholds.
GPU	dcgm_fi_dev_fb_reserved	Reserved frame buffer, in MB.	Shows how much VRAM the system holds back for driver and bookkeeping purposes, explaining differences between raw and usable VRAM.
GPU	dcgm_fi_dev_fb_used_percent	Percentage of frame buffer used: `Used/(Total - Reserved)`. Range 0.0-1.0.	Normalized VRAM usage that is easy to chart and alert on across mixed GPU types when DCGM is your primary telemetry source.
GPU	dcgm_fi_dev_ecc_sbe_vol_total	Total single-bit volatile ECC errors.	Short-term corrected error activity. Spikes help detect environmental or workload-triggered memory issues without being dominated by old history.
GPU	dcgm_fi_dev_ecc_dbe_vol_total	Total double-bit volatile ECC errors.	Recent uncorrectable ECC errors indicate the GPU memory has experienced multi-bit data corruption that the error-correction code could detect but not fix, pointing to a potential system instability.
GPU	dcgm_fi_dev_ecc_sbe_agg_total	Total single-bit aggregate (persistent) ECC errors. Note: monotonically increasing.	Long-term health indicator for GPU memory. Used to rank devices by error history and decide which ones to retire first.
GPU	dcgm_fi_dev_ecc_dbe_agg_total	Total double-bit aggregate (persistent) ECC errors. Note: monotonically increasing.	Tracks how many serious memory failures a GPU has experienced over its life. High counts suggest a history of instability and motivate more aggressive monitoring policies.
GPU	dcgm_fi_dev_uncorrectable_remapped_rows	Number of remapped rows for uncorrectable errors.	DCGM’s view of consumed spare rows. Used to confirm remap activity, inform health scores.
GPU	dcgm_fi_dev_correctable_remapped_rows	Number of remapped rows for correctable errors.	Shows how often the GPU has preemptively repaired marginal cells. Growing values can indicate a part that is starting to degrade.
GPU	dcgm_fi_dev_row_remap_failure	Whether remapping of rows has failed (0 = no failure, 1 = failure).	Failed remaps are a serious reliability concern and usually mean the GPU should be taken out of service.
GPU	dcgm_fi_dev_row_remap_pending	Whether remapping of rows is pending (0 = no pending, 1 = pending).	Pending remaps signal that the GPU has detected problematic rows that have not yet been fully repaired. This is useful for early warning and maintenance planning.
GPU	dcgm_fi_dev_nvlink_bandwidth_total	Total number of NVLink bandwidth counters for all lanes.	Helps quantify how heavily NVLink fabrics are exercised in multi-GPU jobs and validate that topologies are used as intended.
GPU	dcgm_fi_dev_pcie_replay_counter	PCIe replay counter.	Replay events indicate transmission problems on the PCIe bus. Sustained growth suggests cabling, riser, or motherboard signal-integrity issues that can hurt performance.
GPU	dcgm_fi_dev_power_usage	Power usage for the device, in Watts.	This metric uses the same concept as other power-usage metrics, but is sourced directly from DCGM. Use it when you want a single authoritative power reading for policy and billing.
GPU	dcgm_fi_dev_total_energy_consumption	Total energy consumption for the GPU, in mJ, since the driver was last reloaded.	Cumulative energy counters are useful for energy accounting, total cost of ownership (TCO) calculations, and understanding which workloads or tenants consume the most power over time.
GPU	dcgm_fi_dev_enforced_power_limit	Effective power limit that the driver enforces after taking into account all limiters.	Verifies that cluster automation or firmware configured the intended power cap and helps correlate throttling behavior with configured limits.
GPU	dcgm_fi_prof_gr_engine_active	Ratio of time the graphics engine is active.	Relevant for workloads using the graphics pipeline (rendering, visualization). Indicates whether GPUs are used for graphics versus pure compute.
GPU	dcgm_fi_prof_sm_active	Ratio of cycles an SM has at least one warp assigned.	Gives a profiling-grade view of how often SMs were busy during the sampling window, supporting detailed performance analysis.
GPU	dcgm_fi_prof_sm_occupancy	Ratio of warps resident on an SM relative to the theoretical maximum.	Provides fine-grained occupancy data from the profiler subsystem, complementing utilization metrics to explain kernel performance characteristics.
GPU	dcgm_fi_prof_pipe_tensor_active	Ratio of cycles any tensor pipe is active.	Shows how much of the sampling period tensor cores were busy. This is critical for analyzing deep-learning kernels and verifying that libraries are using tensor paths.
GPU	dcgm_fi_prof_dram_active	Ratio of cycles the device memory interface is active.	Indicates how busy DRAM is. High DRAM activity with moderate SM activity suggests a memory-bandwidth-bound workload.
GPU	dcgm_fi_prof_pipe_fp64_active	Ratio of cycles the FP64 pipe is active.	Profiling view of double-precision use. Used in performance studies of HPC workloads and to size clusters for FP64 demand.
GPU	dcgm_fi_prof_pipe_fp32_active	Ratio of cycles the FP32 pipe is active.	Profiling-grade single-precision utilization. Helps performance engineers tune kernels and understand whether workloads are saturating FP32 units.
GPU	dcgm_fi_prof_pipe_fp16_active	Ratio of cycles the FP16 pipe is active (excluding HMMA).	Profiling-grade half-precision utilization. Helps distinguish between tensor-core-driven FP16 and standard FP16 usage for mixed-precision workloads.
GPU	dcgm_fi_prof_pcie_tx_bytes	Number of bytes of active PCIe transmit data from the GPU perspective.	Tracks GPU-to-host or GPU-to-device PCIe egress traffic. Used to identify PCIe bottlenecks and validate that data paths are sized correctly.
GPU	dcgm_fi_prof_pcie_rx_bytes	Number of bytes of active PCIe receive data from the GPU perspective.	Complements PCIe TX bytes. Together they provide a full view of PCIe bandwidth usage for capacity planning and troubleshooting.
GPU	dcgm_fi_prof_nvlink_tx_bytes	Total bytes of active NVLink transmit data, including header and payload.	Helps quantify directional NVLink usage and detect imbalances or hotspots in multi-GPU communication patterns (transmit bytes).
GPU	dcgm_fi_prof_nvlink_rx_bytes	Total bytes of active NVLink receive data, including header and payload.	Complements NVLink TX bytes to give a full picture of NVLink bandwidth usage between GPUs (received bytes).
GPU	dcgm_fi_prof_pipe_tensor_imma_active	Ratio of cycles the tensor (IMMA) pipe is active (off the peak sustained elapsed cycles).	Detailed view of integer tensor-core usage, important for analyzing and tuning quantized inference workloads.
GPU	dcgm_fi_prof_pipe_tensor_hmma_active	Ratio of cycles the tensor (HMMA) pipe is active (off the peak sustained elapsed cycles).	Detailed view of half-precision tensor-core usage for FP16/BF16 training and inference performance analysis.
GPU	dcgm_fi_prof_pipe_tensor_dfma_active	Ratio of cycles the tensor (DFMA) pipe is active (off the peak sustained elapsed cycles).	Profiling-level utilization of double-precision tensor cores, relevant for advanced HPC workloads using FP64 tensor operations.
GPU	dcgm_fi_prof_pipe_int_active	Ratio of cycles the integer pipe is active.	Gives a more granular picture of integer workload intensity than coarse utilization, helping optimize kernels that are heavy in address or integer math.
GPU	dcgm_fi_dev_memory_temp	Memory temperature for the device.	Memory temperature can differ from core temperature. Monitoring it helps avoid memory-specific reliability issues and explains ECC spikes or throttling.
GPU	dcgm_fi_dev_gpu_temp	Current temperature readings for the device, in degrees C.	Provides GPU temperature from DCGM for use in pipelines and ensures all accelerator thermals are available from a single source.
GPU	dcgm_fi_dev_slowdown_temp	Slowdown temperature for the device.	Provides the vendor-defined thermal threshold used by firmware. Operators can use this when designing data center cooling envelopes and alert levels.
GPU	dcgm_fi_dev_gpu_util	GPU utilization.	Canonical utilization metric for capacity planning and SLOs. Used to understand how busy GPUs are over time and to drive autoscaling or scheduling decisions.
GPU	dcgm_fi_dev_mem_copy_util	Memory utilization.	High copy utilization with low SM utilization can indicate I/O-bound workloads or inefficient data-movement patterns that need optimization.
GPU	dcgm_fi_dev_enc_util	Encoder utilization.	Important for video processing and streaming workloads. Helps ensure encoder capacity is sized correctly and identify hotspots where encoders are saturated.
GPU	dcgm_fi_dev_dec_util	Decoder utilization.	Similar to encoder utilization, but for decode. High values show where ingest or preprocessing stages can be bottlenecked by decode throughput.
GPU, XID	dcgm_fi_dev_xid_errors	Count of XID errors in the last check period.	XID errors are severe GPU driver or hardware faults. Counting them is essential for detecting GPUs issues, bad drivers, or firmware issues that warrant investigation.
GPU, NVLINK	dcgm_fi_dev_nvlink_error_dl_crc	Cumulative count of CRC errors detected at the NVLink Data Link (DL) layer, indicating bit-level data corruption in transit across the NVLink fabric.	CRC errors point to data-integrity issues on NVLink links. High counts threaten correctness and performance.
GPU, NVLINK	dcgm_fi_dev_nvlink_error_dl_recovery	Cumulative count of recovery errors at the NVLink Data Link layer where the link had to execute a recovery procedure due to a fault condition.	Used with `dcgm_fi_dev_nvlink_error_dl_replay` to monitor the health and stability of high-speed GPU interconnects. High or increasing counts suggest issues requiring investigation, such as improper cable seating or hardware faults.
GPU, NVLINK	dcgm_fi_dev_nvlink_error_dl_replay	Cumulative count of replay errors at the NVLink Data Link layer where a transmitted packet was not acknowledged and had to be retransmitted.	Used with `dcgm_fi_dev_nvlink_error_dl_recovery` to monitor the health and stability of high-speed GPU interconnects.
CPU	dcgm_fi_dev_cpu_power_util_current	CPU power utilization.	Identifies how much power the host CPU is drawing in real time on a GPU node. This is important because GPU performance and stability are directly affected by CPU behavior in multi-accelerator servers.
CPU	dcgm_fi_dev_cpu_power_limit	CPU power limit.	Identifies the power limit of the host CPU. When the current draw equals the limit, throttling can occur.
GPU, NVLINK	dcgm_fi_dev_nvlink_count_link_recovery_successful_events	Number of times the link went from Up to recovery, succeeded, and the link came back up.	Successful recovery events show that the link had to perform corrective action to recover from faults. Success means the errors were correctable. Persistent growth suggests a degrading cable, thermal changes, connector degradation, and so on. Growth should be monitored.
GPU, NVLINK	dcgm_fi_dev_nvlink_count_link_recovery_failed_events	Number of times the link went from Up to recovery, failed, and the link was declared down.	Failed recovery events show that the link performed corrective action to recover from faults, but the recovery was unsuccessful. This suggests a degrading cable, connector, or GPU NVLink.
GPU, NVLINK	dcgm_fi_dev_nvlink_count_link_recovery_events	Number of times the link went from Up to recovery, regardless of the result.	Recovery events show NVLink issues. This is an overall count.