NVIDIA Mission Control DGX GB200 Measurables | NVIDIA Base Command Manager

Measurables are metrics, health checks, and enummetrics (appendix G of the Administrator Manual).

This section describes the extra DGX GB200 measurables and their parameters that may not have been described in appendix G of the Administrator Manual. The DGX GB200 measurables are grouped in this chapter according to the following themes:

circuit-related metrics (section 12.1)
leak detection measurables (section 12.2)
NVLink measurables (section 12.3)
power shelf measurables (section 12.4)
cooling distribution unit metrics (section 12.5)
GPU measurables (section 12.6)
Prometheus metrics (section 12.7)
Redfish metrics (section 12.8)
Redfish enums (section 12.9)
health checks (section 12.10)

Electrical power in the data center is supplied to the DGX GB200 racks from a circuit that is typically from a remote power panel (RPP) or an overhead busway.

Circuit-related metrics for the DGX GB200 platform are shown in table 12.1.

Table 12.1: Circuit-related DGX GB200 metrics
DGX GB200 Circuit Metric	Parameter	Description
`CircuitCurrent`
`CircuitPhaseCurrent`	phase=1
`CircuitPhaseCurrent`	phase=2
`CircuitPhaseCurrent`	phase=3
`CircuitPower`
`RackCircuitCurrent`^*	circuit=RPP-B12-3
`RackCircuitCurrentLimit`^*	circuit=RPP-B12-3
`RackCircuitPhaseCurrent`^,*	circuit=RPP-B12-3;phase=3
`RackCircuitPower`^*	circuit=RPP-B12-3
`TotalCircuitCurrent`		Total
`TotalCircuitCurrentLimit`		Total
`TotalCircuitPhaseCurrent`		Total
`TotalCircuitPhaseCurrent`	phase=1	Total
`TotalCircuitPhaseCurrent`	phase=2	Total
`TotalCircuitPhaseCurrent`	phase=3	Total
`TotalCircuitPower`		Total

* The parameter set here depends on the listed power circuit name (section 4.8)

** The phase value can be 1, 2, or 3

12.2 Leak Detection DGX GB200 Measurables

Liquid cooling is used in the DGX GB200. The ability to detect leaking coolant is an important enough concern that there are several measurables associated with it.

Leak detection measurables for the DGX GB200 platform are shown in table 12.2:

Table 12.2: Leak detection DGX GB200 measurables
DGX GB200 leak measurable	Parameter	Description
`LeakResponseRackElectricalIsolationStatus`
`LeakResponseRackLiquidIsolationStatus`
`LeakSensorFaultRack`
`DevicesWithLeaks`	`Total`	Number of devices that have detected a leak
`RF_Chassis_0_LeakDetector_0_ColdPlate` ^*
`RF_Chassis_0_LeakDetector_0_Manifold` ^*
`RF_Chassis_0_LeakDetector_0_ColdPlate` ^*	`state`	Enum, not metric
`RF_Chassis_0_LeakDetector_0_Manifold` ^*	`state`	Enum, not metric
`RF_LD_LeakDetection`		Enum
`RF_LeakDetector`	`Chassis_0_LeakDetector_0_ColdPlate` ^*	Enum
`RF_LeakDetector`	`Chassis_0_LeakDetector_0_Manifold` ^*	Enum

* The substring LeakDetector_0 can also take the value LeakDetector_1

12.3 DGX GB200 NVLink Measurables

NVLink-related measurables for the DGX GB200 platform are shown in table 12.3.

The measurables without the RF_ prefix are provided by the ClusterTotal data producer. The measurables with the RF_ prefix are provided via Redfish.

BCM picks up the Redfish measurables and their descriptions automatically. Many measurables still have no associated descriptions. For now, the names of the measurables can be a guide on what they are about.

Table 12.3: DGX GB200 NVLink measurables
DGX GB200 NVLink measurables	Parameter	Description
`NVLinkSwitchesClosed`		Number of NVLink switches not marked as UP or DOWN
`NVLinkSwitchesDown`		Number of NVLink switches marked as DOWN
`NVLinkSwitchesTotal`		Total number of NVLink switches
`NVLinkSwitchesUp`		Number of NVLink switches marked as UP
`RF_NVLink_Port_Nvidia_BitErrorRate`^*
`RF_NVLink_Port_Nvidia_LinkDowned`^*
`RF_NVLink_Port_Nvidia_RXNoProtocol`^*
`RF_NVLink_Port_Nvidia_TXNoProtocol`^*
`RF_NVLink_Port_Nvidia_TXWait`^*
`RF_NVLink_Port_RX`^*
`RF_NVLink_Port_RXFrames_Networking`^*
`RF_NVLink_Port_RX_Total`
`RF_NVLink_Port_TX`^*
`RF_NVLink_Port_TXFrames_Networking`^*
`RF_NVLink_Port_TX_Total`
`RF_NVLink_ResourceHealthRollup_Status`
`RF_NVLink_ResourceHealth_Status`
`RF_NVLink_ResourceState_Status`^*
`RF_NVLink_Resource_CurrentSpeed`^*
`RF_NVLink_Resource_LinkState`^*
`RF_NVLink_Resource_LinkStatus`^*
`RF_NVLink_Resource_MaxSpeed`^*
`RF_NVLink_Resource_Nvidia_RXWidth`^*
`RF_NVLink_Resource_Nvidia_TXWidth`^*

* The parameter here is one of 17 ports from the node to the switch.

It can take a value of acp0 to acp17

12.4 DGX GB200 Power Shelf Measurables

The list of power shelf measurables on a DGX GB200 platform are displayed in table 12.4.

These are produced by the redfish_power_shelf, ClusterTotal, and AggregatePowerShelf data producers.

Table 12.4: Power shelf measurables
Power Shelf Measurable	Parameter	Description
`AveragePowerShelfTemperature`^*		Average power shelf temperature
`PowerShelvesClosed`^**		Number of power shelves not marked as UP or DOWN
`PowerShelvesDown`^**		Number of power shelves marked as DOWN
`PowerShelvesTotal`^**		Total number of power shelves
`PowerShelvesUp`^**		Number of power shelves marked as UP
`RF_Active_PSU`		Number of PSUs of power shelf active
`RF_Health_PSU`[‡]		Are all PSUs of power shelf active?
`RF_PowerShelf_Status`[†]	1	Status of power shelf PSU 1
`RF_PowerShelf_Status`	total	Status of power shelf PSU, total
`RF_Power_Supply_Energy`[†]	1	Energy consumption of PSU 1 since boot
`RF_Power_Supply_Energy`	total	Energy consumption of all PSUs of power shelf since boot, total
`RF_Power_Supply_FanSpeed`[†]	1	Fan speed of PSU 1
`RF_Power_Supply_FanSpeed`	average	Fan speed of all PSUs of power shelf, averaged
`RF_Power_Supply_Id`[†]	1
`RF_Power_Supply_InputCurrent`[†]	1	Input current for PSU 1
`RF_Power_Supply_InputCurrent`	total	Input current for all PSUs of power shelf, total
`RF_Power_Supply_InputPower`[†]	1	Input power for PSU 1
`RF_Power_Supply_InputPower`	total	Input power for all PSUs of power shelf, total
`RF_Power_Supply_InputVoltage`[†]	1	Input voltage for PSU 1
`RF_Power_Supply_LifetimeReading_Energy`[†]	1	Input lifetime reading for energy for PSU 1
`RF_Power_Supply_Name`[†]	1
`RF_Power_Supply_OutputPower`[†]	1	Output power for PSU 1
`RF_Power_Supply_OutputPower`	total	Output power for all PSUs of power shelf, total
`RF_Power_Supply_PowerFactor_Input`[†]	1
`RF_Power_Supply_RailCurrent`[†]	1	Rail Current for PSU 1
`RF_Power_Supply_RailCurrent`	total	Rail Current for all PSUs of power shelf, total
`RF_Power_Supply_RailPower`[†]	1	Rail power for PSU 1
`RF_Power_Supply_RailPower`	total	Rail power for all PSUs of power shelf, total
`RF_Power_Supply_RailVoltage`[†]	1	Rail voltage for PSUs 1
`RF_Power_Supply_Temperature`[†]	1	Temperature for PSU 1
`RF_Power_Supply_Temperature`	average	Temperature for all PSUs of power shelf, averaged
`RF_Power_Supply_code_error`[†]	1	Code error for PSU 1
`RF_Power_Supply_message_error`[†]	1	Message error for PSU 1
`RF_Summary_Metrics_Id`
`RF_Summary_Metrics_Name`
`RF_Summary_Metrics_Power`		Power consumption (Watts)
`RF_Summary_Metrics_Temperature`		Temperature (Celsius)
`RF_Total_PSU`
`TotalPowerShelfActivePSU`^*		Total power shelf PSUs active
`TotalPowerShelfAvailablePSU`^*		Total power shelf PSUs available
`TotalPowerShelfCriticalPSU`[‡]		Are all active PSUs below critical threshold?
`TotalPowerShelfDegradedPSU`[‡]		Are all active PSUs below degraded threshold?
`TotalPowerShelfHealthyPSU`[‡]		Are all PSUs active?
`TotalPowerShelfInputCurrent`^*		Total power shelf input current
`TotalPowerShelfInputPower`^*		Total power shelf input power
`TotalPowerShelfRailCurrent`^*		Total power shelf rail current
`TotalPowerShelfRailPower`^*		Total power shelf rail power
`TotalPowerShelfPSU`^*		Total power shelf PSU

* From AggregatePowerShelf data producer

** From ClusterTotal data producer

† Takes the PSU number as a parameter. On the DGX GB200 it can be from 1 to 6

‡ Health check

12.5 Cooling Distribution Unit Metrics On The DGX GB200

Metrics for the cooling distribution units (CDUs) of the DGX GB200 platform are shown in table 12.5. The data producers for these are:

MonitoringSystem for the metrics in the table beginning with the string CDU
AggregateCDU for the metrics in the table that begin with the strings Average or Total

Table 12.5: DGX GB200 metrics for CDUs
DGX GB200 CDU Metric	Description
`AverageCDULiquidDifferentialPressure`	Average CDU liquid cooling differential pressure (Pa)
`AverageCDULiquidFlow`	Average CDU liquid cooling flow (LPM)
`AverageCDULiquidReturnTemperature`	Average CDU liquid cooling return temperature (°C)
`AverageCDULiquidSupplyTemperature`	Average CDU liquid cooling supply temperature (°C)
`AverageCDULiquidSystemPressure`	Average CDU liquid system pressure (Pa)
`CDUAvailable`	CDU available
`CDULiquidDifferentialPressure`	CDU liquid differential pressure (Pa)
`CDULiquidFlow`	CDU liquid flow (LPM)
`CDULiquidReturnTemperature`	CDU liquid return temperature (°C)
`CDULiquidSupplyTemperature`	CDU liquid supply temperature (°C)
`CDULiquidSystemPressure`	CDU liquid system pressure (Pa)
`CDUStatus`	CDU status (l/s)
`TotalCDUAvailable`	Total CDU available
`TotalCDUStatus`	Total CDU status

12.6 GPU Measurables For The DGX GB200

Some basic GPU measurables are covered in appendix G of the Administrator Manual.

Extra GPU measurables available on the DGX GB200 platform are:

Table 12.6:
GPU Measurables For The DGX GB200	Parameter	Description
`gpu_board_limit_violation`^*	gpu0	Board violation limit
`gpu_board_limit_violation`	total	Throttling duration due to board limits
`gpu_c2c_link_bandwidth`^*	gpu0	GPU C2C link bandwidth
`gpu_c2c_link_count`^*	gpu0	GPU C2C link count
`gpu_c2c_link_status`^*	gpu0	GPU C2C link status
`gpu_correctable_remapped_rows`^*	gpu0	Number of remapped rows for correctable errors
`gpu_dec_utilization`	average	Average GPU decoder utilization
`gpu_dec_utilization`	gpu0	GPU decoding usage
`gpu_ecc_dbe_agg`^*	gpu0	Total double bit aggregate ECC errors
`gpu_ecc_dbe_vol`^*	gpu0	Total double bit volatile ECC errors
`gpu_ecc_sbe_agg`^*	gpu0	Total single bit aggregate ECC errors
`gpu_ecc_sbe_vol`^*	gpu0	Total single bit volatile ECC errors
`gpu_enc_utilization`	average	Average GPU encoder utilization
`gpu_enc_utilization`^*	gpu0	GPU encoding usage
`gpu_enforced_power_limit`^*	gpu0	GPU enforced power limit
`gpu_enforced_power_profile_mask`^*	gpu0	Enforced workload power profile mask
`gpu_fabric_status`^*	gpu0	Status of the nvlink link domain (Enum)
`gpu_fan_speed`^*	gpu0	GPU fan speed percentage
`gpu_hbm_memory_temperature`^*	gpu0	High bandwidth memory (GPU memory) temperature
`gpu_health_driver`^,*	gpu0	Driver-related status
`gpu_health_hostengine`^**		Host engine status
`gpu_health_inforom`^,*	gpu0	Inforom status
`gpu_health_mcu`^,*	gpu0	Microcontroller unit status
`gpu_health_mem`^,*	gpu0	Memory status
`gpu_health_nvlink`^,*	gpu0	NVLINK system status
`gpu_health_nvswitch_fatal`^,*	gpu0	NV switch fatal errors
`gpu_health_nvswitch_non_fatal`^,*	gpu0	NV switch non fatal errors
`gpu_health_overall`^**		Overall system status
`gpu_health_overall`^,*	gpu0	Overall system status for the specified GPU
`gpu_health_pcie`^,*	gpu0	PCIe system status
`gpu_health_pmu`^,*	gpu0	Power management unit status
`gpu_health_power`^,*	gpu0	Power status
`gpu_health_sm`^,*	gpu0	Streaming multiprocessor status
`gpu_health_thermal`^,*	gpu0	Temperature status
`gpu_low_util_violation`^*	gpu0	Low utilization violation limit
`gpu_low_util_violation`	total	Throttling duration due to low utilization limits
`gpu_mem_clock`^*	gpu0	GPU memory clock
`gpu_mem_copy_utilization`	average	Average GPU memory copy utilization
`gpu_mem_copy_utilization`^*	gpu0	Percentage of the GPU memory copy used
`gpu_mem_free`^*	gpu0	Amount of GPU free memory
`gpu_mem_total`^*	gpu0	GPU framebuffer size
`gpu_mem_total`	total	combined GPU total memory
`gpu_mem_used`^*	gpu0	Amount of GPU used memory
`gpu_mem_used`	total	combined GPU memory usage
`gpu_mem_utilization`	average	Average GPU memory utilization
`gpu_mem_utilization`^*	gpu0	Percentage of GPU memory used
`gpu_memory_temp`	average	Average GPU memory temperature
`gpu_memory_temp`^*	gpu0	GPU memory temperature
`gpu_nvlink_crc_data_errors`^*	gpu0	Total nvlink data CRC errors over all lanes
`gpu_nvlink_crc_flit_errors`^*	gpu0	Total nvlink flit CRC errors over all lanes
`gpu_nvlink_total_bandwidth`^*	gpu0	Total nvlink bandwidth used
`gpu_nvlink_total_bandwidth`	total	combined GPU nvlink bandwidth
`gpu_performance_state`^*	gpu0	GPU performance state (0=highest)
`gpu_power_management_limit`^*	gpu0	GPU power management limit
`gpu_power_usage`^*	gpu0	GPU power usage
`gpu_power_usage`	total	combined GPU power usage
`gpu_power_usage_instant`^*	gpu0	GPU power usage at this instant
`gpu_power_usage_instant`	total	combined GPU power usage at this instant
`gpu_power_violation`^*	gpu0	Throttling duration due to power constraints
`gpu_power_violation`	total	Throttling duration due to power constraints
`gpu_reliability_violation`^*	gpu0	Reliability violation limit
`gpu_reliability_violation`	total	Throttling duration due to reliability limits
`gpu_requested_power_profile_mask`^*	gpu0	Requested workload power profile mask
`gpu_row_remap_failure`^*	gpu0	Remapping of rows failures
`gpu_shutdown_temp`^*	gpu0	GPU shutdown temperature
`gpu_slowdown_temp`^*	gpu0	GPU slowdown temperature
`gpu_sm_clock`^*	gpu0	GPU shader multiprocessor clock
`gpu_sync_boost_violation`^*	gpu0	Throttling duration due to sync boost constraints
`gpu_sync_boost_violation`	total	Throttling duration due to sync boost constraints
`gpu_recovery_action`^*	gpu0	Recovery action needed for a GPU (enum)
`gpu_recovery_check`^,*	gpu0	Fail if any GPU needs a recovery action
`gpu_temperature`	average	Average GPU temperature
`gpu_temperature`^*	gpu0	GPU temperature
`gpu_thermal_violation`^*	gpu0	Throttling duration due to thermal constraints
`gpu_thermal_violation`	total	Throttling duration due to thermal constraints
`gpu_total_app_clocks_violation`^*	gpu0	App clock violation limit
`gpu_total_app_clocks_violation`	total	Throttling duration due to application clocks limits
`gpu_total_base_clocks_violation`^*	gpu0	Base clock violation limit
`gpu_total_base_clocks_violation`	total	Throttling duration due to base clocks limits
`gpu_uncorrectable_remapped_rows`^*	gpu0	Number of remapped rows for uncorrectable errors
`gpu_utilization`	average	Average GPU utilization
`gpu_utilization`^*	gpu0	GPU utilization percentage
`gpu_xid_error`^*	gpu0	The value is the specific XID error

* parameter can be gpu0, gpu1…

** health check

12.7 Prometheus Metrics For The DGX GB200

Extra Prometheus metrics available on the DGX GB200 platform are shown in table 12.7.

The data producer for the metrics with a prefix of job_metadata_ is JobMetadataSampler. The remaining metrics are produced by JobSampler.

BCM picks up the measurables and their descriptions automatically. Many measurables still have no associated descriptions. For now, the names of the measurables can be a guide on what they are about.

Table 12.7:
Prometheus Metrics For The DGX GB200	Description
`job_cpu_power_limit`	CPU power limit
`job_cpu_power_usage`
`job_cpu_temperature`
`job_cpuacct_usage_seconds`	Total CPU time consumed by processes
`job_gpu_board_limit_violation`
`job_gpu_c2c_link_count`
`job_gpu_c2c_link_status`
`job_gpu_correctable_remapped_rows`
`job_gpu_dec_utilization`
`job_gpu_ecc_dbe_agg`
`job_gpu_ecc_dbe_vol`
`job_gpu_ecc_sbe_agg`
`job_gpu_ecc_sbe_vol`
`job_gpu_enc_utilization`
`job_gpu_enforced_power_limit`
`job_gpu_enforced_power_profile_mask`
`job_gpu_fabric_status`
`job_gpu_fan_speed`
`job_gpu_low_util_violation`
`job_gpu_mem_clock`
`job_gpu_mem_copy_utilization`
`job_gpu_mem_free`
`job_gpu_mem_total`
`job_gpu_mem_used`
`job_gpu_mem_utilization`
`job_gpu_memory_temp`
`job_gpu_nvlink_crc_data_errors`
`job_gpu_nvlink_crc_flit_errors`
`job_gpu_nvlink_total_bandwidth`	Total nvlink bandwidth used
`job_gpu_performance_state`
`job_gpu_power_management_limit`
`job_gpu_power_usage`
`job_gpu_power_violation`
`job_gpu_reliability_violation`
`job_gpu_requested_power_profile_mask`
`job_gpu_row_remap_failure`
`job_gpu_shutdown_temp`
`job_gpu_slowdown_temp`
`job_gpu_sm_clock`
`job_gpu_sync_boost_violation`
`job_gpu_temperature`
`job_gpu_thermal_violation`
`job_gpu_total_app_clocks_violation`
`job_gpu_total_base_clocks_violation`
`job_gpu_uncorrectable_remapped_rows`
`job_gpu_utilization`
`job_gpu_wasted`	The amount of the allocated GPUs that is not used
`job_gpu_workload_power_profile`	GPU workload power profile status
`job_gpu_xid_error`
`job_memory_cache_bytes`	Page cache, including tmpfs (shmem)
`job_memory_failcnt`	Number of times that the memory limit has reached the value set in memory.limit_in_bytes
`job_memory_mapped_file_bytes`	Size of memory-mapped mapped files, including tmpfs
`job_memory_memsw_failcnt`	Number of times that the memory plus swap space limit has reached the value set in memory.memsw.limit_in_bytes
`job_memory_memsw_usage_bytes`	Maximum amount of memory and swap space used by processes
`job_memory_rss_bytes`	Anonymous and swap cache, not including tmpfs
`job_memory_swap_bytes`	Swap memory usage
`job_memory_usage_bytes`	Total memory usage processes
`job_metadata_allocated_cpu_cores`
`job_metadata_allocated_gpus`
`job_metadata_is_running`
`job_metadata_is_waiting`
`job_metadata_num_cpus`
`job_metadata_num_nodes`
`job_metadata_pending_jobs`
`job_metadata_running_jobs`
`job_metadata_running_seconds`
`job_metadata_waiting_seconds`

12.8 Redfish Metrics For The DGX GB200

Redfish monitoring can be carried out for devices with a Redfish subscription. The redfishsubscriptions command (appendix N.2.4 of the Administrator Manual) can be used to check if a device has a subscription.

Example

[basecm11->device]% redfishsubscriptions node001
hostname      ...initialized ip          kind     port     ssl      subscribed...
------------- ...----------- ----------- -------- -------- -------- ----------
node001       ...true        7.241.2.13  GB200    443      true     true

Redfish firmware metrics that may be available on the DGX GB200 platform are shown in table 12.8.

As is usual with hardware, the metrics can change according to hardware and especially with firmware changes. The table should therefore be regarded as an indication of what metrics are available, and not a list of what metrics must exist.

BCM picks up the metrics and their descriptions automatically. Many metrics have no associated descriptions. For now, the names of the metrics can be a guide on what they are about.

The list of Redfish metrics available can be found by running:

Example

basecm11->monitoring->measurable]% list metric | grep RF_

Table 12.8:
Redfish Metrics For The DGX GB200	Description
`RF_AOC_NIC1Temp`^[1]	AOC NIC 1 Temperature
`RF_AOC_PORTATemp`	AOC PORT A Temperature
`RF_AOC_PORTBTemp`	AOC PORT B Temperature
`RF_AOC_PORTCTemp`	AOC PORT C Temperature
`RF_Active_PSU`^[2]
`RF_Average_Temp`
`RF_BF3_Slot_1_NIC_Temp_0`	BF3 Slot 1 NIC Temperature 0
`RF_BF3_Slot_2_NIC_Temp_0`	BF3 Slot 2 NIC Temperature 0
`RF_BF3_Slot_1_Port_0_Temp`	BF3 Slot 1 Port 0 Temperature
`RF_BF3_Slot_1_Port_1_Temp`	BF3 Slot 1 Port 1 Temperature
`RF_BF3_Slot_2_Port_0_Temp`	BF3 Slot 2 Port 0 Temperature
`RF_BF3_Slot_2_Port_1_Temp`	BF3 Slot 2 Port 1 Temperature
`RF_BF3_Slot_1_Temp`	BF3 Slot 1 Temperature
`RF_BF3_Slot_2_Temp`	BF3 Slot 2 Temperature
`RF_BMCTemp`	BMC Temperature
`RF_BMC_0_DCSCM_Temp_0`	BMC 0 DCSCM Temperature 0
`RF_C2CPU1Temp`	C2 CPU1 Temperature
`RF_C2CPU2Temp`	C2 CPU2 Temperature
`RF_CPU1RearTemp`	CPU1 Rear Temperature
`RF_CPU1Temp`	CPU1 Temperature
`RF_CPU1_VCCHV`	CPU1_VCCHV
`RF_CPU1_VCCIN`	CPU1_VCCIN
`RF_CPU1_VCCON`	CPU1_VCCON
`RF_CPU1_VRMHVTemp`	CPU1_VRMHV Temperature
`RF_CPU1_VRMINTemp`	CPU1_VRMIN Temperature
`RF_CPU1_VRMONTemp`	CPU1_VRMON Temperature
`RF_CPU2RearTemp`	CPU2 Rear Temperature
`RF_CPU2Temp`	CPU2 Temperature
`RF_CPU2_VCCHV`	CPU2_VCCHV
`RF_CPU2_VCCIN`	CPU2_VCCIN
`RF_CPU2_VCCON`	CPU2_VCCON
`RF_CPU2_VRMHVTemp`	CPU2_VRMHV Temperature
`RF_CPU2_VRMINTemp`	CPU2_VRMIN Temperature
`RF_CPU2_VRMONTemp`	CPU2_VRMON Temperature
`RF_CPU_0_CoreUtil_0`^[3]	CPU_0_CoreUtil_0
In the preceding row, the substring `CoreUtil_0` can also take values of
`CoreUtil_1`, `CoreUtil_2`…, up to `CoreUtil_71`
`RF_CPU_0_CpuFreq_0`^[3]	CPU_0_CpuFreq_0
`RF_CPU_0_Energy_0`^[3]	CPU_0_Energy_0
`RF_CPU_0_Power_0`^[3]	CPU_0_Power_0
`RF_CPU_0_Processor_code_error`^[3]
`RF_CPU_0_Processor_message_error`^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_AccumulatedGPUContextUtilizationDuration` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_AccumulatedSMUtilizationDuration` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_EDPViolationState` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_GlobalSoftwareViolationThrottleDuration` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_HardwareViolationThrottleDuration` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_MemoryPageRetirement` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_MemorySpareChannelPresence` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_PCIeRXBytes` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_PCIeTXBytes` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_PerformanceState` ^[3]
`RF_CPU_0_Processor_Metrics__Nvidia_PowerBreakPerformanceState` ^[3]
`RF_CPU_0_Processor_Metrics_PowerLimitThrottleDuration` ^[3]
`RF_CPU_0_Processor_Metrics_ThermalLimitThrottleDuration` ^[3]
`RF_CPU_0_TempAvg_0`^[3]	CPU_0_TempAvg_0
`RF_CPU_0_TempLimit_0`^[3]	CPU_0_TempLimit_0
`RF_Chassis_0_FAN_1_FRONT`^[4]	Chassis 0 FAN 1 FRONT
`RF_Chassis_0_FAN_1_PWM`^[4]	Chassis 0 FAN 1 PWM
`RF_Chassis_0_FAN_1_REAR`^[4]	Chassis 0 FAN 1 REAR
`RF_Chassis_0_Front_IO_Temp_0`	Chassis 0 Front IO Temp 0
`RF_Chassis_0_LeakDetector_0_ColdPlate` ^[5]	Chassis 0 LeakDetector 0
`RF_Chassis_0_LeakDetector_0_Manifold` ^[5]	Chassis 0 LeakDetector 0
`RF_Chassis_0_LeakDetector_1_ColdPlate` ^[5]	Chassis 0 LeakDetector 1
`RF_Chassis_0_LeakDetector_1_Manifold` ^[5]	Chassis 0 LeakDetector 1
`RF_Chassis_0_TotalHSC_Power_0`	Chassis 0 TotalHSC Power 0
`RF_Chassis_Intru_Reading`
`RF_Chassis_Intru_ReadingRangeMax`
`RF_Chassis_Intru_ReadingRangeMin`
`RF_DIMM_SlotPartLocationContext_Location`
`RF_DIMM_SlotState_Status`
`RF_DIMM_Slot_BusWidthBits`
`RF_DIMM_Slot_Capacity`
`RF_DIMM_Slotcode_error`
`RF_DIMM_Slotmessage_error`
`RF_DIMM_Slot_OperatingSpeed`
`RF_DIMM_Slot_DataWidthBits`
`RF_DIMM_Slot_FirmwareRevision`
`RF_DIMM_Slot_Nvidia_RowRemappingFailed`
`RF_DIMM_Slot_Nvidia_RowRemappingPending`
`RF_DIMM_Slot_OperatingSpeed`
`RF_DIMM_Slot_Rank`
`RF_Energy_0`	Energy_0
`RF_FPGATemp`	FPGA Temp
`RF_FPGA_ARTTemp`	FPGA_ART Temp
`RF_GPU_0_DRAM_0_Memory_MetricsCorrectableECCErrorCount_LifeTime` ^[6]
`RF_GPU_0_DRAM_0_Memory_MetricsUncorrectableECCErrorCount_LifeTime` ^[6]
`RF_GPU_0_DRAM_0_Memory_Metrics_BandwidthUtilization` ^[6]
`RF_GPU_0_DRAM_0_Memory_Metrics_CapacityUtilizationPercent` ^[6]
`RF_GPU_0_DRAM_0_Memory_Metrics_OperatingSpeed` ^[6]
`RF_GPU_0_Processor_MetricsCorrectableErrorCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsFatalErrorCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsL0ToRecoveryCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsNAKReceivedCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsNAKSentCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsNonFatalErrorCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsReplayCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsReplayRolloverCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_MetricsUnsupportedRequestCount_PCIeErrors` ^[6]
`RF_GPU_0_Processor_Metrics_BandwidthUtilization` ^[6]
`RF_GPU_0_Processor_Metrics_CoreVoltage` ^[6]
`RF_GPU_0_Processor_Metrics_LifeTime_CorrectableECCError` ^[6]
`RF_GPU_0_Processor_Metrics_LifeTime_UncorrectableECCError` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_AccumulatedGPUContextUtilizationDuration` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_AccumulatedSMUtilizationDuration` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_DMMAUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_FP16ActivityPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_FP32ActivityPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_FP64ActivityPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_GlobalSoftwareViolationThrottleDuration` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_GraphicsEngineActivityPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_HMMAUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_HardwareViolationThrottleDuration` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_IMMAUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_IntegerActivityUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_NVDecUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_NVJpgUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_NVLinkDataRxBandwidth` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_NVLinkDataTxBandwidth` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_NVLinkRawRxBandwidth` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_NVLinkRawTxBandwidth` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_NVOfaUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_PCIeRXBytes` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_PCIeRawRxBandwidth` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_PCIeRawTxBandwidth` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_PCIeTXBytes` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_SMActivityPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_SMOccupancyPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_SMUtilizationPercent` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_SRAMECCErrorThresholdExceeded` ^[6]
`RF_GPU_0_Processor_Metrics_Nvidia_TensorCoreActivityPercent` ^[6]
`RF_GPU_0_Processor_Metrics_OperatingSpeed` ^[6]
`RF_GPU_0_Processor_Metrics_PowerLimitThrottleDuration` ^[6]
`RF_GPU_0_Processor_Metrics_ThermalLimitThrottleDuration` ^[6]
`RF_GPU_0_Processor_Metricscode_error` ^[6]
`RF_GPU_0_Processor_Metricsmessage_error` ^[6]
`RF_GPU_0_Processor_Reset_Metrics_ConventionalResetEntry` ^[6]
`RF_GPU_0_Processor_Reset_Metrics_ConventionalResetExit` ^[6]
`RF_GPU_0_Processor_Reset_Metrics_FundamentalResetEntry` ^[6]
`RF_GPU_0_Processor_Reset_Metrics_FundamentalResetExit` ^[6]
`RF_GPU_0_Processor_Reset_Metrics_IRoTResetExit` ^[6]
`RF_GPU_0_Processor_Reset_Metrics_PF_FLR_ResetEntry` ^[6]
`RF_GPU_0_Processor_Reset_Metrics_PF_FLR_ResetExit` ^[6]
`RF_HGX_BMC_0_Temp_0`	HGX BMC 0 Temp 0
`RF_HGX_Baseboard_0CoreCount_ProcessorSummary`
`RF_HGX_Baseboard_0Count_ProcessorSummary`
`RF_HGX_Baseboard_0Model_ProcessorSummary`
`RF_HGX_Baseboard_0State_Status`
`RF_HGX_Baseboard_0TotalSystemMemoryGiB_MemorySummary`
`RF_HGX_Baseboard_0_Description`
`RF_HGX_Baseboard_0_LastResetTime`
`RF_HGX_Baseboard_0_Nvidia_ISTModeEnabled`
`RF_HGX_Baseboard_0_PowerMode`
`RF_HGX_Baseboard_0_PowerState`
`RF_HGX_CPU_0State_Status`^[3]
`RF_HGX_CPU_0_AssetTag`^[3]
`RF_HGX_CPU_0_PartLocation_ServiceLabel` ^[3]
`RF_HGX_CPU_0_SKU`^[3]
`RF_HGX_CPU_0code_error`^[3]
`RF_HGX_CPU_0message_error`^[3]
`RF_HGX_Chassis_0_TotalGPU_Power_0`	HGX Chassis 0 TotalGPU Power 0
`RF_HGX_GPU_0State_Status`^[6]
`RF_HGX_GPU_0_DRAM_0_Power_0` ^[6]	HGX GPU 0 DRAM 0 Power 0
`RF_HGX_GPU_0_DRAM_0_Temp_0` ^[6]	HGX GPU 0 DRAM 0 Temp 0
`RF_HGX_GPU_0_Energy_0`^[6]	HGX GPU 0 Energy 0
`RF_HGX_GPU_0_MaxPower`^[6]
`RF_HGX_GPU_0_MinPower`^[6]
`RF_HGX_GPU_0_Power_0`^[6]	HGX GPU 0 Power 0
`RF_HGX_GPU_0_SKU`^[6]
`RF_HGX_GPU_0_TEMP_0`^[6]	HGX GPU 0 TEMP 0
`RF_HGX_GPU_0_TEMP_1`^[6]	HGX GPU 0 TEMP 1
`RF_HGX_GPU_0_Voltage_0`^[6]	HGX GPU 0 Voltage 0
`RF_HGX_ProcessorModule_0_Exhaust_Temp_0` ^[12]	HGX ProcessorModule 0 Exhaust
`RF_HGX_ProcessorModule_0_Inlet_Temp_0` ^[12]	HGX ProcessorModule 0
`RF_HGX_ProcessorModule_0_Inlet_Temp_1` ^[12]	HGX ProcessorModule 0
`RF_HotswapTemp`	Hotswap Temp
`RF_IO_Board_0_CX7_0_Port_0_Temp`	IO Board 0 CX7 0 Port 0 Temp
`RF_IO_Board_0_CX7_0_Temp`	IO Board 0 CX7 0 Temp
`RF_IO_Board_0_CX7_0_Temp_0`	IO Board 0 CX7 0 Temp 0
`RF_IO_Board_0_CX7_1_Port_0_Temp`	IO Board 0 CX7 1 Port 0 Temp
`RF_IO_Board_0_CX7_1_Temp`	IO Board 0 CX7 1 Temp
`RF_IO_Board_0_CX7_1_Temp_0`	IO Board 0 CX7 1 Temp 0
`RF_IO_Board_1_CX7_0_Port_0_Temp`	IO Board 1 CX7 0 Port 0 Temp
`RF_IO_Board_1_CX7_0_Temp`	IO Board 1 CX7 0 Temp
`RF_IO_Board_1_CX7_0_Temp_0`	IO Board 1 CX7 0 Temp 0
`RF_IO_Board_1_CX7_1_Port_0_Temp`	IO Board 1 CX7 1 Port 0 Temp
`RF_IO_Board_1_CX7_1_Temp`	IO Board 1 CX7 1 Temp
`RF_IO_Board_1_CX7_1_Temp_0`	IO Board 1 CX7 1 Temp 0
`RF_InletTemp`	Inlet Temp
`RF_InterswitchPort_0_Port_MetricsRXFrames_Networking`
`RF_InterswitchPort_0_Port_MetricsRXMulticastFrames_Networking`
`RF_InterswitchPort_0_Port_MetricsRXUnicastFrames_Networking`
`RF_InterswitchPort_0_Port_MetricsTXDiscards_Networking`
`RF_InterswitchPort_0_Port_MetricsTXFrames_Networking`
`RF_InterswitchPort_0_Port_MetricsTXMulticastFrames_Networking`
`RF_InterswitchPort_0_Port_MetricsTXUnicastFrames_Networking`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_BitErrorRate`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_LinkDowned`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_LinkErrorRecovery`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_MalformedPackets`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_NeighborMTUDiscards`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_QP1Dropped`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_RXRemotePhysicalErrors`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_RXSwitchRelayErrors`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_TXWait`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_VL15Dropped`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_VL15TXBytes`
`RF_InterswitchPort_0_Port_Metrics_Nvidia_VL15TXPackets`
`RF_InterswitchPort_0_Port_Metrics_RXBytes`
`RF_InterswitchPort_0_Port_Metrics_RXErrors`
`RF_InterswitchPort_0_Port_Metrics_TXBytes`
`RF_InterswitchPort_0_ResourceState_Status`
`RF_InterswitchPort_0_Resource_CurrentSpeed`
`RF_InterswitchPort_0_Resource_LinkState`
`RF_InterswitchPort_0_Resource_LinkStatus`
`RF_InterswitchPort_0_Resource_MaxSpeed`
`RF_LD_Chassis_0_LeakDetector_0_ColdPlate` ^[5]	Chassis 0 LeakDetector 0
`RF_LD_Chassis_0_LeakDetector_0_Manifold` ^[5]	Chassis 0 LeakDetector 0
`RF_LD_Chassis_0_LeakDetector_1_ColdPlate` ^[5]	Chassis 0 LeakDetector 1
`RF_LD_Chassis_0_LeakDetector_1_Manifold` ^[5]	Chassis 0 LeakDetector 1
`RF_LD_LeakDetection`^[5]	Leak Detection Systems
`RF_LeakDetection`^[5]	Leak Detection Systems
`RF_M2_SSD1Temp`	M2_SSD1 Temp
`RF_M2_SSD2Temp`	M2_SSD2 Temp
`RF_MB1.8VCC`	MB 1.8VCC
`RF_MB1.8VSB`	MB 1.8VSB
`RF_MB12V`	MB 12V
`RF_MB12VSB`	MB 12VSB
`RF_MB3.3VCC`	MB 3.3VCC
`RF_MB3.3VSB`	MB 3.3VSB
`RF_MB5VCC`	MB 5VCC
`RF_MB5VSB`	MB 5VSB
`RF_MGX_NVSwitch_0_TEMP_0`	MGX NVSwitch 0 TEMP 0
`RF_MGX_NVSwitch_1_TEMP_0`	MGX NVSwitch 1 TEMP 0
`RF_MemCntl_0_Freq_0`	MemCntl_0_Freq_0
`RF_MemCntl_1_Freq_0`	MemCntl_1_Freq_0
`RF_NVLinkManagement_0_Port_MetricsRXFrames_Networking`
`RF_NVLinkManagement_0_Port_MetricsRXMulticastFrames_Networking`
`RF_NVLinkManagement_0_Port_MetricsRXUnicastFrames_Networking`
`RF_NVLinkManagement_0_Port_MetricsTXDiscards_Networking`
`RF_NVLinkManagement_0_Port_MetricsTXFrames_Networking`
`RF_NVLinkManagement_0_Port_MetricsTXMulticastFrames_Networking`
`RF_NVLinkManagement_0_Port_MetricsTXUnicastFrames_Networking`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_BitErrorRate`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_LinkDowned`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_LinkErrorRecovery`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_MalformedPackets`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_NeighborMTUDiscards`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_QP1Dropped`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_RXRemotePhysicalErrors`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_RXSwitchRelayErrors`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_TXWait`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_VL15Dropped`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_VL15TXBytes`
`RF_NVLinkManagement_0_Port_Metrics_Nvidia_VL15TXPackets`
`RF_NVLinkManagement_0_Port_Metrics_RXBytes`
`RF_NVLinkManagement_0_Port_Metrics_RXErrors`
`RF_NVLinkManagement_0_Port_Metrics_TXBytes`
`RF_NVLinkManagement_0_ResourceState_Status`
`RF_NVLinkManagement_0_Resource_CurrentSpeed`
`RF_NVLinkManagement_0_Resource_LinkState`
`RF_NVLinkManagement_0_Resource_LinkStatus`
`RF_NVLinkManagement_0_Resource_MaxSpeed`
`RF_NVLink_0_Port_MetricsRXFrames_Networking` ^[7]
`RF_NVLink_0_Port_MetricsRXMulticastFrames_Networking` ^[7]
`RF_NVLink_0_Port_MetricsRXUnicastFrames_Networking` ^[7]
`RF_NVLink_0_Port_MetricsTXDiscards_Networking` ^[7]
`RF_NVLink_0_Port_MetricsTXFrames_Networking` ^[7]
`RF_NVLink_0_Port_MetricsTXMulticastFrames_Networking` ^[7]
`RF_NVLink_0_Port_MetricsTXUnicastFrames_Networking` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_BitErrorRate` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_EffectiveError` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_LinkDowned` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_LinkErrorRecovery` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_MalformedPackets` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_NVLinkDataRxBandwidth` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_NVLinkDataTxBandwidth` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_NVLinkRawRxBandwidth` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_NVLinkRawTxBandwidth` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_NeighborMTUDiscards` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_QP1Dropped` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_RXNoProtocolBytes` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_RXRemotePhysicalErrors` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_RXSwitchRelayErrors` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_TXNoProtocolBytes` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_TXWait` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_VL15Dropped` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_VL15TXBytes` ^[7]
`RF_NVLink_0_Port_Metrics_Nvidia_VL15TXPackets` ^[7]
`RF_NVLink_0_Port_Metrics_RXBytes` ^[7]
`RF_NVLink_0_Port_Metrics_RXErrors` ^[7]
`RF_NVLink_0_Port_Metrics_TXBytes` ^[7]
`RF_NVLink_0_ResourceState_Status` ^[7]
`RF_NVLink_0_Resource_CurrentSpeed`^[7]
`RF_NVLink_0_Resource_LinkState`^[7]
`RF_NVLink_0_Resource_LinkStatus`^[7]
`RF_NVLink_0_Resource_MaxSpeed`^[7]
`RF_NVLink_0_Resource_Nvidia_RXWidth` ^[7]
`RF_NVLink_0_Resource_Nvidia_TXWidth` ^[7]
`RF_NVME_M2_0_Temp_0`	NVME M2 0 Temp 0
`RF_NVMe_SSDATemp`	NVMe_SSDA Temp
`RF_NVSwitch_0_ResourceState_Status`
`RF_NVSwitch_0_Resource_#Switch.Reset_target`
`RF_NVSwitch_0_Resource_CurrentBandwidth`
`RF_NVSwitch_0_Resource_Enabled`
`RF_NVSwitch_0_Resource_FirmwareVersion`
`RF_NVSwitch_0_Resource_MaxBandwidth`
`RF_NVSwitch_0_Resource_Nvidia_SwitchIsolationMode`
`RF_NVSwitch_1_ResourceState_Status`
`RF_NVSwitch_1_Resource_#Switch.Reset_target`
`RF_NVSwitch_1_Resource_CurrentBandwidth`
`RF_NVSwitch_1_Resource_Enabled`
`RF_NVSwitch_1_Resource_FirmwareVersion`
`RF_NVSwitch_1_Resource_MaxBandwidth`
`RF_NVSwitch_1_Resource_Nvidia_SwitchIsolationMode`
`RF_P1_DIMMA DTemp`	P1_DIMMA D Temp
`RF_P1_DIMME HTemp`	P1_DIMME H Temp
`RF_P2_DIMMA DTemp`	P2_DIMMA D Temp
`RF_P2_DIMME HTemp`	P2_DIMME H Temp
`RF_PCB_Inlet_Temp`	PCB Inlet Temp
`RF_PCH1.05V`	PCH 1.05V
`RF_PCHPVNN`	PCH PVNN
`RF_PCHTemp`	PCH Temp
`RF_PCIe_0_ResourceState_Status`
`RF_PCIe_0_Resource_ActiveWidth`
`RF_PCIe_0_Resource_CurrentSpeed`
`RF_PCIe_0_Resource_MaxSpeed`
`RF_PCIe_0_Resource_Width`
`RF_PCIe_DeviceLanesInUse_PCIeInterface`
`RF_PCIe_DeviceMaxLanes_PCIeInterface`
`RF_PCIe_DeviceMaxPCIeType_PCIeInterface`
`RF_PCIe_DevicePCIeType_PCIeInterface`
`RF_PCIe_DeviceState_Status`
`RF_PCIe_Device_Nvidia_AERCorrectableErrorStatus`
`RF_PCIe_Device_Nvidia_AERUncorrectableErrorStatus`
`RF_PCIe_Device_Nvidia_NVLinkReferenceClockEnabled`
`RF_PCIe_Device_Nvidia_PCIeReferenceClockEnabled`
`RF_PCIe_Devicecode_error`
`RF_PCIe_Devicemessage_error`
`RF_PDB_0_HSC_0_Cur_0`	PDB 0 HSC 0 Cur 0
`RF_PDB_0_HSC_0_Pwr_0`	PDB 0 HSC 0 Pwr 0
`RF_PDB_0_HSC_0_Temp_0`	PDB 0 HSC 0 Temp 0
`RF_PDB_0_HSC_0_Volt_In_0`	PDB 0 HSC 0 Volt In 0
`RF_PDB_0_HSC_0_Volt_Out_0`	PDB 0 HSC 0 Volt Out 0
`RF_PDB_0_HSC_1_Cur_0`	PDB 0 HSC 1 Cur 0
`RF_PDB_0_HSC_1_Pwr_0`	PDB 0 HSC 1 Pwr 0
`RF_PDB_0_HSC_1_Temp_0`	PDB 0 HSC 1 Temp 0
`RF_PDB_0_HSC_1_Volt_In_0`	PDB 0 HSC 1 Volt In 0
`RF_PDB_0_HSC_1_Volt_Out_0`	PDB 0 HSC 1 Volt Out 0
`RF_PDB_0_Inlet_Temp_0`	PDB 0 Inlet Temp 0
`RF_PDB_0_Vreg_0_Temp_0`	PDB 0 Vreg 0 Temp 0
`RF_PDB_0_Vreg_0_Temp_1`	PDB 0 Vreg 0 Temp 1
`RF_PDB_0_Vreg_1_Temp_0`	PDB 0 Vreg 1 Temp 0
`RF_PDB_0_Vreg_1_Temp_1`	PDB 0 Vreg 1 Temp 1
`RF_PDB_TOTAL_CURRENT`
`RF_PDB_TOTAL_PWR`
`RF_PDB_TOTAL_VOLT_IN`
`RF_PDB_TOTAL_VOLT_OUT`
`RF_PS1_Status_Reading`
`RF_PS1_Status_ReadingRangeMax`
`RF_PS1_Status_ReadingRangeMin`
`RF_PWConsumption`	PW Consumption
`RF_PeripheralTemp`	Peripheral Temp
`RF_PowerShelf_Status`^[2],[8]	Power shelf status of PSU
`RF_PowerShelf_Status`^[2],[8]	Power shelf status total
`RF_Power_0`	Power_0
`RF_Power_PowerControl`
`RF_Power_PowerSupplies`
`RF_Power_Redundancy`
`RF_Power_Supply_Energy`^[2],[8]	Energy consumption since boot of PSU
`RF_Power_Supply_Energy`^[9]	Energy consumption since boot of all PSUs
`RF_Power_Supply_FanSpeed`^[2],[8]	Fan speed of PSU
`RF_Power_Supply_FanSpeed`^[10]	Fan speed average of all PSUs
`RF_Power_Supply_Id`^[2],[8]	ID of PSU
`RF_Power_Supply_InputCurrent`^[2],[8]	Input current for PSU
`RF_Power_Supply_InputCurrent`^[9]	Input current of all PSUs
`RF_Power_Supply_InputPower`^[2],[8]	Input power for PSU
`RF_Power_Supply_InputPower`^[9]	Input power total of all PSUs
`RF_Power_Supply_InputVoltage`^[2],[8]	Input voltage for PSU
`RF_Power_Supply_LifetimeReading_Energy`^[2],[8]	Energy consumption of PSU over lifetime
`RF_Power_Supply_Name`^[2],[8]	Name of PSU
`RF_Power_Supply_OutputPower`^[2],[8]	Output power of PSU
`RF_Power_Supply_OutputPower`^[9]	Output power total of all PSUs
`RF_Power_Supply_PowerFactor_Input`^[2],[8]	Input power factor of PSU
`RF_Power_Supply_RailCurrent`^[2],[8]	Rail current of PSU
`RF_Power_Supply_RailCurrent`^[9]	Rail current total of all PSUs
`RF_Power_Supply_RailPower`^[2],[8]	Rail current of PSU
`RF_Power_Supply_RailPower`^[9]	Rail power total of all PSUs
`RF_Power_Supply_RailVoltage`^[2],[8]	Rail voltage of PSU
`RF_Power_Supply_Temperature`^[2],[8]	Temperature of PSU
`RF_Power_Supply_Temperature`^[10]	Temperature average of all PSUs
`RF_Power_Supply_code_error`^[2],[8]	Code error of PSU
`RF_Power_Supply_message_error`^[2],[8]	Message error of PSU
`RF_Power_Voltages`
`RF_ProcessorModule_0_CPU_0_CoreUtil_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_CPU_0_CpuFreq_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_CPU_0_Energy_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_CPU_0_EnforcedEDPc_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_CPU_0_EnforcedEDPp_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_CPU_0_Power_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_CPU_0_TempAvg_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_CPU_0_TempLimit_0` ^[11]	ProcessorModule 0 CPU 0
`RF_ProcessorModule_0_MemCntl_0_Freq_0` ^[11]	ProcessorModule 0 MemCntl 0
`RF_ProcessorModule_0_Vreg_0_CpuPower_0` ^[11]	ProcessorModule 0 Vreg 0
`RF_ProcessorModule_0_Vreg_0_CpuVoltage_0` ^[11]	ProcessorModule 0 Vreg 0
`RF_ProcessorModule_0_Vreg_0_SocPower_0` ^[11]	ProcessorModule 0 Vreg 0
`RF_ProcessorModule_0_Vreg_0_SocVoltage_0` ^[11]	ProcessorModule 0 Vreg 0
`RF_SYS_ART1Temp`	SYS_ART1 Temp
`RF_SYS_ART2Temp`	SYS_ART2 Temp
`RF_StorageBackplane_0_SSD_0_Temp_0`	StorageBackplane 0 SSD 0
`RF_StorageBackplane_0_SSD_2_Temp_0`	StorageBackplane 0 SSD 2
`RF_StorageBackplane_1_SSD_0_Temp_0`	StorageBackplane 1 SSD 0
`RF_StorageBackplane_1_SSD_2_Temp_0`	StorageBackplane 1 SSD 2
`RF_Summary_Metrics_Id`^[2]
`RF_Summary_Metrics_Name`^[2]
`RF_Summary_Metrics_Power`^[2]	Power consumption (Watts)
`RF_Summary_Metrics_Temperature`^[2]	Temperature (Celsius)
`RF_SystemTemp`	System Temp
`RF_Total_PSU`^[2]
`RF_VBAT_Reading`
`RF_VBAT_ReadingRangeMax`
`RF_VBAT_ReadingRangeMin`
`RF_Vreg_0_CpuPower_0`	Vreg_0_CpuPower_0
`RF_Vreg_0_CpuVoltage_0`	Vreg_0_CpuVoltage_0
`RF_Vreg_0_SocPower_0`	Vreg_0_SocPower_0
`RF_Vreg_0_SocVoltage_0`	Vreg_0_SocVoltage_0
`RF_Vreg_1_CpuPower_0`	Vreg_1_CpuPower_0
`RF_Vreg_1_CpuVoltage_0`	Vreg_1_CpuVoltage_0
`RF_Vreg_1_SocPower_0`	Vreg_1_SocPower_0
`RF_Vreg_1_SocVoltage_0`	Vreg_1_SocVoltage_0

[[1]] The substring NIC1Temp can also be NIC2Temp, NIC3Temp…

[[2]] from redfish_power_shelf data producer

[[3]] The substring CPU_0 can also be CPU_1

[[4]] The substring FAN_1 can also be FAN_2, FAN_3…

[[5]] From redfish_leak data producer

[[6]] The substring GPU_0 can also be GPU_1, GPU_2…

[[7]] On the DGX GB200 the value of <number> in ...NVLink_ <number>...can be from 0 to 71

[[8]] Takes the PSU number as a parameter. On the DGX GB200 it can be from 1 to 6

[[9]] Takes total as a parameter. On the DGX GB200 it means the total value of the 6 PSUs

[[10]] Takes average as a parameter. On the DGX GB200 it means the average value of the 6 PSUs

[[11]] On the DGX GB200 the value of <number> in ...ProcessorModule_ <number>... can be 0 or 1

[[12]] The substring ProcessorModule_0 can also be ProcessorModule_1

12.9 Redfish Enums For The DGX GB200

Redfish monitoring can be carried out for devices with a Redfish subscription (here).

Redfish firmware enums that may be available on the DGX GB200 platform are shown in table 12.9.

The list of Redfish enums that are there on the DGX GB200 platform can be found by running:

Example

basecm11->monitoring->measurable]% list enum | grep RF_

Table 12.9:
Redfish Enums For The DGX GB200	Parameter	Description
`RF_BF3_Slot_1_NIC_Temp_0`	state
`RF_BF3_Slot_2_NIC_Temp_0`	state
`RF_BF3_Slot_2_Port_0_Temp`	state
`RF_BF3_Slot_2_Port_1_Temp`	state
`RF_BF3_Slot_2_Temp`	state
`RF_BMC_0_DCSCM_Temp_0`	state
`RF_Chassis_0_FAN_1_FRONT`^[1]	state
`RF_Chassis_0_FAN_1_PWM`	state
`RF_Chassis_0_FAN_1_REAR`	state
`RF_CPU_0_Processor__Nvidia_EDPViolationState`
`RF_CPU_0_Processor__Nvidia_PerformanceState`
`RF_CPU_0_Processor__Nvidia_PowerBreakPerformanceState`
`RF_CPU_1_Processor__Nvidia_EDPViolationState`
`RF_CPU_1_Processor__Nvidia_PerformanceState`
`RF_CPU_1_Processor__Nvidia_PowerBreakPerformanceState`
`RF_DIMM_SlotHealthRollup_Status`	state
`RF_DIMM_SlotHealth_Status`	state
`RF_ERoT_BMC_0_PowerState`
`RF_ERoT_BMC_0HealthRollup_Status`	state
`RF_ERoT_BMC_0Health_Status`	state
`RF_HGX_BMC_0_Temp_0`	state
`RF_HGX_Baseboard_0HealthRollup_Status`	state
`RF_HGX_Baseboard_0Health_Status`	state
`RF_HGX_Baseboard_0_PowerState`
`RF_HGX_CPU_0HealthRollup_Status`	state
`RF_HGX_CPU_0Health_Status`	state
`RF_HGX_CPU_0_PowerState`
`RF_HGX_CPU_1HealthRollup_Status`	state
`RF_HGX_CPU_1Health_Status`	state
`RF_HGX_CPU_1_PowerState`
`RF_HGX_Chassis_0_TotalGPU_Power_0`	state
`RF_HGX_ERoT_BMC_0HealthRollup_Status`	state
`RF_HGX_ERoT_BMC_0Health_Status`	state
`RF_HGX_ERoT_CPU_0HealthRollup_Status`	state
`RF_HGX_ERoT_CPU_0Health_Status`	state
`RF_HGX_ERoT_CPU_0_PowerState`
`RF_HGX_ERoT_CPU_1HealthRollup_Status`	state
`RF_HGX_ERoT_CPU_1Health_Status`	state
`RF_HGX_ERoT_FPGA_0HealthRollup_Status`	state
`RF_HGX_ERoT_FPGA_0Health_Status`	state
`RF_HGX_ERoT_FPGA_1HealthRollup_Status`	state
`RF_HGX_ERoT_FPGA_1Health_Status`	state
`RF_HGX_GPU_0HealthRollup_Status`^[2]	state
`RF_HGX_GPU_0Health_Status`^[2]	state
`RF_HGX_GPU_0_DRAM_0_Power_0`^[2]	state
`RF_HGX_GPU_0_DRAM_0_Temp_0`^[2]	state
`RF_HGX_GPU_0_Energy_0`^[2]	state
`RF_HGX_GPU_0_PowerState`^[2]
`RF_HGX_GPU_0_Power_0`^[2]	state
`RF_HGX_GPU_0_TEMP_0`^[2]	state
`RF_HGX_GPU_0_TEMP_1`^[2]	state
`RF_HGX_GPU_0_Voltage_0`^[2]	state
`RF_HGX_IRoT_GPU_0HealthRollup_Status`^[2]	state
`RF_HGX_IRoT_GPU_0Health_Status`^[2]	state
`RF_HGX_IRoT_GPU_1_PowerState`^[2]
`RF_HGX_ProcessorModule_0_Exhaust_Temp_0`	state
`RF_HGX_ProcessorModule_0_Inlet_Temp_0`	state
`RF_HGX_ProcessorModule_0_Inlet_Temp_1`	state
`RF_HGX_ProcessorModule_1_Exhaust_Temp_0`	state
`RF_HGX_ProcessorModule_1_Inlet_Temp_0`	state
`RF_HGX_ProcessorModule_1_Inlet_Temp_1`	state
`RF_IO_Board_0_CX7_0_Port_0_Temp`	state
`RF_IO_Board_0_CX7_0_Temp`	state
`RF_IO_Board_0_CX7_0_Temp_0`	state
`RF_IO_Board_0_CX7_1_Port_0_Temp`	state
`RF_IO_Board_0_CX7_1_Temp`	state
`RF_IO_Board_0_CX7_1_Temp_0`	state
`RF_IO_Board_1_CX7_0_Port_0_Temp`	state
`RF_IO_Board_1_CX7_0_Temp`	state
`RF_IO_Board_1_CX7_0_Temp_0`	state
`RF_IO_Board_1_CX7_1_Port_0_Temp`	state
`RF_IO_Board_1_CX7_1_Temp`	state
`RF_IO_Board_1_CX7_1_Temp_0`	state
`RF_IRoT_CX7_0HealthRollup_Status`	state
`RF_IRoT_CX7_0Health_Status`	state
`RF_IRoT_CX7_1HealthRollup_Status`	state
`RF_IRoT_CX7_1Health_Status`	state
`RF_IRoT_CX7_2HealthRollup_Status`	state
`RF_IRoT_CX7_2Health_Status`	state
`RF_IRoT_CX7_3HealthRollup_Status`	state
`RF_IRoT_CX7_3Health_Status`	state
`RF_LD_LeakDetection`^[3]
`RF_LeakDetector`^[3]	Chassis_0_LeakDetector_0_ColdPlate
`RF_LeakDetector`^[3]	Chassis_0_LeakDetector_0_Manifold
`RF_LeakDetector`^[3]	Chassis_0_LeakDetector_1_ColdPlate
`RF_LeakDetector`^[3]	Chassis_0_LeakDetector_1_Manifold
`RF_NVLinkManagement_0_ResourceHealthRollup_Status`	state
`RF_NVLinkManagement_0_ResourceHealth_Status`	state
`RF_NVLink_ResourceHealthRollup_Status`	state
`RF_NVLink_ResourceHealth_Status`	state
`RF_NVLink_ResourceState_Status`^[4]	acp0
`RF_NVLink_Resource_LinkState`^[4]	acp0
`RF_NVLink_0_ResourceHealthRollup_Status`	state
`RF_NVLink_0_ResourceHealth_Status`	state
`RF_NVME_M2_0_Temp_0`	state
`RF_NVSwitch_0_ResourceHealthRollup_Status`^[5]	state
`RF_NVSwitch_0_ResourceHealth_Status`^[5]	state
`RF_InterswitchPort_0_ResourceHealthRollup_Status`	state
`RF_InterswitchPort_0_ResourceHealth_Status`	state
`RF_PCIe_0_ResourceHealthRollup_Status`	state
`RF_PCIe_0_ResourceHealth_Status`	state
`RF_PCIe_DeviceHealthRollup_Status`	state
`RF_PCIe_DeviceHealth_Status`	state
`RF_PDB_0_HSC_0_Cur_0`	state
`RF_PDB_0_HSC_0_Pwr_0`	state
`RF_PDB_0_HSC_0_Temp_0`	state
`RF_PDB_0_HSC_0_Volt_In_0`	state
`RF_PDB_0_HSC_0_Volt_Out_0`	state
`RF_PDB_0_HSC_1_Cur_0`	state
`RF_PDB_0_HSC_1_Pwr_0`	state
`RF_PDB_0_HSC_1_Temp_0`	state
`RF_PDB_0_HSC_1_Volt_In_0`	state
`RF_PDB_0_HSC_1_Volt_Out_0`	state
`RF_PDB_0_Inlet_Temp_0`	state
`RF_PDB_0_Vreg_0_Temp_0`	state
`RF_PDB_0_Vreg_0_Temp_1`	state
`RF_PDB_0_Vreg_1_Temp_0`	state
`RF_PDB_0_Vreg_1_Temp_1`	state
`RF_ProcessorModule_0_CPU_0_CpuFreq_0`^[6]	state
`RF_ProcessorModule_0_CPU_0_Energy_0`^[6]	state
`RF_ProcessorModule_0_CPU_0_EnforcedEDPc_0`^[6]	state
`RF_ProcessorModule_0_CPU_0_EnforcedEDPp_0`^[6]	state
`RF_ProcessorModule_0_CPU_0_Power_0`^[6]	state
`RF_ProcessorModule_0_CPU_0_TempAvg_0`^[6]	state
`RF_ProcessorModule_0_CPU_0_TempLimit_0`^[6]	state
`RF_ProcessorModule_0_MemCntl_0_Freq_0`^[6]	state
`RF_ProcessorModule_0_Vreg_0_CpuPower_0`^[6]	state
`RF_ProcessorModule_0_Vreg_0_CpuVoltage_0`^[6]	state
`RF_ProcessorModule_0_Vreg_0_SocPower_0`^[6]	state
`RF_ProcessorModule_0_Vreg_0_SocVoltage_0`^[6]	state
`RF_ProcessorModule_0_CPU_0_CoreUtil_0`^[6]	state
In the preceding row, the substring `CoreUtil_0` can also take values of
`CoreUtil_1`, `CoreUtil_2`…, up to `CoreUtil_71`
`RF_StorageBackplane_0_SSD_0_Temp_0`^[7]	state

[[1]]The number used in the substring Chassis_0 and the substring FAN_ can vary

[[2]] The substring GPU_0 can be GPU_1, GPU_2…

[[3]] Produced by redfish_leak data producer

[[4]]The parameter substring acp0 can be acp1, acp2…

[[5]]The substring NVSwitch_0 can be NVSwitch_1, NVSwitch_2…

[[6]]The substring ProcessorModule_0 can be ProcessorModule_1, ProcessorModule_2…

[[7]]The substring SSD_0 can be SSD_1, SSD_2…, and

the substring StorageBackplane_0 can be StorageBackplane_1, StorageBackplane_2…

12.10 Health Checks For The DGX GB200

Redfish health checks that may be additionally available on the DGX GB200 platform are shown in table 12.10.

As is usual with hardware, the health checks can change according to hardware and especially with firmware changes. The table should therefore be regarded as an indication of the health checks that are available, and not a list of the health checks that must exist.

The list of Redfish health checks that are there can be found by running:

Example

[basecm11->monitoring->measurable]% list healthcheck

Table 12.10: DGX GB200 Health Checks
DGX GB200 Health Check	Parameter	Description
`gpu_health_driver`	gpu0	Driver-related status
`gpu_health_hostengine`		Host engine status
`gpu_health_inforom`	gpu0	Inforom status
`gpu_health_mcu`	gpu0	Microcontroller unit status
`gpu_health_mem`	gpu0	Memory status
`gpu_health_nvlink`	gpu0	NVLINK system status
`gpu_health_nvswitch_fatal`	gpu0	NV switch fatal errors
`gpu_health_nvswitch_non_fatal`	gpu0	NV switch non-fatal errors
`gpu_health_overall`		Overall system status
`gpu_health_overall`	gpu0	Overall system status for GPU0
`gpu_health_pcie`	gpu0	PCIe system status
`gpu_health_pmu`	gpu0	Power management unit status
`gpu_health_power`	gpu0	Power status
`gpu_health_sm`	gpu0	Streaming multiprocessor status
`gpu_health_thermal`	gpu0	Temperature status
`nvsm_pci_health`		Checks health of PCI devices in a DGX system
`nvsm_show_health`		Checks health of DGX system
`NMXController`		Is NMX Controller in the correct state?

12.1 Circuit-Related DGX GB200 Metrics

12.2 Leak Detection DGX GB200 Measurables

12.3 DGX GB200 NVLink Measurables

12.4 DGX GB200 Power Shelf Measurables

12.5 Cooling Distribution Unit Metrics On The DGX GB200

12.6 GPU Measurables For The DGX GB200

12.7 Prometheus Metrics For The DGX GB200

12.8 Redfish Metrics For The DGX GB200

12.9 Redfish Enums For The DGX GB200

12.10 Health Checks For The DGX GB200