Power and Thermals#
Vera-Rubin NVL72 combines CPU/GPU power steering, workload profiles, rack power smoothing, and liquid cooling. Record the active controls when comparing performance: a GPU’s power ceiling alone does not describe the power available to it during a run. Use the supported controls in the qualified firmware, driver, and management stack for the installed system.
Workload Power Profiles#
Workload Power Profiles provide presets for application behavior. Discover the profiles supported by the installed GPU and management software, and check both the requested and enforced profiles; conflicting requests might not all be enforced. A profile exposed by an API is usable only when the deployed platform release supports it.
The public NVIDIA-SMI documentation describes profile discovery and conflict metadata. The Dynamic Power Software Redfish reference documents the Vera-Rubin power model. Check the management release’s support matrix before applying profile policies; support can differ between the GPU, firmware API, and cluster manager.
HBM Clock Tuning For LLM Inference#
The prefills and throughput regions of generation phase in LLMs use large GEMMs because of the large sequence length and large batch sizes respectively.
Here is some additional information about large GEMMs:
The math utilization is high.
The HBM utilization is low.
In these cases, to help with power availability to the Tensor Cores improving overall performance, lower the HBM clock to 3.2GHz. Low latency generation phase inference typically has smaller GEMMs, and these GEMMs are HBM bandwidth limited, so a lower HBM clock will affect the performance.
Power Smoothing#
Intelligent Power Smoothing uses energy storage and state-of-charge awareness to reduce fast changes in rack input power. Capture its enabled state and effective profile with the benchmark. Coordinate smoothing, ramp controls, and administrative overrides with the rack power controller and facility policy. Use the system’s qualified power-shelf and firmware settings.
Measure sustained power and transient peaks over their specified averaging windows. GPU telemetry, tray DC power, and rack AC input describe different measurement points and should not be substituted for one another when sizing electrical or cooling infrastructure.
Power Balancing/Steering#
GPU Base Power is the maximum power the GPU can draw without steering power from another component. Available CPU headroom can let the GPU operate above that base, subject to its ceiling and the active system policy. Base power is not a guaranteed minimum power draw.
Use GPU Base Power for provisioning when directed by the platform power policy, and retain the qualified GPU ceiling to preserve steering headroom. Vera-Rubin uses per-device CPU and GPU controls; do not transfer processor-module power limit recipes from earlier platforms. Ensure the cluster manager and local tools agree on which controller owns each limit.
Cooling And Measurement#
The reference compute and switch trays are fanless and fully liquid cooled. NVIDIA describes operation with up to 45 degrees Celsius liquid inlet temperature; use the system vendor’s validated temperature, coolant, flow, and pressure envelope for the actual rack. Warm-water capability does not replace flow, leak, and thermal-throttling checks.
For each comparison, record CPU/GPU base and ceiling settings where supported, enforced workload profiles, rack power budget, power-smoothing state, coolant conditions, and clock-throttling events. Evaluate application throughput and energy per unit of work as well as peak throughput. See NVIDIA’s Rubin architecture overview for the published rack power and cooling design.