Power and Thermals#

This chapter provides information about CPU power and thermal management settings on Vera.

C-States#

C-States refer to idle CPU power states. Use C-state controls when idle power or wake up latency matters. Deep idle states save more power but add exit latency when work resumes.

Vera has the following C-states:

  • LPI-0 (Low-Power Idle 0): Enhanced clock gated state

    This state is entered when WFI/WFE instructions are executed by the CPU core. In addition to clock gating, the core voltage and frequency are reduced. Core context is preserved.

  • LPI-1: Power gated state

    Core power gated state. Context is lost. Best suited for longer idle durations.

The LPI feature in Advanced Configuration and Power Interface (ACPI) tables provides information about the C-states to the OS. On Linux, the information is used by the cpuidle framework.

Vera supports SMT2 where each CPU core contains two Processing Elements (PE).

LPI states describe a CPU core’s power state. However, the kernel’s cpuidle framework exposes states per PE. This is because each PE independently requests a state using WFI/WFE, so the hardware enters the shallowest LPI state that is common to both PEs on the core. If only one PE on a core is in WFI/WFE and the other PE is active, the physical core remains active, and no LPI state is entered.

For systems that do not use cpuidle governors, CPU cores can still enter the LPI-0 state when WFI/WFE instructions are executed, but no stats will be available.

C-State sysfs Mapping and Metrics#

The following table provides the default mapping of the C-State to the sysfs node directory that contains information on desc, time, and usage statistics for the power state. However, you should always verify the mapping on the end system using the description in the directory.

Default mapping of C-States to sysfs directories#

C-State

(Expected) sysfs node directory

LPI-0

/sys/devices/system/cpu/cpuN/cpuidle/state0/

LPI-1

/sys/devices/system/cpu/cpuN/cpuidle/state1/

To confirm state mapping, run the following command:

cat /sys/devices/system/cpu/cpuN/cpuidle/state0/desc

For systems that have the cpuidle framework enabled, state metrics are collected through the idle framework and can be read through sysfs.

  • To read the number of state0 selections for cpuN, run the following command:

    cat /sys/devices/system/cpu/cpuN/cpuidle/state0/usage

  • To read the time spent (in microseconds) in state0 for cpuN, run the following command:

    cat /sys/devices/system/cpu/cpuN/cpuidle/state0/time

Note

There are some considerations to be made regarding LPI-1.

  • LPI-1 metrics reflect the core level state, not the PE level state, so the metrics for a logical core will only update when the core level state is triggered (for example, when both PEs on a core enter LPI-1). For LPI-0, they reflect the PE level state.

  • The LPI-1 state might cause performance degradation in latency sensitive applications because of the latency required to exit from the state. Refer to Enabling and Disabling C-States for more information about disabling the state.

Enabling and Disabling C-States#

C-state enablement and disablement can be controlled globally and per-CPU.

  • Global control

    State enablement can also be controlled globally with the cpupower tool, and through the disable node in the sysfs directories above.

    • To install the cpupower tool, run the following command:

      sudo apt install linux-tools-common linux-tools-`uname -r`

    • To disable the LPI-1 state on all CPUs, run one of the following commands:

      • sudo cpupower idle-set -d 1

      • echo 1 | sudo tee /sys/devices/system/cpu/cpu*/cpuidle/state1/disable

    • To enable the LPI-1 state on all CPUs, run one of the following commands:

      • sudo cpupower idle-set -e 1

      • echo 0 | sudo tee /sys/devices/system/cpu/cpu*/cpuidle/state1/disable

  • Per-CPU control

    You can enable LPI states per-PE through the disable node in the sysfs directories above.

    • To disable the cpuidle governor from selecting LPI-1 on cpuN, run the following command:

      echo 1 | sudo tee /sys/devices/system/cpu/cpuN/cpuidle/state1/disable

    • To enable the cpuidle governor to select LPI-1 on cpuN, run the following command:

      echo 0 | sudo tee /sys/devices/system/cpu/cpuN/cpuidle/state1/disable

Note

LPI-0 cannot be truly disabled. It can be disabled in the cpuidle subsystem, but it can still be triggered in other ways. If no other valid state is available, the kernel will put PEs into WFI. In these cases, the hardware mechanism for LPI-0 will trigger, which results in a lower voltage and frequency.

P-States#

P-States refer to performance states, and Vera does not offer explicit P-states.

Vera exposes the maximum and minimum performance capabilities through ACPI’s Collaborative Processor Performance Control (CPPC) mechanism. CPPC offers users and operating systems the ability to request any performance within the allowed bounds of a power profile. Refer to CPU Performance and Frequency Management for more information.

CPU Performance and Frequency Management#

Each CPU core has one frequency source that is shared by its two PEs, so the PE frequency cannot be independently set. The final frequency is the maximum of the frequencies requested by the two PEs in a core.

Core frequency can be determined and/or set in the following ways:

  • Using an OS-based frequency governor

  • Hardware-autonomous frequency selection

  • User overrides

The following sections provide more information about these methods.

Hardware-Autonomous Frequency Selection#

Vera supports hardware-based DVFS (HW DVFS) for each core, which allows CPU cores to automatically and independently adjust their frequency based on usage and other factors. The frequency determination and change can be at a timescale that is much faster than OS-based governors, which allows the core to be more efficient. Vera supports CPPC, which can be enabled in hardware autonomous mode. To enable HW DVFS, set both PEs to the same mode. To maintain an efficient frequency control of the core you can, for example, enable or disable autonomous mode on both PEs.

Vera supports HW DVFS through the CPPC Autonomous Selection functionality, and it can be enabled by using the cppc_cpufreq.auto_sel_mode kernel boot arg. When hardware autonomous selection and an OS-based governor are both enabled, hardware autonomous selection takes precedence.

Boot-time configuration:

Here are the valid values for cppc_cpufreq.auto_sel_mode:

  • performance (or 1): override Energy Performance Preference (EPP) to performance (0x0)

  • balance_performance (or 2): override EPP to balance_performance (0x80)

  • default_epp (or 3): preserve EPP value programmed by BIOS/firmware

For example, the following to the kernel command line will enable autonomous selection with an EPP of 0x0 for all cores at boot:

cppc_cpufreq.auto_sel_mode=performance

This sets the initial state of the auto_select knob for all cores before userspace starts, but EPP values can also be modified by users at runtime. Refer to Energy Performance Preference for more information.

Runtime configuration: The following sysfs nodes can be used to control autonomous selection properties:

  • Auto selection

    Enables or disables the hardware-autonomous frequency selection.

    sysfs path:

    • /sys/devices/system/cpu/cpuN/cpufreq/auto_select

    Here are the valid values:

    • 1: Enable Autonomous Selection

    • 0: Disable Autonomous Selection

    This knob can be modified at runtime irrespective of the boot-time configuration.

  • Min and max frequency

    Lower and upper frequency bounds in kHz. Hardware will not scale frequency beyond these values.

    sysfs paths:

    • /sys/devices/system/cpu/cpuN/cpufreq/scaling_min_freq

    • /sys/devices/system/cpu/cpuN/cpufreq/scaling_max_freq

    Here are the valid values:

    • scaling_min_freq: integer value in kHz between scaling_max_freq and cpuinfo_min_freq.

    • scaling_max_freq: integer value in kHz between scaling_min_freq and cpuinfo_max_freq.

  • Energy Performance Preference

    Hints to the hardware how aggressively to favor performance versus energy efficiency. Only takes effect when hardware autonomous selection is enabled (auto_select=1).

    sysfs path:

    • /sys/devices/system/cpu/cpuN/cpufreq/energy_performance_preference_val

    Here are the valid values:

    • Integer between 0 and 255. Values closer to 0 favor performance, values closer to 255 favor energy efficiency.

    Examples:

    • 0: Maximum performance (ramp to high frequency aggressively)

    • 128: Balanced

    • 255: Maximum energy efficiency (favor lowest frequency)

OS-Based Frequency Governors#

Linux’s cpufreq subsystem provides many frequency governors, and here are the most commonly used:

  • Performance governor: Geared towards getting the maximum performance and sets the performance/frequency request of the CPU cores to the maximum possible value.

    The request is kept statically fixed at highest value.

  • Powersave governor: Geared towards lowest power consumption while keeping the core still operational.

    The governor statically sets the frequency to the lowest allowed value.

  • Userspace governor: Bypasses a kernel governor and provides control to the userspace application for frequency control.

    To set the frequency of the cores, a hypervisor, or a higher level software entity, can take input from an application. The kernel does not modify the frequency based on other information but will honor frequency caps based on thermals.

  • Schedutil governor: Uses information from the scheduler to determine the optimal frequency for the core.

    The goal of this governor is to provide best performance while saving power by setting the frequency based on scheduler visible workloads.

  • Ondemand governor: Periodically samples the load on the CPU core to determine the optimal frequency.

    This governor reacts to CPU core load and ramps frequency accordingly.

Refer to https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt for more information.

The default frequency governor in Vera is the performance governor, which sets the frequency to the maximum value for that core. The maximum frequency usually corresponds to the maximum possible performance and is higher than the frequency at which nominal (sustained) performance can be achieved. When running at the maximum frequency violates thermals, the thermal management solution throttles frequency. Refer to Power and Thermal Management for more information.

Each PE in a core has its own CPPC interface, which allows the PE to make an independent frequency request. However, the PE’s final frequency is the maximum of the requested value from both PEs because they share a frequency source.

The ACPI _CPC object generically describes processor performance, which is requested on this abstract scale, and the platform maps the request to real values that are understood by hardware. Refer to ACPI Specification 6.6, section 8.4.6 for more information.

CPPC v4#

Vera supports OSPM Nominal Performance Register functionality from CPPC v4, which extends CPPC with an interface to control processor performance. It is defined in the ACPI _CPC object:

OSPM Nominal Performance Register: Enables setting a lower nominal (sustained) performance point than the platform-defined nominal performance. Performance above this programmed value is boosted, and performance below this value is throttled. This value must be in the range between the lowest performance to nominal performance.

CPPC v4 has the following mutually exclusive operating modes on Vera:

  • ACPI (OS-managed) Mode

    OSPM nominal performance registers can be set through the OS.

  • DPRP (HW-managed) Mode

    The system firmware manages resource provisioning.

Both modes write the same CPPC v4 nominal performance register, so only one mode can be active at a time.

  • When the OS should own this policy, use ACPI mode.

  • When firmware should manage resource provisioning, use DPRP.

  1. To enable CPPC v4 in the UEFI menu, select the following option.

Vera Configuration ->
   Power and Performance ->
      Enable CPPC V4 (ACPI 6.6)
  1. After CPPC v4 is enabled, select one of the following modes.

Vera Configuration ->
   Power and Performance ->
      CPPC V4 Mode ->
         ACPI Mode (OS-Managed)
         or
         DPRP (HW-Managed)

ACPI (OS) mode and DPRP controlled CPPC v4 both write the same CPPC v4 nominal performance register so only one mode can be active at a time.

OS Managed#

Note

Support for CPPC v4 is not yet present in upstream Linux. Support will be available in a future BaseOS version.

In this mode, the OSPM Nominal Performance register is exposed in each PE of a core through the /sys/devices/system/cpu/cpuN/cpufreq/ospm_nominal_freq sysfs node. This node can be written to set the nominal performance for the PE to a value between the lowest and nominal performance values defined in the ACPI _CPC object. The value represents the nominal performance target the OS wants hardware to apply when the system becomes power-limited.

  • To read the OSPM nominal frequency for PE N, run the following command:

    cat /sys/devices/system/cpu/cpuN/cpufreq/ospm_nominal_freq

  • To write a new OSPM nominal frequency value (in kHZ) for PE N, run the following command:

    echo *[frequency in kHz]* > /sys/devices/system/cpu/cpuN/cpufreq/ospm_nominal_freq

This enables priority-aware power management in power-limited conditions. By programming different OSPM nominal values for different cores, the software allows the selected cores to retain a higher operating frequency while other cores are throttled more aggressively when system power limits are exceeded.

During a power-limit event, hardware uses the programmed OSPM nominal value as the temporary limit for the affected core.

  • If HW DVFS is enabled, this value becomes the upper bound for autonomous frequency selection.

  • If HW DVFS is disabled, the core is driven directly to this value.

When the event clears, the temporary bound is removed and frequency selection returns to the typical behavior.

This interface is exposed for each PE, but hardware uses the maximum of the two requested values for both PEs in a core. For predictable behavior, program the same value on both sibling PEs.

Dynamic Power and Resource Partitioning (DPRP)#

DPRP is a firmware control layer for CPPC v4 and works by distributing a chip-wide performance budget across the cores in the system. Cores with stronger performance preference receive more of the budget, and lower-priority cores are more aggressively capped.

When DPRP is enabled, the firmware monitors each core’s programmed CPPC settings, such as EPP and min/max performance limits, every 50 ms, and uses the values as inputs to compute a new value and write to the core’s OSPM nominal performance register.

EPP values fall into one of four priority tiers:

  • Tier 0: 0 <= EPP < 64

  • Tier 1: 64 <= EPP < 128

  • Tier 2: 128 <= EPP < 192

  • Tier 3: 192 <= EPP <= 255

DPRP distributes the performance budget across these tiers to grant higher priority tiers (lower EPP) a larger share while remaining within each core’s minimum and maximum limits.

The same min/max/epp sysfs nodes described in Hardware-Autonomous Frequency Selection can be used to steer DPRP behavior per-core.

User Overrides#

Managing the CPU frequency on Linux can be achieved by using the cpupower tool or directly through the sysfs interface. The next section provides information about the methods to set a fixed frequency and a scaling max frequency. Vera supports SMT, so the commands in the next section need to be run for each PE to take effect.

Note

User overrides will only work if HW DVFS is not enabled

Setting a Fixed Frequency#

This section provides information about setting a fixed frequency.

  • The cpupower tool

    This command allows you to set a fixed frequency for all cores.

    1. To manually set the frequency, switch to the userspace governor (if supported).

      sudo cpupower frequency-set -g userspace

    2. Set the frequency (for example, 3.2 GHz = 3200000 kHz).

      sudo cpupower frequency-set -f 3200000

  • The sysfs method

    Sysfs allows you to complete the following tasks:

    1. Switch to the userspace governor (if supported).

      $ echo userspace | sudo tee \
            /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
      
    2. Set the frequency.

      $ echo <desired frequency in kHz> | sudo tee \
            /sys/devices/system/cpu/cpu*/cpufreq/scaling_setspeed
      

Setting a Scaling Max Frequency#

This section provides information about setting a scaling max frequency.

  • The cpupower tool

    1. Switch to performance governor to limit the max scaling frequency:

      sudo cpupower frequency-set -g performance

    2. Set the scaling max frequency (for example, 3.2 GHz = 3200000 kHz):

      sudo cpupower frequency-set -u 3200000

  • The sysfs method

    1. Switch to the performance governor (if supported).

      $ echo performance | sudo tee \
            /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
      
    2. Limit the maximum scaling frequency (replace [MAX_FREQUENCY] with desired value in kHz).

      $ echo *[MAX_FREQUENCY]* | sudo tee \
            /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
      

The following table provides a list of the general CPU frequency commands to read the currently requested and measured settings.

Note

At least one PE in a core should be active to get an accurate frequency reading for that core.

Reading

Command

Notes

Software requested frequency (kHz) for CPU core X

cat /sys/devices/system/cpu/cpuN/cpufreq/scaling_cur_freq

Measured instantaneous frequency (kHz) for CPU core X

cat /sys/devices/system/cpu/cpuN/cpufreq/cpuinfo_cur_freq

Instantaneous readings are susceptible to error due to hardware clocks used to measure them. For accurate readings of CPU frequency, use the cpuinfo_avg_freq node.

Measured average frequency (kHz) for CPU core X

cat /sys/devices/system/cpu/cpuN/cpufreq/cpuinfo_avg_freq

Reads of this node will fail if performed when the core is not active. Core activity is necessary in order to compute an accurate average frequency.

Firmware-Based Dynamic Voltage and Frequency Scaling#

Vera supports firmware-based Dynamic Voltage and Frequency Scaling (DVFS) for some non-CPU system components. The system firmware monitors activity and adjusts their voltage and frequency.

Firmware-based DVFS is supported for the following components:

  • DRAM

  • UCF SLC

  • C2C Hub

These can be enabled per component through the UEFI menu by navigating to the following:

Vera Configuration ->
   Power and Performance ->
      Enable DRAM DVFS Manager
      Enable UCF SLC DVFS
      Enable C2CHUB DVFS Manager

Note

Currently, enabling DRAM DVFS and C2C Hub DVFS through UEFI has no effect. This functionality will be part of a future SBIOS release.

Performance Determinism Modes#

Performance Determinism Modes define power-provisioning mechanisms and are useful when uniform and deterministic behavior across parts matters more than absolute performance. Vera supports the following performance determinism modes that can be selected through the UEFI menu:

Vera Configuration ->
   Power and Performance ->
      Performance Determinism Mode ->
         Both Disabled (Legacy behavior)
         or
         Power Limit Scaling Only
         or
         Perf Determinism (Full mode)

Mode

Description

Both Disabled (Legacy behavior)

Default. No frequency cap and no power-limit scaling. Highest peak performance but most possible variation.

Power Limit Scaling only

Enables power-limit scaling but leaves frequency uncapped. Normalizes power limit per part for predictable power/thermal behavior across all parts, while allowing cores to boost if there is headroom. System power budget is limited, so sustained performance might be lower than Legacy.

Perf Determinism (Full mode)

Frequencies and power-limits are both scaled. Frequency is limited so it can be achieved across all parts when power is not constrained, and power limits are scaled as described above. Provides the most repeatable performance and power across parts and is useful for performance guarantees at the cost of peak performance.

Power and Thermal Management#

Vera provides the following types of thermal management support:

  • Limit power consumption to stay within the power limit.

  • Thermal sensor (Tj)-based management.

More information about these thermal management types is provided in the following sections.

Power Telemetry#

This section provides information about power telemetries for Vera and guidance to compare Vera power telemetry to x86 power telemetry.

Vera Power Telemetry#

Vera exposes power and related telemetry through hwmon, using the ACPI power meter interface. You can read this telemetry in the following ways:

Reading Power Telemetry#

  • To see hwmon sysfs nodes, set CONFIG_SENSORS_ACPI_POWER=m in your kconfig.

  • To display the name of the power meter (telemetry name).

    This provides information about which value is being reported by the hwmon node X.

    cat /sys/class/hwmon/hwmonX/power1_oem_info

  • To display the power stats interval in milliseconds, on hwmon node X:

    cat /sys/class/hwmon/hwmonX/power1_average_interval

  • To display average value over past interval on hwmon node X:`

    cat /sys/class/hwmon/hwmonX/power1_average

  • To change the interval in milliseconds, on hwmon node X:

    echo <value in ms> | sudo tee /sys/class/hwmon/hwmonX/power1_average_interval

    • Vera supports 50 ms (default) and 1000 ms averaging intervals for the hwmon nodes that report power. Other nodes are fixed at 50 ms.

  • To dump all available ACPI Power Meter telemetry:

    for x in /sys/class/hwmon/hwmon*/power1_average; do \
        info=${x/power1_average/power1_oem_info}; \
        cat "$info" | tr '\n' ' '; \
        echo ": $x = $(cat "$x")"; \
    done
    

Available Power Telemetries#

All telemetries are replicated per-socket. When read from hwmon, the name of each telemetry will include the socket it belongs to. All telemetries are replicated per-socket. When read from hwmon, the name of each telemetry will include the socket to which it belongs. The following table provides information about available power telemetries.

Vera hwmon telemetries#

Type

Hwmon Name

Details

Power

Total Input Power in uW

Total power of the socket. This includes Vera CPU power, SOCAMM power, and regulator loss. This is the Total CPU Power (TCP). This is the domain in which power capping is performed.

Power

AP Input Power in uW

Total power of Vera including regulator loss. This includes the power consumed by all of its rails. This telemetry is not present on all platforms.

Power

AP Output Power (TDP) in uW

Total power of Vera after regulator losses are removed. This is the TDP of Vera, and includes the power consumed by all of its rails.

Power

CPU Rail Input Power in uW

CPU rail power, including regulator loss. This includes the power consumed by the Vera CPU Olympus Cores and the Scalable Coherency Fabric (SCF).

Power

SoC Rail Input Power in uW

SoC rail power, including regulator loss This includes the power consumed by the Vera System Die.

Power

DRAM Input Power in uW

DRAM power, including regulator loss This includes the power consumed by SOCAMM modules.

Power

CPU Rail Output Power in uW

CPU rail power after regulator losses are removed. This includes the power consumed by the Vera CPU Olympus Cores and the SCF.

Energy

Total CPU Energy in uJ

Energy consumed by CPU rail, including regulator losses

Regulator Temperature

CPUVDD Vtg Regulator Temperature in uC

CPU rail regulator temperature

Regulator Temperature

SOCVDD Vtg Regulator Temperature in uC

SoC rail regulator temperature

Regulator Temperature

MSSVDD Vtg Regulator Temperature in uC

MSS rail regulator temperature

Throttle Counter

Chipthrot TCPc Throttle (samples x1000)

Number of telemetry sampling windows (50 ms) where TCPc (continuous TCP) violations caused throttling to occur.

Throttle Counter

Chipthrot TJMAX Throttle (samples x1000)

Number of telemetry sampling windows (50 ms) where TJMAX violations caused throttling to occur

Throttle Counter

Chipthrot TJAVG Throttle (samples x1000)

Number of telemetry sampling windows (50 ms) where TJAVG violations caused throttling to occur

The following graphic shows the topology of power telemetry sensors on Vera:

_images/vera-power-telemetry.png

Comparing Vera and x86 Power Telemetry#

Vera exposes many power telemetries that are comparable to telemetry that x86 exposes.

Note

x86 TDP is equivalent to the AP Output Power telemetry on Vera, so you can use this telemetry for comparisons to x86 TDP.

The following table can be used to map x86 telemetries to their Vera counterparts. References to any Sensors in this table can be resolved by referring to the diagram above (link).

Vera vs x86 power telemetry#

x86 Power Term

What the x86 term covers

Vera Equivalent

Where the Vera term comes from

Processor Base Power / TDP (Intel PBP, AMD Default TDP MSR register)

Package only: cores + uncore + IO. Excludes DRAM and board regulator losses. TDP is a thermal spec for x86, but it includes FIVR losses.

AP Output power

Power of Vera (TDP) as shown in the diagram above

Core Power (Intel RAPL PP0 MSR register)

Cores + caches only

CPU Rail Output Power

CPU Rail Output Power measured on Sensor 5

DRAM Power (RAPL DRAM domain)

Off package DIMMs on the motherboard

DRAM Input Power

On-module LPDDR measured on Sensor 4 ; at the input of the regulator.

No standard x86 term/sensors. Requires board instrumentation

Power drawn by entire socket: die + DRAM + SOC/IO + regulator losses

Total Input Power (TCP)

Total CPU Power including CPU and DRAM measured at input of regulator on Sensor 0. This is what Vera specs and rack sizing are built on.

Socket Power Cap (Intel PL1/PL2, AMD PPT)

Enforced power limit on the package power domain

TCP Cap

Enforced on the Sensor 0 “module”/TCP domain - a larger domain than x86 PL1, so the number is larger. This is the TCP, which includes the CPU and DRAM measured at the input of the regulator.

Comparing Vera and Grace Power Telemetry#

Vera and Grace expose power telemetry through the ACPI power meter interface, but the names of the telemetries are slightly different.

Telemetry name

Description

Grace power meter name

Vera power meter name

Module power

Total power including CPU power, DRAM power, and regulator losses

Grace Power Socket X

Total Input Power in uW Socket X

CPU rail input power

CPU rail input power including regulator losses

CPU Power Socket X

CPU Rail Input Power in uW Socket X

CPU rail output power

CPU rail power after regulator losses are removed

N/A

CPU Rail Output Power in uW Socket X

SoC rail power

SoC rail input power including regulator losses

SysIO Power Socket X

SoC Rail Input Power in uW Socket X

DRAM power

Power consumed by DRAM, including regulator losses

N/A

DRAM Input Power in uW Socket X

Refer to the following Grace power sensor topology diagram for more information.

_images/grace_superchip_telemetry_sensors.png

Using the Throttle and Energy Counters#

Vera’s energy and throttle counters provide insights into system behavior that cannot be achieved only through power metrics. The CPU energy counter can provide an accurate accounting system of energy consumed over time by providing snapshots of accumulated energy consumption. This counter can provide a picture of CPU power consumption without having to continuously sample the power meters.

Average power can be computed by taking energy snapshots before and after running a workload as follows (assuming the CPU energy counter is hwmon0):

cat /sys/class/hwmon/hwmon0/power1_average  # 3164784457000
stress-ng -c 150 -t 10 # run stress-ng CPU stressor on 150 cores, for 10 seconds
cat /sys/class/hwmon/hwmon0/power1_average  # 3169849339000

While this workload was running, the CPU consumed 3169849339000 uJ - 3164784457000 uJ = 5064882000 uJ = 5064.88 J, which is an average of 5064.88 J / 10 s = 506.488 W over the 10s for which it ran.

The throttle counters provide insight into why a stressful workload’s performance might not be as high as expected. If the throttling conditions are met, power to the CPU will be reduced. The following example illustrates TCPc throttling (assuming hwmon2 is “Total Input Power in uW socket 0”, and hwmon0 is “Chipthrot TCPc Throttle (samples x1000) socket 0”):

cat /sys/class/hwmon/hwmon2/power1_cap      # 500000000 - system is power limited to 500 W
cat /sys/class/hwmon/hwmon0/power1_average  # 4454000 (4454)
stress-ng -c 150 -t 10                      # run stress-ng CPU stressor on 150 cores, for 10 seconds
cat /sys/class/hwmon/hwmon0/power1_average  # 4648000 (4648)

While this workload was running, 4648 - 4454 = 194 throttling events took place because of TCPc throttling. The number of “throttling events” is equal to the number of telemetry sampling windows (50 ms), where at least one throttling event occurred. This does not indicate any information about the depth or frequency of the throttling, only that it occurred.

Power Capping#

Power capping limits average power consumption and is usually set based on the thermal power dissipation capability of the system. On Vera, power capping is performed at the TCP level. When TCP (Sensor 0 reading, diagram link) exceeds the power cap, throttling is performed to maintain an average TCP within the power cap.

This setting is exposed through the TCP hwmon nodes (“Total Input Power in uW”), and users can reduce the power limit lower than the default value that was set in the BIOS.

Power capping can be applied only at the TCP level, not at the VDD_CPU or VDD_SOC levels.

To set power limit for socket N, run the following command:

echo <power value in micro Watts> > /sys/class/hwmon/hwmonX/power1_cap

where /sys/class/hwmon/hwmonX/power1_oem_info is “Total Input Power in uW socket N”.

Writes to this sysfs node are not persistent across reboots.

On Vera Rubin (CPU + GPU) systems, capping the Vera CPU power allows the Rubin GPU to draw more power. This can improve the performance of GPU-heavy applications.

CPU Temperature Management#

Thermal management (Tj) on Vera uses telemetry from temperature sensors to ensure that no local hotspots exceed the operating temperature. Power capping ensures that the average power of the socket/module is at or below the thermal capacity of the system. However, this does not account for asymmetric power distribution based on workload distribution across the cores.

On Vera, thermal management occurs in the hardware and firmware. Thermal controllers monitor temperature and perform throttling to ensure temperatures remain within acceptable operating range.

These operating ranges are used by the hardware and firmware and are not exposed to the OS.

ACPI does expose critical temperature limits for thermal zones. If the temperature of a thermal zone exceeds its critical limit, a shutdown is initiated.

Vera has the following thermal zones:

  • Thermal Zone Compute TJMax Socket 0: Max CPU temperature

  • Thermal Zone Compute TJAvg Socket 0: Average CPU temperature

  • Thermal Zone DRAM TJMax Socket 0: Max DRAM temperature

To read the critical limit of thermal zone N:

cat /sys/class/thermal/thermal_zoneN/trip_point_0_type

cat /sys/class/thermal/thermal_zoneN/trip_point_0_temp