6.1.1.1. Power and thermal monitoring on Intel platforms

On Intel platforms, node manager (NM) and running average power limit (RAPL) are the fundamental interface reporting power. Figure 6.1 presents the different components for reporting AC and DC node power on a Lenovo dense ThinkSystem server.

PCH is the platform controller hub (i.e. south bridge), ME is the management engine (embedded in PCH) that runs Intel NM firmware and HSC is the hot swap controller that provides power readings.

On Lenovo ThinkSystem servers, XCC (xClarity Controller) runs the node-level system management. XCC is based on a baseboard management controller (BMC) using a dual-core ARM Cortex A9 service processor. It monitors DC power consumed by the node (as a whole and by the CPU and memory subsystems), it monitors inlet air temperature for the node and it caps DC power consumed by the node as a whole. It also monitors the CPU and memory subsystem throttling caused by node-level throttling and enables or disables power savings for node.

FPC is the fan/power controller for the chassis-level systems management. It monitors AC and DC power consumed by individual power supplies and aggregates to chassis level. It also monitors DC power consumed by individual fans and again aggregates to chassis level.

As shown in Figure 6.1, the information reported by the node-level system management can be reported in-band where the measurement is done on the node itself, or out-band where the measurement is requested by another node such as the cluster manager.

Figure 6.2 presents the reporting frequencies at the different levels from sensors to the application for NM and RAPL power management flows.

For NM, the sensor level sampling rate is 1 KHz and the node power reported at the application level has a 1Hz frequency. For RAPL, the sampling from the energy Model Specific Registers (MSRs) for CPU and memory is 500 Hz and the reporting frequency is 200 Hz.

Therefore, as RAPL only reports CPU and memory power, the DC node power sampling frequency is only 1 Hz. This limits the scope of DC node power measurements to large granularity, such as job level, and is clearly not enough fine grain to measure DC node power at the subroutine or loop level. To achieve such fine grain measurements, other solutions have been implemented on top of NM (Hackenberg et al. 2014; Benini 2018; Libri et al. 2018).

An example of such a solution has been developed by Lenovo for the ThinkSystem SD650 with a new circuit and data flow. Figure 6.3 presents the new data flow and the reported frequency at each level, leading to a 100Hz DC node power frequency.

Figure 6.4 shows the circuits that provide both the usual NM interface and this new high accuracy DC node power measurement through an Intelligent Platform Management Interface (IPMI) raw command (see section 6.1.1.3).

Figure 6.5 presents the percentage of error of the power readings of the different circuits we have presented, depending on the power load in the node. In the light-yellow rectangle on the left side of Figure 6.5, the server is idle or very little used, while on the light-yellow rectangle at the right side of the Figure 6.5 the server is highly loaded. When the node is idle or little used, the error is dominated by the offset error, while when the node is highly loaded the error is dominated by the gain error. The green, blue and orange curves present the error with three different circuits: green and blue curves represent the error for NM readings on an industry standard solution based on circuits such as TI LM25066, Analog Devices ADM1278 or Maxim VT505 (green) and on Lenovo NeXtScale nx360 m5 (blue). The orange curve represents the error on Lenovo SD650 with the high accuracy circuits presented in Table 4.15. This shows that these new circuits report not only higher frequency, but also much better accuracy.

RAPL stands for running average power limit. It is the low-level interface used by Intel to collect energy information from the CPU and memory subsystems. NM consolidates the raw energy data and converts it to power by subtracting two energy readings and dividing by the time between the two readings.

6.1.1.2. Circuits and interface for monitoring power and thermal on NVIDIA GPUs

The basic tool for power and thermal monitoring on NVIDIA graphics processing units (GPUs) is the NVIDIA Management Library (NVML 2019). This is an API, based on C language, for monitoring and managing states of NVIDIA GPU devices, such as GPU utilization rate, running process, clock and performance, state, temperature and fan speed, power consumption and power management. Its sampling rate is ~500 Hz and the power accuracy for the entire board is ±5 W. NVIDIA System Management Interface (NVSMI) is a command line utility based on NVML to aid in the management and monitoring of NVIDIA GPU devices (NVSMI n.d.). NVSMI is a high-level command line interface and has a low sampling rate (~1 Hz) (Ferro et al. 2017).

NVML has query commands to query the state of the different GPU devices such as:

• – ECC error counts: both correctable single bit and detectable double bit errors are reported. Error counts are provided for both the current boot cycle and for the lifetime of the GPU;
• – GPU utilization: current utilization rates are reported for both the compute resources of the GPU and the memory interface;
• – active compute process: the list of active processes running on the GPU is reported, along with the corresponding process name/id and allocated GPU memory;
• – clocks and P-State: max and current clock rates are reported for several important clock domains, as well as the current GPU performance state.
• – temperature and fan speed: the current core GPU temperature is reported, along with fan speeds for non-passive products;
• – power management: for supported products, the current board power draw and power limits are reported.

NVML also has devices commands, like setting the power management limit (nvmlDeviceSetPowerManagementLimit), also called power cap elsewhere in this book, or setting the clock frequencies of the memory of graphics devices (nvmlDeviceSetApplicationsClocks) at which the compute and graphics applications will be running.

6.1.1.3. IPMI

IPMI is the major standard to request and collect information from IT devices (IPMI 2009). IPMI commands are a convenient low-level method to get sensor readings from monitored devices.

Out-of-band commands collect information through an access server or card (such as a BMC), which is connected to a management port of the monitored devices. In-band commands collects information on the device itself. IPMI raw commands are specified in hexadecimal value and are specific to the vendor.

Examples of IPMI commands like “IPMI-DCMI power set limit” to set a power limit in watts or “IPMI-sensors” to get CPU temperature, voltage and fan speed are provided by freeIPMI (2018)¹.

The following IPMI raw command reads the energy from the NM on SD650: “ipmitool raw 0x2e 0x81 0x66 0x4a 0x00 0x20 0x01 0x82 0x0 0x08”.

Such a command has a reporting rate of ~1 Hz with NM on Intel platforms. A similar reporting rate is achieved with NVSMI on NVIDIA GPUs.

The following IPMI raw command will retrieve energy from the high sampling rate sensors on SD650 with a reporting rate of ~100 Hz: “ipmitool raw 0x3a 0x32 4 2 0 0 0”.

For higher sampling and reporting rates (300–500 MHz), RAPL for Intel platforms and NVML for NVIDIA GPU should be used.

6.1.1.4. Power API

IPMI is a low-level interface to get information but it does not abstract the system and the relation between all its components, which is the goal of power API.

Power API (Grant et al. 2016) is the result of collaboration among national laboratories, universities and major HPC vendors to provide a range of standardized power management functions, from application-level control and measurement to facility-level accounting, including real-time and historical statistics gathering.

Power API describes a system by a hierarchical representation of objects (cabinet, chassis, node, board, power plane, core). It defines roles that interact with the system. Each object has certain attributes (e.g. power cap, voltage) that can be accessed depending on the role of the requester. Get/set functions enable basic measurement and control for the exposed object attribute.

Support is already available for Intel and AMD CPUs with the support of different hardware vendors like Cray, HPE, IBM and software vendors like Adaptive Computing.

6.1.1.5. Redfish

Redfish is a standard led by the Distributed Management Task Force (DMTF n.d.). The goal of Redfish is to replace and extend IPMI to deliver simple and secure management for converged, hybrid IT and the Software Defined Data Center, both human readable and machine capable. The main advantage over IPMI is its RESTful API. Redfish v1.0 is supported by all the major IT vendors.

It allows the user to retrieve basic server information and sensor data (like temperatures, fans, power supply) and facilitates remote management tasks such as reset, power cycle and remote console. Until now, the Redfish specification has covered only IT equipment. However, work is in progress to extend its scope to cover power with the Data Center Equipment Schema Bundle².

6.1.2.1. Performance monitoring on Intel platforms

Hardware performance counters are specific registers built in the processor to store the count of hardware events like number of instructions executed, number of floating point/integer instructions executed, number of cycles, number of L1/L2 cache misses and so on. As the number of hardware counters is limited, prior to using the counters, each counter has to be programmed with the index of the event type to be monitored (Intel 2017).

The Performance Application Programming Interface (PAPI, see Terpstra et al. 2010) is an open source machine independent set of callable routines that provides access to the hardware performance counters on most modern processors like x86, ARM, Power and GPUs. It also provides access to RAPL functions for Intel x86.

In Chapter 5, we used cycles per instructions (CPI) [5.1] and gigabytes of memory read or written per second (GBS) [5.2] to characterize application performance. CPI was computed with the PAPI core events as:

[6.1]

GBS was computed with the uncore events (Intel 2012) of the Integrated Memory Controller (IMC) as:

[6.2]

On Intel platforms, the Intel Performance Tuning Utility (ptu or ptumon) is an easy way to collect performance temperature and power metrics reported in Chapter 5, although it is not officially supported by Intel anymore.

High-level tools are also available to visualize and help the developer to analyze the performance of workload running on a system. Some are proprietary like Intel Vtune³ and Intel Advisor⁴ who run on Intel processors and like ARM Forge and ARM Performance Reports⁵ who run on Intel, AMD, ARM, OpenPower processors and NVIDIA GPU accelerators. There are also many open source tools such as gprof, TAU, Vampir, Paraver, Scalasca, Periscope and so on.

6.1.2.2. Performance monitoring on NVIDIA platforms

NVIDIA GPUs also have hardware performance counters to monitor the activity of the multiple functional units (NVIDIA 2015).

The NVIDIA CUDA Profiling Tools Interface (CUPTI) provides performance analysis tools with detailed information about GPU usage in a system. CUPTI is used by performance analysis tools such as nvprof, NVIDIA Visual Profiler, NVIDIA NSight, TAU and Vampir Trace.

PAPI CUDA is a PAPI version available on NVIDIA CUDA platform, which provides access to the hardware counters inside the GPU. PAPI CUDA is based on CUPTI and provides detailed performance counter information regarding the execution of GPU kernels.

NVIDIA Nsight Systems is a system-wide performance analysis tool designed to visualize an application’s algorithms. It helps to identify the largest opportunities to optimize, and tune to scale efficiently across any number of CPUs and GPUs.

Open source high-level tools such as TAU, Vampir and Scalasca are also available on NVIDIA GPUs.

6.2.2.1. Application-specific models

Predicting the performance of a given algorithm on a specific system has been done for many years where the application execution time is split into computation, communication and I/O time. Each of these is then described by a simple equation related to its algorithmic complexity (number of operation to be performed) and some hardware characteristics of the system (time to compute a DP operation, to send a DP or SP word or to read/write one byte or one record to storage, etc.). Many examples of such techniques for specific numerical algorithms exist from early examples (Keyes and Gropp 1986; Brochard 1989) up to more recent examples (Bonfa et al. 2018).

For example, the elapsed time is split as:

where T_total is the total elapsed time, T_comp is the computation time, T_comm the communication time and T_io the I/O time, with no overlapping assumed between the different tasks.

For a matrix multiplication of rank n, computation time in ms would be:

where GFlops is evaluated as in Table 5.2 for the Intel Xeon 6148 taking into account the type of instructions executed. For a matrix multiplication when using a tuned BLAS library (NVIDIA Developer n.d.; Gennady and Shaojuan 2018), it should be GFlops for AVX-512 FMA, while if the code is written by hand with no compiler, optimization GFlops should be taken as SSE2 ADD or MULT. This simple example shows clearly how difficult it is to select the right parameters since they highly depend on the type of instructions executed.

Similar work has been done regarding power consumption by splitting the total power of the device across its different components (CPU, memory, storage, etc.) and eventually splitting each component power into static and dynamic power. But the coefficients cannot be expressed by simple arithmetic formulas. This drawback is corrected by the predictive methods as described in section 6.2.3.

6.2.2.2. Application signature/surrogate-based performance models

Other performance prediction methods have introduced some level of abstraction to characterize the applications (application signature) and the system hardware (system signature) at a higher level than using the application and system characteristics. Examples of such approach are Todi et al. (2007) and Snavely et al. (2013). Another approach is to decompose an application behavior as a linear equation of surrogates, which are small enough such that they have been projected by cycle-accurate simulators and which represent the workloads to be projected (Sameh et al. 2012).

6.2.3.1. Statistical power models

Several authors have worked to build linear power models where coefficients are calculated by best fit (or linear multiregression) statistical methods. With such models, accuracy is limited between 5% and 10%, but given their simplicity they can be used to make real time decisions.

An example of such a model is presented below. Economou et al. (2006) is another example.

As in every statistical method, the events and data used are critical. Looking at the measurements presented in Chapter 5, we saw that CPU power is varying with CPI, the core frequency and the processor temperature of the processors while DIMM power is varying with GBS.

The impact of CPI (CPI = 1/IPC) on CPU power and GBS on DIMM power is visible when we compare the CPU power of the various workloads Turbo OFF. HPL in Figures 5.3–5.6 has a very low CPI (0.45) and the highest possible CPU power of 150 W and a DIMM power of 32 W with a GBS of 70 for the plateau PL1 phase. For STREAM in Figures 5.11–5.16 we note an average CPU power of 125 W with a CPI ~10 and a DIMM power of 43 W with a GBS of 140. We see the same behavior on BQCD where BQCD128 has an average CPU power of 118 W with a CPI of 1.10 and BQCD1K has an average CPU power of 121 W with a CPI of 0.70. Similarly, BQCD128 has a DIMM power of 33 W and BQCD1K of 21 W with a respective average GBS value of 74 and 27.

This correlation of CPU power with 1/CPI and DIMM power with GBS leads to a simple CPI/GBS model, which was first introduced by Brochard et al. (2010) to predict the power and the performance of a workload at any possible frequency f_n on a given system when this workload has run already once at nominal frequency f₀ on the same system with no temperature variation.

Such a simple model enables real time decisions on which frequency to set while the workload is running to minimize power or energy (see section 6.3).

In this model, DC power at frequency f_n is given by:

[6.3]

where GIPS is the number of giga instructions per second and GBS is the number of giga bytes of memory read or written per second with:

[6.4]

The elapsed time of a code at frequency f_n is given by:

[6.5]

As the number of instructions in a code is independent of the frequency at which it is executed, we can write [6.5] at frequency f₀ and from it derive:

[6.6]

where the CPI at frequency fn is given by:

[6.7]

The A(f_n), B(f_n), C(f_n), D(f_n), E(f_n) and F(f_n) are the hardware coefficients of the server computed through the least squares fitting of [6.3] and [6.5] measuring power and elapsed time of a suite of small kernels run at all possible frequency f₀ to f_n (Brochard et al. 2010).

Therefore, with [6.3], [6.6] and [6.7], the elapsed time and power of a code can be computed at frequency fn given its power, time, CPI and GBS at frequency f₀ and the hardware coefficients that have been computed and stored once and for all unless the hardware configuration changes.

This model was later modified to better predict the DC power since [6.3] has no information on the actual power measured at f₀ and [6.3] was replaced by

[6.8]

where TPI is the number of memory transactions per instruction.

This latter model based on [5.12] and [4.12] was used by IBM LoadLeveler Energy Aware Scheduling (Brochard et al. 2011; IBM Knowledge Center n.d.), which will be described in section 6.3.1. The LRZ data center, which we will present in section 7.2, has been using LoadLeveler Energy Aware Scheduler (EAS) from 2012 until 2018 and measured the accuracy of this model on a large selection of real workloads, demonstrating an average error less than 5% (Auweter et al. 2014).

An extension of this model where power and elapsed time can be predicted from any frequency fi instead of only f₀ was introduced and used by Energy Aware Runtime (BSC n.d.).

These models have been developed such that the predicted performance and power can be computed near real time while the application is executing such decisions can be made to change their performance, power and energy either at the job level or at the iteration level (see section 6.3).

6.2.3.2. NN-based models for power prediction

Neural network (NN) methods are a new class of predictive methods that are not using classic statistical methods, but rather NNs that have been trained by ingesting real data gathered from the data center sensors to predict some behavior.

Figure 6.6 presents an example of an NN with x_i inputs, hidden layers and one output P.

Puzovic et al. (2018) use an NN model to predict the power consumption of workloads on three different servers equipped with Intel Xeon E5 v4, IBM Power8 System S822LC and Cavium ThunderX ARMv8 processors and compare it to the linear regression model based on CPI and GBS as described above. For statistical methods, the selection of the data is critical, but also now their raw number as we will see. In this study, they use the following hardware performance counters as the input of the NN:

1. 1) total number of instructions (INST) – the number of instructed retired;
2. 2) cycles (CYC) – number of cycles during the execution time. It should be denoted as INST/CYC = IPC, which we saw has a major impact on the processor power;
3. 3) dispatched/fetched instructions (IFETCH) – the previous metric (IPC) only accounts for instructions that have been retired, but it does not take into account instructions that have been speculatively executed. These instructions still consume power;

1. 4) stalls (STALL) – due to multiple issues and out-of-order execution, contemporary processors stall due to dependencies such as data and resource conflicts. The conflicts draw power and are not accounted for by any of the previous counters.
2. 5) branch hit ratio (BR) – in order to find contribution to power consumed by speculatively executed instructions due to branch misprediction, the percentage of correctly predicted branches during the application execution is measured;
3. 6) floating point instructions (FLOPS) – for HPC applications, the largest contributor toward power consumption are instructions that are utilizing the floating-point unit as they represent the majority of executed instructions;
4. 7) L1 cache hit – due to the fact that the previous counters measure only power that is consumed within the processor, the number of hits in local cache (L1) is measured;
5. 8) last level cache miss – the number of misses in the shared last level cache (LLCM) is also measured to record power from memory.

The above hardware performance counters were extracted from 43 benchmarks to train the model, which comes from the MATLAB Neural Network Toolbox.

The accuracy of the NN model increases with the number of hardware performance counters, the different benchmarks in the training set as well as running multiples copies of the same benchmark. The accuracy of the NN model is also shown to be less than 3% across the three servers, better than the statistical linear regression model based on CPI and GBS.

Another example of an NN model (Gao 2014) predicts the PUE of a data center. This NN utilizes five hidden layers and 50 nodes per hidden layer. The training dataset contains 19 normalized input variables (listed below) and one normalized output variable, the data center PUE:

• 1) total server IT load [kW];
• 2) total Campus Core Network Room (CCNR) IT load [kW];
• 3) total number of process water pumps (PWP) running;
• 4) mean PWP variable frequency drive (VFD) speed [%];
• 5) total number of condenser water pumps (CWP) running;
• 6) mean CWP variable frequency drive (VFD) speed [%];
• 7) total number of cooling towers running;
• 8) mean cooling tower leaving water temperature (LWT) set point [F];
• 9) total number of chillers running;
• 10) total number of dry coolers running;
• 11) total number of chilled water injection pumps running;
• 12) mean chilled water injection pump set point temperature [F];
• 13) mean heat exchanger approach temperature [F];
• 14) outside air wet bulb (WB) temperature [F];
• 15) outside air-dry bulb (DB) temperature [F];
• 16) outside air enthalpy [kJ/kg];
• 17) outside air relative humidity (RH) [%];
• 18) outdoor wind speed [mph];
• 19) outdoor wind direction [deg].

Between predicted and actual PUE measured on a major data center, the model achieved a mean absolute error of 0.004 and standard deviation of 0.005 on the test dataset or 0.4% error for a PUE of 1.1, which is quite amazing and better than what was achieved by descriptive models. Note that the model error generally increases for PUE values greater than 1.14 due to the scarcity of training data corresponding to those values. The model accuracy for those PUE ranges is expected to increase over time as Google collects additional data on its DC operations. After this calibration phase, the model was used for identifying optimization opportunities.