7.2.1.1 Heterogeneous CMP

Heterogeneous CMPs consist of multiple cores of varying size, performance, and complexity on a single die. Since the amount of ILP or TLP varies for different workloads, building a CMP with some large cores with high single-thread performance and some small cores with greater throughput per die area provides an attractive approach for chip design. Research indicates that the best heterogeneous CMPs contain cores customized to a subset of application characteristics (since no single core can be well suited for all embedded applications) resulting in nonmonotonic cores (i.e., cores cannot be strictly ordered in terms of performance or complexity for all the applications) [236]. To achieve high performance, applications are mapped to the heterogeneous cores such that the assigned core best meets an application's resource requirements. Heterogeneous CMPs can provide performance gains as high as 40% but at the expense of additional customization cost [237].

7.2.1.2 Conjoined-Core CMP

Conjoined-core CMPs are multiprocessors that allow topologically feasible resource sharing (e.g., floating-point units (FPUs), instruction, and data caches) between adjacent cores to reduce die area with minimal impact on performance and improve the overall computational efficiency. Since conjoined-core CMPs are topology oriented, the layout must be codesigned with the architecture, otherwise the architectural specifications for resource sharing may not be topologically possible or may incur higher communication costs. In general, the shared resources should be large enough so that the cost of the additional wiring required for sharing may not exceed the area benefits achieved by sharing. Static scheduling is the simplest way to organize resource sharing in conjoined-core CMPs where cores share resources in different nonoverlapping cycles (e.g., one core may use the shared resource during even cycles and the other core may use the shared resource during odd cycles, or one core may share the resource for the first five cycles and the other core for the next five cycles, and so on). Results indicate that conjoined-core CMPs can reduce area requirements by 50% and maintain performance within 9–12% of conventional cores without conjoining [238].

7.2.1.3 Tiled Multicore Architectures

Tiled multicore architectures exploit massive on-chip resources by combining each processor core with a switch to create a modular element called a tile, which can be replicated to create a multicore embedded system with any number of tiles. Tiled multicore architectures contain a high-performance interconnection network that constrains interconnection wire length to no longer than the tile width, and a switch (communication router) interconnects neighboring switches. Examples of tiled multicore architectures include the Raw processor, Intel's Tera-Scale research processor, Tilera TILE64, TILEPro64, and TILE-Gx processor family [239].

7.2.1.4 3D Multicore Architectures

A 3D multicore architecture is an integrated circuit that orchestrates architectural units (e.g., processor cores and memories) across cores in a 3D layout. The architecture provides HPEEC by decreasing the interconnection lengths across the chip, which results in reduced communication latency. Research reveals that 3D multicore processors can achieve 47% performance gain and 20% power reduction on average over 2D multicore processors [240]. The 3D multicore architectures' disadvantages include high power density that exacerbates thermal challenges as well as increased interconnect capacitance due to electrical coupling between different layers [241].

7.2.1.5 Composable Multicore Architectures

The composable multicore architecture is an integrated circuit that allows the number of processors and each processor's granularity to be configured based on application requirements (i.e., large powerful processors for applications (tasks) with more ILP and small less powerful processors for tasks with more TLP). The architecture consists of an array of composable lightweight processors (CLPs) that can be aggregated to form large powerful processors to achieve high performance depending on the task granularity. Examples of composable multicore architectures include TRIPS and TFlex [239].

7.2.1.6 Stochastic Processors

Stochastic processors are processors used for fault-tolerant computing that are scalable with respect to performance requirements and power constraints while producing outputs that are stochastically correct in the worst case. Stochastic processors maintain scalability by exposing multiple functionally equivalent units to the application layer that differ in their architecture and exhibit different reliability levels. Applications select appropriate functional units for a program or program phase based on the program and/or program phase's reliability requirements. Stochastic processors can provide significant power reduction and throughput improvement especially for stochastic applications (applications with a priori knowledge of reliability requirements, such as multimedia applications, where computational errors are considered an additional noise source). Results indicate that stochastic processors can achieve 20–60% power savings in the motion estimation block of H.264 video encoding application [242].

7.2.2.1 Transactional Memory

Transactional memory incorporates the definition of a transaction (a sequence of instructions executed by a single process with the following properties: atomicity, consistency, and isolation) in parallel programming to achieve lock-free synchronization efficiency by coordinating concurrent threads. A computation within a transaction executes atomically and commits on successful completion, making the transaction's changes visible to other processes, or aborts, causing the transaction's changes to be discarded. A transaction ensures that concurrent reads and writes to shared data do not produce inconsistent or incorrect results. The isolation property of a transaction ensures that a transaction produces the same result as if no other transactions were running concurrently [243]. In transactional memories, regions of code in parallel programming can be defined as a transaction. Transactional memory benefits from hardware support that ranges from complete execution of transactions in hardware to hardware-accelerated software implementations of transactional memory [239].

7.2.2.2 Cache Partitioning

One of the major challenges in using multicore embedded systems for real-time applications is timing unpredictability due to core contention for on-chip shared resources (e.g., level two (L2) or level three (L3) caches, interconnect networks). Worst-case execution time (WCET) estimation techniques for single-core embedded systems are not directly applicable to multicore embedded systems because a task running on one core may evict useful L2 cache contents of another task running on another core. Cache partitioning is a cache space isolation technique that exclusively allocates different portions of shared caches to different cores to avoid cache contention for hard real-time tasks, thus ensuring a more predictable runtime. Cache partitioning-aware scheduling techniques allow each task to use a fixed number of cache partitions ensuring that a cache partition is occupied by at most one scheduled task at any time [244]. Cache partitioning can enhance performance by assigning larger portions of shared caches to cores with higher workloads as compared to the cores with lighter workloads.

7.2.2.3 Cooperative Caching

Cooperative caching is a hardware technique that creates a globally managed shared cache using the cooperation of private caches. Cooperative caching allows remote L2 caches to hold and serve data that would not fit in the local L2 cache of a core and, therefore, improves average access latency by minimizing off-chip accesses [245]. Cooperative caching provides three performance enhancing mechanisms: cooperative caching facilitates cache-to-cache transfers of unmodified data to minimize off-chip accesses, cooperative caching replaces replicated data blocks to make room for unique on-chip data blocks called singlets, and cooperative caching allows eviction of singlets from a local L2 cache to be placed in another L2 cache. Cooperative caching implementation requires placement of cooperation-related information in private caches and the extension of cache coherence protocols to support data migration across private caches for capacity sharing. Results indicate that for an eight-core CMP with 1 MB L2 cache per core, cooperative caching improves the performance of multithreaded commercial workloads by 5–11% and 4–38% as compared to shared L2 cache and private L2 caches, respectively [246].

7.2.2.4 Smart Caching

Smart caching focuses on energy-efficient computing and leverages cache set (way) prediction and low-power cache design techniques [18]. Instead of waiting for the tag array comparison, way prediction predicts the matching way prior to the cache access. Way prediction enables faster average cache access time and reduces power consumption because only the predicted way is accessed if the prediction is correct. However, if the prediction is incorrect, the remaining ways are accessed during the subsequent clock cycle(s), resulting in a longer cache access time and increased energy consumption as compared to a cache without way prediction. The drowsy cache is a low-power cache design technique that reduces leakage power by periodically setting the unused cache line's SRAM cells to a drowsy, low-power mode. A drowsy cache is advantageous over turning off cache lines completely because the drowsy mode preserves the cache line's data, whereas turning off the cache line loses the data. However, drowsy mode requires transitioning the drowsy cache line to a high-power mode before accessing cache line's data. Research reveals that 80–90% of cache lines can be put in drowsy mode with less than a 1% performance degradation and result in a cache static and dynamic energy reduction of 50–75% [247].

7.2.3.1 Interconnect Topology

One of the most critical interconnection network parameters is the network topology, which determines the on-chip network cost and performance. The number of hops that must be traversed by a message and the interconnection length are dictated by the interconnect topology. Therefore, the interconnect topology determines the communication latency and energy dissipation (since message traversal across links and through routers dissipates energy). Furthermore, the interconnect topology determines the number of alternate paths between computing nodes, which affects reliability (since messages can route around faulty paths) as well as the ability to evenly distribute network traffic across multiple paths, which affects the effective on-chip network bandwidth and performance. The interconnect topology cost is dictated by the node degree (the number of links at each computing node) and length of the interconnecting wires.

Examples of on-chip interconnection network topologies include buses (linear 1D array or ring), 2D mesh, and hypercube. In bus topology, the processor cores share a common bus for exchanging data. Buses are the most prevalent interconnect network in multicore embedded systems due to the bus's low cost and ease of implementation. Buses provide lower costs than other interconnect topologies because of a lower node degree: the node degree for a bus interconnect is 2, for a 2D mesh is 4, and for a hypercube is where is the total number of computing nodes. However, buses do not scale well as the number of cores in the CMP increases. The 2D mesh interconnect topology provides short channel lengths and low router complexity; however, the 2D mesh diameter is proportional to the perimeter of the mesh, which can lead to energy inefficiency and high network latency (e.g., the diameter of mesh is 18 hops) [248]. The hypercube topology is a special case of a -dimensional mesh (a -dimensional mesh has a node degree of 2) when .

7.2.3.2 Packet-Switched Interconnect

Packet-switched interconnection networks replace buses and crossbar interconnects as scalability and high-bandwidth demand increase for multicore embedded systems. Packet-switched networks connect a router to each computing node, and routers are connected to each other via short-length interconnect wires. Packet-switched interconnection networks multiplex multiple packet flows over the interconnect wires to provide highly scalable bandwidth [239]. Tilera's TILE architectures leverage the packet-switched interconnection network.

7.2.3.3 Photonic Interconnect

As the number of on-chip cores in a CMP increases, global on-chip communication plays a prominent role in overall performance. While local interconnects scale with the number of transistors, the global wires do not because the global wires span across the entire chip to connect distant logic gates and the global wires' bandwidth requirements increase as the number of cores increases. A photonic interconnection network—consisting of a photonic source, optical modulators (rates exceed 12.5 Gbps), and symmetrical optical waveguides—can deliver higher bandwidth and lower latencies with considerably lower power consumption than an electronic signaling based interconnect network.

In photonic interconnects, once a photonic path is established using optical waveguides, data can be transmitted end-to-end without repeaters, regenerators, or buffers as opposed to the electronic interconnects that require buffering, regeneration, and retransmission of messages multiple times from source to destination [249]. The photonic interconnection network is divided into zones each with a drop point such that the clock signal is optically routed to the drop point where the optical clock signal is converted to the electrical signal. Analysis reveals that power dissipation in an optical clock distribution is lower than an electrical clock distribution [241].

The photonic interconnection networks can benefit several classes of embedded applications, including real-time and throughput-intensive applications (especially applications with limited data reuse such as streaming applications). However, even though photonic interconnection networks provide many benefits, these networks have several drawbacks such as delays associated with the rise and fall times of optical emitters and detectors, losses in the optical waveguides, signal noise due to waveguides coupling, limited buffering, and signal processing [241].

7.2.3.4 Wireless Interconnect

Wireless interconnect is an emerging technology that promises to provide high bandwidth, low latency, and low-energy dissipation by eliminating lengthy wired interconnects. Carbon nanotubes are a good candidate for wireless antennas due to a carbon nanotube's high aspect ratio (virtually a 1D wire), high conductance (low losses), and high current-carrying capacity (109 A/cm, which is much higher than silver and copper) [241]. Wireless interconnect can deliver high bandwidth by providing multiple channels and using time-division, code-division, frequency-division, or some hybrid of these multiplexing techniques.

Experiments indicate that a wireless interconnect can reduce the communication latency by 20–45% as compared to a 2D-mesh interconnect while consuming a comparable amount of power [248]. A wireless interconnect's performance advantage increases as the number of on-chip cores increases. For example, a wireless interconnect can provide a performance gain of 217%, 279%, 600% over a 2D-mesh interconnect when the number of on-chip cores is equal to 128, 256, and 512, respectively [250].

7.2.4.1 Leakage Current Reduction

As advances in the chip fabrication process reduces the feature size, the CMOS leakage current and the associated leakage power have increased. Leakage current reduction techniques include back biasing, silicon-on-insulator technologies, multithreshold MOS transistors, and power gating [18].

7.2.4.2 Short-Circuit Current Reduction

Short-circuit current flows in a CMOS gate when both nMOSFET and pMOSFET are on, which causes a large amount of current to flow through transistors and can result in increased power dissipation or even transistor burnout. The short-circuit effect is exacerbated as the clock period approaches the transistor switching period due to increasing clock frequencies. The short-circuit current can be reduced using low-level design techniques that aim to reduce the time during which both nMOSFET and pMOSFET are on [18].

7.2.4.3 Peak Power Reduction

Peak power reduction not only increases power supply efficiency but also reduces packaging, cooling, and power supply cost as these costs are proportional to the peak power dissipation rather than the average power dissipation. Adaptive processors can reduce peak power by centrally managing architectural component configurations (e.g., instruction and data caches, integer and floating-point instruction queues, reorder buffers, load–store execution units, integer and floating-point registers, register renaming) to ensure that not of all these components are maximally configured simultaneously. Adaptive processors incur minimal performance loss and high peak power reduction by restricting maximum configuration to a single resource or a few resources (but not all) at a time. Research reveals that adaptive processors reduce peak power consumption by 25% with only a 5% performance degradation [251].

7.2.4.4 Interconnection Length Reduction

The interconnecting wire length increases as the number of on-chip devices increases, resulting in both increased power dissipation and delay. An energy-efficient design requires reduced interconnection wire lengths for high switching activity signals and use of placement and routing optimization algorithms for reduced delay and power consumption [18]. Chip design techniques (e.g., 3D multicore architectures) and various interconnect topologies (e.g., 2D mesh, hypercube) help in reducing interconnection wire lengths.

7.2.4.5 Instruction and Data Fetch Energy Reduction

Hardwired ASICs typically provide 50 more efficient computing as compared to general-purpose programmable processors; however, architecture-level energy consumption analysis can help in energy-efficient design of programmable processors [5]. Previous work indicates that the programmable processors spend approximately 70% of the total energy consumption fetching instructions (42%) and data (28%) to the arithmetic units, whereas performing the arithmetic consumes a small fraction of the total energy (around 6%). Moreover, the instruction cache consumes the majority of the instruction fetch energy (67%) [5]. Research indicates that reducing instruction and data fetch energy can reduce the energy-efficiency gap between ASICs and programmable processors to 3 . Specifically, instruction fetch techniques that avoid accessing power-hungry caches are required for energy-efficient programmable processors (e.g., the Stanford efficient low-power microprocessor (ELM) fetches instructions from a set of distributed instruction registers rather than the cache) [5].

7.3.3.1 Power Gating

Power gating is a power management technique that reduces leakage power by switching off the supply voltage to idle logic elements after detecting no activity for a certain period of time. Power gating can be applied to idle functional units, cores, and cache banks [18].

7.3.3.2 Per-Core Power Gating

Per-core power gating is a fine-grained power gating technique that individually switches off idle cores. In conjunction with DVFS, per-core power gating provides more flexibility in optimizing performance and power dissipation of multicore processors running applications with varying degrees of parallelism. Per-core power gating increases single-thread performance on a single active core by increasing the active core's supply voltage while power gating the other idle cores, which provides additional power- and thermal-headroom for the active core. Experiments indicate that per-core power gating in conjunction with DVFS can increase the throughput of a multicore processor (with 16 cores) by 16% on average for different workloads exhibiting a range of parallelism while maintaining the power and thermal constraints [256].

7.3.3.3 Split Power Planes

Split power planes is a low-power technique that allows different power planes to coexist on the same chip and minimizes both static and dynamic power dissipation by removing power from idle portions of the chip. Each power plane has separate pins, a separate (or isolated) power supply, and independent power distribution routing. For example, Freescale's MPC8536E PowerQUICC III processor has two power planes: one plane for the processor core (e500) and L2 cache arrays, and a second plane for the remainder of the chip's components [257].

7.3.3.4 Clock Gating

Clock gating is a low-power technique that allows gating off the clock signal to registers, latches, clock regenerators, or entire subsystems (e.g., cache banks). Clock gating can yield significant power savings by gating off the functional units (e.g., adders, multipliers, and shifters) not required by the currently executing instruction, as determined by the instruction decode unit. Clock gating can also be applied internally for each functional unit to further reduce power consumption by disabling the functional unit's upper bits for small operand values that do not require the functional unit's full-bit width. The granularity at which clock gating can be applied is limited by the overhead associated with the clock to enable signal generation [18].

7.3.4.1 Hyper-Threading

Hyper-threading leverages simultaneous multithreading to enable a single processor to appear as two logical processors and allows instructions from both of the logical processors to execute simultaneously on the shared resources [245]. Hyper-threading enables the OS to schedule multiple threads to the processor so that different threads can use the idle execution units. The architecture state, consisting of general-purpose registers, interrupt controller registers, control registers, and some machine state registers, is duplicated for each logical processor. However, hyper-threading does not offer the same performance as a multiprocessor with two physical processors.

7.3.4.2 Helper Threading

Helper threading leverages special execution modes to provide faster execution by reducing cache miss rates and miss latency [245]. Helper threading accelerates performance of single-threaded applications using speculative pre-execution. This pre-execution is most beneficial for irregular applications where data prefetching is ineffective due to the difficulty in prediction of data addresses. The helper threads run ahead of the main thread and reduce cache miss rates and miss latencies by pre-executing regions of the code that are likely to incur many cache misses. Helper threading can be particularly useful for applications with multiple control paths where helper threads pre-execute all possible paths and prefetch the data references for all paths instead of waiting until the correct path is determined. Once the correct execution path is determined, all the helper threads executing incorrect paths are aborted.

7.3.4.3 Speculative Threading

Speculative threading approaches provide high performance by removing unnecessary serialization in programs. We discuss two speculative approaches: speculative multithreading and speculative synchronization.

Speculative multithreading divides a sequential program into multiple contiguous program segments called tasks and execute these tasks in parallel on multiple cores. The architecture provides hardware support for detecting dependencies in a sequential program and rolling back the program state on misspeculations. Speculative multithreaded architectures exploit high transistor density by having multiple cores and relieve programmers from parallel programming, as is required for conventional CMPs. Speculative multithreaded architectures provide instruction windows much larger than conventional uniprocessors by combining the instruction windows of multiple cores to exploit distant TLP as opposed to the nearby ILP exploited by conventional uniprocessors [239].

Speculative synchronization removes unnecessary serialization by applying thread-level speculation to parallel applications and preventing speculative threads from blocking at barriers, busy locks, and unset flags. Hardware monitors detect conflicting accesses and roll back the speculative threads to the synchronization point prior to the access violation. Speculative synchronization guarantees forward execution using a safe thread that ensures that the worst-case performance of the order of conventional synchronization (i.e., threads not using any speculation) when speculative threads fail to make progress.

7.3.6.1 Temperature Determination for DTM

DTM requires efficient chip thermal profiling, which can be done using sensor-based, thermal model-based, or performance counters-based methods. Sensor-based methods leverage physical sensors to monitor the temperature in real time. DTM typically uses one of the two sensor placement techniques: global sensor placement monitors global chip hotspots and local sensor placement places sensors in each processor component to monitor local processor components. Thermal model-based methods use thermal models that exploit the duality between electrical and thermal phenomena by leveraging lumped-RC (resistor/capacitor) models. Thermal models can be either low level or high level. Low-level thermal models estimate temperature accurately and report the steady state as well as provide transient temperature estimation; however, these low-level thermal models are computationally expensive. High-level thermal models leverage a simplified lumped-RC model that can only estimate the steady-state temperature; however, these high-level thermal models are computationally less expensive than the low-level thermal models. Performance counters-based methods estimate the temperature of different on-chip functional units using temperature values read from specific processor counter registers. These counter readings can be used to estimate the access rate and timing information of various on-chip functional units.

7.3.6.2 Techniques Assisting DTM

DVFS is one of the major technique that helps DTM in maintaining a chip's thermal balance and alleviates a core's thermal emergency by reducing the core voltage and frequency. DVFS can be global or local. Global DVFS provides less control and efficiency as a single core's hotspot could result in unnecessary stalling or scaling of all the remaining cores. Local DVFS controls each core's voltage and frequency individually to alleviate thermal emergency of the affected cores; however, the local DVFS makes the design more complex. A hybrid local–global thermal management approach has the potential to provide better performance than local DVFS while maintaining the simplicity of global DVFS. The hybrid approach applies global DVFS across all the cores but specializes the architectural parameters (e.g., instruction window size, issue width, fetch throttling/gating) of each core locally. Research reveals that the hybrid approach achieves a 5% better throughput than the local DVFS [253]. Although DVFS can help DTM to maintain thermal balance, there exist other techniques to assist DTM. (For example, Zhou et al. [261] suggested that adjusting microarchitectural parameters such as instruction window size and issue width has relatively lower overhead than DVFS-based approaches.)

7.3.7.1 -Modular Redundancy

The process variation, technology scaling (deep submicron and nanoscale devices), and computational energy approaching thermal equilibrium leads to high error rates in CMPs, which necessitates redundancy to meet reliability requirements. Core-level NMR runs program copies on different cores and can meet high reliability goals for multicore processors. Each core performs the same computation, and the results are voted (compared) for consistency. Voting can either be time-based or event-based. Based on the voting result, program execution continues or rolls back to a checkpoint (a previously stored, valid architectural state). A multicore NMR framework can provide either static or dynamic redundancy. Static redundancy uses a set of statically configured cores, whereas dynamic redundancy assigns redundant cores during runtime based on the application's reliability requirements and environmental stimuli [262]. Static redundancy incurs high area requirement and power consumption due to the large number of cores required to meet an application's reliability requirements, whereas dynamic redundancy provides better performance, power, and reliability trade-offs.

The dependable multiprocessor (DM) is an example of a DHPEEC platform that leverages NMR. The DM design includes a fault-tolerant embedded message passing interface (FEMPI) (a lightweight fault-tolerant version of the message passing interface (MPI) standard) for providing fault tolerance to parallel embedded applications [11]. Furthermore, DM can leverage HPEC platforms such as the TilePro64 [173].

7.3.7.2 Dynamic Constitution

Dynamic constitution, an extension of dynamic redundancy, permits an arbitrary core on a chip to be a part of an NMR group, which increases dependability as compared to the static NMR configuration by scheduling around cores with permanent faults. For example, if an NMR group is statically constituted and the number of cores with permanent faults drops below the threshold to meet the application's reliability requirements, the remaining nonfaulty cores in the NMR group are rendered useless. Dynamic constitution can also be helpful in alleviating thermal constraints by preventing NMR hotspots [263].

7.3.7.3 Proactive Checkpoint Deallocation

Proactive checkpoint deallocation is a high-performance extension for NMR that permits cores participating in voting to continue execution instead of waiting on the voting logic results. After a voting logic decision, only the cores with correct results are allowed to continue further execution.

7.4.2.1 Task Scheduling

The task scheduling problem can be defined as determining an optimal assignment of tasks to cores that minimizes the power consumption while maintaining the chip temperature below the DTM enforced ceiling temperature with minimal or no performance degradation given the total energy budget. Task scheduling applies for both DPM and DTM and plays a pivotal role in extending battery life for portable embedded systems, alleviating thermal emergencies, and enabling long-term savings from reduced cooling costs. Task scheduling can be applied in conjunction with DVFS to meet real-time task deadlines as a higher processing speed results in faster task execution and shorter scheduling lengths, but at the expense of greater power consumption. Conversely, the decrease in processor frequency reduces power consumption but increases the scheduling length, which may increase the overall energy consumption. Since the task scheduling overhead increases as the number of cores increases, hardware-assisted task scheduling techniques are the focus of emerging research (e.g., thread scheduling in graphics processing units (GPUs) is hardware-assisted). Experiments indicate that hardware-assisted task scheduling can improve the scheduling time by 8.1% for CMPs [265].

7.4.2.2 Task Migration

In a multithreaded environment, threads periodically and/or aperiodically enter and leave cores. Thread migration is used both as a DPM and DTM technique that allows a scheduled thread to execute, preempt, or migrate to another core based on the thread's thermal and/or power profile. The OS or thread scheduler can dynamically migrate threads running on cores with limited resources to the cores with more resources as resources become available. Depending on the executing workloads, there can be a substantial temperature variation across cores on the same chip. Thread migration-based DTM periodically moves threads away from the hot cores to the cold cores based on this temperature differential to maintain the cores' thermal balance. A thread migration technique must take into account the overhead incurred due to thread migration communication costs and address space updates. Temperature determination techniques (e.g., performance counter-based, sensor-based) assist thread management techniques in making migration decisions.

Thread migration techniques can be characterized as rotation-based, temperature-based, or power-based [266]. The rotation-based technique migrates a thread from core to core where denotes the total number of processor cores. The temperature-based technique orders cores based on the cores' temperature', and the thread on core is swapped with the thread on core (i.e., the thread on the hottest core is swapped with the thread on the coldest core, the thread on the second hottest core is swapped with the thread on the second coldest core, and so on). The power-based technique orders cores based on the cores' temperature in ascending order and orders threads based on the threads' power consumption in descending order. The power-based technique then schedules thread to core (e.g., the most power-hungry thread is scheduled to the coldest core).

Thread migration can be applied in conjunction with DVFS to enhance performance. Research indicates that thread migration alone can improve performance by 2 on average, whereas thread migration in conjunction with DVFS can improve performance by 2.6 on average [252].

7.4.2.3 Load Balancing and Unbalancing

Load balancing techniques distribute a workload equally across all the cores in a multicore embedded system. Load unbalancing can be caused by either extrinsic or intrinsic factors. Extrinsic factors are associated with the OS and hardware topology. For example, the OS can schedule daemon processes during the execution of a parallel application, and an asymmetric hardware topology can result in varying communication latencies for different processes. Intrinsic factors include imbalanced parallel algorithms, imbalanced data distribution, and changes in the input dataset. An unbalanced task assignment can lead to a performance degradation because cores executing light workloads may have to wait/stall for other cores executing heavier workloads to reach a synchronization point. Load balancing relies on efficient task scheduling techniques as well as balanced parallel algorithms. Cache partitioning can assist load balancing by assigning more cache partitions to the cores executing heavier workloads to decrease the cache miss rate and increase the core's execution speed, and thus reduce the stall time for the cores executing light workloads [267].

Although load balancing provides a mechanism to achieve high performance in embedded systems, load balancing may lead to high power consumption if not applied judiciously because load balancing focuses on utilizing all the cores even for a small number of tasks. A load unbalancing strategy that considers workload characteristics (i.e., periodic or aperiodic) can achieve better performance and lower power consumption as compared to a load balancing or a load unbalancing strategy that ignores workload characteristics. A workload-aware load unbalancing strategy assigns repeatedly executed periodic tasks to a minimum number of cores and distributes aperiodic tasks that are not likely to be executed repeatedly to a maximum number of cores. We point out that the critical performance metric for periodic tasks is deadline satisfaction rather than faster execution (a longer waiting time is not a problem as long as the deadline is met), whereas the critical performance metric for aperiodic tasks is response time rather than deadline satisfaction. The periodic tasks not distributed over all the cores leave more idle cores for scheduling aperiodic tasks, which shortens the response time of aperiodic tasks. Results on an ARM11 MPCore chip demonstrate that the workload-aware load unbalancing strategy reduces power consumption and the mean waiting time of aperiodic tasks by 26% and 82%, respectively, as compared to a load balancing strategy. The workload-aware load unbalancing strategy reduces the mean waiting time of aperiodic tasks by 92% with similar power efficiency as compared to a workload unaware load unbalancing strategy [268].