The compute nodes on the Blue Gene/Q system comprise 17 cores on a single chip with 16 GB of dedicated physical memory. Applications run on 16 of the cores with the 17th core reserved for system software. Nearly the full 16 GB of physical memory is dedicated to application usage.

Within each job, processes (also known as tasks) are distributed to all of the compute nodes. Each node runs separate instantiations of the Compute Node Kernel (CNK). Each CNK can run multiple tasks or processes. This chapter describes some of the characteristics of those processes and how to configure them.

When submitting a job to the Blue Gene/Q system with the runjob command or another job scheduler command, it is important to decide how many processes or tasks to run on each node. This decision significantly impacts performance and the memory that is allocated to each process, for example:

When a job is submitted to the Control System, the user specifies the number of processes to create. This value is used in the configuration of memory and number of cores assigned to the process.

Jobs are submitted to the CNK through the Control System using the functional network path through the I/O node to load the application. A job can be a subset of the block. Multiple jobs comprising one or more node jobs can exist per block. However, a single job cannot span blocks. From a CNK perspective, jobs comprise one or more processes to the node. With each job, the Control System provides information that describes the environment of the job (environment variables, program arguments, and so on).

For statically linked applications, the CNK loads the applications into memory at process creation time. To ensure that the application load scales to large system configurations, the application is loaded from a job leader node and broadcast to the other nodes in the job.

For dynamically linked applications, the CNK loads the dynamic interpreter (ld.so) into memory. The interpreter then pulls the target application and associated dynamic libraries into main memory by using standard system calls.

Processes are only created at the time of job initialization. The CNK does not support the fork system call and therefore cannot dynamically spawn processes.

When a job is submitted to the Control System, the user specifies the number of processes to create. This value is used to configure memory and the number of cores that are assigned to the process. To specify the process count, use the runjob -p (or --ranks-per-node) option.

When the user specifies the process count per node, the memory on that node is divided evenly among the processes. Generally, each process has roughly the same amount of memory, although there can be slight variations. Variations can be because of the size of the application text, shared memory size, kernel storage, and so on. The CNK configures the hardware to avoid memory page translations while the application is running. This configuration is known as a static memory map.

To achieve a static memory map, the process count must be a power of two. Thus, the valid numbers are 1, 2, 4, 8, 16, 32, or 64 processes per node. User submitted jobs run on 16 of the 17 cores.

The CNK allocates a number of cores to a process. Therefore, as shown in Table 3-1, the number of threads that each process can have active at a given moment is dictated by the processes per node value.

The best configuration of processes per node depends on the type of application, the memory requirement, and the parallel paradigm that is implemented. There might be several options for applications that use a hybrid paradigm with both MPI and OpenMP or pthreads, depending on the memory footprint. Hybrid applications that support a high degree of threading might work well with a single process per node, but scenarios with 2, 4, 8, or 16 processes per node are more common. For single‑threaded applications, the memory requirement per process is the main consideration. If possible, use all 16 of the cores with 16, 32, or 64 processes per node.

One trade off to consider is that each additional process that is running on a node has a fixed amount of overhead. Overhead consists of a replicated data segment (for example, the global storage for the process), storage for the main stack, and storage for the heap.

The default mode for the runjob command is one process per node. To specify other values for processes per node, use the following commands:

The Blue Gene/Q system only supports processes compiled for the 64-bit PowerPC application binary interface (ABI). 32-bit processes are not supported:

There are various identifiers related to the process objects on the compute node. Each of the 16 physical cores that support the application processes is assigned a processor core identifier. Each of the four hardware threads within that core is assigned a processor thread identifier. There is also a unique identifier for each hardware thread within the node termed the processor identifier. Table 3-2 describes the interrelationship between these hardware identifiers. The first 16 cores are used to run applications. The 17th core is reserved for system use.

The process identifier (PID) is a 4-byte signed number that identifies a process. Each process on a compute node has a unique PID. The PID value is not unique across compute nodes. The PID can be passed to various pthreads, signal APIs, and system calls as a thread group identifier (TGID) for the process. The thread identifier (TID) that corresponds to the primary thread of the process is the same value as the PID for the process.

The thread identifier (TID) is a number assigned by the kernel used to uniquely identify a thread within the node. The TID of the process’ main thread is the same as the PID of the process. See 3.7.1, “Process identifier” on page 16 for more information about PID number generation.

The thread group identifier (TGID) is an input parameter on several pthread and signal APIs and system calls. The PID (that is, the main thread TID of the process) serves as a valid TGID.

Multiple processes within a node for a given job are assigned a “T” coordinate value. That value can be used with the A, B, C, D, and E coordinates to uniquely identify a specific process or rank in the block or sub‑block. The “T” coordinates begin at 0 and are assigned in sequential order to a value that is equal to the number of processes minus one. For example, if the node is configured to contain four processes, the “T” coordinates range from 0 to 3. The coordinate T = 0 corresponds to the first process containing processor IDs 0 - 15. The coordinate T = 3 corresponds to the last process containing processor IDs 48 - 63. Each unique rank within a job is identified by a corresponding set of A, B, C, D, E, T coordinates.

The compute node kernel supports sub‑block jobs within a node. A sub-node, sub‑block job is known as a sub-node job. A sub-node job occupies a subset of the 16 cores available for assignment to applications. Jobs that are running in a subset of the 16 cores in the node can be started and ended asynchronously. Only one core per sub-node job is supported. Only one process per node in a sub-node job is supported. Sub‑node jobs are restricted to a single user per node.

The CNK provides a threading model based on the Native Portable Operating System Interface (POSIX) Thread Library (NPTL) available in the glibc library. The NPTL package is the default threading package for Linux applications. The NPTL threading package implements the POSIX pthread API. The NPTL package allows the same POSIX pthread API library that Linux uses (-lpthreads) to function on the CNK without special parameters.

More than one pthread can be assigned to a given hardware thread. These additional pthreads are supported by additional kernel thread structures (that is, an M:N threading model of 1:1). By default, five pthreads can be assigned to one hardware thread. In the Blue Gene/Q threading model, pthreads have absolute affinity to the hardware threads they are associated with. There is no time-quantum driven preemption of pthreads running on a hardware thread. After a pthread begins to run on a hardware thread, it continues to run until one of the following occurs:

The kernel scheduler runs on each hardware thread independently. Each local dispatcher handles the dispatching of the software threads assigned to the one hardware thread that it controls. There is no global dispatcher. Therefore, no global locks or blocking conditions are required to manage the dispatching of threads.

A pthread is preempted when and only when a pthread with a strictly higher software priority is available to be run on the same hardware thread. This scenario can occur for the following reasons:

When a pthread executes a pthread_yield() function and another pthread with the same software priority is available to be dispatched, the current thread relinquishes control. The other thread is dispatched. If there is no other runnable thread of equal or higher‑priority, control returns to the thread that executed the pthread_yield() function.

A thread relinquishes control due to a yield or a futex wait. If there are other pthreads with the same software priority, those pthreads are selected over the current thread for the next dispatch in a round-robin order. In other words, when there are multiple equal-priority pthreads on a hardware thread, and each pthread issues frequent yields, each of the pthreads makes progress. There is no guarantee that each thread will make equal progress. Interrupt conditions presented to the hardware thread might cause unbalanced thread dispatching within the scheduler's simple, light-weight, round-robin algorithm.

When a pthread is created within a process, the CNK must select a hardware thread for the pthread. The kernel supports two layout algorithms for assigning pthreads to hardware threads. The number of hardware threads that are available to the process is dependent on the number of processes in the node. See 3.3, “Processes per node” on page 14. The layout types in the following sections can be activated through the use of an environment variable, BG_THREADLAYOUT. If required, additional layout algorithms can be added. When possible, the even-numbered processor IDs within the process are assigned before the odd-numbered processor IDs because of the configuration limitations that are imposed by the hardware universal performance counter implementation.

Breadth-first is the default thread layout algorithm. This algorithm corresponds to BG_THREADLAYOUT = 1. With breadth-first assignment, the hardware thread-selection algorithm progresses across the cores that are defined within the process before selecting additional threads within a given core.

This algorithm corresponds to BG_THREADLAYOUT = 2. With depth-first assignment, the hardware thread-selection algorithm progresses within each core before moving to another core defined within the process.

Controlling the placement of pthreads on the existing hardware threads is supported by the kernel through the sched_setaffinity() system call. The target of the affinity operation must be one and only one hardware thread. The interface to specify the target hardware thread is defined by the glibc structure, cpu_set_t. The CPU numbers to be specified by the caller correspond to the processor IDs 0 - 63. The caller must be aware of the range of valid processor IDs for the current process. For a configuration where there is one process on the node, all processor IDs are owned by the process. However, on a system that has four processes in the node, the first process owns processor IDs 0 - 15. The second process owns processor IDs 16 - 31. Determining what processor IDs are controlled by a given process can be accomplished by using the Kernel_ThreadMask(T) and the Kernel_MyTcoord() SPIs. After the T coordinate is obtained using the Kernel_MyTcoord system programming interface (SPI), supply it to the Kernel_ThreadMask(T) SPI. The SPI returns a 64-bit mask representing the processor IDs owned by the currently running process.

There are two methods to set the affinity of a pthread. The first method is at pthread creation time through the pthread attributes structure. The second method is explicitly through the set_affinity system call.

Example 3-1 shows how to set affinity with the pthread attributes at pthread creation.

The following code example shows how to set explicit affinity with the system call.

Extended thread affinity control is a facility that allows a process to place, using set affinity, software threads on hardware threads that were not originally allocated to that process. This feature is useful in application environments where an application might enter different phases of execution that require a larger number of threads to be used by a subset of the processes in a node while other processes in the node are not actively using their threads.

An environment variable is used to enable the extended thread affinity control facility. If the BG_THREADMODEL environment variable is set to the value 2, set affinity APIs can be used to place a pthread onto a hardware thread that is not configured as a hardware thread owned by the current process. See Table D-5 on page 142 for more information about the BG_THREADMODEL variable.

For example, if the application must transition between 16 active processes, each using four hardware threads, to four active processes, each using 16 hardware threads, the application is started with 16 processes configured. Each process creates its pthreads using the pthread_create() function. Each of the four processes can use the pthread_create() function to create up to a total of 16 threads and use the setaffinity API interfaces to place these pthreads on hardware threads outside its configured set of hardware threads. When the application reaches the end of its first phase, 12 of the 16 processes block using a standard POSIX synchronization mechanism, such as a shared mutex, condition, or barrier. Then the four remaining processes begin running their additional pthreads using the additional hardware threads that were previously used by the now blocked 12 processes. When this phase of the application completes, the processes are unblocked and the application returns to its original behavior of having 16 active processes each using four hardware threads.

The following restrictions apply to the extended thread affinity control facility:

The application can use any of the following CNK supported interprocess synchronization mechanisms to transition into and out of actively running pthreads that are executing on a hardware thread that is not configured to a pthread’s process:

The application can also use its own synchronization mechanisms as long as the blocked threads are not waiting on the completion of a function shipping system call.

The software thread priority can be set within a pthread through either the pthread attribute structure or through an explicit system call. The priority values that are supported depend on the scheduling policy that is set for the pthread. Thread priorities are evaluated in the scheduler when a condition occurs within a hardware thread that causes the scheduler to select a potentially different pthread for dispatching. Because the Blue Gene/Q system has absolute hardware thread affinity, the relative pthread priorities of pthreads on different hardware threads has little consequence. The relative thread priorities are important for pthreads assigned to the same hardware thread.

There are conditions in which a communication thread might require control only when no other application threads are running. Because of this requirement, communication threads can specify a priority that is lower than any application thread. Conversely, there are situations when a communication thread might need to be the highest priority software thread on the hardware thread. Therefore, communication threads are allowed to set a priority value that is more favored than any application thread priority. This widened range of priorities is supported by the use of a special scheduling policy, SCHED_COMM. See 3.12.1, “Setting priority through the pthread attribute” on page 22.

Thread priority can be modified dynamically, for example, a pthread might want to raise or lower its priority before relinquishing control. A priority change results in a call to the kernel within the target thread. At that time, the relative priorities of the software threads on the hardware thread are re-evaluated. The most eligible software thread is dispatched.

Priority can be set through the pthread attribute structure supplied to pthread_create. The following API must first be issued to have the priority information in the attribute used. The following code sets the inherit attribute:

pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);

Setting the priority within the pthread attribute can be done by either specifying just the priority or by specifying both the policy and priority. To specify both the policy and priority, see 3.12, “Thread priority” on page 21. To specify just the priority, use the following API:

If this API is used, the policy is inherited from the caller if the pthread_attr_setschedpolicy() was not called, even though the PTHREAD_EXPLICIT_SCHED value is specified on the pthread_attr_setinheritsched() API is specified.

To determine the priority range for a given scheduler policy, use the following APIs:

Table 3-3 outlines the priorities for the Blue Gene/Q system. However, these ranges are subject to change. The application code must avoid making assumptions regarding the valid priority range for a given policy and use the previously mentioned APIs.

Priority can be set explicitly through the use of the pthread_setschedparam() API. Example 3-2 on page 20 shows an example of explicitly setting affinity.

The hardware thread priority represents the relative proportion of available core cycles that are to be given to hardware threads that share the same core. There are seven PowerPC architected hardware thread priorities. However, the Blue Gene/Q system internally implements two priority levels. Based on internal configuration settings, the mapping between the architected and available priority levels are shown in Table 3-4.

Applications can use the low and medium Blue Gene/Q hardware thread priorities. The following statement brings in the required header file that contains the inline interfaces:

The interfaces control hardware thread priorities.

The following inlines can be used to set hardware thread priorities:

The ThreadPriority_Medium() and ThreadPriority_MediumLow() interfaces both map to the medium Blue Gene/Q priority level. The ThreadPriority_Low() sets the low Blue Gene/Q priority level.

The current priority of the hardware thread can be obtained by reading the PPR32 register. Table 3-4 describes the mapping of the priority levels. The thread priority can also be set by writing to this register. This approach can be useful when restoring a previously saved priority after priority modification. Example 3-4 on page 24 demonstrates a sequence of saving, modifying, and restoring the hardware priority of a thread.