Blue Gene/Q personality
System calls that provide access to certain hardware or system features can be accessed by applications. This appendix illustrates how to obtain hardware-related information.
Personality of Blue Gene/Q nodes
The personality of a Blue Gene/Q node is the static data given to every compute node and I/O node at boot time by the Control System. This data contains information that is specific to the node, with respect to the block that is being booted.
The personality is a set of C language structures that contains such items as the node coordinates on the torus network. This information can be useful if the application programmer wants to determine, at run time, where the tasks of the application are running. It can also be used to tune certain aspects of the application at run time. For example, it can be used to determine which set of tasks shares the same I/O node and then optimize the network traffic from the compute nodes to that I/O node.
Examples of retrieving Blue Gene/Q personality information
Example B-1 illustrates how to invoke and print selected hardware features.
Example B-1 personality.c program
#include <stdio.h>
#include <mpi.h>
#include <spi/include/kernel/location.h>
int main(int argc, char * argv[])
{
uint64_t Nflags;
char procname[128];
Personality_t pers;
int rank, procid, core, hwthread, namelen;
int Anodes, Bnodes, Cnodes, Dnodes, Enodes;
int Acoord, Bcoord, Ccoord, Dcoord, Ecoord;
int Atorus, Btorus, Ctorus, Dtorus, Etorus;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Get_processor_name(procname, &namelen);
procid = Kernel_ProcessorID(); // 0-63
core = Kernel_ProcessorCoreID(); // 0-15
hwthread = Kernel_ProcessorThreadID(); // 0-3
Kernel_GetPersonality(&pers, sizeof(pers));
Anodes = pers.Network_Config.Anodes; Acoord = pers.Network_Config.Acoord;
Bnodes = pers.Network_Config.Bnodes; Bcoord = pers.Network_Config.Bcoord;
Cnodes = pers.Network_Config.Cnodes; Ccoord = pers.Network_Config.Ccoord;
Dnodes = pers.Network_Config.Dnodes; Dcoord = pers.Network_Config.Dcoord;
Enodes = pers.Network_Config.Enodes; Ecoord = pers.Network_Config.Ecoord;
Nflags = pers.Network_Config.NetFlags;
if (Nflags & ND_ENABLE_TORUS_DIM_A) Atorus = 1; else Atorus = 0;
if (Nflags & ND_ENABLE_TORUS_DIM_B) Btorus = 1; else Btorus = 0;
if (Nflags & ND_ENABLE_TORUS_DIM_C) Ctorus = 1; else Ctorus = 0;
if (Nflags & ND_ENABLE_TORUS_DIM_D) Dtorus = 1; else Dtorus = 0;
if (Nflags & ND_ENABLE_TORUS_DIM_E) Etorus = 1; else Etorus = 0;
if (rank == 0) {
printf("block shape : <%d,%d,%d,%d,%d>
",
Anodes,Bnodes,Cnodes,Dnodes,Enodes);
printf("torus links enabled : <%d,%d,%d,%d,%d>
",
Atorus,Btorus,Ctorus,Dtorus,Etorus);
}
printf("rank %d has processor name %s
", rank, procname);
printf("rank %d location <%d,%d,%d,%d,%d> core %d hwthread %d procid = %d
",
rank,Acoord,Bcoord,Ccoord,Dcoord,Ecoord,core,hwthread,procid);
MPI_Finalize();
return 0;
}
The following command is an example command that is used to build a personality test program from personality.c with the GNU compiler for Blue Gene/Q:
/bgsys/drivers/ppcfloor/comm/gcc/bin/mpicc personality.c -o personality
Example B-2 shows part of the output that is generated by running the personality program with the default (ABCDET) ordering using a 128-node block and two ranks per node. See Appendix A, “Mapping” on page 115. The output is sorted by MPI rank for readability, in practice; however, output is not ordered.
Example B-2 Output with default (ABCDET) order, a 128-node block, and two ranks per node
$ runjob --block R02-M1-N00-128 --ranks-per-node 2 --np 256 --env_all --cwd $PWD : personality
block shape : <2,2,4,4,2>
torus links enabled : <0,0,1,1,1>
rank 0 has processor name Task 0 of 256 (0,0,0,0,0,0) R02-M1-N00-J00
rank 0 location <0,0,0,0,0> core 0 hwthread 0 procid = 0
rank 1 has processor name Task 1 of 256 (0,0,0,0,0,1) R02-M1-N00-J00
rank 1 location <0,0,0,0,0> core 8 hwthread 0 procid = 32
rank 2 has processor name Task 2 of 256 (0,0,0,0,1,0) R02-M1-N00-J07
rank 2 location <0,0,0,0,1> core 0 hwthread 0 procid = 0
rank 3 has processor name Task 3 of 256 (0,0,0,0,1,1) R02-M1-N00-J07
rank 3 location <0,0,0,0,1> core 8 hwthread 0 procid = 32
rank 4 has processor name Task 4 of 256 (0,0,0,1,0,0) R02-M1-N00-J12
rank 4 location <0,0,0,1,0> core 0 hwthread 0 procid = 0
...
Example B-3 shows output from the personality program with TEDCBA mapping. The output is sorted by MPI rank for readability.
Example B-3 Output with TEDCBA order, a 128-node block, and two ranks per node
$ runjob --block R02-M1-N00-128 --ranks-per-node 2 --np 256 --mapping TEDCBA --env_all --cwd $PWD : personality
block shape : <2,2,4,4,2>
torus links enabled : <0,0,1,1,1>
rank 0 has processor name Task 0 of 256 (0,0,0,0,0,0) R02-M1-N00-J00
rank 0 location <0,0,0,0,0> core 0 hwthread 0 procid = 0
rank 1 has processor name Task 1 of 256 (1,0,0,0,0,0) R02-M1-N00-J29
rank 1 location <1,0,0,0,0> core 0 hwthread 0 procid = 0
rank 2 has processor name Task 2 of 256 (0,1,0,0,0,0) R02-M1-N00-J03
rank 2 location <0,1,0,0,0> core 0 hwthread 0 procid = 0
rank 3 has processor name Task 3 of 256 (1,1,0,0,0,0) R02-M1-N00-J30
rank 3 location <1,1,0,0,0> core 0 hwthread 0 procid = 0
rank 4 has processor name Task 4 of 256 (0,0,1,0,0,0) R02-M1-N00-J01
rank 4 location <0,0,1,0,0> core 0 hwthread 0 procid = 0
…
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.