• Data layout in memory, aligned and packed. Sixteen values have to be collected into each input register. If the data is laid out in memory in the same linear order and aligned on 512-bit (64 bytes) boundary, then a simple high performance vector load will suffice. If not, the first consideration should be whether data could be laid out in order and alignment. Data that is not packed and aligned optimally will require more instructions and cache or memory accesses to collect and organize in registers in order to use the vector operations. The extra instructions and data accesses reduce performance. The compilers offer alignment directives and command line options, covered later in this chapter, to force or specify alignment of variables.

• Data locality:

− Fetch data from cache not memory. Data eventually comes from memory, but at the time a vector load instruction is issued it is much faster if the data has been fetched into the closest (level 1, known as L1 cache) prior to the load instruction. On an Intel Xeon Phi coprocessor data actually travels from memory to L2 cache to L1 cache. Optimally this can be done by a prefetch of data from memory to L2, later followed by a prefetch from L2 to L1 cache, and later by the load instruction from L1 cache. The prefetches can be initiated by hardware or by software prefetches specified by the compiler automatically or manually by the programmer. L1 Prefetch instructions can ask for memory data to L1, but there are far fewer L1 prefetch operations allowed to be outstanding at the same time than the capacity for L2 prefetches to be outstanding. The compiler inserts prefetches automatically, and there are directives to help the compiler with hints and there are mm_prefetch intrinsics to do the prefetching manually. These are covered in a later section in this chapter.

− Data reuse. If a program is fetching data into the cache that will be accessed more than once, then those accesses should occur close together. This is called temporal locality of reference. Ensuring that data is reused quickly helps reduce the chance that data will be evicted before it is needed again—resulting in a new fetch. Rearranging a program to increase temporal reuse is often a rewarding optimization. Intel MKL (Chapter 11) uses blocking in the library routines, which is a big help. For code we write explicitly, the challenge of blocking generally falls to the programmer. The good news is that cache blocking has been studied for decades for processors, and Intel Xeon Phi coprocessor relies on the same programming techniques. Before we decide to curse caches, it is worth noting that caches exist to lower power consumption and increase performance. The fact that our programming style can align with the cache designs to maximize performance is the price we pay to have more performance and power efficiency than would be possible without caches.

− Streaming stores. If a program is writing data out that it will not use again, and that occupies memory linearly without gaps, then making sure streaming stores are being generated is important. Streaming stores increase cache utilization for data that matters by preventing the data being written from needlessly using up cache space.

• Directs compiler to align the start of objects A, B to 32 bytes

• Does not work for objects inside common blocks or derived types with sequence attribute; can be used to align start of common block itself

• Tells the compiler that A and B have been aligned

• Useful if the compiler wouldn’t otherwise know

• Invites the compiler to vectorize a loop using aligned loads for all arrays, ignoring efficiency heuristics, provided that it is safe to do so.

• Roughly like !dir$ attributes align:32 wherever possible

• where arrays are declared, not where they are used

• can appear together, but should not be necessary

• Most obvious use is for an incoming subroutine argument

• Cannot be used for global objects declared elsewhere or sequential objects (risk of conflicts) such as within common blocks or within modules or within structures

• Applies to all the arrays in the following loop

• No impact on array attributes or other loops

Compiler prefetches

When using the Intel compiler with normal or elevated optimization levels (-O2 or greater) then prefetching is automatically set to opt-prefetch=3. Compiler prefetching is enabled with the -opt-prefetch=n option with n being a value 1 to 4 on Linux (Windows: /Qopt-prefetch:n). The higher the value, the more aggressive the compiler is with issuing prefetch instructions. If algorithm is well blocked to fit in L2 cache, prefetching is less critical but generally still useful for Intel Xeon Phi coprocessors.

In general, we recommend using the compiler to issue prefetches. If this is not perfect, there are a number of ways to give the compiler additional hints or refinements but still have the compiler generate the prefetches. We consider manual insertion of prefetch directives to be a last resort if compiler prefetching cannot be utilized for a particular need.

The first tuning to try is to use different second-level prefetch (vprefetch0) distances. We recommend trying n=1,2,4,8 with -mP2OPT_hlo_use_const_second_pref_dist=n. In this case, n=2 means use a prefetch distance of two vectorized iterations ahead (if the loop is vectorized). Instead of the compiler option, prefetch pragmas can be utilized in the source code.

Compiler prefetch controls (prefetching via pragmas/directives)

Instead of relying only on the compiler option, prefetch pragmas can be utilized in the source code. The compiler can be given prefetch directives to modify default behavior. The passing of addition information through directives can improve the performance of an application. The format for C/C++ and Fortran are respectively:

#pragma prefetch var:hint:distance

!DIR$ prefetch var:hint:distance

The hint value can be 0 (L1 cache [and L2 due to inclusion]), 1 (L2 cache, not L1). Figure 5.7 shows usage of prefetch directives and Figure 5.8 shows the roughly equivalent manual prefetches that we get from the compiler automatically with the simpler Figure 5.7 example syntax. Figures 5.9, 5.10A, 5.10B, and 5.11 give additional usage examples to examine.

Manual prefetches (using vprefetch0 and vprefetch1)

Figures 5.12 and 5.13 show examples of using manual prefetch directives. We recommend only doing this if the compiler cannot generate prefetches sufficiently even with hint covered in the prior section. These intrinsics work on processors and coprocessors both and are not specific to Intel Xeon Phi coprocessors. The intrinsics for Fortran and C/C++, respectively, are:

MM_PREFETCH (address [, hint])

void _mm_prefetch(char const*address, int hint)

The intrinsic issues a software prefetch to load one cache line of data located at address. The value hint specifies the type of prefetch operation. Values of _MM_HINT_T0 or 0 (for L1 prefetches) and _MM_HINT_T1 or 1 (for L2 prefetches) are most common and are the interesting ones for use with Intel Xeon Phi coprocessors.

If we do decide to insert prefetches manually, we will want to disable automatic compiler insertion of prefetch instructions. Doing both will generally create competition for the limited prefetch capabilities of the hardware, and will result in reduced performance. If we are going through the effort to control prefetching manually, we should have already determined the compiler prefetching was not optimal. This generally happens for more complex data processing than the compiler can determine automatically.

Compiler prefetching is disabled with the -opt-prefetch=0 or -no-opt-prefetch option on Linux (Windows: /Qopt-prefetch:0 or /Qopt-prefetch-).

When streaming stores will be generated for Intel® Xeon Phi™ coprocessors

The compiler is smart enough not to generate prefetches for lines for the streaming store, so as to avoid negating the value of the streaming stores. The compiler will also inject L2 cache line eviction instructions when using streaming stores. The Intel compiler generates streaming store instructions for an Intel Xeon Phi coprocessor only when:

Compiler generation of clevicts

On streaming operations, the use of clevicts instructions allows the compiler to tell the hardware to not fetch data that will entirely be discarded (overwritten). This effectively avoids wasted prefetching efforts. Compiler has heuristics that determine whether a store is streaming. Usually this heuristic will kick in only when the loop has a large number of iterations specified and that is known at compile time. This is very limiting, so for applications where the calculation for the number of iterations is symbolic, there is a nontemporal pragma that allows us to mark streaming stores:

#pragma vector nontemporal

For processors, this pragma generates the nontemporal hints on stores. For Intel Xeon Phi coprocessors the compiler generates clevicts instead when the compiler knows the store addresses are aligned. Feature is controlled via the option:

-mGLOB_default_function_attrs=”clevict_level=N”

where N=0, 1, 2 or 3 (default is 3)

• If a loop is part of a loop nest, it must be the inner loop. Outer loops can be parallelized using OpenMP or autoparallelization (-parallel), but they cannot be vectorized unless the compiler is able either to fully unroll the inner loop, or to interchange the inner and outer loops. Additional high level loop transformations such as these may require -O3.

• The loop must contain straight-line code (a single basic block). There should be no jumps or branches, but masked assignments are allowed. A “masked assignment” simply means an assignment controlled by a conditional (such as an IF statement).

• The loop must be countable, which means that the number of iterations must be known before the loop starts to execute although it need not be known at compile time. Consequently, there must be no data-dependent exit conditions.

• There should be no backward loop-carried dependencies. For example, the loop must not require statement 2 of iteration 1 to be executed before statement 1 of iteration 2 for correct results. This allows consecutive iterations of the original loop to be executed simultaneously in a single iteration of the unrolled, vectorized loop. Figures 5.14 and 5.15 illustrate this requirement. The examples only make sense when thinking of doing multiple iterations at once due to vectorization. That is what would make Figure 5.15 illustrate a barrier to vectorization.

• Both reductions and vector assignments to arrays are allowed.

• Try to avoid mixing vectorizable data types in the same loop (except for integer arithmetic on array subscripts). Vectorization of type conversions may be either unsupported or inefficient. Support for the vectorization of loops containing mixed data types may be extended in a future version of the Intel compiler.

• Try to access contiguous memory locations. (So loop over the first array index in Fortran, or the last array index in C). While the compiler may sometimes be able to vectorize loops with indirect or non-unit stride memory addressing, the cost of gathering data from or scattering back to memory is often too great to make vectorization worthwhile.

• The ivdep pragma or directive may be used to advise the compiler that there are no loop-carried dependencies that would make vectorization unsafe.

• The vector always pragma or directive may be used to override the compiler’s heuristics that determine whether vectorization of a loop is likely to yield a performance benefit.

• To see whether a loop was or was not vectorized, and why, look at the vectorization report. This may be enabled by the command line switch /Qvec-report3 (Windows) or -vec-report3 (Linux or Mac OS X).

• The vec-report option, when used with offloading, can be easier to read if the reports for the host and the coprocessor are not interspersed. The section “Vec-report Option Used with Offloads” in Chapter 7 explains how to do this.

Requirements to vectorize with SIMD directives (Intel® Compiler)

The SIMD directives ask the compiler to relax some of its requirements and to make every possible effort to vectorize a loop. If an ASSERT clause is present, the compilation will fail if the loop is not successfully vectorized. This has led it to be sometimes called the “vectorize or die” directive.

The directive (#pragma simd in C/C++, or !DIR$ SIMD in Fortran) behaves somewhat like a combination of #pragma vector always and #pragma ivdep, but is more powerful. The compiler does not try to assess whether vectorization is likely to lead to performance gain, it does not check for aliasing or dependencies that might cause incorrect results after vectorization, and it does not protect against illegal memory references. #pragma ivdep overrides potential dependencies, but the compiler still performs a dependency analysis, and will not vectorize if it finds a proven dependency that would affect results. With #pragma simd, the compiler does no such analysis, and tries to vectorize regardless. It is the programmer’s responsibility to ensure that there are no backward dependencies that might impact correctness. The semantics of #pragma simd are rather similar to those of the OpenMP #pragma omp parallel for. It accepts optional clauses such as REDUCTION, PRIVATE, FIRSTPRIVATE, and LASTPRIVATE. SIMD-specific clauses are VECTORLENGTH (implies the loop unroll factor), and LINEAR, which can specify different strides for different variables. Pragma SIMD allows a wider variety of loops to be vectorized, including loops containing multiple branches or function calls. It is particularly powerful in conjunction with the vector functions of Intel Cilk Plus.

Nevertheless, the technology underlying the SIMD directives is still that of the compiler vectorizer, and some restrictions remain on what types of loop can be vectorized:

A number of the requirements detailed in the prior section “Requirements for a Loop to Vectorize (Intel® Compiler)” are relaxed for SIMD directives, in addition to the above-mentioned ones relating to dependencies and performance estimates. Loops that are not the innermost loop may be vectorized in certain cases; more mixing of different data types is allowed; function calls are possible and more complex control flow is supported. Nevertheless, the advice in the prior section should be followed where possible, since it is likely to improve performance.

It is worth noting that with SIMD directives, loops are vectorized under the “fast” floating-point model, corresponding to /fp:fast (-fp-model=fast). The command line option /fp:precise (-fp-model precise) is not respected by a loop vectorized with a SIMD directive; such a loop might not give identical results to a loop without the directive. For further information about the floating-point model, see “Consistency of Floating-Point Results using the Intel® Compiler” (listed in additional reading at the end of the chapter).

SIMD directive clauses

A SIMD directive can be modified by additional clauses, which control chunk size or allow for some C/C++ programmers’ fondness for bumping pointers or indices inside the loop. The SIMD directives come in two forms for C/C++ and Fortran, respectively:

#pragma simd [clause[ [,] clause]…]

!DIR$ SIMD [clause[[,] clause]…]

If we specify the SIMD directive with no clause, default rules are in effect for variable attributes, vector length, and so forth. The VECTORLENGTH and VECTORLENGTHFOR clauses are mutually exclusive. We cannot use the VECTORLENGTH clause with the VECTORLENGTHFOR clause, and vice versa.

If we do not explicitly specify a VECTORLENGTH or VECTORLENGTHFOR clause, the compiler will choose a VECTORLENGTH using its own cost model. Misclassification of variables into PRIVATE, FIRSTPRIVATE, LASTPRIVATE, LINEAR, and REDUCTION, or the lack of appropriate classification of variables, may lead to unintended consequences such as runtime failures and/or incorrect results.

We can only specify a particular variable in at most one instance of a PRIVATE, LINEAR, or REDUCTION clause. If the compiler is unable to vectorize a loop, a warning occurs by default. However, if ASSERT is specified, an error occurs instead. If the vectorizer has to stop vectorizing a loop for some reason, the fast floating-point model is used for the SIMD loop. A SIMD loop may contain one or more nested loops or be contained in a loop nest. Only the loop preceded by the SIMD directive is processed for SIMD vectorization.

The vectorization performed on this loop by the SIMD directive overrides any setting we may specify for options -fp-model (Linux OS and OS X) and /fp (Windows) for this loop.

VECTORLENGTH (n1 [, n2]…)

n is a vector length (VL). It must be an integer that is a power of 2 (16 or less for C/C++); the value must be 2, 4, 8, or 16. If we specify more than one n, the vectorizor will choose the VL from the values specified. This clause causes each iteration in the vector loop to execute the computation equivalent to n iterations of scalar loop execution. Multiple VECTORLENGTH clauses will cause a syntax error.

VECTORLENGTHFOR (data-type)

data-type is one of the following intrinsic data types in Fortran or built-in integer, pointer, float, double or complex types in C/C++. This clause causes each iteration in the vector loop to execute the computation equivalent to n iterations of scalar loop execution where n is computed from size_of_vector_register/sizeof(data_type).

For example, VECTORLENGTHFOR (REAL (KIND=4)) or vectorlengthfor(float) results in n=4 for SSE2 to SSE4.2 targets (packed float operations available on 128-bit XMM registers) and n=8 for AVX target (packed float operations available on 256-bit YMM registers) and n=16 for Intel Xeon Phi coprocessors. The VECTORLENGTHFOR and VECTORLENGTH clauses are mutually exclusive. We cannot use the VECTORLENGTHFOR clause with the VECTORLENGTH clause, and vice versa. Multiple VECTORLENGTHFOR clauses cause a syntax error.

LINEAR (var1:step1 [, var2:step2]…)

var is a scalar variable; step is a compile-time positive, integer constant expression. For each iteration of a scalar loop, var1 is incremented by step1, var2 is incremented by step2, and so on. Therefore, every iteration of the vector loop increments the variables by VL*step1, VL*step2, …, to VL*stepN, respectively. If more than one step is specified for a var, a compile-time error occurs. Multiple LINEAR clauses are merged as a union. A variable in a LINEAR clause cannot appear in a REDUCTION, PRIVATE, FIRSTPRIVATE, or LASTPRIVATE clause.

REDUCTION (oper:var1[, var2]…)

oper is a reduction operator (+, *,−, .AND., .OR., .EQV., or .NEQV.); var is a scalar variable. Applies the vector reduction indicated by oper to var1, var2, …, varN. A SIMD directive can have multiple reduction clauses using the same or different operators. If more than one reduction operator is associated with a var, a compile-time error occurs. A variable in a REDUCTION clause cannot appear in a LINEAR, PRIVATE, FIRSTPRIVATE, or LASTPRIVATE clause.

[NO]ASSERT

Directs the compiler to assert (produce an error) or not to assert (produce a warning) when the vectorization fails. The default is NOASSERT. If this clause is specified more than once, a compile-time error occurs.

PRIVATE (var1 [, var2]…)

var is a scalar variable. Causes each variable to be private to each iteration of a loop. Its initial and last values are undefined upon entering and exiting the SIMD loop. Multiple PRIVATE clauses are merged as a union. A variable that is part of another variable (for example, as an array or structure element) cannot appear in a PRIVATE clause. A variable in a PRIVATE clause cannot appear in a LINEAR, REDUCTION, FIRSTPRIVATE, or LASTPRIVATE clause.

FIRSTPRIVATE (var1 [, var2]…)

var is a scalar variable. Provides a superset of the functionality provided by the PRIVATE clause. Variables that appear in a FIRSTPRIVATE list are subject to PRIVATE clause semantics. In addition, its initial value is broadcast to all private instances for each iteration upon entering the SIMD loop. A variable in a FIRSTPRIVATE clause can appear in a LASTPRIVATE clause. A variable in a FIRSTPRIVATE clause cannot appear in a LINEAR, REDUCTION, or PRIVATE clause.

LASTPRIVATE (var1 [, var2]…)

var is a scalar variable. Provides a superset of the functionality provided by the PRIVATE clause. Variables that appear in a LASTPRIVATE list are subject to PRIVATE clause semantics. In addition, when the SIMD loop is exited, each variable has the value that resulted from the sequentially last iteration of the SIMD loop (which may be undefined if the last iteration does not assign to the variable). A variable in a LASTPRIVATE clause can appear in a FIRSTPRIVATE clause. A variable in a LASTPRIVATE clause cannot appear in a LINEAR, REDUCTION, or PRIVATE clause.

[NO]VECREMAINDER

Directs the compiler to vectorize (or not to vectorize) the remainder loop when the original loop is vectorized. If !DIR$ VECTOR ALWAYS is specified, the following occurs: if neither the VECREMAINDER or NOVECREMAINDER clause is specified, the compiler overrides efficiency heuristics of the vectorizer and it determines whether the loop can be vectorized. If VECREMAINDER is specified, the compiler vectorizes remainder loops when the original main loop is vectorized. If NOVECREMAINDER is specified, the compiler does not vectorize the remainder loop when the original main loop is vectorized.

Use SIMD directives with care

The SIMD directives are most powerful in forcing vectorization, but come with the danger that the implications of vectorization need to be understood to avoid changing the program behavior in unexpected ways. They should be used with care. First-time users generally make mistakes and learn from them. We advise working on loops where the output values can be tested during development to provide rapid feedback. Some instructors advise “stick in SIMD directives and start debugging.” We find that suggestion a little scary, but it seems to work with many developers quite well. Just be advised that it is hard to see the changes in a loop that are unexpected and problematic. On the other hand, SIMD directives free us from another evil: everything looks good, but we cannot figure out how to get the compiler to do the vectorization. Take your pick: SIMD to force the compiler to vectorize but we have to be careful, or auto-vectorization where the burden is on the compiler to keep things correct to the limits of its ability to analyze an application. The VECTOR and IVDEP directives give us some “in between” options, which share the burden by retaining some compiler checking but giving us more control.

Use VECTOR directives with care

The VECTOR directive should be used with care. Overriding the efficiency heuristics of the compiler should only be done if we are absolutely sure the vectorization will improve performance.

For instance, the compiler normally does not vectorize loops that have a large number of non-unit stride references (compared to the number of unit stride references). In the following example, vectorization would be disabled by default, but the directive overrides this behavior:

!DIR$ VECTOR ALWAYS

 do i = 1, 100, 2

  ! two references with stride 2 follow

  a(i) = b(i)

 enddo

There may be cases where we want to explicitly avoid vectorization of a loop; for example, if vectorization would result in a performance regression rather than an improvement. In these cases, we can use the NOVECTOR directive to disable vectorization of the loop.

In the following example, vectorization would be performed by default, but the directive overrides this behavior:

!DIR$ NOVECTOR

 do i = 1, 100

  a(i) = b(i) + c(i)

 enddo

IVDEP example in fortran

In the following example, the IVDEP directive provides more information about the dependences within the loop, which may enable loop transformations to occur:

  !DIR$ IVDEP

   DO I=1, N

    A(INDARR(I)) = A(INDARR(I)) + B(I)

   END DO

In this case, the scalar execution order follows:

IVDEP directs the compiler to initially assume that when steps 1 and 5 access a common memory location, step 1 always accesses the location first because step 1 occurs earlier in the execution sequence. This approach lets the compiler reorder instructions, as long as it chooses an instruction schedule that maintains the relative order of the array references.

IVDEP examples in C

The loop in this example will not vectorize without the ivdep pragma, since the value of k is not known; vectorization would be illegal if k<0:

void ignore_vec_dep(int *a, int k, int c, int m)

{

 #pragma ivdep

 for (int i = 0; i < m; i++)

  a[i] = a[i + k] * c;

}

The pragma binds only the for loop contained in current function. This includes a for loop contained in a sub-function called by the current function.

The following loop requires the parallel option in addition to the ivdep pragma to indicate there are no loop-carried dependencies:

#pragma ivdep

for (i=1; i<n; i++)

{

 e[ix[2][i]] = e[ix[2][i]]+1.0;

 e[ix[3][i]] = e[ix[3][i]]+2.0;

}

The following loop requires the parallel option in addition to the ivdep pragma to ensure there is no loop-carried dependency for the store into a():

#pragma ivdep

for (j=0; j<n; j++)

{

 a[b[j]] = a[b[j]] + 1;

}

Remainder loop

A remainder loop is created to execute the remaining iterations when the number of loop iterations (trip count) for a vectorized loop is not a multiple of the vector length. While this is unavoidable in many cases, having a large amount of time spent in remainder loops will lead to performance inefficiencies. For example, if the vectorized loop trip count is 20 and the vector length is 16, it means every time the kernel loop gets executed once, the remainder 4 iterations have to be executed in the remainder-loop. Though the Intel Xeon Phi compiler may vectorize the remainder-loop (as reported by -vec-report6), it won’t be as efficient as the kernel loop. For example, the remainder-loop will use masks, and may have to use gathers/scatters instead of unit-strided loads/stores (due to memory-fault-protection issues). The best way to address this is to refactor the algorithm/code in such a way that the remainder-loop is not executed at runtime (by making trip counts a multiple of vector length) and/or making the trip count large compared to the vector length (so that the overhead of any execution in the remainder loop is low).

The compiler optimizations also take into account any knowledge of actual trip count values. So if the trip count is 20, compiler usually makes better decisions if it knows that the trip count is 20 (trip count is a constant known statically to the compiler) as opposed to a trip count of n (symbolic value) that happens to have a value of 20 at runtime (maybe it is an input value read in from a file). In the latter case, we can help the compiler by using a #pragma loop_count (20) in C/C++, or CDEC$ LOOP COUNT (20) in Fortran, before the loop.

Also take into account any unrolling of the vector-loop done by the compiler (by studying output from -vec-report6 option). For example, if the compiler vectorizes a loop (of trip count n and vector length 16) and unrolls the loop by 2 (after vectorization), each kernel loop is designed to execute 32 iterations of the original src loop. If the dynamic trip count happens to be 20, the kernel loop gets skipped completely and all execution will happen in the remainder loop. If we encounter this issue, we can use the #pragma nounroll in C/C++ or CDEC$ NOUNROLL in Fortran to turn off the unrolling of the vector loop. (We can also use the loop_count pragma described earlier instead to influence the compiler heuristics).

If we want to disable vectorization of the remainder loop generated by the compiler, use #pragma vector novecremainder in C/C++ or CDEC$ vector noremainder in Fortran pragma/directive before the loop (using this also disables vectorization of any peel loop generated by the compiler for this loop). We can also use the compiler internal option -mP2OPT_hpo_vec_remainder=F to disable remainder loop vectorization (for all loops in the scope of the compilation). This is typically useful if we are analyzing the assembly code of the vector loop, and we want to identify clearly the vector-kernel loop from the line numbers (otherwise we have to carefully sift through multiple versions of the loop in the assembly—kernel/remainder/peel to identify which one we are looking at).

Peel loop

The compiler generates dynamic peel loops typically to align one of the memory accesses inside the loop. The peel loop peels a few iterations of the original src loop until the candidate memory access gets aligned. The peel loop is guaranteed to have a trip count that is smaller than the vector length. This optimization is done so that the kernel vector loop can utilize more aligned load/store instructions—thus increasing the performance efficiency of the kernel loop. But the peel loop itself (even though it may be vectorized by the compiler) is less efficient (study the-vec-report6 output from the compiler). The best way to address this is to refactor the algorithm/code in such a way that the accesses are aligned and the compiler knows about the alignment following the vectorizer alignment BKMs. If the compiler knows that all accesses are aligned (say if the user correctly uses #pragma vector aligned before the loop so that the compiler can safely assume all memory accesses inside the loop are aligned), then there will be no peel loop generated by the compiler.

We can also use the loop_count pragma described earlier to influence the compiler decision of whether or not to create a peel loop.

We can instruct the compiler to not generate a dynamic peel loop by adding #pragma vector unaligned in C/C++ or CDEC$ vector unaligned in Fortran pragma/directive before the loop in the source.

We can use the vector pragma/directive with the novecremainder clause (as mentioned above) to disable vectorization of the peel loop generated by the compiler. We can also use the compiler internal option -mP2OPT_hpo_vec_peel=F to disable peel-loop vectorization (for all loops in the scope of the compilation).

% cat -n t2.c

#include <stdio.h>

void foo1(float *a, float *b, float *c, int n)

{

 int i;

 #pragma ivdep

 for (i=0; i<n; i++) {

  a[i] *= b[i] + c[i];

 }

}

void foo2(float *a, float *b, float *c, int n)

{

 int i;

 #pragma ivdep

 for (i=0; i<20; i++) {

  a[i] *= b[i] - c[i];

 }

}

For the loop in function foo1, the compiler generates a kernel-vector loop (unrolled after vectorization by a factor of 2), a peel loop and remainder loop, both of which are vectorized. For the loop in function foo2, the compiler takes advantage of the fact that the trip count is a constant (20) and generates a kernel loop that is vectorized (and not unrolled). The remainder loop (of 4 iterations) is completely unrolled by the compiler (and not vectorized). There is no peel loop generated.

1. Align the data

2. Use pragmas/directives in performance critical regions where the data is used to tell the compiler that the arguments are aligned

Step 1: Aligning the data

It is important for performance to align data. It is also important for optimization to inform the compiler of the alignment information in critical regions of the code. If we align data but do not tell the compiler, we can end up getting less optimal code and/or longer compilation time.

How to define aligned STATIC arrays

Here is an example for statically declaring a 1000-element single-precision floating-point array A on a 64-byte boundary, optimal on Windows C/C++:

__declspec(align(64)) float A[1000];

on Linux or OS X C/C++:

float A[1000] __attribute__((aligned(64)));

For Fortran simple arrays, the easiest way to get array data aligned is to use compiler option -align array64byte to get all arrays, static or dynamic, aligned on a 64-byte boundary. This option does not align data in COMMON blocks, nor elements within derived types. We can also align arrays with a directive. Directives in our code remove the need to remember the -align array64byte compiler option by explicitly calling out the alignment in our code where the variable is declared. Here is an example:

real :: A(1000)

!dir$ attributes align: 64:: A

For Fortran COMMON data, use -align zcommons to align all common block entities on 32-byte boundaries by adding padding bytes as needed. This is not the ideal data alignment for Intel Xeon Phi coprocessors, but is ideal for AVX. For the coprocessor, we have a 50/50 chance of full 64-byte alignment. Note: padding bytes may break many legacy applications that assume COMMON entities are packed and have no padding. So be sure to check results from our application to insure correctness with this compiler option.

For a Fortran derived-type data element, use -align recnbyte. To align data elements contained within a derived type on the 64-byte boundary optimal for the coprocessor, use the compiler option -align rec64byte. This option aligns components of derived types and fields within record structures on the boundary specified by (n). Each derived type element after the first is stored on either the size of the member type or n-byte boundaries, whichever is smaller. Fortran module data:

module mymod

real, allocatable :: a(:), b(:)

!dir$ attributes align:64 :: a

!dir$ attributes align:64 :: b

…

end module mymod

For alignment of dynamic data in C/C++ we replace malloc() and free() with alignment-specified replacements _mm_malloc() and _mm_free(). The arguments are identical. These _mm_ replacements provided by the Intel C++ Composer XE use the same argument and return types as malloc() and free(). The returned data from _mm_malloc() will be 64-byte aligned.

_aligned_malloc()

_mm_malloc() and _mm_free()

For alignment of dynamic data in Fortran we use the -align arraynbyte and -align recnbyte compiler options as discussed previously.

Step 2: Inform the compiler of the alignment

Now that we have aligned our data, it is necessary to inform the compiler that this data is aligned where that data is actually used in the program. For example, when we pass data as arguments to a performance-critical function or subroutine, how does the compiler know if the arguments are aligned or unaligned? This information must be provided by us since the compiler has no information on the arguments.

Here’s an example in C/C++: for a specific variable, use the __assume_aligned macro to inform the compiler that a particular variable or argument is aligned. For example, to inform the compiler that an array passed in as an argument or in global data is aligned we would do the following:

void myfunc( double p[] ) {

 __assume_aligned(p, 64);

 for (int i=0; i<N; i++){

  p[i]++;

 }

}

void myfunc2( double *p2, double *p3, double *p4, int n) {

 for (int j=0; j<n; j+=8) {

  __assume_aligned(p2, 64);

  __assume_aligned(p3, 64);

  __assume_aligned(p4, 64);

  p2[j:8] = p3[j:8] * p4[j:8];

 }

}

Here’s an example in Fortran: use the directive ASSUME_ALIGNED. The general syntax is:

cDEC$ ASSUME_ALIGNED address1:n1 [, address2:n2]…

If we specify more than one address:n item, they must be separated by a comma. If address is a Cray POINTER or it has the POINTER attribute, it is the POINTER and not the pointee or the TARGET that is assumed aligned. If we specify an invalid value for n, an error message is displayed.

!DIR$ ASSUME_ALIGNED A: 64

do i=1, N

A(I) = A(I) + 1

end do

!DIR$ ASSUME_ALIGNED A: 64

A = A + 1

How to tell the vectorizer all memory references are nicely aligned for the target

A more general pragma or directive can be put in front of a loop to tell the compiler that all data in the loop is aligned. In this way, we do not have to specify each variable using the methods above.

Example in C/C++

#pragma vector aligned

for (i=0; i<n; i++){

 A[i] = B[i] * C[i] + D[i];

}

//Add pragma just before an array-notation stmt to

//specify alignment for arrays used

#pragma vector aligned

A[0:n] = B[0:n] * C[0:n] + D[0:n];

Examples in Fortran:

!DIR$ VECTOR ALIGNED

do I=1, N

 A(I) = B(I) * C(I) + D(I)

end do

!DIR$ VECTOR ALIGNED

A = B * C + D

Some notes: these clauses override the efficiency heuristics in the compiler vectorizer. These clauses cause the compiler to use aligned data movement instructions for all array references. These clauses disable all the advanced alignment optimizations of the compiler, such as determining alignment properties from the program context or using dynamic loop peeling to make references aligned. Be careful when using these clauses. Instructing the compiler to implement all array references with aligned data movement instructions will cause a runtime exception if some of the access patterns are actually unaligned.

Mixing aligned and unaligned accesses: how to tell the vectorizer all RHS memory references are 64-Byte aligned

Data alignment is a method to force the compiler to create data objects in memory on specific byte boundaries. This is done to increase efficiency of data loads and stores to and from the processor. For The Intel® Many Integrated Core Architecture (Intel® MIC Architecture) products such as the Intel Xeon Phi coprocessor, memory movement is optimal when the data starting address lies on 64-byte boundaries.

In addition to creating the data on aligned boundaries, the compiler is able to make optimizations when the data is known to be aligned by 64 bytes. Thus, we must also inform the compiler of this alignment via pragmas (C/C++) or directives (Fortran) so that the compiler can generate optimal code. The one exception is that Fortran module data receives alignment information at USE sites.

This example shows alignment being asserted to help the compiler optimize as much as possible:

float *p, *p1;

__assume_aligned(p,64);

__assume_aligned(p1,64);

__assume(n1%16==0);

__assume(n2%16==0);

__assume(n3%16==0);

__assume(n4%16==0);

__assume(n5%16==0);

for(i=0;i<n;i++){

 q[i] = p[i]+p[i+n1]+p[i+n2]+p[i+n3]+p[i+n4]+p[i+n5];

}

for(i=0;i<n;i++){

 q1[i] =

  p1[i]+p1[i+n1]+p1[i+n2]+p1[i+n3]+p1[i+n4]+p1[i+n5];

}

1. The temporary array tmp, which was a single scalar value in the original code, is now a large array. This data is reused several times in the algorithm, but does not fit in the cache and must be reloaded from memory. It may even lead to stack allocation problems.

2. The array B has the same problem: it is reused but does not fit in the cache.

3. The size S array section operations are much larger than hardware vector length. The compiler must decide how to break them up for efficient vectorization.

Subscript triplets

A subscript triplet is a set of three values representing the lower bound of the array section, the upper bound of the array section, and the increment (stride) between them. It takes the following form [first-bound] : [last-bound] [:stride], where first-bound is a scalar integer (or other numeric) expression representing the first value in the subscript sequence. If omitted, the declared lower bound of the dimension is used, last-bound is a scalar integer (or other numeric) expression representing the last value in the subscript sequence. If omitted, the declared upper bound of the dimension is used. When indicating sections of an assumed-size array, this subscript must be specified. Here stride is a scalar integer (or other numeric) expression representing the increment between successive subscripts in the sequence. It must have a nonzero value. If it is omitted, it is assumed to be 1.

If the stride is positive, the subscript range starts with the first subscript and is incremented by the value of the stride, until the largest value less than or equal to the second subscript is attained. If the first subscript is greater than the second subscript, the range is empty. If the stride is negative, the subscript range starts with the value of the first subscript and is decremented by the absolute value of the stride, until the smallest value greater than or equal to the second subscript is attained. If the second subscript is greater than the first subscript, the range is empty. If a range specified by the stride is empty, the array section has a size of zero.

A subscript in a subscript triplet need not be within the declared bounds for that dimension if all values used to select the array elements are within the declared bounds. For example, if an array has been declared as A(15), the array section specified as A(4:16:10) is valid. The section is a rank-one array with shape (2) and size 2. It consists of elements A(4) and A(14).

If the subscript triplet does not specify bounds or stride, but only a colon (:), the entire declared range for the dimension is used. If all subscripts are omitted, the section defaults to the entire extent in that dimension.

Vector subscripts

A vector subscript is a one-dimensional (rank one) array of integer values (within the declared bounds for the dimension) that selects a section of a whole (parent) array. The elements in the section do not have to be in order and the section can contain duplicate values. An array section with a vector subscript that has two or more elements with the same value is called a many-one array section. A many-one section must not appear on the left of the equal sign in an assignment statement, or as an input item in a READ statement.

Implications for array copies, efficiency issues

Unlike the array section definition for C/C++ with Cilk Plus, the Fortran language semantics sometimes require the compiler to make a temporary copy of an array or array slice. Situations where this can occur include:

By default, these temporary values are created on the stack and, if large, may result in a “stack overflow” error at runtime. The size of the stack can be increased, but with limitations dependent on the operating system. Use of the /heap-arrays (Windows) or -heap-arrays (Linux) compiler option tells the compiler to use heap allocation, rather than the stack, for such temporary copies. Heap allocation adds a small amount of overhead when creating and removing the temporary values, but this is usually inconsequential in comparison to the rest of the code.

Performance can be further improved by eliminating the need for a temporary copy entirely. For the first case above, passing a noncontiguous array to a procedure expecting a contiguous array, enabling the /check: arg_temp_created (Windows OS) or -check arg_temp_created (Linux OS) compiler option, will produce a runtime informational message when the compiler determines that the argument being passed is not contiguous. A runtime test is made and, if the argument is contiguous, no copy is made. However, this option will not issue a diagnostic for other uses of temporary copies.

One way to avoid temporary copies for array arguments is to change the called procedure to declare the array as assumed-shape, with the DIMENSION(:) attribute. Such procedures require an explicit interface to be visible to the caller. This is best provided by placing the called procedure in a module or a CONTAINS section. As an alternative, an INTERFACE block can be declared.

Use of POINTER arrays makes it difficult for the compiler to know if a temporary value can be avoided. Where possible, replace POINTER with ALLOCATABLE, especially as components of derived types. The language definition allows the compiler to assume that ALLOCATABLE arrays are contiguous and that they do not overlap other variables, unlike POINTERs.

Another situation where the temporary values can be created is for automatic arrays, where an array’s bounds are dependent on a routine argument, use or host associated variable, or COMMON variable, and the array is a local variable in the procedure. As above, these automatic arrays are created on the stack by default; the /heap-arrays (Windows) or -heap-arrays (Linux) compiler option will create them on the heap. Consider making such arrays ALLOCATABLE instead; local ALLOCATABLE variables that do not have the SAVE attribute are automatically deallocated when the routine exits. For example, replace:

SUBROUTINE SUB (N)

INTEGER, INTENT(IN) :: N

REAL :: A(N)

with:

SUBROUTINE SUB(N)

INTEGER, INTENT(IN) :: N

REAL, ALLOCATABLE :: A(:)

ALLOCATE (A(N))

Specifying array sections

An array section operator is written as one of the following:

[first:length:stride]

[first:length]

[:]

where:

All three of these values must be integers, but they can be computed at runtime (they do not need to be known at compile time). The notation expr[:] is a shorthand for a whole array dimension if expr has array type before decay (conversion to pointer type) and the size of the array is known. If either first or length is specified, then both must be specified.

Operations on array sections

Most C and C++ scalar operations act element-wise on array sections and return an element-wise result. For example, a[10:n]-b[20:n] returns an array section of length n where the j element is a[10+j]-b[20+j]. Each operand must have the same shape, unless it is a scalar operand. Scalar operands are reshaped by replication to match the shape of the non-scalar operands. Function calls are also applied element-wise. The few operators that are not applied element-wise or have peculiar rank rules are:

Note that pointer arithmetic follows the element-wise rule just like other arithmetic. A consequence is that a[i] is not always the same as *(a+i) when array sections are involved. For example, if both a and i have rank one, then a[i] has rank two, but *(a+i) has rank one, because it is element-wise unary * applied to the result of element-wise addition.

Historical note: In the design of array notation, an alternative was explored that preserved the identity ∗(a + i) ≡ a[i], but it broke the identity (a + i) + j ≡ a + (i + j) when a is a pointer type, and made the rank of a+i dependent on the type (not just the rank) of a. It turns out that array notation must break one of the two identities, and breaking associativity was deemed the worse of two evils.

Reductions on array sections

There are built-in operations for efficient reductions of array sections. For example, _sec_reduce_add(a[0:n]) sums the values of array section a[0:n]. Table 5.2 is a summary of the built-in operations. The last column shows the result of reducing a zero-length section.

The “−∞” and “∞” are shorthand for the minimum and maximum representable values of the type. The result of a reduction is always a scalar. For most of these reductions, the rank of a can be one or greater. The exception is that the rank of a must be one for _sec_reduce_max_ind and _sec_reduce_min_ind.

Avoid partial overlap of array sections

In C and C++, the effect of a structure assignment *p=*q is undefined if *p and *q point to structures that partially overlap in memory. The assignment is well defined if *p and *q are either completely disjoint or are aliases for exactly the same structure. Cilk Plus extends this rule to array sections. Examples:

extern float a[15];

a[0:4] = a[5:4]; // Okay, disjoint

a[0:5] = a[4:5]; // WRONG! Partial overlap

a[0:5] = a[0:5]+1;   // Okay, exact overlap

a[0:5:2] = a[1:5:2]; // Okay, disjoint, no locations shared

a[0:5:2] =  a[1:5:3]; // WRONG! Partial overlap (both share a [4])

a[0:5] = a[5:5]+a[6:5]; // Okay, reads can partially overlap

The last example shows how partial overlap of reads is okay. It is partial overlap of a write with another read or write that is undefined. This definition for array notation, which makes partial overlap ill defined, is different from the “well-defined but inefficient” choice made by APL and Fortran. Experience showed that doing so required a compiler to often generate temporary arrays, so it could fully evaluate the right side of a statement before doing an assignment. These temporary arrays hurt performance and caused unpredictable space consumption, both at odds with the C++ philosophy of providing abstractions with minimal performance penalty. So the specification was changed to match the rules for structure assignment in C/C++. Perhaps future compilers will offer to insert partial overlap checks into code for debugging.

Elemental functions

An elemental function is a scalar function with markup that tells the compiler to generate extra versions of it optimized to evaluate multiple iterations in parallel. These are well-suited to converting legacy code where such functions already exist and can be converted without changing the rest of the program. It is a convenient notation that is useful to consider in writing new code as well because it holds up well over time as hardware vector lengths change.

When we call an elemental function from a parallel context, the compiler can call the parallel version instead of the serial version, even if the function is defined in a different source file than the calling context. The steps for using an elemental function are:

The examples in Figures 5.23 and 5.24 show definition and use respectively of an elemental function. This code will likely perform better than a program where the function is not marked as elemental, particularly when the function is defined in a separate file. Writing in this manner exposes the opportunity explicitly instead of hoping that a super-optimizing compiler will discover the opportunity, which is particularly important in examples less trivial than this one.

Specifying unit-stride accesses inside elemental functions

If an elemental function accesses memory in unit-stride, these are the two ways we can write the function to achieve good performance, as shown in Figure 5.25.

Symptom 1: usage of unaligned loads and stores

Loading or storing unaligned data in a loop can result in the compiler using two instructions for each memory access. Generally, the compiler will try to dynamically align accesses using peel/remainder loops as discussed above, but in some situations this may not be possible. If the vloadunpackld/hd (low data/high data) or vstoreunpackld/hd instruction pairs are observed in a main loop body (not a peel or remainder loop), it indicates compiler was not able to enforce alignment dynamically and this is likely impacting performance. To address this issue, you should both align your data accesses and tell the compiler they are aligned. Both are important. Alignment is discussed in the section “Data Layout, Alignment, Prefetching, and So On” earlier in this chapter. In an illustrative example, a simple for loop (Figure 5.28) results in unaligned loads and stores (Figure 5.29). After telling the compiler that alignment is enforced using the #pragma vector aligned, the vmovaps instruction can be used just once for each data access (Figure 5.30) for a much more efficient loop.

Symptom 2: usage of %k1 mask

If the vector instructions for the Xeon Phi coprocessor are all masked, this may indicate inefficiencies. Masks are used for marking which elements in a vector register to which to apply the operation. Instructions that are generated by the compiler with a %k0 mask apply to every element in a vector register – mask %k0 is automatically all 1 s. The compiler can use a different mask with an instruction to indicate that it should apply to less than a full vector (to fewer than all elements in a 512bit vector register). For a thorough explanation of how the masks are used, see section 2.1.2 of the Intel® Xeon Phi™ Coprocessor Instruction Set Architecture Reference Manual.

If scalar code (not vectorized) is being generated, typically the compiler will still use a vector instruction with a mask of %k1. Earlier in the code the compiler will have set %k1 to have a value of 1 – so, the vector instruction being used will only apply to 1 element in the vector register. Seeing the %k1 mask being used is not a definite sign of an issue – but, it is likely that less than full registers are being operated on and so vectorization is not as efficient as it could be. Again, this is something to be concerned about when looking at the assembly code for the main loop body, which should have aligned memory accesses and operate on full vector registers. Peel and remainder loops will have instructions operation on fewer elements and so may use masking effectively.

If this symptom is present, examine the compiler vectorization report for the loops in question. Using the highest level of diagnostic information (level 6) gives information on which loops were not vectorized and why. Figure 5.31 shows instructions not operating on full vectors. Instructions operating on full vectors are shown in Figure 5.32. Or, depending on the format, the %k0 mask may be left out to indicate no mask is being given as shown in Figure 5.33.

Symptom 3: usage of scatters or gathers

Like symptoms 1 and 2, usage of scatter or gather instructions, see Figure 5.34, should cause concern only if observed in a main loop body and not a peel or remainder loop. Scatter and gather instructions are used to load or store multiple elements at a time. Gathers load elements into a vector register to be operated on, and scatters store elements back to multiple memory locations (cachelines) at once. They are useful instructions in the case where the application code is performing true indirect memory accesses, such as in a sparse data structure. However, due to their nature of loading/storing many elements they can have a long execution latency. In some cases, such as strided accesses, the application code can be restructured to have stride 1 access and avoid long latency scatter/gather operations. (When memory is accessed with a stride of 1, a single cacheline can hold a full vector register’s worth of data to be operated on.)

A typical case of strided accesses is an array of structures, often seen in molecular dynamics codes. These arrays holding structs (X, Y, and Z coordinates for example) are usually accessed in a strided fashion—moving through memory accessing all the X elements for example. Restructuring the code to access a structure of arrays—with all the Xs in one array, the Ys in another array, and so on, can change the access pattern to stride 1, which is far more beneficial. Section 5 of the Guide to Vectorization with Intel C++ Compilers (see “For More Information” at end of this chapter) discusses the structure of arrays concept with an example.

Symptom 4: lack of prefetch instructions

Software prefetch is an important strategy for improving performance on the Intel Xeon Phi coprocessor. Within loops, the compiler will usually insert prefetch instructions into code for you. One prefetch methodology used by the compiler is to prefetch data first into the local L2 cache with a vprefetch1 instruction, and then into the L1 cache with a vprefetch0 instruction. Typically you will see 1 or 2 prefetch instructions prior to each data load in a loop, as in Figure 5.35, and these could be a mixture of vprefetch0 and vprefetch1, and sometimes vprefetche1 or 2 (for stores). The absence of these prefetch instructions could indicate that the compiler is using an alternate prefetching methodology. If you see few or no prefetch instructions, and are seeing a poor L1 hit rate (see Chapter 13), it may be worth trying some of the compiler pragmas governing prefetch. A number of prefetch options are configurable and may yield better performance. The earlier section “Prefetching” in this chapter discusses these options.

Symptoms: quick inspections

The quick inspections we have shown give some things to look for in assembly code that may point to less than optimal code being generated for the Intel Xeon Phi coprocessor. These inspections can be useful tool in searching for top performance, and do not require great expertise in assembly language code.