11.2.2.1.1 Notation

There are three main linkage methods in hierarchical agglomeration schedules, as shown in Fig. 11.9: the nearest-neighbor or single-linkage, the furthest-neighbor or complete-linkage, and the between-groups or average-linkage.

Table 11.11 illustrates the distance to be considered in each clustering stage, based on the linkage method chosen.

Table 11.11

Distance to be Considered Based on the Linkage Method

Linkage Method	Illustration	Distance (Dissimilarity)
Single (Nearest-Neighbor or Single-Linkage)		d23
Complete (Furthest-Neighbor or Complete-Linkage)		d15
Average (Between-Groups or Average-Linkage)		$\frac{d_{13}+d_{14}+d_{15}+d_{23}+d_{24}+d_{25}}{6}$

The single-linkage method favors the shortest distances (thus, the nomenclature nearest neighbor) so that new clusters can be formed at each clustering stage through the incorporation of observations or groups. Therefore, applying it is advisable in cases in which the observations are relatively far apart, that is, different, and we would like to form clusters considering a minimum of homogeneity. On the other hand, its analysis may be hampered when there are observations or clusters just a little farther apart from each other, as shown in Fig. 11.10.

The complete-linkage method, on the other hand, goes in the opposite direction, that is, it favors the greatest distances between the observations or groups so that new clusters can be formed (hence, the name furthest neighbor) and, in this regard, using it is advisable in cases in which there is no considerable distance between the observations, and the researcher needs to identify the heterogeneities between them.

Finally, in the average-linkage method, two groups merge based on the average distance between all the pairs of observations that are in these groups (hence, the name average linkage). Accordingly, even though there are changes in the calculation of the distance measures between the clusters, the average-linkage method ends up preserving the order of the observations in each group, offered by the single-linkage method, in case there is a considerable distance between the observations. The same happens with the sorting solution provided by the complete-linkage method, if the observations are very close to each other.

Johnson and Wichern (2007) proposed a logical sequence of steps in order to facilitate the understanding of a cluster analysis, elaborated through a certain hierarchical agglomerative method:

1. 1. If n is the number of observations in a dataset, we must start the agglomeration schedule with exactly n individual groups (stage 0), such that we will initially have a distances (or similarities) matrix D₀ formed by the distances between each pair of observations.
2. 2. In the first stage, we must choose the smallest distance among all of those that form matrix D₀, that is, the one that connects the two most similar observations. At this exact moment, we will not have n individual groups any longer, we will have (n − 1) groups, and one of them is formed by two observations.
3. 3. In the following clustering stage, we must repeat the previous stage. However, we now have to take the distance between each pair of observations, and between the first group already formed, and each one of the other observations into consideration, based on one of the linkage methods adopted. In other words, we will have, after the first clustering stage, matrix D₁ with dimensions (n − 1) × (n − 1), in which one of the rows will be represented by the first grouped pair of observations. Consequently, in the second stage, a new group will be formed by the grouping of two new observations or by adding a certain observation to the first group previously formed in the first stage.
4. 4. The previous process must be repeated (n − 1) times, until there is only a single group formed by all the observations. In other words, in the stage (n − 2) we will have matrix D_n-2 that will only contain the distance between the last two remaining groups, before the final fusion.
5. 5. Finally, from the clustering stages and the distances between the clusters formed, it is possible to develop a tree-shaped diagram that summarizes the clustering process, and explains the allocation of each observation in each cluster. This diagram is known as a dendrogram or a phenogram.

Therefore, the values that form the D matrices of each one of the stages will be a function of the distance measure chosen and of the linkage method adopted. In a certain clustering stage s, imagine that a researcher groups two clusters M and N formed previously, containing observations m and n, respectively, so that cluster MN can be formed. Next, he intends to group MN with another cluster W, with w observations. Since we know that the decision to choose the next cluster will always be the smallest distance between each pair of observations or groups in the hierarchical agglomerative methods, the agglomeration schedule will be essential in order for the distances that will form each matrix D_s to be analyzed. Using this logic and based on Table 11.11, let’s discuss the criterion to calculate the distance between the clusters MN and W, inserted in matrix D_s, based on the linkage method:

• • Nearest-Neighbor or Single-Linkage Method:

$d_{\left(\mathit{MN}\right)W}=min\left\{d_{\mathit{MW}};d_{\mathit{NW}}\right\}$

where d_MW and d_NW are the distances between the closest observations in clusters M and W and in clusters N and W, respectively.

• • Furthest-Neighbor or Complete-Linkage Method:

$d_{\left(\mathit{MN}\right)W}=max\left\{d_{\mathit{MW}};d_{\mathit{NW}}\right\}$

where d_MW and d_NW are the distances between the farthest observations in clusters M and W and in clusters N and W, respectively.

• • Between-Groups or Average-Linkage Method:

$d_{\left(\mathit{MN}\right)W}=\frac{\sum _{p=1}^{m+n}\sum _{q=1}^w{d}_{\mathit{pq}}}{\left(m+n\right)\cdot \left(w\right)}$

where d_pq represents the distance between any observation p in cluster MN and any observation q in cluster W, and m + n and w represent the number of observations in clusters MN and W, respectively.

In the following section, we will present a practical example that will be solved algebraically, and from which the concepts of hierarchical agglomerative methods will be established.

11.2.2.1.2 A Practical Example of Cluster Analysis With Hierarchical Agglomeration Schedules

Imagine that a college professor, who is very concerned about his students’ capacity to learn the subject he teaches, Quantitative Methods, is interested in allocating them to groups with the highest homogeneity possible, based on the grades they obtained on the college entrance exams in subjects considered quantitative (Math, Physics, and Chemistry).

In order to do that, the professor collected information on these grades, which vary from 0 to 10. In addition, since he will carry out a cluster analysis, first, in an algebraic way, he decided, for pedagogical purposes, to only work with five students. This dataset can be seen in Table 11.12.

Based on the data obtained, the chart in Fig. 11.11 is constructed, and, since the variables are metric, the dissimilarity measure known as Euclidian distance will be used for the cluster analysis. Besides, since all the variables have values in the same unit of 0 measure (grades from 0 to 10), in this case, it will not be necessary to standardize them through Z-scores.

In the following sections, hierarchical agglomeration schedules based on the Euclidian distance will be elaborated through the three linkage methods being studied.

11.2.2.1.2.1 Nearest-Neighbor or Single-Linkage Method

At this moment, from the data presented in Table 11.12, let’s develop a cluster analysis through a hierarchical agglomeration schedule with the single-linkage method. First of all, we define matrix D₀, formed by the Euclidian distances (dissimilarities) between each pair of observations, as follows:

It is important to mention that at this initial moment each observation is considered an individual cluster, that is, in stage 0, we have 5 clusters (sample size). Highlighted in matrix D₀ is the smallest distance between all the observations and, therefore, in the first stage, observations Gabriela and Ovidio will initially be grouped, and will now be a new cluster.

We must construct matrix D₁ so that we can go to the next clustering stage, in which the distances between the cluster Gabriela-Ovidio and the other observations are calculated. Observations that are still isolated. Thus, by using the single-linkage method and based on the Expression (11.23), we have:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}\right)\mathrm{Luiz\; Felipe}}=min\left\{10.132;10.290\right\}=10.132$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovídio}\right)\mathrm{Patricia}}=min\left\{8.420;6.580\right\}=6.580$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovídio}\right)\mathrm{Leonor}}=min\left\{4.170;5.474\right\}=4.170$

Matrix D₁ can be seen:

In the same way, in matrix D₁ the smallest distance between all of them is highlighted. Therefore, in the second stage, observation Leonor is inserted into the already-formed cluster Gabriela-Ovidio. Observations Luiz Felipe and Patricia still remain isolated.

We must construct matrix D₂ so that we can take the next step, in which the distances between the cluster Gabriela-Ovidio-Leonor and the two remaining observations are calculated. Analogously, we have:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)\mathrm{Luiz\; Felipe}}=min\left\{10.132;8.223\right\}=8.223$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)\mathrm{Patricia}}=min\left\{6.580;6.045\right\}=6.045$

Matrix D₂ can be written as:

In the third clustering stage, observation Patricia is incorporated into the cluster Gabriela-Ovidio-Leonor, since the corresponding distance is the smallest among all the ones presented in matrix D₂. Therefore, we can write matrix D₃, which comes next, taking into consideration the following criterion:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}-\mathrm{Patricia}\right)\mathrm{Luiz\; Felipe}}=min\left\{8.223;7.187\right\}=7.187$

Finally, in the fourth and last stage, all the observations are allocated to the same cluster, thus, concluding the hierarchical process. Table 11.13 presents a summary of this agglomeration schedule constructed by using the single-linkage method.

Table 11.13

Agglomeration Schedule Through the Single-Linkage Method

Stage	Cluster	Grouped Observation	Smallest Euclidian Distance
1	Gabriela	Ovidio	3.713
2	Gabriela-Ovidio	Leonor	4.170
3	Gabriela-Ovidio-Leonor	Patricia	6.045
4	Gabriela-Ovidio-Leonor-Patricia	Luiz Felipe	7.187

Based on this agglomeration schedule, we can construct a tree-shaped diagram, known as a dendrogram or phenogram, whose main objective is to illustrate the step by step of the clusters and to facilitate the visualization of how each observation is allocated to each stage. The dendrogram can be seen in Fig. 11.12.

Through Figs. 11.13 and 11.14, we are able to interpret the dendrogram constructed.

First of all, we drew three lines (I, II, and III) that are orthogonal to the dendrogram lines, as shown in Fig. 11.13, which allow us to identify the number of clusters in each clustering stage, as well as the observations in each cluster.

Therefore, line I “cuts” the dendrogram immediately after the first clustering stage and, at this moment, we can verify that there are four clusters (four intersections with the dendrogram’s horizontal lines), one of them formed by observations Gabriela and Ovidio, and the others, by the individual observations.

On the other hand, line II intersects three horizontal lines of the dendrogram, which means that, after the second stage, in which observation Leonor was incorporated into the already formed cluster Gabriela-Ovidio, there are three clusters.

Finally, line III is drawn immediately after the third stage, in which observation Patricia merges with the cluster Gabriela-Ovidio-Leonor. Since two intersections between this line and the dendrogram’s horizontal lines are identified, we can see that observation Luiz Felipe remains isolated, while the others form a single cluster.

Besides providing a study of the number of clusters in each clustering stage and of the allocation of observations, a dendrogram also allows the researcher to analyze the magnitude of the distance leaps in order to establish the clusters. A high magnitude leap, in comparison to the others, can indicate that a certain observation or cluster, a considerably different one, is incorporated into already formed clusters, which offers subsidies for the establishment of a solution regarding the number of clusters without the need for a next clustering stage.

Although we know that setting an inflexible, mandatory number of clusters may hamper the analysis, at least giving an idea of this number, given the distance measure used and the linkage method adopted, may help researchers better understand the characteristics of the observations that led to this fact. Moreover, since the number of clusters is important for constructing nonhierarchical agglomeration schedules, this piece of information (considered an output of the hierarchical schedule) may serve as input for the k-means procedure.

Fig. 11.14 presents three distance leaps (A, B, and C), regarding each one of the clustering stages, and, from their analysis, we can see that leap B, which represents the incorporation of observation Patricia into the cluster that had already been formed Gabriela-Ovidio-Leonor, is the greatest of the three. Therefore, in case we intend to set the ideal number of clusters in this example, the researcher may choose the solution with three clusters (line II in Fig. 11.13), without the stage in which observation Patricia is incorporated, since it possibly has characteristics that are not so homogeneous and that make it unfeasible to include it in the previously formed cluster, given the large distance leap. Thus, in this case, we would have a cluster formed by Gabriela, Ovidio, and Leonor, another one formed only by Patricia, and a third one formed only by Luiz Felipe.

When using dissimilarity measures in methods clustering, a very useful criterion for identifying the number of clusters consists in identifying a considerable distance leap (whenever possible), and defining the number of clusters formed in the clustering stage immediately before the great leap, since very high leaps may incorporate observations with characteristics that are not so homogeneous.

Furthermore, it is also important to mention that, if the distance leaps from a stage to another are small, due to the existence of variables with values that are too close to the observations, which can make it difficult to read the dendrogram, the researcher may use the squared Euclidean distance, so that the leaps can become clearer and better explained, making it easier to identify the clusters in the dendrogram, and providing better arguments for the decision making process.

Software such as SPSS shows dendrograms with rescaled distance measures, in order to facilitate the interpretation of the allocation of each observation and the visualization of the large distance leaps.

Fig. 11.15 illustrates how clusters can be established after the single-linkage method is elaborated.

Next, we will develop the same example. However, now, let’s use the complete- and average-linkage methods, so that we can compare the order of the observations and the distance leaps.

11.2.2.1.2.2 Furthest-Neighbor or Complete-Linkage Method

Matrix D₀, shown here, is obviously the same, and the smallest Euclidian distance, the one highlighted, is between observations Gabriela and Ovidio that become the first cluster. It is important to emphasize that the first cluster will always be the same, regardless of the linkage method used, since the first stage will always consider the smallest distance between two pairs of observations, which are still isolated.

In the complete-linkage method, we must use Expression (11.24) to construct matrix D₁, as follows:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}\right)\mathrm{Luiz\; Felipe}}=max\left\{10.132;10.290\right\}=10.290$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}\right)\mathrm{Patricia}}=max\left\{8.420;6.580\right\}=8.420$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}\right)\mathrm{Leonor}}=max\left\{4.170;5.474\right\}=5.474$

Matrix D₁ can be seen and by analyzing it, we can see that observation Leonor will be incorporated into the cluster formed by Gabriela and Ovidio. Once again, the smallest value, among all the ones shown in matrix D₁, is highlighted.

As verified when using the single-linkage method, here, observations Luiz Felipe and Patricia also remain isolated at this stage. The differences between the methods start arising now. Therefore, we will construct matrix D₂ using the following criteria:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)\mathrm{Luiz\; Felipe}}=max\left\{10.290;8.223\right\}=10.290$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)\mathrm{Patricia}}=max\left\{8.420;6.045\right\}=8.420$

Matrix D₂ can be written as follows:

In the third clustering stage, a new cluster is formed by the fusion of observations Patricia and Luiz Felipe, since the furthest-neighbor criterion adopted in the complete-linkage method makes the distance between these two observations become the smallest among all the ones calculated to construct matrix D₂. Therefore, notice that at this stage differences related to the single-linkage method appear, in terms of the sorting and allocation of the observations to groups.

Hence, to construct matrix D₃, we must take the following criterion into consideration:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)\left(\mathrm{Luiz\; Felipe}-\mathrm{Patricia}\right)}=max\left\{10.290;8.420\right\}=10.290$

In the same way, in the fourth and last stage, all the observations are allocated to the same cluster, since there is the clustering between Gabriela-Ovidio-Leonor and Luiz Felipe-Patricia. Table 11.14 shows a summary of this agglomeration schedule, elaborated by using the complete-linkage method.

Table 11.14

Agglomeration Schedule Through the Complete-Linkage Method

Stage	Cluster	Grouped Observation	Smallest Euclidian Distance
1	Gabriela	Ovidio	3.713
2	Gabriela-Ovidio	Leonor	5.474
3	Luiz Felipe	Patricia	7.187
4	Gabriela-Ovidio-Leonor	Luiz Felipe-Patricia	10.290

This agglomeration schedule’s dendrogram can be seen in Fig. 11.16. We can initially see that the sorting of the observations is different from what was observed in the dendrogram seen in Fig. 11.12.

Analogous to what was carried out in the previous method, we chose to draw two vertical lines (I and II) over the largest distance leap, as shown in Fig. 11.17.

Thus, if the researcher chooses to consider three clusters, the solution will be the same as the one achieved previously through the single-linkage method, one formed by Gabriela, Ovidio, and Leonor, another one by Luiz Felipe, and a third one by Patricia (line I in Fig. 11.17). However, if he chooses to define two clusters (line II), the solution will be different since, in this case, the second cluster will be formed by Luiz Felipe and Patricia, while in the previous case, it was formed only by Luiz Felipe, since observation Patricia was allocated to the first cluster.

Similar to what was done in the previous method, Fig. 11.18 illustrates how the clusters can be established after the complete-linkage method is carried out.

Defining the clustering method can be based on the application of the average-linkage method, in which two groups merge based on the average distance between all the pairs of observations that belong to these groups. Therefore, as we have already discussed, if the most suitable method is the single linkage because there are observations considerably far apart from one another, the sorting and allocation of the observations will be maintained by the average-linkage method. On the other hand, the outputs of this method will show consistency with the solution achieved through the complete-linkage method as regards the sorting and allocation of the observations, if they are very similar in the variables in study.

Thus, it is advisable for the researcher to apply the three linkage methods when elaborating a cluster analysis through hierarchical agglomeration schedules. Therefore, let’s move on to the average-linkage method.

11.2.2.1.2.3 Between-Groups or Average-Linkage Method

First of all, let’s show the Euclidian distance matrix between each pair of observations (matrix D₀), once again, highlighting the smallest distance between them.

By using Expression (11.25), we are able to calculate the terms of matrix D₁, given that the first cluster Gabriela-Ovidio has already been formed. Thus, we have:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}\right)\mathrm{Luiz\; Felipe}}=\frac{10.132+10.290}{2}=10.211$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}\right)\mathrm{Patricia}}=\frac{8.420+6.580}{2}=7.500$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}\right)\mathrm{Leonor}}=\frac{4.170+5.474}{2}=4.822$

Matrix D₁ can be seen and, through it, we can see that observation Leonor is once again incorporated into the cluster formed by Gabriela and Ovidio. The smallest value among all the ones presented in matrix D₁ has also been highlighted.

In order to construct matrix D₂, in which the distances between the cluster Gabriela-Ovidio-Leonor and the two remaining observations are calculated, we must perform the following calculations:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)\mathrm{Luiz\; Felipe}}=\frac{10.132+10.290+8.223}{3}=9.548$

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)\mathrm{Patr}í\mathrm{cia}}=\frac{8.420+6.580+6.045}{3}=7.015$

Note that the distances used to calculate the dissimilarities to be inserted into matrix D₂ are the original Euclidian distances between each pair of observations, that is, they come from matrix D₀. Matrix D₂ can be seen:

As verified when the single-linkage method was elaborated, here, observation Patricia is also incorporated into the cluster already formed by Gabriela, Ovidio and Leonor, and observation Luiz Felipe remains isolated. Finally, matrix D₃ can be constructed from the following calculation:

$d_{\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}-\mathrm{Patricia}\right)\mathrm{Luiz\; Felipe}}=\frac{10.132+10.290+8.223+7.187}{4}=8.958$

Once again, in the fourth and last stage, all the observations are in the same cluster. Table 11.15 and Fig. 11.19 present a summary of this agglomeration schedule and the corresponding dendrogram, respectively, resulting from this average-linkage method.

Table 11.15

Agglomeration Schedule Through the Average-Linkage Method

Stage	Cluster	Grouped Observation	Smallest Euclidian Distance
1	Gabriela	Ovidio	3.713
2	Gabriela-Ovidio	Leonor	4.822
3	Gabriela-Ovidio-Leonor	Patricia	7.015
4	Gabriela-Ovidio-Leonor-Patricia	Luiz Felipe	8.958

Despite having other distance values, we can see that Table 11.15 and Fig. 11.19 show the same sorting and the same allocation of observations in the clusters as those presented in Table 11.13 and in Fig. 11.12, respectively, obtained when the single-linkage method was elaborated.

Hence, we can state that the observations are significantly different from the variables studied, fact proven by the consistency of the answers obtained from the single- and average-linkage methods. If the observations were more similar, fact that has not been observed in the diagram seen in Fig. 11.11, the consistency of answers would occur between the complete- and average-linkage methods, as already discussed. Therefore, when possible, the initial construction of scatter plots may help researchers, even if in a preliminary way, choose the method to be adopted.

Hierarchical agglomeration schedules are very useful and offer us the possibility to analyze, in an exploratory way, the similarity between observations based on the behavior of certain variables. However, it is essential for researchers to understand that these methods are not conclusive by themselves and more than one answer may be obtained, depending on what is desired and on the data behavior.

Besides, it is necessary for researchers to be aware of how sensitive these methods are to the presence of outliers. The existence of a very discrepant observation may cause other observations, not so similar to one another, to be allocated to the same cluster because they are extremely different from the observation considered an outlier. Hence, it is advisable to apply the hierarchical agglomeration schedules with the linkage method chosen several times, and, in each application, to identify one or more observations considered outliers. This procedure will make the cluster analysis become more reliable, since more and more homogeneous clusters may be formed. Researchers are free to characterize the most discrepant observation as the one that ended up becoming isolated after the penultimate clustering stage, that is, if it happens before the total fusion. Nonetheless, many are the methods to define an outlier. Barnett and Lewis (1994), for instance, mention almost 1000 articles in the existing literature on outliers, and, for pedagogical purposes, in the Appendix of this chapter, we will discuss an efficient procedure in Stata for detecting outliers when a researcher is carrying out a multivariate data analysis.

It is also important to emphasize, as we have already discussed in this section, that different linkage methods, when elaborating hierarchical agglomeration schedules, must be applied to the same dataset, and the resulting dendrograms, compared. This procedure will help researchers in their decision-making processes with regard to choosing the ideal number of clusters, and also to sorting the observations and allocating each one of them to the different clusters formed. This will even allow researchers to make coherent decisions about the number of clusters that may be considered input in a possible nonhierarchical analysis.

Last but not least, it is worth mentioning that the agglomeration schedules presented in this section (Tables 11.13, 11.14, and 11.15) provide increasing values of the clustering measures because a dissimilarity measure was used (Euclidian distance) as a comparison criterion between the observations. If we had chosen Pearson’s correlation between the observations, a similarity measure also used for metric variables, as we discussed in Section 11.2.1.1, the values of the clustering measures in the agglomeration schedules would be decreasing. The latter is also true for cluster analyses in which similarity measures are used, as the ones studied in Section 11.2.1.2, to assess the behavior of observations based on binary variables.

In the following section we will develop the same example, in an algebraic way, using the nonhierarchical k-means agglomeration schedule.

11.2.2.2.1 Notation

As the one developed in Section 11.2.2.1.1, we now present a logical sequence of steps, based on Johnson and Wichern (2007), in order to facilitate the understanding of the cluster analysis (k-means procedure):

1. 1. We define the initial number of clusters and the respective centroids. The main objective is to divide the observations from the dataset into K clusters, such that those within each cluster are the closest to each other if compared to any other that belongs to a different cluster. For such, the observations need to be allocated arbitrarily to the K clusters, so that the respective centroids can be calculated.
2. 2. We must choose a certain observation that is closer to a centroid and reallocate it to this cluster. At this moment, another cluster has just lost that observation, and, therefore, the centroids of the cluster that receives it and of the cluster that loses it must be recalculated.
3. 3. We must continue repeating the previous step until it is no longer possible to reallocate any observation due to its close proximity to a centroid from another cluster.

Centroid coordinate $\overline{x}$ must be recalculated whenever including or excluding a certain observation p in the respective cluster, based on the following expressions:

${\overline{x}}_{\mathit{new}}=\frac{N\cdot \overline{x}+x_p}{N+1},\text{if observation}p\text{is inserted into the cluster under analysis}$

${\overline{x}}_{\mathit{new}}=\frac{N\cdot \overline{x}-x_p}{N-1},\text{if observation}p\text{is excluded from the cluster under analysis}$

where N and $\overline{x}$ refer to the number of observations in the cluster and to its centroid coordinate before the reallocation of that observation, respectively. In addition, x_p refers to the coordinate of observation p, which changed clusters.

For two variables (X₁ and X₂), Fig. 11.20 shows a hypothetical situation that represents the end of the k-means procedure, in which it is no longer possible to reallocate any observation because there are no more close proximities to centroids of other clusters.

The matrix with distances between observations does not need to be defined at each step, different from hierarchical agglomeration schedules, which reduces the requirements in terms of technological capabilities, allowing nonhierarchical agglomeration schedules to be applied to considerably larger dataset than those traditionally studied through hierarchical schedules.

In addition, bear in mind that the variables must be standardized before elaborating the k-means procedure, and in the hierarchical agglomeration schedules too, if the respective values are not in the same unit of measure.

Finally, after concluding this procedure, it is important for researchers to analyze if the values of a certain metric variable differ between the groups defined, that is, if the variability between the clusters is significantly higher than the internal variability of each cluster. The F-test of the one-way analysis of variance, or one-way ANOVA, allows us to develop this analysis, and its null and alternative hypotheses can be defined as follows:

• H₀: the variable under analysis has the same mean in all the groups formed.
• H₁: the variable under analysis has a different mean in at least one of the groups in relation to the others.

Therefore, a single F-test can be applied for each variable, aiming to assess the existence of at least one difference among all the comparison possibilities, and, in this regard, the main advantage of applying it is the fact that adjustments in the discrepant dimensions of the groups do not need to be carried out to analyze several comparisons. On the other hand, rejecting the null hypothesis at a certain significance level, does not allow the researcher to know which group(s) is(are) statistically different from the others in relation to the variable being analyzed.

The F statistical expression, corresponding to this test, is given by the following expression:

$F=\frac{\text{variability between the groups}}{\text{variability within the groups}}=\frac{\frac{\sum _{k=1}^K{N}_k\cdot {\left({\overline{X}}_k-\overline{X}\right)}^2}{K-1}}{\frac{\sum _{\mathit{ki}}{\left(X_{\mathit{ki}}-{\overline{X}}_k\right)}^2}{n-K}}$

where N is the number of observations in the k-th cluster, ${\overline{X}}_k$ is the mean of variable X in the same k-th cluster, $\overline{X}$ is the general average of variable X, and X_ki is the value that variable X takes on for a certain observation i present in the k-th cluster. In addition, K represents the number of clusters to be compared, and n, the sample size.

By using the F statistic, researchers will be able to identify the variables whose means most differ between the groups, that is, those that most contribute to the formation of at least one of the K clusters (highest F statistic), as well as those that do not contribute to the formation of the suggested number of clusters, at a certain significance level.

In the following section, we will discuss a practical example that will be solved algebraically, and from which the concepts of the k-means procedure may be established.

11.2.2.2.2 A Practical Example of a Cluster Analysis With the Nonhierarchical K-Means Agglomeration Schedule

To solve the nonhierarchical k-means agglomeration schedule algebraically, let’s use the data from our own example, which can be found in Table 11.12 and are shown in Table 11.16.

Software packages such as SPSS use the Euclidian distance as the standard dissimilarity measure, reason why we will develop the algebraic procedures based on this measure. This criterion will even allow the results obtained to be compared to the ones found when elaborating the hierarchical agglomeration schedules in Section 11.2.2.1.2, as, in those situations, the Euclidian distance was also used. In the same way, it will not be necessary to standardize the variables through Z-scores, since all of them are in the same unit of measure (grades from 0 to 10). Otherwise, it is crucial for researchers to standardize the variables before elaborating the k-means procedure.

Using the logical sequence presented in Section 11.2.2.2.1, we will develop the k-means procedure with K = 3 clusters. This number of clusters may have come from a decision made by the researcher and based on a certain preliminary criterion, or it was chosen based on the outputs of the hierarchical agglomeration schedules. In our case, the decision was made based on the comparison of the dendrograms that had already been constructed, and by the similarity of the outputs obtained by the single- and average-linkage methods.

Thus, we need to arbitrarily allocate the observations to three clusters, so that the respective centroids can be calculated. Therefore, we can establish that observations Gabriela and Luiz Felipe form the first cluster, Patricia and Ovidio, the second, and Leonor, the third. Table 11.17 shows the arbitrary formation of these preliminary clusters, as well as the calculation of the respective centroid coordinates, which makes the initial step of the k-means procedure algorithm possible.

Table 11.17

Arbitrary Allocation of the Observations in K = 3 Clusters and Calculation of the Centroid Coordinates—Initial Step of the K-Means Procedure

	Centroid Coordinates
Cluster	Variable
	Grade in Mathematics	Grade in Physics	Grade in Chemistry
Gabriela	$\frac{3.7+7.8}{2}=5.75$	$\frac{2.7+8.0}{2}=5.35$	$\frac{9.1+1.5}{2}=5.30$
Luiz Felipe
Patricia	$\frac{8.9+7.0}{2}=7.95$	$\frac{1.0+1.0}{2}=1.00$	$\frac{2.7+9.0}{2}=5.85$
Ovidio
Leonor	3.40	2.00	5.00

Based on these coordinates, we constructed the chart seen in Fig. 11.21, which shows the arbitrary allocation of each observation to its cluster and the respective centroids.

Based on the second step of the logical sequence presented in Section 11.2.2.2.1, we must choose a certain observation and calculate the distance between it and all the cluster centroids, assuming that it is or it is not reallocated to each cluster. Selecting the first observation (Gabriela), for example, we can calculate the distances between it and the centroids of the clusters that have already been formed (Gabriela-Luiz Felipe, Patricia-Ovidio, and Leonor) and, after that, assume that it leaves its cluster (Gabriela-Luiz Felipe), and is inserted into one of the other two clusters, forming the cluster Gabriela-Patricia-Ovidio or Gabriela-Leonor. Thus, from Expressions (11.26) and (11.27), we must recalculate the new centroid coordinates, simulating that, in fact, the reallocation of Gabriela to one of the two clusters takes place, as shown in Table 11.18.

Table 11.18

Simulating the Reallocation of Gabriela and Calculating the New Centroid Coordinates

		Centroid Coordinates
Cluster	Simulation	Variable
		Grade in Mathematics	Grade in Physics	Grade in Chemistry
Luiz Felipe	Excluding Gabriela	$\frac{2\cdot \left(5.75\right)-3.70}{2-1}=7.80$	$\frac{2\cdot \left(5.35\right)-2.70}{2-1}=8.00$	$\frac{2\cdot \left(5.30\right)-9.10}{2-1}=1.50$
Gabriela	Including Gabriela	$\frac{2\cdot \left(7.95\right)+3.70}{2+1}=6.53$	$\frac{2\cdot \left(1.00\right)+2.70}{2+1}=1.57$	$\frac{2\cdot \left(5.85\right)+9.10}{2+1}=6.93$
Patricia
Ovidio
Gabriela	Including Gabriela	$\frac{1\cdot \left(3.40\right)+3.70}{1+1}=3.55$	$\frac{1\cdot \left(2.00\right)+2.70}{1+1}=2.35$	$\frac{1\cdot \left(5.00\right)+9.10}{1+1}=7.05$
Leonor

Obs.: Note that the values calculated for the Luiz Felipe centroid coordinates are exactly the same as this observation’s original coordinates, as shown in Table 11.16.

Thus, from Tables 11.16, 11.17, and 11.18, we can calculate the following Euclidian distances:

• • Assumption that Gabriela is not reallocated:

$d_{\mathrm{Gabriela}-\left(\mathrm{Gabriela}-\mathrm{Luiz\; Felipe}\right)}=\sqrt{{\left(3.70-5.75\right)}^2+{\left(2.70-5.35\right)}^2+{\left(9.10-5.30\right)}^2}=5.066$

$d_{\mathrm{Gabriela}-\left(\mathrm{Patricia}-\mathrm{Ovidio}\right)}=\sqrt{{\left(3.70-7.95\right)}^2+{\left(2.70-1.00\right)}^2+{\left(9.10-5.85\right)}^2}=5.614$

$d_{\mathrm{Gabriela}-\mathrm{Leonor}}=\sqrt{{\left(3.70-3.40\right)}^2+{\left(2.70-2.00\right)}^2+{\left(9.10-5.00\right)}^2}=4.170$

• • Assumption that Gabriela is reallocated:

$d_{\mathrm{Gabriela}-\mathrm{Luiz\; Felipe}}=\sqrt{{\left(3.70-7.80\right)}^2+{\left(2.70-8.00\right)}^2+{\left(9.10-1.50\right)}^2}=10.132$

$d_{\mathrm{Gabriela}-\left(\mathrm{Gabriela}-\mathrm{Patricia}-\mathrm{Ovidio}\right)}=\sqrt{{\left(3.70-6.53\right)}^2+{\left(2.70-1.57\right)}^2+{\left(9.10-6.93\right)}^2}=3.743$

$d_{\mathrm{Gabriela}-\left(\mathrm{Gabriela}-\mathrm{Leonor}\right)}=\sqrt{{\left(3.70-3.55\right)}^2+{\left(2.70-2.35\right)}^2+{\left(9.10-7.05\right)}^2}=2.085$

Since Gabriela is the closest to the Gabriela-Leonor centroid (the shortest Euclidian distance), we must reallocate this observation to the cluster initially formed only by Leonor. So, the cluster in which observation Gabriela was at first (Gabriela-Luiz Felipe) has just lost it, and now Luiz Felipe has become an individual cluster. Therefore, the centroids of the cluster that receives it and the one that loses it must be recalculated. Table 11.19 shows the creation of the new clusters and the calculation of the respective centroid coordinates too.

Table 11.19

New Centroids With the Reallocation of Gabriela

	Centroid Coordinates
Cluster	Variable
	Grade in Mathematics	Grade in Physics	Grade in Chemistry
Luiz Felipe	7.80	8.00	1.50
Patricia	7.95	1.00	5.85
Ovidio
Gabriela	$\frac{3.7+3.4}{2}=3.55$	$\frac{2.7+2.0}{2}=2.35$	$\frac{9.1+5.0}{2}=7.05$
Leonor

Based on these new coordinates, we can construct the chart shown in Fig. 11.22.

Once again, let’s repeat the previous step. At this moment, since observation Luiz Felipe is isolated, let’s simulate the reallocation of the third observation (Patricia). We must calculate the distances between it and the centroids of the clusters that have already been formed (Luiz Felipe, Patricia-Ovidio, and Gabriela-Leonor) and, afterwards, assume that it leaves its cluster (Patricia-Ovidio) and is inserted into one of the other two clusters, forming the cluster Luiz Felipe-Patricia or Gabriela-Patricia-Leonor. Also based on Expressions (11.26) and (11.27), we must recalculate the new centroid coordinates, simulating that, in fact, the reallocation of Patricia to one of these two clusters happens, as shown in Table 11.20.

Table 11.20

Simulation of Patricia’s Reallocation—Next Step of the K-Means Procedure Algorithm

		Centroid Coordinates
Cluster	Simulation	Variable
		Grade in Mathematics	Grade in Physics	Grade in Chemistry
Luiz Felipe	Including Patricia	$\frac{1\cdot \left(7.80\right)+8.90}{1+1}=8.35$	$\frac{1\cdot \left(8.00\right)+1.00}{1+1}=4.50$	$\frac{1\cdot \left(1.50\right)+2.70}{1+1}=2.10$
Patricia
Ovidio	Excluding Patricia	$\frac{2\cdot \left(7.95\right)-8.90}{2-1}=7.00$	$\frac{2\cdot \left(1.00\right)-1.00}{2-1}=1.00$	$\frac{2\cdot \left(5.85\right)-2.70}{2-1}=9.00$
Gabriela	Including Patricia	$\frac{2\cdot \left(3.55\right)+8.90}{2+1}=5.33$	$\frac{2\cdot \left(2.35\right)+1.00}{2+1}=1.90$	$\frac{2\cdot \left(7.05\right)+2.70}{2+1}=5.60$
Patricia
Leonor

Obs.: Note that the values calculated of the Ovidio centroid coordinates are exactly the same as this observation’s original coordinates, as shown in Table 11.16.

Similar to what was carried out when simulating Gabriela’s reallocation, based on Tables 11.16, 11.19, and 11.20, let’s calculate the Euclidian distances between Patricia and each one of the centroids:

• • Assumption that Patricia is not reallocated:

$d_{\mathrm{Patricia}-\mathrm{Luiz\; Felipe}}=\sqrt{{\left(8.90-7.80\right)}^2+{\left(1.00-8.00\right)}^2+{\left(2.70-1.50\right)}^2}=7.187$

$d_{\mathrm{Patricia}-\left(\mathrm{Patricia}-\mathrm{Ovidio}\right)}=\sqrt{{\left(8.90-7.95\right)}^2+{\left(1.00-1.00\right)}^2+{\left(2.70-5.85\right)}^2}=3.290$

$d_{\mathrm{Patricia}-\left(\mathrm{Gabriela}-\mathrm{Leonor}\right)}=\sqrt{{\left(8.90-3.55\right)}^2+{\left(1.00-2.35\right)}^2+{\left(2.70-7.05\right)}^2}=7.026$

• • Assumption that Patricia is reallocated:

$d_{\mathrm{Patricia}-\left(\mathrm{Luiz\; Felipe}-\mathrm{Patricia}\right)}=\sqrt{{\left(8.90-8.35\right)}^2+{\left(1.00-4.50\right)}^2+{\left(2.70-2.10\right)}^2}=3.593$

$d_{\mathrm{Patricia}-\mathrm{Ovidio}}=\sqrt{{\left(8.90-7.00\right)}^2+{\left(1.00-1.00\right)}^2+{\left(2.70-9.00\right)}^2}=6.580$

$d_{\mathrm{Patricia}-\left(\mathrm{Gabriela}-\mathrm{Patricia}-\mathrm{Leonor}\right)}=\sqrt{{\left(8.90-5.33\right)}^2+{\left(1.00-1.90\right)}^2+{\left(2.70-5.60\right)}^2}=4.684$

Bearing in mind that the Euclidian distance between Patricia and the cluster Patricia-Ovidio is the shortest, we have to reallocate it to another cluster and, at this moment, let’s maintain the solution presented in Table 11.19 and in Fig. 11.22.

Next, we will develop the same procedure, however, simulating the reallocation of the fourth observation (Ovidio). Analogously, we must, therefore, calculate the distances between this observation and the centroids of the clusters that have already been formed (Luiz Felipe, Patricia-Ovidio, and Gabriela-Leonor) and, after that, assume that it leaves its cluster (Patricia-Ovidio) and is inserted into one of the other two clusters, forming the cluster Luiz Felipe-Ovidio or Gabriela-Ovidio-Leonor. Once again by using Expressions (11.26) and (11.27), we can recalculate the new centroid coordinates, simulating that, in fact, the reallocation of Ovidio to one of these two clusters takes place, as shown in Table 11.21.

Table 11.21

Simulating Ovidio’s Reallocation—New Step of the K-Means Procedure Algorithm

		Centroid Coordinates
Cluster	Simulation	Variable
		Grade in Mathematics	Grade in Physics	Grade in Chemistry
Luiz Felipe	Including Ovidio	$\frac{1\cdot \left(7.80\right)+7.00}{1+1}=7.40$	$\frac{1\cdot \left(8.00\right)+1.00}{1+1}=4.50$	$\frac{1\cdot \left(1.50\right)+9.00}{1+1}=5.25$
Ovidio
Patricia	Excluding Ovidio	$\frac{2\cdot \left(7.95\right)-7.00}{2-1}=8.90$	$\frac{2\cdot \left(1.00\right)-1.00}{2-1}=1.00$	$\frac{2\cdot \left(5.85\right)-9.00}{2-1}=2.70$
Gabriela	Including Ovidio	$\frac{2\cdot \left(3.55\right)+7.00}{2+1}=4.70$	$\frac{2\cdot \left(2.35\right)+1.00}{2+1}=1.90$	$\frac{2\cdot \left(7.05\right)+9.00}{2+1}=7.70$
Ovidio
Leonor

Obs.: Note that the values calculated of the Patricia centroid coordinates are exactly the same as this observation’s original coordinates, as shown in Table 11.16.

Next, we can see the calculations of the Euclidian distances between Ovidio and each one of the centroids, defined from Tables 11.16, 11.19, and 11.21:

• • Assumption that Ovidio is not reallocated:

$d_{\mathrm{Ovidio}-\mathrm{Luiz\; Felipe}}=\sqrt{{\left(7.00-7.80\right)}^2+{\left(1.00-8.00\right)}^2+{\left(9.00-1.50\right)}^2}=10.290$

$d_{\mathrm{Ovidio}-\left(\mathrm{Patricia}-\mathrm{Ovidio}\right)}=\sqrt{{\left(7.00-7.95\right)}^2+{\left(1.00-1.00\right)}^2+{\left(9.00-5.85\right)}^2}=3.290$

$d_{\mathrm{Ovidio}-\left(\mathrm{Gabriela}-\mathrm{Leonor}\right)}=\sqrt{{\left(7.00-3.55\right)}^2+{\left(1.00-2.35\right)}^2+{\left(9.00-7.05\right)}^2}=4.187$

• • Assumption that Ovidio is reallocated:

$d_{\mathrm{Ovidio}-\left(\mathrm{Luiz\; Felipe}-\mathrm{Ovidio}\right)}=\sqrt{{\left(7.00-7.40\right)}^2+{\left(1.00-4.50\right)}^2+{\left(9.00-5.25\right)}^2}=5.145$

$d_{\mathrm{Ovidio}-\mathrm{Patricia}}=\sqrt{{\left(7.00-8.90\right)}^2+{\left(1.00-1.00\right)}^2+{\left(9.00-2.70\right)}^2}=6.580$

$d_{\mathrm{Ovidio}-\left(\mathrm{Gabriela}-\mathrm{Ovidio}-\mathrm{Leonor}\right)}=\sqrt{{\left(7.00-4.70\right)}^2+{\left(1.00-1.90\right)}^2+{\left(9.00-7.70\right)}^2}=2.791$

In this case, since observation Ovidio is the closest to the centroid of Gabriela-Ovidio-Leonor (the shortest Euclidian distance), we must reallocate this observation to the cluster formed originally by Gabriela and Leonor. Therefore, observation Patricia becomes an individual cluster. Table 11.22 shows the centroid coordinates of clusters Luiz Felipe, Patricia, and Gabriela-Ovidio-Leonor.

We will not carry out the procedure proposed for the fifth observation (Leonor), since it had already fused with observation Gabriela in the first step of the algorithm. We can consider that the k-means procedure is concluded, since it is no longer possible to reallocate any observation due to closer proximity to another cluster’s centroid. Fig. 11.23 shows the allocation of each observation to its cluster and their respective centroids. Note that the solution achieved is equal to the one reached through the single- (Fig. 11.15) and average-linkage methods, when we elaborated the hierarchical agglomeration schedules.

As we have already discussed, we can see that the matrix with the distances between the observations does not need to be defined at each step of the k-means procedure algorithm, different from the hierarchical agglomeration schedules, which reduces the requirements in terms of technological capabilities, allowing nonhierarchical agglomeration schedules to be applied to dataset significantly larger than the ones traditionally studied through hierarchical schedules.

Table 11.23 shows the Euclidian distances between each observation of the original dataset and the centroids of each one of the clusters formed.

We would like to emphasize that this algorithm can be elaborated with another preliminary allocation of the observations to the clusters besides the one chosen in this example. Reapplying the k-means procedure with several arbitrary choices, given K clusters, allows the researcher to assess how stable the clustering procedure is, and to underpin the allocation of the observations to the groups in a consistent way.

After concluding this procedure, it is essential to check, through the F-test of one-way ANOVA, if the values of each one of the three variables considered in the analysis are statistically different between the three clusters. To make the calculation of the F statistics that correspond to this test easier, we constructed Tables 11.24, 11.25, and 11.26, which show the means per cluster and the general mean of the variables mathematics, physics, and chemistry, respectively.

Table 11.24

Means per Cluster and General Mean of the Variable mathematics

Cluster 1	Cluster 2	Cluster 3
$X_{\mathrm{Luiz\; Felipe}}=7.80$	$X_{\mathtt{Patricia}}=8.90$	$X_{\mathtt{Gabriela}}=3.70$
		$X_{\mathtt{Ovidio}}=7.00$
		$X_{\mathtt{Leonor}}=3.40$
${\overline{X}}_1=7.80$	${\overline{X}}_2=8.90$	${\overline{X}}_3=4.70$
$\overline{X}=6.16$

Table 11.25

Means per Cluster and General Mean of the Variable physics

Cluster 1	Cluster 2	Cluster 3
$X_{\mathrm{Luiz\; Felipe}}=8.00$	$X_{\mathtt{Patricia}}=1.00$	$X_{\mathtt{Gabriela}}=2.70$
		$X_{\mathtt{Ovidio}}=1.00$
		$X_{\mathtt{Leonor}}=2.00$
${\overline{X}}_1=8.00$	${\overline{X}}_2=1.00$	${\overline{X}}_3=1.90$
$\overline{X}=2.94$

Table 11.26

Means per Cluster and General Mean of the Variable chemistry

Cluster 1	Cluster 2	Cluster 3
$X_{\mathtt{Luiz\; Felipe}}=1.50$	$X_{\mathtt{Patricia}}=2.70$	$X_{\mathtt{Gabriela}}=9.10$
		$X_{\mathtt{Ovidio}}=9.00$
		$X_{\mathtt{Leonor}}=5.00$
${\overline{X}}_1=1.50$	${\overline{X}}_2=2.70$	${\overline{X}}_3=7.70$
$\overline{X}=5.46$

So, based on the values presented in these tables and by using Expression (11.28), we are able to calculate the variation between the groups and within them for each one of the variables, as well as the respective F statistics. Tables 11.27, 11.28, and 11.29 show these calculations.

Table 11.27

Variation and F Statistic for the Variable mathematics

Variability between the groups	$\frac{{\left(7.80-6.16\right)}^2+{\left(8.90-6.16\right)}^2+3{\left(4.70-6.16\right)}^2}{3-1}=8.296$
Variability within the groups	$\frac{{\left(3.70-4.70\right)}^2+{\left(7.00-4.70\right)}^2+{\left(3.40-4.70\right)}^2}{5-3}=3.990$
F	$\frac{8.296}{3.990}=2.079$

Note: The calculation of the variability within the groups only took cluster 3 into consideration, since the others show variability equal to 0, because they are formed by a single observation.

Table 11.28

Variation and F Statistic for the Variable physics

Variability between the groups	$\frac{{\left(8.00-2.94\right)}^2+{\left(1.00-2.94\right)}^2+3{\left(1.90-2.94\right)}^2}{3-1}=16.306$
Variability within the groups	$\frac{{\left(2.70-1.90\right)}^2+{\left(1.00-1.90\right)}^2+{\left(2.00-1.90\right)}^2}{5-3}=0.730$
F	$\frac{16.306}{0.730}=22.337$

Note: The same as the previous table.

Table 11.29

Variation and F Statistic for the Variable chemistry

Variability between the groups	$\frac{{\left(1.50-5.46\right)}^2+{\left(2.70-5.46\right)}^2+3{\left(7.70-5.46\right)}^2}{3-1}=19.176$
Variability within the groups	$\frac{{\left(9.10-7.70\right)}^2+{\left(9.00-7.70\right)}^2+{\left(5.00-7.70\right)}^2}{5-3}=5.470$
F	$\frac{19.176}{5.470}=3.506$

Note: The same as Table 11.27.

Now, let’s analyze the rejection or not of the null hypothesis of the F-tests for each one of the variables. Since there are two degrees of freedom for the variability between the groups (K – 1 = 2) and two degrees of freedom for the variability within the groups (n – K = 2), by using Table A in the Appendix, we have F_c = 19.00 (critical F at a significance level of 0.05). Therefore, only for the variable physics can we reject the null hypothesis that all the groups formed have the same mean, since F calculated F_cal = 22.337 > F_c = F_2,2,5% = 19.00, So, for this variable, there is at least one group that has a mean that is statistically different from the others. For the variables mathematics and chemistry, however, we cannot reject the test’s null hypothesis at a significance level of 0.05.

Software such as SPSS and Stata do not offer the F_c for the defined degrees of freedom and a certain significance level. However, they offer the F_cal significance level for these degrees of freedom. Thus, instead of analyzing if F_cal > F_c, we must verify if the F_cal significance level is less than 0.05 (5%). Therefore:

If Sig. F (or Prob. F) < 0.05, there is at least one difference between the groups for the variable under analysis.

The F_cal significance level can be obtained in Excel by using the command Formulas → Insert Function → FDIST, which will open a dialog box as the one shown in Fig. 11.24.

As we can see in this figure, sig. F for the variable physics is less than 0.05 (sig. F = 0.043), that is, there is at least one difference between the groups for this variable at a significance level of 0.05. An inquisitive researcher will be able to carry out the same procedure for the variables mathematics and chemistry. In short, Table 11.30 presents the results of the one-way ANOVA, with the variation of each variable, the F statistics, and the respective significance levels.

The one-way ANOVA table still allows the researcher to identify the variables that most contribute to the formation of at least one of the clusters, because they have a mean that is statistically different from at least one of the groups in relation to the others, since they will have greater F statistic values. It is important to mention that F statistic values are very sensitive to the sample size, and, in this case, the variables mathematics and chemistry ended up not having statistically different means among the three groups, mainly because the sample is small (only five observations).

We would like to emphasize that this one-way ANOVA can also be carried out soon after the application of a certain hierarchical agglomeration schedule, since it only depends on the classification of the observations within groups. The researcher must be careful about only one thing, when comparing the results obtained by a hierarchical schedule to the ones obtained by a nonhierarchical schedule, to use the same distance measure in both situations. Different allocations of the observations to the same number of clusters may happen if different distance measures are used in a hierarchical schedule and in a nonhierarchical schedule. Therefore, different values of the F statistics in both situations can be calculated.

In general, in case there are one or more variables that do not contribute to the formation of the suggested number of clusters, we recommend that the procedure be reapplied without it (or them). In these situations, the number of clusters may change and, if the researcher feels the need to underpin the initial input regarding the number of K clusters, he may even use a hierarchical agglomeration schedule without those variables before reapplying the k-means procedure, which will make the analysis cyclical.

Moreover, the existence of outliers may generate considerably disperse clusters, and treating the dataset in order to identify extremely discrepant observations becomes an advisable procedure, before elaborating nonhierarchical agglomeration schedules. In the Appendix of this chapter, an important procedure in Stata for detecting multivariate outliers will be presented.

As with hierarchical agglomeration schedules, the nonhierarchical k-means schedule cannot be used as an isolated technique to make a conclusive decision about the clustering of observations. The data behavior, sample size, and criteria adopted by the researcher may be extremely sensitive to the allocation of observations and the formation of clusters. The combination of the outputs found with the ones coming from other techniques can more powerfully underpin the choices made by the researcher, and provide higher transparency in the decision-making process.

At the end of the cluster analysis, since the clusters formed can be represented in the dataset by a new qualitative variable with terms connected to each observation (cluster 1, cluster 2, ..., cluster K), other exploratory multivariate techniques can be elaborated from it, as, for example, a correspondence analysis, so that, depending on the researcher’s objectives, we can study a possible association between the clusters and the categories of other qualitative variables.

This new qualitative variable, which represents the allocation of each observation, may also be used as an explanatory variable of a certain phenomenon in confirmatory multivariate models as, for example, multiple regression models, as long as it is transformed into dummy variables that represent the categories (clusters) of this new variable generated in the cluster analysis, as we will study in Chapter 13. On the other hand, such a procedure only makes sense when we intend to propose a diagnostic regarding the behavior of the dependent variable, without aiming at having forecasts. Since a new observation does not have its place in a certain cluster, obtaining its allocation is only possible when we include such observation into a new cluster analysis, in order to obtain a new qualitative variable and, consequently, new dummies.

In addition, this new qualitative variable can also be considered dependent on a multinomial logistic regression model, allowing the researcher to evaluate the probabilities each observation has to belong to each one of the clusters formed, due to the behavior of other explanatory variables not initially considered in the cluster analysis. We would also like to highlight that this procedure depends on the research objectives and construct established, and has a diagnostic nature as regards the behavior of the variables in the sample for the existing observations, without a predictive purpose.

Finally, if the clusters formed present substantiality in relation to the number of observations allocated, by using other variables, we may even apply specific confirmatory techniques for each cluster identified, so that, possibly, better adjusted models can be generated.

Next, the same dataset will be used to run cluster analyses in SPSS and Stata. In Section 11.3, we will discuss the procedures for elaborating the techniques studied in SPSS and their results too. In Section 11.4, we will study the commands to perform the procedures in Stata, with the respective outputs.