20.1.1 Intuition

It is very intuitive to think that the base models of an ensemble should be accurate so that the ensemble is also accurate. The importance of diversity among the base models usually requires extra thought, but it also matches intuition. After all, if the base models of an ensemble make the same mistakes, then the ensemble will make the same mistakes as the individual base models and its error will be no better than the individual errors. On the other hand, ensembles composed of diverse models can compensate for the mistakes of certain models through the correct predictions given by the other models.

Table 20.1 shows an illustrative example of how diversity can help in reducing the error of prediction systems. For simplicity, this example considers a classification problem where the predictions made by the models can be either correct (dark gray) or incorrect (white). Table 20.1a, 20.1b show the prediction error of two ensembles and their base models on ten hypothetical data points. The ensemble shown in Table 20.1a is composed of three nondiverse base models, whereas the ensemble shown in Table 20.1b is composed of three diverse base models. All base models have fairly good accuracy, obtaining the same classification error of 30%, i.e., they gave incorrect predictions in 30% of the cases.

Table 20.1

An Illustrative Example of the Importance of Ensemble Diversity

Each ensemble is composed of the three base models shown in its corresponding table, and its prediction is the majority vote among the predictions of these base models.

The predictions given by the ensembles in this example are the majority vote among the predictions of their base models. So, as long as the majority of the base models gives a correct prediction, the ensemble also gives a correct prediction. As the base models from Table 20.1a make the same mistakes, the ensemble also makes the same mistakes and has the same classification error of 30% as its base models. On the other hand, the diverse ensemble shown in Table 20.1b achieves a better classification error of 10%, because the mistakes made by each base model were frequently compensated by the correct predictions given by the other base models.

20.1.2 Theoretical foundation

Many studies in the literature show that the accuracy of ensembles is influenced not only by the accuracy of its base models, but also by their diversity [100, 244, 414]. For classification problems, there is still no generally accepted measure to quantify diversity, making the development of a theoretically well-grounded framework to explain the relationship between diversity and accuracy of classification ensembles difficult. Nevertheless, there are very clear theoretical frameworks explaining the role of base models' accuracy and diversity in regression ensembles. The ambiguity decomposition [241] and the bias-variance-covariance decomposition [336] provide a solid quantification of diversity and formally explain the role of diversity in ensemble learning. This section will concentrate on the ambiguity decomposition, because we believe it to be easier to understand. In fact, basic knowledge of algebra is enough to understand the general idea of the framework. So, even though this section can be skipped should the reader wish to, we encourage the reader to read it through to gain a deeper understanding on why ensembles can be useful and why their base models should be diverse.

Assume the regression task of learning a function based on examples of the format (x, y) where x =[x₁, , x_k] are the input features and y is the target real valued output, which is a function of x. The examples are drawn randomly from the distribution p(x). Consider an ensemble whose output is the weighted average of the outputs of its base models

$f_{\mathit{ens}}\left(\mathbf{x}\right)={\displaystyle \sum _{i=1}^N{w}_i{f}_i\left(\mathbf{x}\right)},$

where N is the number of base models, f_i(x) is the output of the base model i for the input features x, w_i is the weight associated to the base model i, and the weights are positive and sum to one. The weights can be seen as our belief on the respective base models. For example, in bagging (Section 20.2), all weights are equal to 1/N.

The quadratic error of the ensemble on the example (x, y) is given by

$e_{\mathit{ens}}\left(\mathbf{x}\right)={\left(f_{\mathit{ens}}\left(\mathbf{x}\right)-y\right)}^2.$

Based on some algebra manipulation using the fact that the weights of the base models sum to one, Krogh and Vedelsby have shown that [241]

$e_{\mathit{ens}}\left(\mathbf{x}\right)=\underset{\text{weighted}\text{error}}{\underbrace{{\displaystyle \sum _i{w}_i{\left(f_i\left(x\right)-y\right)}^2}}}-\underset{\text{ambiguity}\text{term}}{\underbrace{{\displaystyle \sum _i{w}_i{\left(f_i\left(x\right)-f_{\mathit{ens}}\left(x\right)\right)}^2}}}.$

This equation is over a single example, but can be generalized for the whole distribution by integrating over p(x). This equation shows that the quadratic error of the ensemble can be decomposed into two terms. The first term is the weighted average error of the base models. The second term is called the ambiguity term. It measures the amount of disagreement among the base models. This can be seen as the amount of diversity in the ensemble. This equation shows that the error of an ensemble depends not only on the error/accuracy of its base models, but also on the diversity among its base models.

Note that the ambiguity term is never negative. As it is subtracted from the first term, the ensemble is guaranteed to have error equal to or lower than the average error of its own base models. The larger the ambiguity term, the larger the ensemble error reduction. However, as the ambiguity term increases, so does the first term. Therefore, diversity on its own is not enough—the right balance between diversity (ambiguity term) and individual base model's accuracy (average error term) needs to be used in order to achieve the lowest overall ensemble error [61].

Bootstrap aggregating (bagging) [46] is an ensemble method that uses a single type of base learner to produce different base models. Figure 20.1 illustrates how it works. Consider a training data set D of size |D|. In the case of SEE, each of the |D| training examples that composes D could be a completed project with known required effort. The input features of this project could be, for example, its functional size, development type, language type, team expertise, etc. The target output would be the true required effort of this project. An example of illustrative training set is given in Table 20.2. Consider also that one would like to create an ensemble with N base models, where N is a predefined value, by using a certain base learning algorithm. The procedure to create the ensemble's base models is as follows. Generate N bootstrap samples S_i (1 ≤ i ≤ N) of size |D| by sampling training examples uniformly¹ with replacement from D. For example, let's say that D is the training set shown in Table 20.2 and N = 15. An example of fifteen bootstrap samples is given in Table 20.3. The procedure then uses the base learning algorithm to create each base model i using sample S_i as the training set.

After the ensemble is created, then it can start being used to make predictions for future instances based on their input features. In the case of SEE, future instances are new projects to which we do not know the true required effort and wish to estimate it. In the case of regression tasks, where the value to be estimated is numeric, the predictions given by the ensemble are the simple average of the predictions given by its base models. This would be the case for SEE. However, bagging can also be used for classification tasks, where the value to be predicted is a class. An example of classification task would be to predict whether a certain module of a software is faulty or nonfaulty. In the case of classification tasks, the prediction given by the ensemble is the majority vote, i.e., the class most often predicted by its base models.

20.2.2 When and why bagging works

The success of bagging depends on its base learning algorithm being unstable, where unstable means that a small change in the training sample can result in a large change in the predictions given by the resulting base model. It has been shown that bagging is able to improve the performance of good but unstable base learning algorithms. On the other hand, bagging could even slightly degrade the performance of stable base learning algorithms [46]. It has been pointed out that classification and RTs, neural networks, and subset selection in linear regression are typically unstable, whereas k-nearest neighbor is stable [46, 47].

The reason why bagging needs good but unstable base learning algorithms is related to accuracy and diversity (Section 20.1). If a stable base learning algorithm is used, then the predictions given by different base models created with different bootstrap samples of the training set will be very similar. A good but unstable base learning algorithm will create accurate but diverse base models, making bagging successful. In terms of the ambiguity decomposition explained in Section 20.1.2, good but unstable base learning algorithms would increase the ambiguity term, helping to reduce the error of the ensemble. Even if each of the base models is slightly less accurate than a single model trained on the whole training set, the reduction of the error caused by the ambiguity term would compensate for that. On the other hand, bagging using a stable base learning algorithm would not get its error reduced because the ambiguity term would be very small—in the extreme case, zero. At the same time, its base models may be slightly less accurate than single models trained on the whole data set would be. So, the error of the bagging ensemble could be slightly worse than single models trained on the whole training set if its base learners are stable.

20.2.3 Potential advantages of bagging for SEE

Bagging is a simple and well-known ensemble learning approach that has three main features that make it potentially beneficial for SEE: (1) several ready implementations of bagging can be easily found on the Web, (2) it does not require extensive human participation in the process of building ensembles, and (3) it has theoretically and empirically shown to improve performance in comparison to the base learning algorithm provided that its base learning algorithm is unstable enough [46].

20.4.1 Choice of data sets and preprocessing techniques

The first point in our framework is the choice of data sets and preprocessing techniques to be used in the study. Considering the aim of providing a general evaluation of a certain SEE approach in comparison to others, the data sets should cover a wide range of problem features, such as number of projects, types of features, countries, and companies. In this work, this is achieved by using five data sets from the PRedictOr Models In Software Engineering Software (PROMISE) Repository [300] and eight data sets based on the International Software Benchmarking Standards Group (ISBSG) Repository [271] Release 10. Sections 20.4.1.1, 20.4.1.2 provide their description and preprocessing.

20.4.2 Choice of learning machines

The second point to be considered when designing a ML experiment is the approaches that will be included in the analysis. As explained in the beginning of Section 20.4, our design will concentrate on the aim of evaluating bagging + RTs against several other fully automated ML approaches. With that aim, three ensemble and three single fully automated learning machines were chosen to be used:

• Single learning machines:

• REPTree RTs [156, 444];

• Radial Basis Function (RBFs) networks [32]; and

• MultiLayer Perceptrons (MLPs) [32].

• Ensembles of learning machines:

• Bagging [46] with MLPs (bagging + MLPs), with RBFs (bagging + RBFs) and with RTs (bagging + RTs);

• Ensembles based on randomization [156, 444] of MLPs (randomizer + MLPs); and

• NCL [263, 264] with MLPs (NCL + MLPs).

RTs were chosen because they present several potential advantages for SEE, including their specific type of locality, as explained in Section 20.3. A study comparing different locality approaches, including the traditional SEE by analogy approach k-NN [390], and three clustering approaches, has shown that even though RTs obtained similar average ranking across data sets to the other locality approaches, when they were not the best, they rarely performed considerably worse than the best [326], being generally more adequate for SEE than the others.

RBFs were included because they are also learners based on locality, due to the RBFs (typically Gaussians) used in their hidden nodes. When the inputs are fed into the hidden layer, their distances (e.g., Euclidean distances) to the center of each neuron are calculated. After that, each node applies a RBF to the distance, which will produce lower/higher values for larger/shorter distances. Each hidden neuron i is connected to each output neuron j with weight w_ji, and output neurons usually compute a linear function of their inputs when working with regression problems. Due to the use of a RBF, changes in the weights connecting a given hidden neuron to the output neurons do not affect input values that are far from the center of this neuron.

Even though the choice of RTs and RBFs was based on their relation to locality, MLPs are not local learning machines and were included as an example of a nonlocal approach. They were chosen for being popular learning machines that can approximate any continuous function [32]. They have also been investigated in the SEE context [25, 45, 171, 242, 403, 422, 445]. For instance, Tronto et al. [422] showed that they can outperform linear regression because they are able to model observations that lie far from the best straight line. MLPs can be easily combined using several different ensemble approaches, such as bagging, random ensembles, and NCL. Other nonlocal approaches that are not restricted to certain function shapes were not chosen because they do not perform well for regression tasks (e.g., NaiveBayes [135]) or have not been so much used for SEE (e.g., support vector machines).

Bagging + RBFs and bagging + MLPs were also included to check whether other types of base learners could achieve similar performance to bagging + RTs.

Even though bagging is a well-known and accessible ensemble approach, each of its base learning machines is trained with only about 63.2% of the unique examples from the available original training set [26]. As the training data sets used for SEE are typically small, it is important to check whether using samples of the training set would not affect bagging badly in comparison to ensemble methods that train each of their base models with 100% of the training examples. So, our evaluation also included ensembles based on randomization of MLPs. They are based on the fact that different initial conditions may cause different neural networks to converge into different local minima of the cost function. They turn an apparent disadvantage, local minima in training neural networks, into something useful by averaging (in the case of regression) the predictions of base learning machines trained using different random seeds. Despite its simplicity, this procedure works surprisingly well in many, but not all, cases [92, 467, 468]. Their weakness is that a good level of diversity is not always achieved just by using different random seeds [264]. Note also that this approach can only be used with base learners that rely on random seeds, and are typically used with neural networks [421]. So, we used MLPs as their base learners in this study.

NCL [263, 264] is an ensemble approach that has strong theoretical foundations for regression problems, explicitly controlling diversity through the error function of the base learning machines [62]. So, it was also chosen to be included in the analysis. Its disadvantage is that its base models usually have to be very strong. Also, it can only be used with neural networks. Other learning machines such as RTs cannot be currently used. So, we used it with MLPs as the base learners in this study.

All the learning machines but NCL were based on the WEKA implementation [156, 444]. The RTs were based on the REPTree implementation available from Weka. We recommend the software WEKA [156, 444] should the reader wish to get more details about the implementation and parameters. The software used for NCL is available upon request.

20.4.3 Choice of evaluation methods

The third point in the experimental framework is the choice of evaluation methods. The evaluation was based on 30 runs for each data set. In each run, for each data set, 10 examples were randomly picked for testing and the remaining were used for the training of all the approaches being compared. Holdout of size 10 was suggested by Menzies et al. [292] and allows the largest possible number of projects to be used for training without hindering the testing. For sdr, half of the examples were used for testing and half for training, due to the small size of this data set.

The approaches are compared against each other based on their performance on the test set. Foss et al. [133] show that performance measures based on magnitude of the relative error (MRE) are potentially problematic due to their asymmetry, biasing toward prediction models that underestimate. That includes a performance measure very popular in the SEE literature: mean MRE (MMRE). Another measure, mean absolute error (MAE), does not present asymmetry problems and is not biased. However, it is difficult to interpret, as the residuals are not standardized. So, measures such as MMRE have continued to be widely used by most researchers in the area. However, Shepperd and Mc Donell [389] very recently proposed a new measure called standardized accuracy (SA), defined as follows:

$\mathit{SA}=1-\frac{\mathit{MA}{E}_{P_i}}{\mathit{MA}{E}_{P_0}},$

where $\mathit{MA}{E}_{P_i}$ is the MAE of the prediction model P_i and ${\overline{\mathit{MAE}}}_{P_0}$ is the mean value of a large number, typically 1000, runs of random guessing. This is defined as predicting $\widehat{y}$ for the example t by randomly sampling over the remaining n − 1 examples and taking ${\widehat{y}}_t=y_r$, where r is drawn randomly with equal probability from 1 … n ^ r ≠ t. Even though this measure is a ratio, such as MMRE, this is not problematic because we are interested in a single direction—better than random [389]. SA can be interpreted as the ratio of how much better P_i is than random guessing, giving a very good idea of how well the approach does.

The following measures of performance were used in this work [242, 292, 389]: MMRE, percentage of estimates within 25% of the actual values (PRED(25)), MAE, and SA. MMRE and PRED(25) were included for reference purposes, as they were reported in previous work evaluating ensembles [321]. In addition, verifying whether different results are achieved when using different performance measures can also provide interesting insights into the behavior of the approaches, as will be discussed in Section 20.5.4.

However, we will consider MAE and SA as more reliable performance measures. Unless stated otherwise, the analysis will always refer to the measures calculated on the test set.

As warned by Shepperd and Mc Donell [389], some papers in the SEE literature have been comparing approaches that do not perform better than random guess. To judge the effect size of the differences in performance against random guess, the following measure has been suggested:

$\text{Δ}=\frac{{\overline{\mathit{MAE}}}_{P_0}-\mathit{MA}{E}_{P_i}}{S_{P_0}},$

where $s_{P_0}$ is the sample standard deviation of the random guessing strategy. The values of Δ can be interpreted in terms of the categories proposed by Cohen [80] of small (≈ 0.2), medium (≈ 0.5), and large (≈ 0.8). We use this measure to show how big the effect of the difference in MAE between each approach and random guess is likely to be in practice, validating our comparisons.

In order to determine whether the performance of the learning machines in terms of MAE is statistically significantly different from each other considering multiple data sets, we use the Friedman statistical test. Friedman was recommended by Demšar [96] as an adequate test for comparing multiple learning machines across multiple data sets. This statistical test also provides a ranking of algorithms as follows. Let r^j_i be the rank of the jth of k algorithms on the ith of N data sets. The average rank of algorithm j is calculated as $R_j=\frac{1}{N}{\displaystyle {\sum }_i{r}_i^j}$. The rounded average ranks can provide a fair comparison of the algorithms given rejection of the null hypothesis that all the approaches are equivalent. Nevertheless, Wilcoxon sign-rank tests [442] are typically used to compare a particular model to other models across multiple data sets after rejection of the Friedman null hypothesis, where necessary. Holm-Bonferroni corrections can be used to avoid high Type-I error due to the multiple tests performed.

As data sets are very heterogeneous in SEE and the performance of the approaches may vary greatly depending on the data set, it is also important to check what approaches are usually among the best and, when they are among the worst, whether the magnitude of the performance is much worse than the best approach for that data set or not. This type of analysis also helps to identify for what type of data sets certain approaches behave better. It is worth noting that statistical tests such as Friedman and Wilcoxon are based on the relative ranking of approaches, thus not considering the real magnitude of the differences in performance when used for comparison across multiple data sets. So, even if a Friedman test accepts the null hypothesis that all approaches perform similarly across multiple data sets, it is still valid to check what approaches are most often among the best, on what type of data sets, and the magnitude of the differences in performance. In this work, we check the magnitude of the differences in performance between an approach and the best approach for a given data set by determining whether this approach has SA worse than the best in more than 0.1 units.

So, building upon previous work [96, 321, 389], an evaluation framework based on the following three steps is proposed by Minku and Yao (2012) [326]:

1. Friedman statistical test and ranking across multiple data sets to determine whether approaches behave statistically significantly differently considering several data sets.

2. Determine which approaches are usually among the first and second highest ranked approaches and, possibly, identify to which type of data sets approaches tend to perform better.

3. Check how much worse each approach is from the best approach for each data set.

A good approach would be highly ranked by Friedman and statistically significantly different from lower ranked approaches, would be more frequently among the best, and would not perform too bad in terms of SA when it is not among the best.

20.4.4 Choice of parameters

The fourth point considered in our framework is the choice of parameters for the approaches used in the experiments. The choice of parameters is a critical step in ML experiments, as results can vary greatly depending on it. For instance, a learning machine that would have better performance under certain choices could have worse performance under other choices. It is important that the method used for choosing the parameters is made clear in papers using ML, so that differences in the results obtained can be better understood.

In order to choose the parameters, we performed five preliminary runs using all combinations of the parameters shown in Table 20.9 for each data set and learning approach. The parameters providing the lowest MMRE for each data set were chosen to perform the thirty runs used in the comparison analysis. In this way, each approach enters the comparison using the parameters that are most likely to provide the best results for each particular data set. These parameters were omitted due to the large number of combinations of approaches and data sets used. The performance measure MMRE was chosen for being used in all previous work evaluating existing automated ensembles.

20.5.1 Friedman ranking

As a first step of the evaluation, the Friedman ranking of the ensembles, single learning machines, and random guess in terms of MAE was determined and the Friedman statistical test was used to check whether these approaches have statistically significantly different MAE. The ranking generated by the test is shown in Table 20.11. The Friedman test rejects the null hypothesis that all approaches perform similarly (statistic F_f = 12.124 > F(8, 96) = 2.036). As we can see from the table, random guess was always ranked last. An additional test was performed without including random guess and the null hypothesis was also rejected (statistic F_f = 4.887 > F(7, 84) = 2.121).

All bagging approaches performed comparatively well, being on average ranked third. RTs were ranked on average just below, as fourth. As the null hypothesis that all approaches perform similarly was rejected, Wilcoxon sign-rank tests with Holm-Bonferroni corrections were performed to determine whether the RT's MAE is similar or different from bagging + RT's, bagging + MLP's, and bagging + RBF's across multiple data sets. Table 20.12 shows the p-values. The tests reveal that ensembles in general do not necessarily perform better than an adequate locality approach such as RTs. Bagging + MLP and bagging + RBF obtained statistically similar performance to RTs. However, bagging + RT, which is an ensemble of locality learning machines, obtained higher and statistically significantly different Friedman ranking in terms of MAE from single RTs. The number of win/tie/loss of bagging + RT versus RT is 10/1/2, further confirming the good performance of bagging + RTs, and demonstrating that RTs do not provide enough instability to allow bagging to improve upon its base learning algorithm only very rarely.

Table 20.12

Wilcoxon Sign-Rank Tests for Comparison of RT's MAE to Bagging + RT's, Bagging + MLP's, and Bagging + RBF's Across Multiple Data Sets

Adapted from [326].

The p-value in light grey background represents statistically significant difference with Holm-Bonferroni corrections at the overall level of significance of 0.05.

Even though single RBFs use locality, they did not perform so well. A possible reason for that is that, even though locality is used in the hidden nodes, the output nodes are based on linear functions.

20.5.2 Approaches most often ranked first or second in terms of MAE, MMRE and PRED(25)

As a second step of the evaluation, the learning machines most often ranked as first or second in terms of MAE, MMRE, and PRED(25) were determined. Some results in terms of SA are also presented in order to provide more easily interpretable results.

In order to check how valid this analysis is, we first compare the SA and effect size Δ of the approach ranked as first, second, and last in terms of MAE for each data set. Table 20.10 shows these values. As we can see, the approaches ranked first or second frequently achieved SA at least 0.20 higher (better) than the worst approaches. That is reflected in the overall average SA considering all data sets, which improves from 0.2251 (approaches ranked last) to 0.4599 and 0.4712 (approaches ranked first and second). The effect size Δ in relation to random guess was frequently changed from small/medium to large, emphasizing the importance of using higher ranked approaches. Wilcoxon rank-sum statistical tests using Holm-Bonferroni corrections at the overall level of significance of 0.05 for comparing the approaches ranked first and second against random guess detected statistically significant difference for all cases. The p-values were very low, ranging from 1.6492 · 10⁻⁴ to 5.6823 · 10⁻¹⁹, confirming that the performance of these approaches is indeed better than random guessing.

Table 20.13a shows the learning machines ranked most often as first or second in terms of MAE. The results show that bagging + RTs, bagging + MLPs, and RTs are singled out, appearing among the first two ranked approaches in 27%, 23%, and 23% of the cases, whereas all other approaches together sum up to 27% of the cases. It is worth noting that bagging + RBF, which achieved high Friedman rank, does not appear so often among the best two approaches. One might think that bagging + RBFs rank for each data set could be more median, but more stable. However, the standard deviation of the ranks (Table 20.11) is similar to bagging + RTs and bagging + MLPs. So, the latter approaches are preferable over bagging + RBFs.

The table also shows that RTs are comparatively higher ranked for PROMISE than for ISBSG, achieving similar ranking to bagging approaches for PROMISE, but lower ranking for ISBSG. Even though RTs perform statistically similarly to bagging + MLPs and bagging + RBFs (Section 20.5.1), the difference in performance is statistically significant in comparison to bagging + RTs. Further, Wilcoxon sign-rank tests to compare RTs and bagging + RTs considering separately PROMISE and ISBSG data show that there is no statistically significant difference considering PROMISE on its own (p-value 1), but there is considering ISBSG (p-value 0.0078). So, approaches joining the power of bagging ensembles to the locality of RTs may be particularly helpful for more heterogeneous data.

Table 20.13b shows the two learning machines most often ranked as first and second in terms of MMRE. Both RTs and bagging + MLPs are very often among the first two ranked learning machines according to MMRE. The trend can be observed both in the PROMISE and ISBSG data sets. For PROMISE, RTs, or bagging + MLPs appear among the first two ranked approaches in 70% of the cases, whereas all other learning machines together sum up to 30%. For ISBSG, RTs, or bagging + MLPs appear among the first two ranked in 62.5% of the cases, whereas all other learning machines together sum up to 37.5%. Wilcoxon sign-rank tests show that these two approaches perform similarly in terms of MMRE (p-value of 0.2439).

The analysis considering PRED(25) shows that both RTs and bagging + MLPs are again frequently among the first two ranked (Table 20.13c), but randomizer + MLP becomes more competitive. If we consider PROMISE data by itself, ensembles such as bagging + MLPs are more frequently ranked higher than single learning machines, even though that is not the case for ISBSG. However, a Wilcoxon sign-rank test across multiple data sets shows that bagging + MLPs and RTs are statistically similar in terms of PRED(25) (p-value of 0.7483). Tests considering PROMISE and ISBSG data separately do not find statistically significant difference either (p-value of 0.8125 and 0.3828, respectively).

As we can see, RTs and bagging + MLPs are singled out as more frequently among the best both in terms of MMRE, PRED(25) and MAE and they perform statistically similarly independent of the performance measure. Other approaches such as randomizer + MLP and bagging + RTs become more or less competitive depending on the performance measure considered. Bagging + RBFs, differently from the Friedman ranking, is not singled out, i.e., it is not frequently among the best. Our study also shows that bagging + RTs outperform RTs in terms of MAE mainly for ISBSG data sets, which are likely to be more heterogeneous.

20.5.3 Magnitude of performance against the best

In this section, we analyze how frequently an approach is worse than the best MAE approach in more than 0.1 units of SA. There are 34 cases in which approaches are worse than the best MAE approach of the data set in more than 0.1 units of SA. p-values of Wilcoxon tests to compare these approaches to the best MAE are below 0.05 in 28 cases. If we apply Holm-Bonferroni corrections considering all the 34 comparisons, the difference is statistically significant in 20 out of 34 cases (59%). So, it is reasonable to consider the number of times that each approach is worse than the best in order to evaluate its performance.

Table 20.14 shows how many times each approach is worse than the best MAE approach in more than 0.1 units of SA. As we can see, RTs and bagging + RTs are rarely worse than the best MAE approach in more than 0.1 SA. They are worse in only 1 in 13 data sets. Bagging + MLPs, bagging + RBFs, and MLPs behave slightly worse, with the difference in SA higher than 0.1 SA in 3 data sets.

20.5.4 Discussion

The analyzes presented in Sections 20.5.1, 20.5.2, and 20.5.3 show that bagging + RTs present a very good behavior for SEE. They have higher average (Friedman) rank, are more frequently among the best in terms of MAE, and are rarely worse than the best MAE approach for each data set in more than 0.1 units of SA. This is an example of how joining the power of bagging ensembles to a good locality approach can help improving SEE. Bagging + RTs were particularly helpful to improve performance in terms of MAE for ISBSG data sets, which are likely to be more heterogeneous. Future research on improving SEE using automated learning machines may benefit from further exploiting the advantages of ensembles and locality together.

As no approach is always the best independent of the data set, ideally, an organization should perform experiments following a principled framework in order to choose a model for their particular data set. Nevertheless, if an organization has no resources to perform such experiments, bagging + RTs are more likely to perform comparatively well across different data sets and are rarely worse than the best approach for a particular data set in more than 0.1 SA. Even though single RTs may perform slightly worse than bagging + RTs, they perform comparatively well in comparison to other approaches and may be used if the practitioners would like to understand the rules used by the learning machine to perform estimations.

The conclusions above are mainly drawn based on MAE and SA, which are considered to be more adequate and reliable performance measures than MMRE and PRED(N). In terms of behavior independent of the performance measure used, RTs and bagging + MLPs are always singled out as being frequently first or second ranked, whereas bagging + RT is less frequently among the best in terms of MMRE and PRED(25). Such a difference in the evaluation based on MAE and MRE-based measures gives us an interesting insight. It suggests that the slightly worse performance of RTs and bagging + MLPs in comparison to bagging + RTs in terms of MAE may be related to the fact that RTs and bagging + MLPs suffer more from underestimations. This issue should be further analyzed as future work and could be helpful for improving the performance of these approaches for SEE.

It is also worth making a note on the preprocessing of the data sets. As explained in Section 20.4.1, data sets were preprocessed using k-means for outliers removal. Outliers are examples different from usual and that cannot be well estimated by the models generated. Experiments done using RTs and bagging + MLPs reveal that the error obtained when attempting to use the models to estimate solely outlier projects is worse than the average obtained using nonoutlier examples, both in terms of MAE and PRED(25). Interestingly, the error in terms of MMRE is not much affected. As MMRE tends to bias models toward underestimations [389], these outliers are likely to be projects underestimated by the models.

We have also repeated the experiments using bagging + RTs, bagging + MLPs, and RTs, but without performing this part of the preprocessing, i.e., allowing the train/test sets to contain outliers. The experiments reveal that the SAs obtained are considerably reduced. So, in practice, the use of a clustering approach such as k-means is recommendable to remove outliers from the training sets and to identify whether a test example is an outlier.