Distribution and Dynamic Linking

Of special interest to you is whether the predictors are useful in predicting the values of Diagnosis, the dependent variable.

To get some initial insight on this issue, return to your distribution report. In the graph for Diagnosis, click on the bar corresponding to M. This selects all rows in the data table for which Diagnosis has the value M. These rows are dynamically linked to all plots, and so, in the 30 histograms corresponding to predictors, areas that correspond to the rows where Diagnosis is M are highlighted.

Scroll across the histograms to see that malignant masses tend to have high values for all of the Mean and Max variables except for Mean Fractal Dim and Max Fractal Dim. Five of the distributions for Max variables are shown in Exhibit 10.4. The malignant observations are highlighted. Max Fractal Dim is shown in the last histogram.

Next, click on the bar for Diagnosis equal to B and scroll across the histograms. You conclude that there are clear relationships between Diagnosis and the 30 potential predictors. You should be able to build good models for classifying a mass as malignant or benign based on these predictors.

Correlations and Scatterplot Matrix

You are also interested in how the 30 predictors relate to each other. To see bivariate relationships among these 30 continuous predictors, you obtain correlations and construct scatterplots.

The report shows a Correlations outline where you see the correlations between the predictors. It also shows a 30 × 30 Scatterplot Matrix. This is a matrix of scatterplots for all pairs of variables. The 6 × 6 portion of the matrix corresponding to the first six Mean variables is shown in Exhibit 10.5.

You scroll through the matrix and notice that some variables are highly related. For example, the Mean and Max values of the size variables, Perimeter, Area, and Radius, are strongly related, as you might expect. Also, as you might expect, Perimeter and Area have a nonlinear relationship. Weaker, but noticeable, relationships exist among other pairs of variables as well.

You would like to see how Diagnosis fits into the scatterplot display. James had shown you that in many plots you can simply right-click to color or mark points by the values in a given column.

You notice that the colored markers now appear in your data table next to the row numbers. Also, an asterisk appears next to Diagnosis in the Columns panel. When you click on it, it indicates that Diagnosis has the Value Colors column property. When you click on Value Colors, it takes you to the Diagnosis Column Info dialog, where you see that the colors have been applied to B and M as you specified.

The colors and markers appear in the scatterplots as well, and a legend is inserted to the right of the matrix as part of the report. You can use this legend to select points. But you have also created a Legend window, shown in Exhibit 10.7. (Script is Colors and Markers.)

Move the Legend window into the upper right of your scatterplot matrix, as shown in Exhibit 10.7. Click on the B level for Diagnosis. This selects all rows where Diagnosis is B and highlights these points in plots, as is shown in the portion of the scatterplot matrix shown in Exhibit 10.7. To unselect the B rows, hold the Control key as you click on the B level in the legend.

Viewing the scatterplot matrix, it is clear that Diagnosis is associated with patterns of bivariate behavior.

Pairwise Correlations

You are interested in which variables are most strongly correlated with each other, either in a positive or negative sense. The scatterplot certainly gives you some clear visual information. However, a quick numerical summary showing correlations for all pairs of variables would be nice.

Because you have 30 variables, each of which can be paired with 29 other variables, there are combinations of two variables. Because these combinations double-count the correlations, there are distinct pairwise correlations. These 435 correlations are sorted in descending order. Exhibit 10.8 shows the largest 20 and the smallest 10 correlations.

The report shows the high positive correlations among the Mean size variables, the Max size variables, and the SE size variables. All of these are 0.9377 or higher. This is expected. Scanning further down the report, you observe that the Mean and Max variables for the same characteristics tend to have fairly high correlations. This is also expected.

Many of the large negative correlations involve fractal dimension and the size measures. Recall that the larger the fractal dimension, the more irregular the contour. Six such correlations are shown in the bottom part of Exhibit 10.8. This suggests that larger cells tend to have more regular contours than smaller cells. How interesting!

A Better Look at Mean Fractal Dimension

To get a better feeling for relationships between size variables and fractal dimension, you decide to explore Mean Radius and Mean Fractal Dim.

The combined collection of benign and malignant cells exhibits a decreasing relationship between Mean Radius and Mean Fractal Dim. Given a value of Mean Fractal Dim, Mean Radius is generally higher for malignant cells than for benign cells. The fitted lines and confidence curves suggest that there is a big difference between the two diagnosis groups based on how Mean Radius and Mean Fractal Dim are related.

Given values on these two variables, you could make a fairly good guess as to whether a mass is benign or malignant. But to help distinguish the two groups in the murky area between the fitted lines, you will need additional variables. This is encouraging—it should be possible to devise a good classification scheme.

In examining the scatterplot matrix further, you observe that there may be a few bivariate outliers. If this were your marketing data set, you would attempt to obtain more background on these records to get a better understanding of how to deal with them. For now, you decide not to take any action relative to these points. However, you make a mental note that you might want to revisit their inclusion in model-building since they have the potential to be influential.

Distribution of the Three Analysis Data Sets

At this point, you should check that the assignment of rows to the three analysis sets worked out as you expected. Run Distribution on Validation (Exhibit 10.11). You observe that the proportion of rows in each set is about what you wanted.

Fitting a Decision Tree Model

Decision Tree is the method that JMP fits by default when you launch the Partition platform.

The Partition Report

The report shown in Exhibit 10.19 appears. The initial node is shown, indicating under Count that there are 327 rows of data. These are the observations in your training set.

In the bar graph in the node and the graph above the node, the color red indicates malignant values (red triangles) and blue indicates benign values (blue circles). The colors and markers are those that you assigned in the script Colors and Markers. Currently there is only one node (All Rows), and the graph above the node shows the points within a node with random jitter in the horizontal direction.

You would like JMP to display the proportions of malignant and benign records in each node. To see these, select Display Options > Show Split Prob from the red triangle at the top of the report. The diagram updates to show the Rate and Prob columns for the two levels of Diagnosis in the initial node, and the report will show these in all subsequent nodes as well. The Rate statistic gives the proportion of observations that are in each response level. The Prob value is the predicted value for the response level, calculated in such a way that the predicted probability is always nonzero.

Now the initial node appears as shown in Exhibit 10.20. You see that the benign proportion is 0.6514 and the malignant proportion is 0.3486. The horizontal line separating the benign points from the malignant points in the graph at the top of the report is at 0.6514.

Splitting

To get a sense of what Partition does, click once on the Split button. JMP determines the best variable on which to split and the best cutting value for that variable.

The report, after the first split, is shown in Exhibit 10.21. The first split is on the variable Max Concave Points, and the observations are split at the value where Max Concave Points = 0.15. Of the 98 observations where Max Concave Points ≥ 0.15 (the leftmost node), 97.96 percent are malignant. Of the 229 for which Max Concave Points < 0.15, 92.14 percent are benign. The plot at the top updates to incorporate the information about the first split, suggesting that the first split has done very well in discriminating between benign and malignant tumors.

You can continue to split manually by clicking the Split button or you can fit a model automatically by clicking the Go button.

If you continue to split manually, splitting continues until splitting is stopped by the Minimum Size Split. To see the default value for Minimum Size Split, select Minimum Size Split from the red triangle menu next to Partition for Diagnosis. The minimum default size is five. So you will be able to split until no further splits of size five or larger are possible. You can set a larger value for Minimum Size Split if you think the value of five results in overfitting.

Automatic Splitting

If you split using the Go button, the Partition platform splits until the RSquare of the validation set no longer improves. Specifically, splitting is continuous until the RSquare of the validation set fails to improve for the next ten splits.

The summary report immediately beneath the graph and above the tree shows that four splits occurred. (See Exhibit 10.22.) The RSquare for the training set is 0.890 and the RSquare for the Validation set is 0.596. Since the tree is trained to fit the training set, it is not surprising that the RSquare for the validation set is lower than the RSquare for the training set. The RSquare for the test set is also given, but it is not of interest to us now.

Split History

When you clicked the Go button, a report called Split History, shown in Exhibit 10.23, appeared beneath the tree. For each of the training, validation, and test sets, this report shows the RSquare value plotted against the number of splits. The plot only shows RSquare values for up to seven splits. This is because the Minimum Size Split rule, with the default minimum size for a node set at five, ends splitting at seven splits.

You see that the RSquare value for the training set (the top line, shown in blue) continues to increase over the seven splits. This is expected, since the models keep fitting finer features of the training set.

The RSquare value for the validation set (the middle line, shown in red) begins to decrease after the fourth split. This is an indication that, after the fourth split, models are fitting nuances of the training set that are not shared by the independent validation set. This suggests that the model has moved from fitting structure to fitting noise. Since the maximum RSquare for the validation set occurs for a model with four splits, the four-split model is selected by the stopping rule and displayed in the tree.

Confusion Matrices

The graph in Exhibit 10.22 indicates that the nodes do a good job of separating the malignant from the benign cases. Select Show Fit Details from the report's red triangle menu to obtain the Fit Details report, which shows numerical results (Exhibit 10.24). The first part of the report shows various measures and summary information. The Misclassification Rate for the Training set is 0.0275. The Misclassification Rate for the Validation set is 0.0789.

The Confusion Matrix report shows the nature of the misclassifications. The matrix for the validation set shows that 61 benign tumors were correctly classified as benign, while 3 benign tumors were misclassified as malignant. It also shows that 6 malignant tumors were misclassified as benign while 44 malignant tumors were correctly classified as malignant. This gives the misclassification rate of .

For all three sets, there are more cases of malignant tumors being misclassified as benign than of benign tumors being misclassified as malignant. Misclassifying malignant tumors as benign may be a more serious error than misclassifying benign tumors as malignant. This is something you want to keep in mind in your model selection.

Decision Tree and Options

Now you go back to your decision tree to see which variables were important (Exhibit 10.25). The terminal nodes, that is, nodes where no further splitting occurs, involve Max Concave Points, Max Area, Max Texture, and Max Concavity. So, concavity, area, and texture variables are involved. It's interesting that these involve the maximum values of the related measurements, and not their means or standard deviations. But you need to keep in mind that these selections might be different with a different validation set, especially because some of the predictors are highly correlated. The Decision Tree method tends to be sensitive to the nuances of the training and validation sets and to correlations among predictors.⁶

You notice that there are many options available in the report's red triangle menu. These include Leaf Report, Column Contributions, K-Fold Cross Validation, ROC Curve, Lift Curve, and so on. ROC and lift curves are used to assess model fit. When a tree is large, the Small Tree View option provides a plot that condenses the information in the large tree. This small schematic shows the split variables and the split values in the tree. For reasons of space, we will not discuss these additional options. We encourage you to explore these using the documentation in Help.

Saving the Decision Tree Prediction Formula

At this point, you want to save the prediction equation to the data table. You do this by selecting Save Columns > Save Prediction Formula from the red triangle at the top of the report (CellClassification.jmp, script is Decision Tree Prediction Formula). This saves three new columns to the data table, Prob(Diagnosis==B), Prob(Diagnosis==M), and Most Likely Diagnosis.

By clicking on the plus sign to the right of Prob(Diagnosis==M) in the Columns panel, you are able to view its formula, shown in Exhibit 10.26. The formula has the effect of placing a new observation into a terminal node based on its values for the split variables and then assigning to that new observation a probability of malignancy equal to the proportion of malignant outcomes observed for training data records in that node. It follows that the classification rule consists of determining the terminal node into which a new observation falls and classifying it into the class with the higher sample proportion in that node.

The Most Likely Diagnosis column contains the classification for each observation. Click on the plus sign to the right of Most Likely Diagnosis in the Columns panel. The formula shows that an observation is classified into the group “B” or “M” for which the probability of diagnosis into that group is larger.

With the prediction formula saved to the data table, you are ready to move on to your next fitting technique.

Fitting a Boosted Tree Model

The Boosted Tree Report

The Boosted Tree report is shown in Exhibit 10.28. It echoes the model specifications, gives overall measures for the three analysis sets, and provides a confusion matrix for each analysis set.

The Specifications report gives you a summary of various aspects of the model fit you requested. Note that 43 layers were fit before the Early Stopping Rule terminated fitting.

The Overall Statistics report is similar to the report shown in Exhibit 10.24 for the Decision Tree fit. Note that the Misclassification Rate in the Training set is 0.0 and in the Validation set is 0.0526. The Confusion Matrix report shows the nature of the misclassifications.

Cumulative Validation Plot

The Cumulative Validation graph (Exhibit 10.29) shows various statistics plotted over models with various numbers of layers. You can obtain the exact values in the plot by selecting Save Columns > Save Cumulative Details from the red triangle menu. The RSquare statistic achieves its maximum value for a 33-layer model. From that point, ten more layers are constructed. Since none of these models improve on the 33-layer model, fitting stops with the 43-layer model and that model becomes the final model.

You can see the small tree fits by selecting Show Trees > Show names categories estimates from the report's red triangle menu. A Tree Views outline appears at the bottom of the report. Click its disclosure icon. There are reports for all 43 layers. Each layer is a small decision tree that has been fit to the residuals of the tree constructed from the layers that precede it. The residuals are weighted using the Learning Rate.

Open the reports for Layer 1, Layer 24, and Layer 43. Notice how different predictors enter the splits in an attempt to explain the residuals. Note that the trees show Estimates for each terminal node. The option Save Columns > Save Tree Details saves the estimates to a separate data table.

Column Contributions Report

To obtain a sense of which predictors are important in this boosted tree model, select Column Contributions from the red triangle menu for the report. The report, shown in Exhibit 10.30, measures a predictor's contribution using the number of times it defined a split and the likelihood ratio chi-square (G^2) for a test that the dichotomized predictor is significant. The main contributors are Max and Mean predictors that measure size, smoothness, concavity, and texture. Recall that for your simple Decision Tree model, the four splits were on Max Concave Points, Max Area, Max Texture, and Max Concavity.

Saving the Boosted Tree Prediction Equation

As you did for the Decision Tree model, you save the prediction formula for the Boosted Tree model to the data table by selecting Save Columns > Save Prediction Formula from the red triangle at the top of the report (CellClassification.jmp, script is Boosted Tree Prediction Formula). This saves three new columns to the data table, Prob(Diagnosis==M) 2, Prob(Diagnosis==B) 2, and Most Likely Diagnosis 2. The suffix “2” is added to these column names to distinguish them from the columns with the same names saved as the Decision Tree prediction formulas.

Click on the plus sign to the right of Prob(Diagnosis==M) 2 in the Columns panel to view its formula. The formula is a logistic transformation of a constant (−0.6251) plus 43 “If” functions, each one corresponding to a small tree. Each “If” function assigns to an observation the Estimate for that observation shown in the corresponding tree. To see this, take the first “If” function and compare it to the tree shown in the Tree Report for Layer 1.

Note that many predictors are involved in the prediction formula. These have entered through the Layers. It is difficult to judge the relative importance of the predictors by viewing the prediction formula. The Column Contributions report (Exhibit 10.30) is very helpful in assessing which predictors are important for the boosted tree model.

The Most Likely Diagnosis 2 column contains the classification for each observation. As with the Decision Tree prediction formula, the formula for Most Likely Diagnosis 2 classifies an observation into the group for which the probability of diagnosis into that group is larger.

With the prediction formula saved to the data table, you are ready to move on to your next fitting technique.

Contribution of Predictors

All three predictors (Max Concave Points, Max Radius, and Max Texture) are significant. As in the Decision Tree model, all three predictors are “Max” measurements. Recall that, for the Decision Tree model, the order of contribution was Max Concave Points, Max Area, Max Texture, and Max Concavity. For the boosted tree model, Max Concave Points, Max Radius, and Max Texture were among the first six predictors in terms of contribution (Exhibit 10.30).

Confusion Matrices

The Confusion Matrix report shows correct and incorrect classification counts. For the Training set, 9 out of 327 observations were misclassified. For the Validation set, 4 out of 114 observations were misclassified. Keep in mind that the choice of predictors by Stepwise, and hence the choice of model, made integral use of the training set. The Test set confusion matrix gives you an idea of how this model will predict for independent data. For the Test set, 6 of the 124 observations are misclassified. Four of the misclassifications result from predicting a benign mass when the mass is in fact malignant.

Specifying the Generalized Regression Model

Generalized Regression is a Personality that you access from the Fit Model specification window. When you click Run in the Model Specification window, you can then launch various fits using the Model Launch outline.

A Generalized Regression report appears. The Model Launch panel, with the Advanced Controls outline open, is shown in Exhibit 10.35.

The Model Launch window lets you choose one of five estimation methods. One of these is Maximum Likelihood, which allows you to compare a penalized fit to a traditional fit. The Adaptive Lasso model is the default. Note that you can also select Forward Selection, Elastic Net, and Ridge.

The Advanced Controls outline enables you to make selections regarding the tuning parameter grid. It also provides a way to force terms into the model. You will fit models accepting the defaults in the Advanced Controls panel.

Several Validation Methods are available. Because you included a Validation column in the Model Specification window, the Validation Method is set to Validation Column.

A viable alternative approach to cross-validation would be to combine your Training and Validation sets and conduct KFold cross-validation on the 441 observations in those sets. However, for simplicity, consistency, and continuity, in this section you will fit penalized regression models using the Validation column for cross-validation. (Note that while Partition calculates a KFold cross-validation statistic as a report option, it does not actually fit models using a KFold approach.)

The Lasso with Validation Column Validation Report

The report shown in Exhibit 10.36 appears. The Parameter Estimates for Original Predictors outline, which is closed in Exhibit 10.36, is shown in Exhibit 10.37.

The Model Summary outline provides descriptive information about the model you selected, followed by measures of fit for the model for each of the Training, Validation, and Tests sets. Since the statistics under Measure depend on the sample size, they cannot be compared across the three sets. But they can be used to compare models built on the Training set. The Estimation Details outline echoes the selections under Advanced Controls.

The Solution Path outline shows two plots:

• The plot on the left shows the scaled parameter estimates as a function of the Magnitude of Scaled Parameter Estimates. The parameter estimates are for a model given in terms of centered and scaled predictors. The Magnitude of Scaled Parameter Estimates is the sum of the absolute values of the scaled parameter estimates for the model (excluding the intercept, dispersion, and zero-inflation parameters). It reflects the amount of shrinkage, or bias, used in obtaining the given parameter estimates. Estimates with large magnitudes are close to the maximum likelihood estimate. Those with small values are heavily penalized.
• The plot on the right shows the Scaled-LogLikelihood as a function of the Magnitude of Scaled Parameter Estimates. The Scaled-LogLikelihood is the statistic used to assess model fit when holdback validation is performed. Smaller values indicate better fit than larger values.

Contribution of Predictors

To get a sense of which predictors are important, open the Parameter Estimates for Centered and Scaled Predictors report. Centering and scaling the predictors gives them comparable ranges of values and so, in a sense, puts them on equal footing. Comparing the absolute values of the parameter estimates for the transformed predictors gives you a sense of which are most important relative to the predictive model.

Right-click in the Parameter Estimates for Centered and Scaled Predictors report and select Sort by Column. In the Select Columns window, select Estimate and check the Ascending box, and then click OK. The report now appears as shown in Exhibit 10.38.

Since all estimates except for the intercept are negative, the report now lists the predictors in order of the absolute value of their estimates. (This was an expedient way to list the predictors in that order. In general, you would right-click on the table, select Make into Data Table, create a column of absolute values for Estimate in the data table, and then sort that column.)

The variable Max Radius is the largest contributor to the model, followed by Max Texture. Compare Exhibit 10.38 with Exhibit 10.30 to see that the order of contribution is fairly consistent with the Boosted Tree model. Another way to assess sensitivity to predictors is to use the Variable Importance feature found in the profiler. Variable Importance measures sensitivity to predictors in a way that does not depend on the model type, but only on the prediction equation. The method is based on simulating predictor values. You are given several choices for how to do the simulation. See the JMP documentation for details.⁸

Saving the Lasso Prediction Equation

You now save the prediction equation for your Lasso model by selecting Save Columns > Save Prediction Formula from the red triangle menu next to Lasso with Validation Column Validation (CellClassification.jmp, script is Lasso Prediction Formula). Three columns are saved to the data table: Probability(Diagnosis=B), Probability(Diagnosis=M), and Most Likely Diagnosis 4.

To see how well this model predicts, you can use Tabulate to construct tables that are analogous to confusion matrices.

The resulting table appears as shown in Exhibit 10.40. Three observations are misclassified in the Validation set.

The Elastic Net with Validation Column Validation Report

The Model Summary and Solution Path reports are shown in Exhibit 10.41. The Solution Path report shows that the optimal solution occurs at the right end of the Magnitude of Scaled Parameter Estimates range. This is where separation begins to be a problem. The plot indicates that there are more non-zero estimates involved in the solution for the Elastic Net than for the Lasso. The Parameter Estimates report indicates that only nine terms have zero as their estimate, leaving 21 terms in the model.

The -LogLikehood values for the Training, Validation, and Test sets are smaller for the Elastic Net model than for the Lasso model. The Bayesian Information Criterion (BIC) and corrected Akaike's Information Criterion (AICc) values are all larger for the Elastic Net model than for the Lasso model. This is because the Elastic Net tends to include all predictors within a correlated group, whereas the Lasso tends to only choose one.

Contribution of Predictors

To get a sense of which predictors are important, open the Parameter Estimates for Centered and Scaled Predictors report. Recall that centering and scaling the predictors puts them on equal footing. You will compare the absolute values of the parameter estimates for the transformed predictors to get a sense of which are most important in the predictive model.

The first three columns in your new data table now appear as shown in Exhibit 10.42. Note that eight of the ten “Max” predictors appear first on the list. The predictors Max Radius and Max Texture were the top two predictors for the Lasso model, they were two of the three predictors selected by stepwise for the Logistic model, and they were among the highest contributors for the Boosted Tree model. Keep in mind, though, that many of the predictors are highly correlated, making it difficult to compare your models in terms of which predictors are important.

Saving the Elastic Net Prediction Equation

You now save the prediction equation for your Elastic Net model by selecting Save Columns > Save Prediction Formula from the red triangle menu next to Elastic Net with Validation Column Validation (CellClassification.jmp, script is Elastic Net Prediction Formula).

Three columns are saved to the data table: Probability(Diagnosis=B) 2, Probability(Diagnosis=M) 2, and Most Likely Diagnosis 5.

To see how well this model predicts, you can use Tabulate, as described in the section Saving the Lasso Prediction Equation, to construct tables that are analogous to Confusion Matrices. You will obtain the table shown in Exhibit 10.43. Again, three observations are misclassified in the Validation set.

Neural Platform in JMP

The Neural Net algorithm in JMP uses a ridge-type penalty to help minimize overfitting. Because of its similarity to a ridge-regression penalty function, the neural net overfitting penalty not only addresses overfitting, but also helps mitigate the effects of multicollinearity.¹¹

You can set the number of nodes in the hidden layer. Note that a small number of hidden nodes can lead to underfitting and a large number can lead to overfitting, but small and large are relative to the specific problem under study. For a given number of hidden nodes, each application of the fitting algorithm has a random start, and JMP refers to these individual fits as tours. Based on experience, about 16 to 20 tours are recommended in order to find a useful optimum.

JMP provides four validation methods to help you select a neural net model that predicts well for new data:

• Excluded Rows Holdback
• Holdback
• KFold
• Validation Column

Excluded Rows Holdback, Holdback, and Validation Column are holdback methods. Excluded Rows Holdback treats excluded rows as a holdback sample. Holdback allows you to set a holdback proportion and then randomly selects a sample of that size to use as a holdback sample. If you specify a Validation set in the model launch window, then Validation Column treats your validation set as the holdback sample.

In the holdback methods, a sample of the observations is withheld (the holdback sample) while the remaining observations are used to train a neural net. JMP constructs a neural net model for the training sample. Then it applies the model to the holdback sample and calculates a LogLikelihood value for that sample. The LogLikelihood of the holdback sample determines when the algorithm stops fitting.

You can vary the number of hidden nodes and other aspects of the model in an attempt to find a fit that generalizes well to the holdback sample. This method works well for large numbers of observations, where one can easily fit a model to 75 percent or fewer of the observations. JMP uses two-thirds of the complete data set by default.

The fourth method is called KFold cross-validation. Here, a neural net model is fit to all of the data to provide starting values. Then, the observations are divided randomly into K groups (or folds). In turn, each of these groups is treated as a holdback sample. For each of the K groups, a model is fit to the data in the other (K − 1) folds, using the starting values from the full fit. The model is then extended to the holdback group. The LogLikelihood is calculated for each holdback sample, and these are averaged to give an average LogLikelihood that represents how the model might perform on new (Validation) observations. The starting values from the full fit are used because the function being optimized is multimodal, and this practice attempts to bias the estimates for the submodels to the mode of the overall fit.

Fitting the Neural Net 1 Model

You will first fit this model, and then you will be able to obtain a diagram of its structure.

The Model NTanH(3) Report

The Model NTanH(3) report appears (Exhibit 10.45), giving Measures, Confusion Matrices, and a Diagram plot. Because fitting a neural net involves a random component, your results will differ from those shown in the exhibit. If you want to replicate the results in Exhibit 10.45, run the script Neural Net Model 1. The script controls the random component by specifying a value for the random seed random seed.

Note that the Misclassification Rates across the data sets are small, on the order of 0.03. The Confusion Matrices give more detail on the performance of your model relative to misclassification.

The Diagram plot, shown in Exhibit 10.46, shows the 30 input variables, the three hidden nodes (H1, H2, and H3), and their combination in the prediction of the probability of Diagnosis classes. The final model is a logistic function of a linear combination of three models, each of which relates one of the hidden nodes, H1, H2, and H3, to the 30 predictors.

From the red triangle next to Model NTanH(3), select Estimates. The Estimates report provides a list of parameters estimated. Note that a total of 97 parameters have been estimated. To see this, right-click in the report and select Make into Data Table. The table has 97 rows. (Note that because of the random starts, when you run this report on your own, your results will differ slightly from those shown here.)

To get some insight on the 30-predictor model, you explore a few other options provided by the Neural Net report. From the red triangle for Model NTanH(3), select Categorical Profiler. This adds a Prediction Profiler plot, a portion of which is shown in Exhibit 10.47.

The dotted vertical red lines represent settings for each of the predictors. The trace shown in the profiler cell above a given predictor represents the cross section of the fitted model for Prob(Diagnosis=B) for that variable, at the given settings of the other variables. When you change one variable's value, you can see the impact of the change on the surface for all other variables. Note that the surfaces appear to be fairly smooth with some steep peaks, but no very jagged areas.

For another way to visualize the surface, select Surface Profiler from the top red triangle menu. This gives a three-dimensional view of the effect of predictor variables, taken two at a time, on Prob(Diagnosis=B) and Prob(Diagnosis=M).

Saving the Neural Net 1 Prediction Equation

You now save the prediction equation for your Neural Net model by selecting Save Profile Formulas from the red triangle menu next to ModelNTanH(3) (CellClassification.jmp, script is Neural Net 1 Prediction Formula).

Three columns are saved to the data table: Probability(Diagnosis=B) 3, Probability(Diagnosis=M) 3, and Most Likely Diagnosis 6.

If you had selected Save Formulas, you would have obtained six formulas: the three formulas described above; and the formulas for the three hidden nodes, called H1_1, H1_2, and H1_3. The Save Formulas option shows the Probability(Diagnosis=B) and Probability(Diagnosis=M) formulas in terms of the three hidden nodes. Probability(Diagnosis=B) applies a logistic function to the estimated linear function of the hidden nodes. If you would like some insight on what a neural net fit is like, explore these formulas.

Fitting the Neural Net 2 Model

The Neural Net report allows you to fit multiple models in the single report window. To do this, go to the top of the report and open the Model Launch panel. Alternatively, you can click the red triangle next to Neural and select Script > Relaunch Analysis and click Go in the window that appears.

The Model NTanH(3)NBoost(10) Report

The report shown in Exhibit 10.48 appears. This second model has a lower misclassification rate than the first model on the Training and Validation data, but a higher misclassification rate on the Test data.

The Diagram report for this model, shown in Exhibit 10.49, shows 30 activation functions. There are three nodes, and ten models were fit as part of the boosting process.

This model requires the estimation of 961 parameters. Again, to see this, select Estimates from the red triangle menu for Model NTandH(3)NBoost(10). Then right-click in the report and select Make into Data Table.

Saving the Neural Net 2 Prediction Equation

You now save the prediction equation for your second Neural Net model. To save the prediction formula, select Save Profile Formulas from the red triangle menu next to Model NTanH(3)NBoost(10) (CellClassification.jmp, script is Neural Net 2 Prediction Formula).

Three columns are saved to the data table: Probability(Diagnosis=B) 4, Probability(Diagnosis=M) 4, and Most Likely Diagnosis 7.

For simplicity and continuity in the exposition, we have fit neural nets using the Validation column. However, your data set, containing 569 observations, is a moderate-sized data set and KFold cross-validation might provide better models. You might want to explore the use of Kfold cross-validation in fitting neural net models to these data.