Description of Data

The target variable for the predictive models is a binary variable indicating survival (or death) five years postsurgery for patients with CRC. The input variables used to build the predictive models consisted of age, race, and different types of variables that described tumor state, tumor differentiation, and the location of the tumor. Two CRC bio-marker variables were also available. There were 500 observations and 107 input variables.

Using all the described variables, a logistic regression, decision tree, and ANN will be built using SAS Enterprise Miner 6.1. The following is a brief description of each model type.

Data Mining Models

Logistic Regression A logistic regression model was built to model the probability of survival (or not) five years postsurgery for CRC. Logistic regression is used to model data when the target variable is binary (e.g., survive—yes/no; recurrence—yes/no). The probability of an outcome is related to a set of predictor variables by an equation of the form

where p is the probability of survival five years postsurgery for CRC, β₀ is an intercept term, β₁, …, β_k are the coefficients associated with each variable, X₁, …, X_k are the values of the predictor variables, and k is a unique subscript denoting each variable. The standard assumption is that the predictor variables are related in a linear fashion to the log odds {log[p/(1 – p)]} of the outcome of interest.

Decision Trees The type of decision tree used in this analysis was a classification and regression tree (CART).¹⁶ the settings in SAS Enterprise Miner can be adjusted to create such a tree. CART is an algorithm that is used to split the data into smaller segments called nodes that are homogeneous with respect to the outcome variable. At each node the algorithm examines all values of the predictor variables with respect to determining the best predictor variable and a value of that predictor variable that will best separate the data into more homogeneous subgroups with respect to the outcome variable. In other words, each node is a classification question, and the branches of the tree are partitions of the data set into different classes (those patients who will survive/not survive five years postsurgery). This process repeats itself in a recursive, iterative manner until no further separation of the data is feasible. Therefore, the terminal nodes at the end of the branches of the decision tree represent the different classes.

The second part of the algorithm is known as pruning. Pruning is applied to the decision tree to ensure that the algorithm does not overfit the training data. At each subsequent node, smaller amounts of observations are available. Toward the end of the splitting algorithm, idiosyncrasies of the training observations at a particular node can display a pattern that is specific only to those observations that can become meaningless and detrimental for prediction when applied to larger populations. Pruning removes smaller branches that failed to generalize using the validation data set.

Artificial Neural Networks The original development of the neural network was inspired by the way the brain recognizes patterns.¹⁷ The goal of an ANN is the same as in logistic regression, predicting an outcome based on the values of predictor variables; however, the approach used in developing the neural network model is quite different from that for logistic regression.

ANNs have the ability to “learn” mathematical relationships between a series of input (predictor) variables and the corresponding output (outcome) variables. This is achieved by “training” the network with a training data set that consists of the predictor variables and a known outcome variable. Once the ANN has been trained, the model can be used for classification on a validation data set.

Figure 14.1 illustrates an ANN that has been trained to predict the probability of a patient dying of CRC five years postsurgery based on only two predictor variables: age and race. ANNs are often represented in diagrams such as the one in Figure 14.1. The circles in the diagram are known as nodes. A typical neural network consists of three layers of nodes: input, hidden, and output nodes. The values of the predictor variables reside in the input node. The output node contains the predicted output of the network. The hidden nodes in the diagram contain a function known as the activation function that allows the network to model complex nonlinear associations between the predictor variables and the outcome.

Each input node is connected to each hidden node, and each hidden node is connected to the output node. In this example, there are two input nodes where the values of age (X₁) and race (X₂) are input into the network along with a bias weight, which is the equivalent to an intercept term found in a regression model.

Figure 14.1 Diagram of neural network trained to predict the probability of a CRC patient’s survival five years postsurgery on the basis of age (X₁) and race (X₂).

The input nodes are connected to the hidden nodes by a connection weight (the connection weights are the lines in Figure 14.1 connecting the input and hidden nodes). The connection weights can be thought of as the neural network equivalent of the β coefficients in a logistic regression model. At each hidden node the connection weights are passed to an activation function, most commonly the sigmoid function. The activation function uses the connection weights to model any nonlinear relationship among the predictor variables and the outcome variable. Another set of connection weights is then passed from the hidden node to the output node to obtain the output of the network. This output of the network corresponds to the predicted probability of the outcome variable.

In the ANN analysis performed there were as many input nodes as predictor variables: three hidden nodes (the default setting in SAS Enterprise Miner) and one output node (probability of survival five years postsurgery for patients with CRC).

Each box represents an input node in which the predictor variables are input into the network. Each line represents a connection weight. Each circle in the middle of the diagram represents the hidden layers where the relationship between the predictor variables and the outcome is modeled. The last circle at the end of the diagram is where the probability of survival is output from the network.

Assessment of Data Mining Models

Kolmogorov–Smirnov Statistic The Kolmogorov–Smirnov (KS) statistic was used as the measure to evaluate model performance. The KS statistic measures the difference between two distributions, where the actual KS statistic is the maximum difference between two different distributions. In this case, the two distributions of interest are the estimated probabilities of belonging to the survival or nonsurvival groups produced by the models. If the two distributions are the same, this implies the model did not effectively separate between survivors and nonsurvivors (implying a small KS statistic). On the other hand, significantly different distributions suggest good separation between the two groups (implying a larger KS statistic). The KS statistic has a known theoretical probability distribution, so a p-value was computed to determine if the two distributions are significantly different.

The predictive models were built on the training data set, and the validation data set was used to obtain the KS statistics and the corresponding p-value.

Area under the Receiver Operating Characteristic Curve Another common predictive model diagnostic is the area under the receiver operating characteristic curve (AUC). For every given probability cutoff, the confusion matrix in Table 14.1 is computed.

The receiver operating characteristic (ROC) curve is a plot of the false positive rate (x-axis) and true positive rate (y-axis) for every single probability cutoff. The ROC curve gives an indication of how well the models are separating between those patients who died and those who survived. A good predictive model will go up fairly steep and then start to level off. There is a 45° diagonal reference line on the ROC plot, as shown in Figure 14.2. The diagonal line represents where the false-positive and true positive rates are the same. If the ROC curve falls below this diagonal line, the model is no better than randomly assigning patients as dead or alive five years postsurgery. A model that has an ROC curve above this line is considered to be a good predictive model. Hence, the higher the AUC value, the better is the predictive model.

ROC curves have been shown to be valuable tests in evaluating the detection of certain types of cancer.¹⁸ With the appropriate use of ROC curves, investigators of cancer detection tests can improve their research. In many cases, ROC curves help the medical community focus on classification rules with low false-positive rates, which are most important for the detection of cancer. However, these ROC curves should always be put in perspective, because a good classification rule for the detection of cancer does not guarantee that cancer screening will reduce cancer mortality.

Figure 14.2 ROC plots for the three predictive models.

Three Data Mining Models for Colorectal Cancer

Figure 14.3 displays the analytical process flow used in SAS Enterprise Miner 6.1.

Once the data are brought into Enterprise Miner, the first issue that needs to be addressed is splitting the modeling data set into training and validation data sets.

A critical step in prediction is choosing among competing models. Given a set of training data, you can easily generate models that very accurately predict a target value from a set of input predictor variables. Unfortunately, these predictions might be accurate only for the training data themselves. Attempts to generalize the predictions from the training data to an independent but similarly distributed sample can result in substantial reductions in accuracy.

To avoid this pitfall, SAS Enterprise Miner is designed to use a validation data set as a means of independently gauging model performance. Typically, the validation data set is created by partitioning the raw analysis data. Observations selected for training are used to build the model, and observations selected for validation are used to tune and compare models. For this modeling exercise 60% of the data were allocated to the training data set and 40% to the validation data set.

Figure 14.3 Analysis flow for the three predictive models in SAS^® Enterprise Miner™ 6.1.

The first analysis built was the decision tree. Decision tree models are advantageous because they are conceptually easy to understand, yet they readily accommodate nonlinear associations between input variables and one or more target variables. They also handle missing values without the need for imputation. CART was the specific decision tree used in this study. Enterprise Miner help files discuss how to set up the decision tree node as a CART.

The next node used in the analysis was the transform node. Sometimes, input data are more informative on a scale other than that on which they were originally collected. For example, variable transformations can be used to stabilize variance, remove nonlinearity, improve additivity, and counter nonnormality. Therefore, for many models, transformations of the input data (either dependent or independent variables) can lead to a better model fit. These transformations can be functions of either a single variable or of more than one variable. For this analysis, many of the input variables were highly skewed. Several different transformations were examined. The best transformation, in terms of making the input variables more symmetric, was the log.

After the transform node was used, the imputation node was used for the logistic and ANN models. In SAS Enterprise Miner, however, models such as regressions and neural networks ignore observations altogether that contain missing values, which reduces the size of the training data set. Less training data can substantially weaken the predictive power of these models. To overcome this obstacle of missing data, imputation of the missing values can be performed before the models are fit. For this analysis, the imputation technique chosen was the mean.

After the models are run, the model comparison node allows the user to obtain diagnostic statistics such as KS and AUC.

Figure 14.3 displays the ROC curves for each of the three models. Based on the AUC values in Table 14.2, the ANN is outperforming the other two models. In terms of which model is best at separating those patients who will and will not die five years postsurgery, the ANN is also superior based on the KS statistics displayed in Table 14.3. Also note the p-value reported for each of the KS statistics; all indicate that there is a significant difference between the distribution of the survival and nonsurvival populations. This result indicates that all the models are doing a good job of separating these two populations. Overall, the ANN is the champion model for this particular data set.

Text Mining Healthcare Alerting Examples

Text mining is a data mining methodology intended to extract meaningful information from unstructured textual data. Text and data mining have much in common; underlying each is the assumption that knowledge lies buried in a large mass of data. Data mining primarily relies on statistical methods to uncover trends in structured data, whereas text-mining techniques seek to make sense of information that is unstructured, such as a doctor’s notes on a patient’s chart or discharge record. Other forms of unstructured data include text files, HTML files, chat messages, emails, images, or handwritten notes. One recent study reported that an estimated 80% of a healthcare organization’s information is contained in an unstructured textual format.¹⁹ Text mining converts the unstructured data into a structured resource that can be used for analysis purposes.

In many cases, unstructured text remains the best option for health-care providers to capture the depth of detail required, for example, in a clinical setting, or to preserve productivity by incorporating dictation and transcription into the workflow. Unstructured text records contain valuable narratives about a patient’s health and the reasoning behind healthcare decisions.

Several recently published studies illustrate the use of text mining in the healthcare field to alert healthcare professionals to potential epidemics, hospital readmissions, and drug safety. Some examples are given in the text that follows.

Data gathered from social media websites such as Facebook or Twitter fit into the “V” for Volume of the big data definition. Although data from Facebook or Twitter posts are unstructured, they contain a wealth of information useful in alerting healthcare professionals to potential health epidemics. One recent study described how to utilize Twitter post data to effectively track an epidemic in real time.²⁰ the authors discuss how Twitter posts from October 1, 2009 through May 20, 2010 were searched using text mining techniques; the frequency of words such as h1n1, flu, swine, and influenza was counted. The information derived from the frequency of the words was incorporated into a predictive model that assisted the researchers in predicting on a weekly basis whether an influenza epidemic was present or not in the United States.²⁰

It is well known that billions of dollars are spent annually in the United States when patients return to the hospital because of a lack of appropriate follow-up care.²¹ A hospital readmission is defined as an admission to a hospital or a healthcare setting within a certain time frame, following an original hospital stay. A readmission can occur at either the same hospital or a different hospital, and may involve planned or unplanned surgical or medical treatments. One recent change to the U.S. Healthcare Affordable Care Act is that healthcare providers will be subject to a financial penalty if their readmission rates are too high. Many providers are looking to healthcare text analytics to help address this challenge.

One way would be to analyze a hospital’s discharge instructions. Discharge planning is the development of an individualized discharge plan for the patient before he or she leaves the hospital, to ensure that patients are discharged at an appropriate time and with provision of adequate postdischarge services. The discharge planning process seeks to determine the appropriate level of services required by the patient, and then matches the patient to an appropriate site of care.

Five investigators from the Mayo Clinic presented a unique research project that used automated text mining to determine whether a hospital discharge summary contained a follow-up clinic appointment.²¹ A dataset consisting of discharge records was manually reviewed to determine whether the records contained follow-up appointment instructions. The same dataset was evaluated for the same criteria using SAS Text Miner software. The two assessments were compared to determine the accuracy of text mining.

Of the 6,481 discharge records reviewed, 3,576 (55.2%) were identified as containing all criteria for follow-up appointment instructions through manual review, 113 (3.2%) of which were missed through text mining. Text mining incorrectly identified 107 (3.7%) follow-up appointments that were not considered valid through manual review. Therefore, the text mining analysis concurred with the manual review in 96.6% of the appointment findings.

The Mayo Clinic researchers concluded that text mining of medical records can accurately detect whether elements of follow-up appointment instructions are documented in hospital discharge notes. The results also suggest that text mining software can be used to identify specific appointment criteria in a large number of textual medical records, thus saving considerable resources required for manual evaluation in quality-related research and performance assessment.

The results of this research project provide a platform for the Mayo Clinic to automatically track its clinic appointment follow-up after hospitalization, develop new strategies for improvement, and reduce its hospital readmission rate.

Today, most automated electronic medical records contain a substantial amount of information that is stored as text. Data mining provides a powerful tool to now use this data for operations improvements; in particular, it may be used to ensure drug safety.

Adverse drug events result in more than 100,000 deaths per year and are the fourth leading cause of death, ranking ahead of pulmonary disease, diabetes, AIDS, pneumonia, accidents, and automobile deaths.²² When a new drug is brought to market, it is tested for adverse reactions. However, given the infinite number of ways that drugs can interact, it is not practical to test every possible drug interaction. Iyer et al. reported on how text mining has been used to alert healthcare professionals to patient drug safety. The study²³ analyzed more than 50 million clinician notes that are part of the electronic health record (EHR) using text mining techniques as well as the structured component of the EHR. The study included 1,165 drugs and 14 adverse events. The researchers developed a predictive model, using both structured and unstructured data, to determine if a patient was likely to have an adverse reaction. This information would assist the clinical decision support team by allowing the healthcare workers to intervene in the patient’s care if an adverse drug affect was predicted. The study demonstrated that by incorporating unstructured data, patient care can be improved and lives saved by implementing this type of warning system.

Conclusion

In conclusion, this chapter demonstrates the usefulness of data and text mining analytic methods in the healthcare industry. When considering CRC, predictive models and their ability to separate between survival and nonsurvival for CRC patients five years postsurgery is a promising approach for developing a diagnostic evaluation. This initial case study reveals the strong potential for data mining methods for CRC. Particularly, this study has shown the promise of using ANNs for decision support among researchers and clinicians in the CRC community.

In this case study, the ANN outperformed the more well-known and understood models, such as the decision tree and logistic regression. In particular, ANNs have been shown to be very powerful and superior over decision trees and logistic regression in predicting a clinical outcome in patients with CRC.^24–26

As described in the preceding text, a series of recent reports in the literature have shown that data mining techniques such as ANN provide higher predictive accuracy than familiar, traditional models such as logistic regression.^27–30 ANNs and other data mining tools are often called “black box” techniques because the logic used to determine the final model is not transparent. This lack of transparency is the greatest disadvantage that medical researchers find when they use ANNs for decision support. Yet, for this one disadvantage, there are many advantages for the medical community to use ANNs for clinical decision support. Claimed advantages of ANNs include:

1. Ease of optimization with the advancement of easy-to-use software such as SAS Enterprise Miner

2. Accuracy for predictive inference, with potential support for clinical decision making

As shown in this chapter, data mining models, especially ANNs, can be developed that accurately predict the outcome of a medical condition. These predictive models can be valuable tools in medicine. They can be used to assist in determining successful treatments, prognosis, or interventions. However, there are areas of concern in the development of ANNs as data mining models for use as alerting systems in the medical community. The U.S. Food and Drug Administration (FDA) has issued a guidance document for software for medical support. The document has a section on using ANNs. Some of the points highlighted by this document are that ANNs can behave in a nondeterministic way. The medical researchers must be able to justify and explain the choices made for the ANN model and the topology. The researchers must also be able to describe how over-fitting is avoided (e.g., using training and validation data sets). The document specifies additional data sets to be processed through the ANN to ensure that performance remains as expected.

Although data mining can provide useful information and support to the medical community by identifying patterns that may not be readily apparent, there are limitations to what data mining can do. Not all patterns found are interesting. For a pattern to be interesting, it should be logical and actionable. Data mining requires human intervention to exploit the extracted knowledge. For example, data mining can provide assistance in making the diagnosis or prescribing the treatment, but it still cannot replace the physician’s intuition and interpretive skills.⁶

Data and text mining methods used as alerting systems in health-care are capable of identifying patterns and discovering relationships in large medical databases, but without cooperation and feedback from the medical community, the results are useless. The patterns found via data mining methods should be evaluated by medical professionals with many years of experience in the problem domain to decide whether the patterns are logical, actionable, and novel enough to be directed in new clinical research directions. Data mining should not aim to replace medical professionals and researchers, but to complement their invaluable efforts to save more human lives by appropriate intervention using some of the healthcare alerting systems described in this chapter. Intervention and decision support can be found more quickly with data and text mining techniques.