Chapter 12
Analyzing Qualitative Data
Keenan (Kenni) Crane
In This Chapter
This chapter explores various ways to analyze data from interviews, focus groups, case studies, and other qualitative data collection methods without having to be a qualitative researcher. Upon completion of this chapter, you will be able to
- describe the importance of qualitative data to support and give credibility to the monetary data
- describe three types of validity in qualitative data
- apply a four-step approach to organize your descriptive data.
The Importance of Using Qualitative Data
Qualitative data are typically defined as information that can be captured that is nonnumerical. These data are obtained through interviews, focus groups, observations, case studies, and document reviews. They can be captured through pictures, videos, emails, text messages, and social networking sites. By their very nature qualitative data are subjective with a potential for researcher’s bias. Although argument about which is better—qualitative or quantitative—often ensues, the fact is that both types of data are important in describing the success of training programs.
Unfortunately for evaluators, good analysis of qualitative data often requires more time and effort than do most quantitative research analyses. One reason for this is that to get greater insight into the significance of qualitative data, codes, themes, and even numerical values are often assigned to these data. Once the assignment is made, then the data are manipulated to achieve a better of meaning of the data. Too often, however, evaluators do not want to go these lengths to analyze their qualitative data, and thereby, threaten the reliability of their story of program success and force a greater reliance on more believable data that are quantitative in nature.
However, without qualitative data, quantitative data are limited in what they reveal about program success. Quantitative data are known to “show” the results of an evaluation, whereas qualitative data are known to “tell” the story. Table 12-1 presents a comparison of quantitative and qualitative analysis objectives as described by Mach and others (2005).
Without strong qualitative analysis, the evaluation researcher provides a skewed look at results—either by lack of balance (quantitative and qualitative) or by weak analysis. What is required after gathering case studies, interviews, and focus group data is a creative yet structured way of observing the data, sorting and coding the data, noticing patterns in the data, and reobserving and revisiting the data. In the final presentation and analysis, this synthesized information can help explain quantitative data and return-on-investment (ROI) findings to the customer or client.
Table 12-1. Comparison of Quantitative and Qualitative Objectives of Analysis
| Quantitative | Qualitative |
| Seek to confirm hypotheses about phenomena | Seek to explore phenomena |
| Instruments use rigid style of eliciting and categorizing response to questions | Instruments use more flexible, iterative style of eliciting and categorizing responses to questions |
| Use highly structured methods such as questionnaires, surveys, and structured observation | Use semi-structured methods such as in-depth interviews, focus groups, and participant observation |
Source: Mach and others (2005).
A Word About Validity in Qualitative Research
Because of their inherent subjectivity, it is important to consider the perceived validity of qualitative data. In reporting evaluation findings and linking data points with less tangible qualitative data, we need to concern ourselves with being plausible, credible, trustworthy, and therefore defensible to our customers and clients. Our ultimate goal is to explain our quantitative findings and contribute to the theory on which we base future evaluation projects.
Types of Validity
Researcher bias, or finding what you want to find, can be a criticism of quantitative as well as qualitative findings. And although some qualitative researchers reject the framework of validity that is found in quantitative research (Trochim, 2006), we want to do everything possible to avoid this kind of criticism and to know that we have taken as many precautions as necessary to report the clearest and most valid picture to our clients.
Three types of validity are important to think about when working with qualitative data. The first type of validity is descriptive, which answers the question, “How accurate is the information—for example, behaviors, settings, and times—that is reported?” The second type of validity is called interpretive validity and answers the question, “Has this information—thoughts and viewpoints, for instance—been accurately understood and reported?” The third type of validity is theoretical validity, which answers the question, “Have we adequately considered all possible explanations for the results we are reporting and not just fit the data to our bias?”
Approaches to Managing Validity
We can promote credibility in reporting qualitative data by taking a few common sense approaches with each of the three types of validity. Johnson (1999) suggests the approaches described in table 12-2.
Techniques to Analyze Qualitative Data
There are various ways to analyze qualitative data. Donald Ratcliff’s compilation provides a description of the following techniques:
- typology
- taxonomy
- constant comparison/grounded theory
- analytic induction
- logical analysis/matrix analysis
- quasi-statistics
- event analysis/microanalysis
- metaphorical analysis
- domain analysis
- hermeneutical analysis
- discourse analysis
- semiotics
- content analysis
- heuristic analysis
- narrative analysis.
Table 12-2. Approaches to Managing Validity
| Type of Validity | Approach |
| Descriptive | Use multiple observers or researchers to interpret, cross-check, and explain the data. |
| Interpretive | Ask for participants’ feedback about the interpretations being made to clear up any misunderstandings. Use actual words and phrases in direct quotations in the report. |
| Theoretical | Search for alternative explanations that do not fit your predictions and discuss with colleagues and peers. |
Although this long list is impressive, there are still other ways in which qualitative data can be analyzed, ranging anywhere from nominal group technique (Phillips and Phillips, 2007) to card sorting and affinity diagrams (Straker, 1997). The key is to recognize that analyzing qualitative data is an iterative process, not a linear process. Good analysis requires evaluators to review, reflect, code, and then review, reflect, and code again, until ultimately a story emerges. By following the four steps that follow, a relatively structured approach to analyzing qualitative data can be used, thereby saving evaluators time while still providing a reliable story of program success.
Four Steps to Qualitative Analysis
Whether data are collected using interviews, transcribed notes, case studies, or questionnaires, the best place to begin your analysis is to organize the data. Once organized, read and read multiple times. Begin to identify common ideas and patterns through a process of coding or finding words and phrases to represent consistent themes. Remember, while preset codes are a starting point, the fun in qualitative analysis is watching the themes emerge through the data. So be ready to add to your predetermined codes or change them as the story unfolds. The following four steps may help bring some structure to a by-design unstructured process.
Step 1: Observing and Searching
- Read through all the qualitative data—case studies, questionnaires, and comments—without judgment.
- Search and be objective in your observations (have others do the same).
- Keep in a “discovery” mindset rather than prejudging the data.
Step 2: Sorting and Coding
- Use condensed words or phrases to identify key categories.
- Be consistent throughout.
- Underline and highlight.
Step 3: Discovering and Coding
- Step back and notice patterns in how you sorted the information.
- Find similarities.
- Find differences.
Step 4: Reobserve and Reread
- Notice if you have a linear map or a three-dimensional map.
- Rethink: if you have found parts and have a linear map, now look at the whole picture and find a three-dimensional picture.
- Look at data again for new insights.
- Finally, involve others in discussion and brainstorm for alternative insights.
Analysis comes within the process and as a result of the process. It is impossible to remove all subjectivity. Having multiple people, however, go through the data in the same way can improve reliability. Common threads will emerge.
Final Thoughts
Evaluation results need to inform, not just with important facts about quantitative data, but also describe other findings that emerge through the analysis. Qualitative analysis is an iterative process and the results that come out of it help evaluators describe what people say, what they mean or need, what they do, and the culture in which they work. How qualitative results are presented both in written report and verbal report form is important. The key to success is to be clear about what you are trying to achieve through your communication. Chapter 18, Reporting Evaluation Results, describes the importance of communicating evaluation results in detail.
Qualitative results can place emphasis on areas of importance. They can describe scenarios and situations that demonstrate a point. They can explain how the quantitative data presented were derived and how to improve a program’s overall success. The argument should not be about which is better, quantitative versus qualitative, but rather how best to use them both to demonstrate the success of training.
Knowledge Check: Analyzing Qualitative Data
Now that you have read this chapter, let’s see what you learned about analyzing qualitative data. Check your answers in the appendix.
1. Why are qualitative data important to training measurement and evaluation?
2. List three types of validity you need to be concerned with in qualitative analysis. What does each type of validity address?
3. List four steps for analyzing qualitative data.
About the Author
Keenan (Kenni) Crane, PhD, is an internationally experienced consultant in individual, group, and organization development using a systems approach to help managers improve performance and strengthen business relationships. For 20 years, Crane has coached leaders in Fortune 500 corporations, family businesses, and not-for-profit organizations to develop leadership and interpersonal and social skills while facilitating a smooth transition during corporate and cultural changes.
In addition to her doctorate in organization and group development, she holds a master’s degree in experimental psychology, a master’s degree in counseling/ human relations, and a BA in psychology and pre-med (magna cum laude). She is presently a full-time faculty member in the e-Business School at Hamdan Bin Mohammed e-University in Dubai, United Arab Emirates. She can be reached at [email protected] or [email protected].
References
Johnson, R. B. (1999). Examining the Validity Structure of Qualitative Research. In A. K. Milinki ed. Cases in Qualitative Research: Research Reports for Discussion and Evaluation. Los Angeles: Pyrczak Publishing, 160–65.
Mach, N., C. Woodsong, K. M. MacQueen, G. Guest, and E. Namey. (2005). Qualitative Research Methods: A Data Collector’s Field Guide. Research Triangle, NC: Family Health International.
Phillips, P. P. and J. J. Phillips. (2007). A Strategic Approach to Retention Improvement: Southeast Corridor Bank. In Proving the Value of HR. Birmingham, AL: ROI Institute.
Ratcliff, D. (undated). 15 Methods of Data Analysis in Qualitative Research, available at http://qualitativeresearch.ratcliffs.net/15methods.pdf.
Straker, D. (1997). Rapid Problem Solving with Post-It Notes. Cambridge, MA: Da Capo Press.
Trochim, W. M. (2006). The Research Methods Knowledge Base, 2nd ed., available at www .socialresearchmethods.net/kb/.
Additional Reading
Creswell, J. W. (2008). Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research. Upper Saddle River, NJ: Pearson/Merrill Prentice.
Denzin, N. K. and Y. S. Lincoln. eds. (2005). The Sage Handbook of Qualitative Research. Thousand Oaks, CA: Sage, available at http://www.loc.gov/catdir/toc/ecip053/2004026085.html.
Miles, M. B., and A. M. Huberman. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage.
Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. Thousand Oaks, CA: Sage.
Rossi, P. H., M. W. Lipsey, and H. E. Freeman. (2007). Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage.
Shadish, W. R., T. D. Cook, and L. C. Leviton. (1992). Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA: Sage.
Spencer, L. M. and S. M. Spencer. (1993). Competence at Work: Models for Superior Performance. New York: Wiley.