Interviews

Interviews are structured or unstructured conversations focusing on needs. Swanson (1994) points out that conducting effective interviews requires a good deal of skill and preparation. The author lists these skills:

• The ability to develop questions that will get meaningful answers.
• The ability to ask open-ended questions spontaneously.
• The ability to create an atmosphere of trust—not defensiveness.
• The ability to take complete and accurate notes without infusing one's own ideas (p. 80).

Instructional designers should usually focus these conversations on key managers' perceptions about the performance problem and the planned instruction to solve it. A key advantage of interviews is that they allow instructional designers the flexibility to question knowledgeable people, probing for information as necessary (Holstein and Gubrium 2001). A key disadvantage of interviews is that they may be time-consuming and expensive to carry out, especially if travel is required. To plan interviews, instructional designers should:

• Prepare a list of general topics or questions.
• Identify people knowledgeable about training needs.
• Meet with the knowledgeable people and pose questions about training needs.
• Take notes during or immediately following the interview.

Direct Observations

Direct observations of work are, as the phrase implies, first-hand examinations of what workers do to perform and how they do it. They may be planned or unplanned, they may rely on specialized forms to record the actions or results of performers, and they may even focus on behavior (Thompson, Felce, and Symons 1999).

Indirect Observations

Indirect examinations of performance or productivity measures are called indirect because they are unobtrusive and do not require instructional designers to observe workers performing; rather, they judge performance from such tangible results or indicators of results as production records, quality control rejects, scrap rates, work samples, or other records about the quantity or quality of work performed. Indirect examinations may be structured (in which results of observations are recorded on checklists) or unstructured (in which the researcher's feelings and perceptions about results are recorded).

Questionnaires

Questionnaires, sometimes called mail or web-based surveys, consist of written questions about instructional needs. They solicit opinions about needs from performers, their supervisors, or other stakeholders. They are sometimes developed from interview results to cross-check how many people share similar opinions or perceptions about needs (Dillman 1999). They may be structured (and use scaled responses) or unstructured (and use open-ended essay responses). In recent years, many people have moved from so-called paper-and-pencil questionnaires to web-based or web-supported questionnaires.

Anecdotal evidence suggests that response rates as poor as 5 percent are not uncommon, and such low response rates are not helpful for drawing generalizations—although they may provide intriguing information for subsequent data collection efforts. Due to these shortcomings, a designer using a questionnaire to collect data should be prepared to:

• Gain the support of the target group's management so questionnaire recipients understand that their management is supporting the data collection process.
• Keep the questionnaire to a reasonable length. Response rates are likely to be negatively affected if the instrument is perceived to be overwhelming.
• Reassure the recipients (and their management) that the data will be held in strict confidence and that no one individual's responses will be divulged.
• Follow up with the responders after the data has been collected and analyzed so they have a general idea of what was discovered because of the process and how the information will be used.

Key Informant Groups or Focus Groups

Key informant groups or focus groups rely on highly knowledgeable people or committees composed of representatives from different segments of stakeholders (Bader and Rossi 2002; Krueger and Casey 2000). Key informant groups are especially knowledgeable about a performance problem or possible instructional needs; focus groups are committees, usually created informally, that pinpoints instructional needs through planned participation of representatives from key stakeholder groups.

Nominal Group Technique

The nominal group technique (NGT) takes its name from the formation of small groups in which the participants do not, during the earliest stages of data collection, actively interact. Hence, they are groups in name only—they are only nominal groups. To use NGT in data collection, instructional designers should:

• Form a panel of people representative of the targeted learners (or their organizational superiors).
• Call a meeting of the panel.
• Ask each panel member to write opinions about training needs on slips of paper.
• Permit no discussion as the opinions are being written.
• Record items on a whiteboard or by using Post-It notes for subsequent panel discussion.
• Combine similar responses.
• Solicit discussion from panel members about what they have written.
• Ask panel members to vote to accept or reject the opinions about training needs recorded earlier.

The Delphi Procedure

The Delphi procedure takes its name from the famed Delphic Oracle, well-known during ancient Greek times. Similar in some ways to NGT, the Delphi procedure substitutes written questionnaires for small-group interaction to collect information about training needs. To use the Delphi procedure to collect data, instructional designers should:

• Form a panel of people representative of the target group.
• Develop a written questionnaire based on the training needs or human performance problems to be investigated. Posing open-ended questions is acceptable at the outset.
• Send copies of the questionnaire to panel members.
• Compile results from the initial round of questionnaires and create scales to assess levels of agreement among the experts.
• Prepare a second questionnaire and send it and the results of the first round to the panel members.
• Compile results from the second round.
• Continue the process of feedback and questionnaire preparation until opinions converge, usually after three rounds.

The Critical Incident Method

The critical incident method takes its name from collecting information about critically important (critical) performance in special situations (incidents). Critical incidents were first used as a method of collecting information about the training needs of pilots during World War II and were subsequently used to identify special training needs of CIA agents (Johnson 1983).

To use the critical incident method, instructional designers should:

• Identify experts such as experienced performers or their immediate supervisors.
• Interview the experts about performance that is critical to success or failure in performing a job.
• Ask the experts to relate anecdotes (stories) from their first-hand experience about situations in which performers are forced to make crucially important decisions.
• Compare stories across the experts to identify common themes about what performers must know.
• Use this information to identify training needs.

Alternative approaches to this critical incident process may be used and may focus on the most difficult situations encountered, common daily work challenges, or the most common human performance problems observed with newcomers.

Root Cause Analysis

The root cause analysis process was popularized by the quality movement that drew a lot of attention in the 1980s. The strength of the methodology is that it not only attempts to answer the what and how questions, but also the why question. If we know why something happened, then we can get to the source of the problem and address the underlying causes so the issue will not occur again.

There are several ways to depict the process, but, as Rooney and Vanden Heuvel (2004) point out, the process comprises four main steps:

• Step One: Data collection. All information related to the problem is collected so it can be analyzed to uncover the eventual cause.
• Step Two: Causal factor charting. A chart is created that displays the sequence of events that led up to the problem occurrence and the conditions that surrounded the event.
• Step Three: Root cause identification. Now that the potential causal factors have been identified, the investigators can now analyze the causes to identify the reason(s) for each cause.
• Step Four: Recommendation generation and implementation. In this final step, the investigators present recommendations for preventing the problem from occurring again.

Okes (2008) points out that effective root cause analysis can often be negatively influenced by human emotional characteristics. These include:

• Recency bias. If the same or similar problem occurred recently, the current situation resulted from the same causes that affected the first problem.
• Availability bias. In data collection, we tend to collect the data that's easy to obtain rather than the data we should collect.
• Anchoring bias. Latching onto the first piece of data we collect while ignoring other, more relevant data.
• Confirmation bias. Collecting only evidence that supports our theory of what caused the problem rather than looking for evidence that might disprove our theory.

Root cause analysis has proven to be an effective tool, when done properly, for uncovering the source of performance problems and correcting the causes to prevent future problems from arising.

Competency Assessment

Competency assessment has been growing in popularity in recent years (Rothwell and Graber 2010; Rothwell, Graber, Dubois, Zabellero, Haynes, Alkhalaf, and Sager 2015; Rothwell and Lindholm, 1999). Its purpose, according to one of many views, is to identify and isolate the characteristics of ideal (exemplary) performers (Dubois and Rothwell 2000). Those characteristics become a foundation for preparing instruction designed to raise average performers to ideal performers. A major advantage of competency assessment is that it is targeted toward achieving ideal performance more than rectifying individual performance problems or deficiencies. But a major disadvantage is that needs assessments using this form of data collection may be expensive and time-consuming to do if they are to be legally defensible.

To use the competency assessment method, instructional designers should:

• Form a panel of managers or experienced performers.
• Identify the characteristics of ideal performers. (In this context, characteristics may mean behaviors, results achieved, or both.)
• Pose the following questions to the panel members: What characteristics should be present in competent performers? How much should they be present? Answering these questions may involve behavioral events interviewing in which exemplary performers are asked to relate a significant work-related story from their experience and describe exactly what they did, how they felt as they did it, and even what they thought as they did it.
• Devise ways to identify and measure the characteristics.
• Compare characteristics of actual performers to those described in the competency model.
• Identify differences that lend themselves to corrective action through planned instruction.

Numerous alternatives to this approach exist. The reason: Views about what should be the basis for competencies may differ. According to one view, for instance, competencies are derived by studying the results (outputs) produced by performers; according to another view, competencies are derived from examining common characteristics shared by exemplary performers.

Assessment Centers

An assessment center is not a place; rather, it is a method of collecting information (Rothwell 2015; Thornton 1992). They are used to screen higher-level job applicants (those applying for senior leadership positions) or to assess the potential of existing employees for promotions. The process can range from a few assessment measures all the way to multiple measures and methods requiring days or even weeks to complete.

Assessment centers are expensive to design and operate, which is a major disadvantage of this approach to data collection. However, their results are detailed, individualized, and job-related, and that is a chief advantage of the assessment center method. To use the assessment center, instructional designers may have to rely on the skills of those who specialize in establishing them. The basic steps in preparing an assessment center are, however, simple enough. They require a highly skilled specialist, familiar with employee selection methods and testing validation, to:

• Conduct an analysis of each job category to be assessed.
• Identify important responsibilities for each job.
• Use the results of Step 2 to develop games or simulations based on the knowledge and skills needed to perform the job successfully.
• Train people to observe and judge the performance of participants in the assessment center.
• Provide each individual who participates in the assessment center with specific feedback from observers about training needs.

Performance Records

Marrelli (2005a and 2005b) points out that existing performance records can also serve as a way of informing critical incidents. Examples include exit interviews, performance appraisals, and work diaries.

Exit interviews are planned or unplanned conversations carried out with an organization's terminating employees to record their perceptions of employee training needs in their job categories or work groups. Exit interviews are relatively inexpensive to do and have high response rates. However, they may yield biased results because they highlight perceptions of employees who left the organization.

Many instructional designers wonder when to choose one or more of these data collection methods. While there is no simple way to decide about choosing a method, several important issues identified by Newstrom and Lilyquist (1979, 56) in their classic treatment of this topic are still relevant:

• Incumbent involvement: How much does the data collection approach allow learners to participate in identifying needs?
• Management involvement: How much does the data collection approach allow managers in the organization to participate in identifying needs?
• Time required: How long will it take to collect and compile the data?
• Cost: What will be the expense of using a data collection method?
• Relevant quantifiable data: How much data will be produced? How useful will it be? How much will it lend itself to verifiable measurement?

In considering various data collection methods, instructional designers are advised to weigh these issues carefully. Not all data collection methods share equal advantages and disadvantages.

Work Samples

The outputs or actual products of work can be useful in the data collection and analysis phase because they represent the directly observable outcome of the work being studied that are collected. Because they represent the output of work, they have high validity, unlike the data collected using focus groups or interviews in which the information is subject to filtering and individual perceptions.

Process Mapping

Process mapping is used to identify the steps taken in sequence to produce a work output. The output of this process is a matrix or a process map. Such a map may also identify the time required to complete each step, a measure of success for each step, and the conditions under which the tasks are performed. The target group for process mapping is a representative group of performers or a small subset of top performers. Marrelli (2005b) points out that process mapping “…can be an excellent approach to identifying the content that should be included in an instructional course, manual, or job aid intended to help workers execute a process” (41).

Unobtrusive Measures

While our goal as instructional designers is to collect valid and relevant data we can use to make sound decisions, not every environment can be said to support this outcome. In some organizations, labor and management conspire, either knowingly or unknowingly, to produce the desired output through less-than-desirable behaviors. This concept is “shadowboxing with data,” in which work conditions or other impediments may be hidden by what appears to be desirable outcomes. One example that is much discussed today in education is the concept of “teaching to the test.” While scores may meet a standard, the behavior behind these results may be less than ideal. Winiecki (2009) describes the concept this way:

“In shadowboxing with data, individuals who know what sorts of measured outputs are desired (by themselves or their organization) may modify their practice so as to produce ‘good numbers’ rather than what we consider to be ‘good performance’” (32).

To address this phenomenon, we might consider unobtrusive measures described as follows. Marrelli (2007) points out that many of the commonly used data collection methods (surveys, focus groups, questionnaires, interviews) are inherently biased by their altering the situation being studied, they are reactive and the participants, by agreeing to participate, will naturally impose their own biases and perceptions. These limitations make it “…difficult to distinguish between typical, real behaviors and behaviors induced by the measurement” (44).

To compensate for these limitations, unobtrusive measures correct for this limitation. They can be grouped into three categories: (1) physical traces; (2) archives; and (3) observations. An education program that encourages recycling of used paper can be assessed by the volume of recycled paper before and after the program (physical traces). Data stored or collected by an organization not intended for analysis can be a rich source of information (archives). Target groups performing a job or task under analysis can be observed without their knowledge (observations) to gather valid and reliable information.

Specifying Instruments and Protocols

What instruments should be used during the needs assessment, and how should they be used? What approvals or protocols are necessary for conducting the needs assessment, and how will the instructional designer interact with members of the organization? These questions must be addressed in a needs assessment plan. The first has to do with specifying instruments; the second has to do with specifying protocol.

Many instruments may be used in needs assessment. Common methods of collecting information about instructional needs rely on commercially available or tailor-made questionnaires, interview guides, observation guides, tests, and document review guides. Commercially available instruments and online data collection methods have been prepared for widespread applications, although some consideration of how to use an instrument or groupware program in one organizational setting is usually necessary and should be described in the needs assessment.

Tailor-made instruments are prepared by instructional designers or others for assessing instructional needs in one organization or one job classification. Developing a valid, reliable questionnaire may require substantial work in its own right, and this process should be described in the needs assessment plan. Using groupware necessitates establishing an approach to data collection. Table 3.3 summarizes methods of needs assessment.

Source: J. Newstrom and J. Lilyquist. Reprinted from Training and Development Journal (1979), 56. Copyright © 1979. The American Society for Training and Development. Reprinted with permission. All rights reserved.

Protocol means diplomatic etiquette and must be considered in planning needs assessment. It stems from organizational culture—the unseen rules guiding organizational behavior. “Rules” should be interpreted as the means by which instructional designers will carry out the needs assessment, interact with the client, deliver results, interpret them, and plan action based on them. In developing the needs assessment plan, instructional designers should seek answers to such questions as these:

• With whom in the organization should the instructional designer interact during the needs assessment? (How many people? For what issues?)
• Whose approval is necessary to collect information? (For example, must the plant manager at each site grant approval for administering a questionnaire?)
• To whom should the results of the needs assessment be reported? To whom should periodic progress reports be provided, if desired at all?
• How have previous consultants interacted with the organization? What did they do well, or what mistakes did they make, according to managers in the organization?
• What methods of delivering results are likely to get the most serious consideration? (For instance, will a lengthy written report be read?)

Instructional designers should always remember that the means by which needs assessment is carried out can influence the results and the willingness of the client to continue the relationship. Use effective interpersonal skills.