Appendix B
Solutions to Case Studies
Solution to Exercise 2-3: The Efficient Electric Company Case
1. Who is the client?
In this scenario, no client was identified. Carolyn had experienced the program and took it upon herself to initiate the training.
2. Given the learning objectives, how would you measure learning (Level 2)?
You cannot currently measure at Level 2 based on the objectives because the objectives do not lend themselves to Level 2 assessment.
How could you improve the objectives?
We need to restate the objectives in measurable terms, based on course content and desired outcomes (information that was not provided in the case). We would want to specifically determine what the participants should know and be able to do (perform). Realistically, a needs assessment should have been done prior to any training to help formulate the objectives as part of the design and development process. The needs assessment would have determined if there was any indication of issues with working relationships or communication, and whether training was the right solution. It could also have helped determine how supervisors would demonstrate a change in behavior to address these issues.
3. How could you collect Level 3 data?
You may be able to conduct interviews or focus groups. However, given the time lapse from now to when the course was given (remember it was a year long effort), it would be difficult to specifically link the ideas from the course to use of those ideas on the job. If Carolyn tries this method, she will need to create a protocol. She may be able to get some information from those who made positive comments, assuming she remembers who they were.
4. What is the business metric?
No business metric was identified. To conduct a Level 4 evaluation (impact and ROI), a metric must be identified, tracked, and converted to dollar terms.
5. How could you perform a Level 4 evaluation on this program?
Realistically, you could not perform a Level 4 evaluation because there is not an identified metric, a value, or a way to track an impact. An attempt to establish benefits and a value of those benefits now, without data tracking any improvements, would call into question the integrity of the entire effort. Even if possible, it is doubtful that the effort would justify the cost of such analysis.
6. What do you recommend for improvements?
• Conduct a needs assessment; this would also indicate who would need the training instead of all supervisors.
• Secure an internal client.
• Establish measurable objectives.
• Design and develop the course with instructional strategies for measuring knowledge, skills, and transfer to the job.
• Pilot it with members from the target audience.
• Develop an evaluation plan with the client and to the level the client wants.
• Make sure the training course has instructional strategies to support transfer.
• Ensure the training course as assessments for knowledge acquisition and demonstration of new behaviors.
• If doing a Level 4 evaluation, establish the metric with initial data (baseline) and a method to track the data, determine the value of the change in the metric, and determine total program costs.
Solution to Exercise 5-1: Doing an Item Analysis
1. Complete the results chart. See the completed chart below.
2. Identify potential issues and possible causes.
Items 2, 12, and 19: Possible reasons why many participants arrived at the wrong answer could be that the item is confusing, or the content was not covered adequately during the training.
3. Determine which participants would pass the learning assessment if the criterion reference passing score were 85 percent.
A score of 17 or better would be required to pass. Therefore, participants 1, 2, and 3 would pass.
4. Determine how well the class as a whole performed on the assessment.
The class as a whole scored 98 correct responses out of a possible 120 (81.67 percent).
Solution to Exercise 6-1: Case Study on Transfer
1. Who made the decision for training?
• Scenario A: manager
• Scenario B: trainer
2. Was a needs assessment conducted?
No. Had an analysis been done, factors other than skill deficiency requiring training may, and probably would, be discovered. How often is training offered with the assumption that the issue is a skill deficiency, but without being supported by thorough analysis?
3. How do we know if training will solve the problem?
We don’t. In fact, because only the growth in new customers has flattened (sales of additional services remains strong), this indicates that they had good sales in the past and so training isn’t actually the solution.
4. What are some potential causes for the decline in sales, other than performance?
• competition
• market saturation
• Kimbel reputation
• cyclical impact
• issues with the services
• leadership issues resulting in low job satisfaction and motivation
• incentive issues.
5. Which barriers to transfer could be affected? Why?
• Pressure from peers to resist changes: Possibly.
• Interference from immediate (work) environment: The program was scheduled independent of their workload or the time of month or year.
• Trainees’ perception of irrelevant training content: It is a generic vendor program, not a needs assessment or job or task analysis.
• Lack of reinforcement on the job: Supervisors were not consulted or included in the process. They already think the telemarketers are good, so it is unlikely that they will support the use of training.
• Waste of time: All telemarketers were required to attend, even those whose sales remained strong.
• Trainees’ perception of impractical training programs: Due to their heavy workload, participants most likely won’t be able to implement the new skills. Also, they had no involvement in the program and there wasn’t a needs assessment.
• Trainees’ discomfort with change and associated effort: Human nature, change driven by people outside their unit who “don’t understand or know” the situation or work.
• Nonsupportive organizational culture: They get paid based on the number of calls and sales; lack of incentives.
• Trainees’ perception of poorly designed and delivered training: There wasn’t any customization or analysis of the program to ensure that sound design and adult learning principles were used. In scenario B, the critical components for transfer (practice exercises and role plays) were removed to shorten the program.
• Possible leadership and follower issues result in low job satisfaction and morale, thus resulting in low performance.
Solution to Exercise 7-2: Case Study: Use of Estimates
What is the annualized dollar impact of sales training for all stores based on the responses from the sales associates and store managers?
• Sales associates’ estimate: | $2,254,503.20 |
• Sales managers’ estimate: | $2,114,415.08 |
The store managers’ estimate is $140,088.12 less than the sales associates.
To determine the annualized dollar impact of sales training for all stores, first calculate the sales impact per store per month, which is $3,298 (or $27,482 × 12 percent margin). Then determine the average research results for the sales associates and store managers by multiplying the average impact per store per month by the percent impact by confidence level by 12 months by the 207 stores that received the training. For example, in the sales training row of Table 1:
$3,298 × 0.32 = $1,055.36 × 0.86 = $907.61 × 12 = $10,891.32 × 207 = $2,254,503.20
Table 1: Sales Associates
*May be some differences due to rounding.
Table 2: Store Managers
*May be some differences due to rounding.
Solution to Exercise 7-3: Calculating ROI: A Case Study
Program Costs
|
Meeting Management Training Program Costs |
Cost per Session |
|
Program design and development was $17,000 |
$8,500 (prorated) |
|
Facilitation (assume no preparation time and an annual fully loaded salary of $65,000) |
$271 ($65,000 ÷ 240 days) |
|
Participant time |
$8,320 ($100,000 ÷ 240 days = $416 per day x 20 participants) |
|
Materials ($35 per participant) |
$700 ($35 x 20 participants) |
|
Administrative support (annual fully loaded salary of $45,000) |
$375 ($45,000 ÷ 240 days x 2 days) |
|
Total Cost |
$18,166* |
*May be some differences due to rounding.
Benefit Analysis:
• 20 managers were involved in the training and analysis.
• Each manager was spending 20 hours per month in meetings.
• Salary plus benefits was $100,000 per manager.
• Unit value was $52 per hour ($100,000 ÷ 1,920 hours per year).
• Cost of meetings was $1,040 per manager ($52 × 20 hours).
• The annualized cost per manager was $12,480 ($1,040 × 12 months).
• The total annualized cost for the 20 managers was $249,600 ($12,480 × 20 managers).
The program was a one-day meeting management skills workshop, with the following results:
• Each manager is now spending eight hours per month in meetings.
• The new cost of meetings is $416 per month per manager ($52 per hour × 8 hours).
• The annualized cost per manager is $4,992 ($416 × 12 months).
• The total annualized cost for the 20 managers is now $99,840 ($4,992 × 20 managers).
• The benefit savings is $624 per month per manager ($1,040 − $416) or $7,488 per manager per year ($624 × 12 months).
• The total annualized benefit for the 20 managers is $149,760 ($7,488 × 20 managers).
or
• Benefit of 12 hours per manager per month (20 hours – 8 hours).
• The dollar value of the benefit is $149,760 ($52 per hour × 12 hours × 12 months × 20 managers).
What is the session’s ROI?
What is the cost-benefit ratio?
Solution to Exercise 7-4: Calculating ROI: Weston Marketing
1. What factors go into determining the cost of turnover?
• low productivity of others
• low morale
• recruiting costs
• training costs and time
• possible overtime of others
• severance
• supervisor time
• lower quality
• relocation
• loss of revenue.
2. What is the program’s ROI?
2a. What does turnover cost the company? $341,250
• Assume the cost of turnover is a fully loaded annual salary, so $26,250 × 1.3 = $34,125.
• If there are 50 agents in the call center and there’s a 20 percent annual turnover, that means 10 people leave each year (50 × 20 percent = 10).
• Annual cost is 10 people × $34,125 salary = $341,250.
2b. What is the cost of the training program? $20,623
• Vendor materials | $4,000 | |
| ||
• Customization | $3,500 | |
• Facilitation | $2,200 | |
| ||
• Facilitator coaching ($1,100 per day × 5) | $5,500 | |
| ||
• Participant time in training | $3,368 | |
| ||
| ||
| ||
• Participant time in coaching | $1,684 | |
| ||
• Lunches | $119 | |
| ||
• Jim’s time at lunches | $95 | |
| ||
| ||
| ||
• Administrative support | $157 | |
| ||
| ||
| Total | $20,623 | |
$800 × 5 days2c. What is the cost of post-program turnover? $136,500
• 50 agents × 8 percent (the new turnover rate) = 4 agents
• Cost of turnover is $136,500 (4 × $34,125)
2d. What is the benefit of lower turnover? $204,750
• Initial turnover cost minus new turnover cost ($341,250 – $136,500)
2e. What is the program’s ROI?
3. What cost factors were not addressed?
• facilities and overhead
• evaluation costs
• needs assessment
4. What other things could be affecting turnover?
• other companies stealing their employees
• no career path
• differing values of individuals versus corporate culture
• relocation of a spouse
• retirement
• lack of job challenge
• internal turnover or promotion
• job burnout
• feelings of exclusion
5. How could you factor that impact in?
The impact could be factored in using estimates with reliability factor as discussed in chapter 7.