7

Level 4: Impact and ROI

What’s Inside This Chapter

This chapter links back to the initial design phase when the business metric was established for the evaluation. It ends with three case studies to help you practice determining return on investment. You’ll also learn:

• a working definition of Level 4 evaluation

• the challenges and benefits of conducting Level 4 evaluations

• questions clients ask about Level 4 evaluation

• organizational drivers for ROI

• tips to measure training ROI

• guidelines for Level 4 evaluation

• the steps for conducting an impact and ROI analysis

• alternative measures of impact.

7

Level 4: Impact and ROI

A Working Definition of Level 4 Evaluation

Level 4 evaluation is the process of determining the final business results that occurred as a result of the training. Level 4 evaluation shows you how much the training contributed to the shift in the business metric identified in the initial business analysis and recorded on the evaluation plan. The impact is the actual shift in the business metric. For example, if the training is designed to increase unit sales, the impact is the change in unit sales attributable to the training program. It is important to remember that it’s unlikely that all sales increases are due to the training program. Other factors within and outside the organization may have also played a role. Thus, impact is calculated to show only the increase in sales due to the participants’ involvement in the training program and the transfer of their knowledge, skills, and abilities to the job.

ROI is the return on the investment in the training. In other words, it is a gauge of how much additional net dollar benefit accrued to the organization for each dollar spent on the training. ROI is usually calculated on an annual basis. If the training program’s ROI is 100 percent, it means that for every dollar invested in the training program, one dollar is returned to the organization in net benefits. In this case, the program paid for itself but did not add any additional benefits to the organization. If the ROI is above 100 percent, the training made a profit; an ROI less than 100 percent means a loss. So, an ROI of 150 percent means there is a return of $1.50 for every dollar the program costs, or the organization receives a net benefit of 50 cents for each dollar spent.

ROI calculations are based on two assumptions:

• The business metric can be converted to monetary value: For example, you need to know the dollar value to the organization for each additional unit sold in the example at the beginning of this chapter.

• You can determine the total cost of designing, developing, delivering, and evaluating the entire training program: This includes any pre-course work or planning sessions, the actual training sessions, and any follow-up sessions or training for reinforcement.

In other words, the ROI is the comparison of the net impact (in dollars) with the total program costs, and expressing the ratio as a percentage:

The ROI calculation is usually done for a single delivery of a single course, but that calculation can be used to study multiple deliveries.

The Challenges of Level 4

Conducting a Level 4 evaluation is not an easy undertaking for several reasons. There is a considerable time lag between the training and the evaluation because you must wait a suitable time before measuring transfer and tracking the data. Sustaining this can be difficult. Then, the business metric must be measured. Many times that metric is measured by the organization (not you) and it is only taken periodically; in many instances, it is six to nine months before a Level 4 evaluation is conducted. As a result, it takes quite a bit of expertise to plan and implement a Level 4 evaluation. Level 4 evaluations also require more resources (time and money) than Levels 1-3.

It is also difficult to separate the many variables that could contribute to the shift in the business metric; that is, to isolate the effects of a single variable such as training. In most cases, training is only one factor among several that may have affected any given business metric.

Placing a dollar value on the benefit derived from the training is not easy. Likewise, it can be difficult to align specific returns with specific costs. A detailed cost worksheet aids in this process.

Benefits of Conducting Level 4 Evaluations

An impact and ROI analysis provides several benefits to the training organization and to the client. Level 4 evaluation identifies the contribution that training makes to the client and the organization. This provides evidence to the client, management, and other stakeholders of the value of the training. By consistently implementing talent development solutions that positively affect client and organizational needs while providing a positive ROI, the talent development function is able to make a bottom-line contribution. Level 4 evaluations also demonstrate some business savvy on the part of the training function, which adds credibility with your client and line management.

The ROI analysis also allows you to determine whether the cost of the training program is justified by the derived benefits. A positive ROI is a good measure for this. Similarly, an ROI analysis can help determine whether the training initiative was the most cost-effective solution. Here, the ROI figure is important for comparisons, but your total cost figure is also important. Can you get the same or comparative results in a more cost-effective manner? This links to another benefit: the ability to know the total program costs. By knowing these costs, you can manage the project and find ways to be more cost-effective.

The ROI can help prioritize the development (by forecasting ROI) and continuation of training programs. Generally, programs with the highest forecasted ROI will receive continued funding and support.

Questions Clients Ask

You’ll likely recall from an earlier chapter that one of the excuses for not doing extensive evaluations was that nobody was asking for them. The assumption is that if nobody verbalizes the request or asks questions, then no questions exist. This is probably a false assumption. Your clients spend considerable resources supporting the training initiatives of their company. Therefore, your clients are probably asking some of the following questions (although they may not verbalize them to the training organization):

• What did I get for my money (investment)?

• Did my people learn anything?

• What difference is the training making to the achievement of my (the client’s) objectives?

• Are my people using their new knowledge and skills?

• What is the training costing the organization?

• Did we use the most cost-effective delivery system? Would online or virtual classroom be more cost-effective?

• What is the ROI?

• Did the training meet its stated objectives?

• How much of the training is actually being used on the job?

• Does the job environment support the use of the training on the job?

• What are any barriers to applying the training to the job?

• What tangible benefits did the organization derive from the training program?

• Was the training delivered in the most cost-effective way?

• How long is the training program? Why does it need to be that long?

• Is the training customized to our organization or is it off-the-shelf?

• Who owns the training material?

The responses to these questions can come from the evaluation of your courses. Realize that most of the these questions are directed at Level 4 evaluation in that they have impact, ROI, and cost implications.

Organizational Drivers for ROI

Why is it important that you be able to conduct a Level 4 evaluation? What are the organizational drivers that make this important to the training organization?

As budgets shrink, it becomes more important for the training function to demonstrate its value to clients and the organization. Helping your clients achieve their objectives is a good way to demonstrate value, as is providing a realistic ROI. By demonstrating value, you reduce the chance of losing precious resources, including organizational budget dollars. Realize, there is internal competition for those dollars. So, as budgets shrink, demonstrating value is critical.

Whether voiced or not, management needs the results of your training programs. You must respond to these requests. Also, as the training organization becomes more linked to the business strategies of its clients, it comes under increasing pressure for accountability for results. The continued trend toward outsourcing also increases pressure for accountability. As companies focus on their core competencies, their support organizations must also be accountable for cost-effective delivery of services that support those core competencies.

There is now a greater recognition that the only sustainable competitive advantage is an organization’s human capital. The development and management of human capital is critical for long-term success. This is squarely in the arena of talent development. The training function must not only help the organization build its bench strength, but also must help build and manage corporate human competence.

Tips to Measure Training ROI

Garvey (2012) provides some pointers for measuring ROI:

• Don’t go overboard in calculating ROI. Prove beyond a reasonable doubt that your program is cost-justified, but only evaluate to the level the client wants.

• Shift from a quality mindset to an impact and results mindset. ROI is a way of thinking. Don’t assume that simply focusing on the quality of training will ultimately result in a positive impact. Quality doesn’t go far enough toward proving that your training program positively affects the business.

• Calculate ROI continuously so you always know how much benefit your program is generating. There are two ways to waste training dollars: train people who don’t need it or train people who don’t use it. When you continuously calculate ROI, you’ll always have an understanding of what is working and what is not.

• Build your case for ROI step by step. Getting to ROI is like building a court case. You make arguments and then present facts to support them.

• The more data points you have, the better. You have to be able to explain how you reached your conclusion. Validate your findings with as much data, from as many different perspectives, as possible. That means conducting evaluations at all four levels.

• ROI isn’t just about money. Analyze results by executing the evaluation plan across all four levels of evaluation.

• Be conservative. Self-reported scores should be factored down to compensate for bias. Use the job impact number reported in the follow-up survey, rather than what was predicted immediately following the class.

• Know the investment outlay. Calculate the program’s investment or costs. This can be done by using the worksheet provided in this chapter. Then, calculate the return.

• Communicate the story behind the numbers. This is part of the communication plan. Restate the goals of your course as agreed upon with your client, the challenges faced in conducting the evaluations and how you addressed them, and the difference the program has made for the business.

• Don’t be discouraged by a low ROI. You may be able to be reduce costs or you could encourage more use in the field.

Guidelines for Level 4 Evaluation

There are a few guidelines to follow in conducting a Level 4 evaluation. Some of these are:

• Conduct all previous levels of evaluation (Levels 1-3).

• Allow enough time for the environment to take effect and for the achievement of business results.

• Demonstrate the causal link between the business metric from the business analysis or needs assessment and the training.

• If you can, measure both before and after the training. The before measurement gives you the baseline performance data, and the after measurement provides the change in the business metric.

• Continue to track the movement of the business metric.

• Conduct no more analysis than is needed to make your decisions.

• Isolate the variables; that is, determine the effects of training vis-à-vis other factors on the change in the business metric.

• If possible, use control groups to help isolate the variables.

• Determine the monetary value of the change in the business metric.

• Determine total program costs.

• Put the business metric (the benefit) in monetary terms.

• Use appropriate statistical analysis.

• Select appropriate programs for Level 4 evaluation (Table 7-1).

Table 7-1. Training Program Criteria for Level 4 Evaluation

Criterion	What You Need to Consider
Long Life	How long will this training course be offered? The longer the time, the stronger the case for Level 4 evaluation.
Very Important	How important is this program in meeting the organization’s goals? Is the course part of a strategic initiative? If so, you may want to consider Level 4 evaluation.
Link to Program Objectives	Do the program’s learning objectives state what is to be implemented and the change in the business metric? You want alignment among business metrics, statement of objectives, and the measurement.
High Cost	The higher the cost of program design, development, and implementation, the greater the need for Level 4 evaluation.
Highly Visible	How visible is the program to senior management? The greater the visibility, the greater the likelihood that it is a good candidate for Level 4 evaluation.
Large Target Audience	The larger the size of the target audience, the stronger the case for Level 4 evaluation.
Data Readily Available	Is the business metric currently being tracked? Would it be a simple matter to collect the data for Level 4 evaluation? If the data are not readily available or very difficult or expensive to obtain, you may not want to conduct a Level 4 evaluation.
Mandatory Participation	Everyone in the organization being required to attend supports the need for Level 4 evaluation.
Executive Request	If an executive is requesting the information, you will need to respond.
Levels 1-3 Evaluation Already Conducted	Have previous Levels (1-3) of evaluation been carried out? If not, you will need to conduct a comprehensive evaluation of the training course.
Direct Cause and Effect	How direct is the connection or link between the training program and the business metric? The more the direct linkage, the stronger the case for Level 4 evaluation.
Easy Conversion to Monetary Value	If it would be straightforward to convert the business metric to a monetary value, this fact builds a case for Level 4 evaluation.

Case Study

A large Fortune 100 company used its inside sales professionals to sell advertisements for placement in the company’s various media. The unit made cold calls as well as taking inbound sales calls. The organization’s customer service department dealt with any customer complaints around ad placement, correct verbiage and images, and timing. It was also responsible for making the financial adjustments.

Company management noticed a continual rise in adjustments being made and the financial impact they were having. Realizing that a recent program based on the idea that “the customer is always right” may have gone too far, it developed a negotiation skills program for the agents working the adjustments. However, the situation did not change.

The organization decided to analyze the negotiation skills program to Level 4. However, as the analysis began, the analyst realized that there were no Level 2 or 3 evaluations for the program. And while the Level 1 evaluation indicated that it was a “good” course and the agents enjoyed it, it was difficult to align some of the examples and exercises to the situation.

The company was able to provide data on the agents’ past and current financial adjustments, which ranged from before the negotiation skills course to the present (almost a year later). After analyzing the data, adjusting for other influences, and determining the costs, the analyst concluded that the program had a negative ROI.

The organization was not happy. How could this be?

After further investigation, the organization discovered that the problem was not a training issue at all. Remember the sales department? It had been experiencing very high turnover and the new sales professionals were not being adequately trained. As a consequence, the input side was making significantly more mistakes, which resulted in more adjustments for the customer service agents.

A couple lessons:

• Don’t assume training is the answer; conduct a needs assessment.

• Look at the entire system to identify possible performance issues.

• Be sure that the training program is of high quality, with instructional strategies that align to the jobs of the participants.

• Ensure that the training program’s design and development has the capability for evaluation at the levels of interest to the organization.

Steps in Conducting an Impact and ROI Analysis

There are eight steps to completing an impact and ROI analysis (Figure 7-1). Some of these steps were mentioned previously, but they will be discussed here in somewhat greater detail and in the overarching context of Level 4 evaluation.

Figure 7-1. Steps in Conducting an Impact and ROI Analysis

Reprinted with permission from Performance Advantage Group, 2016.

Step 1: Conduct Business or Needs Analysis

The actual impact and ROI analysis process begins with the business or needs analysis, the identification of the business metric, determining the value of that metric, and setting the learning objectives. Barksdale and Lund (2006) provide several examples of business metrics, some of which are listed in Table 7-2.

Step 2: Develop an Evaluation Plan

For Level 4 evaluation, you need to complete the evaluation plan for all four levels. Be as complete as possible. Refer to the section in chapter 3 on “Developing the Evaluation Plan” for details.

Table 7-2. Common Measures to Determine Effectiveness

Step 3: Design and Develop an Instruments and Data Collection Plan

In this step, you actually design and develop the instruments and methods for data collection. In some cases, the instruments—action plans, performance contracts, and so forth—are already part of the program design.

In other cases, you must develop the instruments (surveys, questionnaires, interview and focus group protocols, for example). There may be some nuances related to Level 4 evaluation to facilitate the collection of information about costs, perceived value of the training program, and so forth. For example, what if you elect to develop a survey or a questionnaire? At Level 4, you may want to include a reliability or confidence level, which represents the degree of confidence that the responders’ estimates provided for the organizational impact are accurate. (This is discussed in more detail in step 5.) The respondents indicate their degree of confidence by assigning a percentage to the confidence they have in their estimate. A 100 percent confidence level indicates that they are completely confident their estimate is accurate.

Another option is collecting data through interviews and focus groups. Again, you may want to include a discussion of other factors that might affect the results and use the estimate and confidence methods previously mentioned. Other methods include the follow-up on action plans, performance contracts, observation, and performance tracking.

Performance tracking allows you to determine a baseline of performance prior to training. The performance is continually tracked until you conduct your Level 4 evaluation. At that time, the difference between the pre-training baseline and the current performance is the effect of training. This is simply a pre- and post-measurement. Again, you must take other factors into consideration that could have influenced the performance (isolation of variables).

A variation of performance tracking is the trend-line approach. Using previous information (collected at least six months prior to the training session if possible), draw (plot) a baseline of the data and extend it into the future (beyond the point of the training session). After the training session, plot the actual performance data and compare it with the original extended line. The difference between the two lines is the change in performance, some or all of which is likely attributable to training. Again, you need to isolate the variables to determine exactly how much of the performance change is due to training.

You may elect to use a control group when you conduct a Level 4 evaluation. This is another decision that is part of the evaluation plan. In a control group design, the experimental group receives the training, while the control group does not. You then compare the groups’ performance following training. Ideally, the groups need to:

• Be demographically similar.

• Have similar environmental influences or working conditions.

• Be isolated from each other (no communication).

• Have participants of the groups randomly selected.

Step 4: Collect Data

You’ve made your data collection decisions regarding instruments and methods, identified and developed the instruments, and identified the sources of information, the timing, and the people who are responsible. Now it’s time to implement your data collection plan.

Step 5: Analyze and Interpret Data

Step 5 has two parts. The first part is to determine the effects of training. To do this, you must isolate the variables; that is, identify other nontraining factors influencing the change in the business metric. The second part involves converting the training benefits to monetary value; that is, determining the dollar value of the benefit.

Isolation of variables is one of the most difficult tasks of evaluation. Some methods include the use of control groups, estimation, and trend-line analysis. Phillips (1997) goes into some detail about how to determine confidence levels and isolate variables.

Control Groups

Using the control group design, the experimental group receives the training and the control group does not. When you compare the groups’ performance following the training, the benefit is the difference in performance between the two:

Net Benefit = Average Trained Performance – Average Untrained [Control] Performance

The result is then standardized in terms of percentage of job performance improvement. Calculate this by dividing the net benefit by the average untrained group’s performance:

Standard Training Impact = Net Benefit ÷ Average Untrained [Control] Performance × 100

An example may help clarify the use of control groups. Suppose there are two groups, Group A and Group B. Group A is the control, and Group B received the training. The comparison of the two groups’ performance before and after the training program is shown in Table 7-3.

Table 7-3. Using a Control Group to Measure Impact

With these data, you can now calculate the net benefit of training and the standard training impact:

Because the production of Group A remained the same, you can assume that no factors other than training influenced the results.

In some instances, production may go up in both groups during the training. This is because something influenced the groups other than training. Table 7-4 provides an example.

Table 7-4. Using a Control Group: Isolation of Variables

In this case, something influenced the control group (Group A) and caused the production to increase by 200 units. That “something” cannot be the training solution because Group A did not receive it. Because of the nature of control groups, you must assume that whatever affected Group A also affected Group B. This then reduces the impact of the training program because something other than training clearly influenced production. You can identify nontraining influences on production by discovering the other causal factors(s). This then allows you to reinforce these other factors, providing even greater increases in production. It is also possible that these other influences may be more cost-effective than training.

Use of Estimates

For some reason, many talent development professionals don’t put much faith in people’s estimates. Yet, it is a fairly common practice. When estimates are based on individuals’ experience and expertise, they are informed estimates. For example, organizations make sales estimates (forecasts), estimates of inventory, even retirement planning estimates. The key is to have the people closest to the data with the greatest relevant perspectives make the estimates. These individuals are usually the participants, their managers, internal experts, and others.

Estimates can be used in conjunction with surveys, interviews, or focus groups. When having people make them, you should ask the following questions:

• What percentage of the improvement is due to the training?

• How reliable is this figure? What degree of confidence do you have in your estimate?

• What is the basis for your estimate?

• What other factors might have contributed to the improvement?

Asking the respondents to provide a percentage figure representing their degree of confidence in their estimate improves the reliability of your calculations. Keep in mind that 100 percent means being fully confident. Asking them what other factors could have contributed to their improvement can help you isolate the variables.

For example, in a focus group you may state that based on the data, turnover in your call center dropped from nine people per quarter to five people per quarter—a decrease of four people per quarter, or 16 people per year. Be sure to annualize the data because ROI is calculated on an annual basis. You would then:

1. Ask the respondents about any major factors other than training that might have influenced the decrease in turnover. The process is isolation of variables. Include an “other” category for minor factors affecting the results.

2. Ask them to indicate what percentage of the increase was related to each major factor. These percentages should add up to 100 percent.

3. Have them indicate how sure (confident) they are in their estimates. Again, these figures are reported as percentages. Because each factor is rated independently, the confidence levels do not need to add up to 100 percent.

You could provide a table for them to complete (Table 7-5).

Table 7-5. Confidence Levels for Decrease in Turnover

Factor	Estimated Contribution of Factor to the Decrease in Turnover	Confidence in Estimate
Training	40%	85%
Career Path and Coaching	58%	90%
Other	2%	98%
Total	100%	---

In this case, the respondent thinks that training was 40 percent responsible for the decrease in turnover, and is quite confident about that estimate (85 percent). The new benefit is then:

or, in this case:

This table also tells you that another factor had a significant impact on the decrease in turnover. In this case, there was an initiative to help representatives work with coaches to develop their career paths, identify future positions, and prepare for a promotion. According to the respondent, the career path and coaching had a greater effect on the decrease in turnover than the training initiative.

Trend-Line Analysis

To track performance, you may use a trend-line approach by which you plot the data over time; plot the performance data on the x-axis and the timeline on the y-axis (Figure 7-2).

The first step is plotting the business metric using extant data, which reflects the movement of the business metric up to the point of the training (line A to A1). It is good to have at least six months of prior data on which to base your trend line, so you may need to use historical data for this step. Then, statistically determine the line of best fit and extend that line into the future, up to the point when you will conduct your Level 4 evaluation (line A1 to A2). This line represents the continuation of the business metric if no training initiative had occurred. Line A1 to A2 is simply the extension of line A to A1.

Figure 7-2. Example of Trend-Line Analysis

Following the training initiative, continue to track and plot the business metric. At the time of the Level 4 evaluation, statistically determine the line of best fit for the data points from the time of the training initiative to the time of the evaluation (line A1 to B). The vertical difference between the points at A2 and B represents the difference due to training—the training impact. Again, you must isolate training’s effect from the effect of other variables that could have influenced the business metric. A review of statistical tools and methods is beyond the scope of this book. You can get SPSS software packages that will help with your analysis. Also see Montgomery, Jennings, and Kulahci’s 2016 book, Introduction to Time Series Analysis and Forecasting.

The second part of step 5 is converting the training benefits to monetary value by determining the dollar value of the benefit. This should be straightforward, based on the business analysis, in which you and the client determined the business metric and the dollar value of that metric. Let’s go back to the call center scenario in the previous section, if you’ll remember:

• The call center is experiencing turnover of nine people per quarter.

• The talent development department conducted a training program to reduce turnover (giving feedback, coaching, setting objectives, resolving complaints and grievances, and evaluating performance).

• After the program, turnover dropped to five people per quarter, representing a savings of four turnovers per quarter (benefit).

• The other factor affecting turnover was the implementation of a career path supported by coaching in the call center.

In the business analysis, the talent development department determined that each representative leaving the call center represents a total cost of $22,500. Using this information, you can now establish the annual savings accruing to the call center by reducing the number of annual turnovers by four. As you can see from Table 7-6, the career path and coaching contributed more toward the reduction in turnovers than did the training. But does this mean that the training didn’t achieve a positive ROI? Should the call center put more resources into career path development and coaching? At this point, you don’t know the answer because you don’t have the cost information to determine ROI.

Table 7-6. Estimated Impact of Different Variables on Turnover Costs for the Call Center

Step 6: Tabulate Program Costs

Tabulating program costs involves determining all the costs of the training program and totaling them to get a total cost. But, what constitutes the training program? Does it include pre-training materials? Does it include post-training follow-up for reinforcement? Does it include job aids? Does it include overhead? Does it include the salaries and benefits of managers for the time spent in the program and any follow-up? For online, blended, or virtual classroom training, does it include time spent in peer threaded discussions or team work? The answer to all these questions is yes. Table 7-7 provides a cost worksheet that provides guidance in determining the costs. Be sure to allocate all costs related to the design and development, delivery, and evaluation of the training program. In some cases, your figures may include hourly calculations and you may notice some duplication. This allows you to better manage your cost side of training.

Table 7-7. Program Cost Worksheet

Needs Assessment Costs*	Cost
Salaries plus benefits of talent development staff	$
Salaries plus benefits of participants involved in needs assessment
Contractor’s fees plus expenses
Travel and living
Office supplies
Printing expenses
Shipping
Equipment expense
Overhead allocation
Miscellaneous expenses
Subtotal	$
Design and Development Costs
Salaries plus benefits of talent development staff	$
Salaries plus benefits of participants, peers, and managers involved in design and development
Travel and living of participants, peers, and managers involved
Software packages for programming for online programming, conferencing tools, assessments, and so on.*
Contractor’s fees plus expenses
Office supplies
Participants’ materials
Manual
Tests and instruments
Pre-read materials
Job aids
Post-program materials and readings
Facilitator guide
Media
Videos and YouTube
PowerPoint
Films
Audiotapes
CDs
Wallboards
Flipcharts
Artwork
Copyrights
Royalties
Shipping
Equipment expenses
Other services
Overhead allocation
Miscellaneous expenses
Subtotal	$
Delivery Costs
Facilitators’ salaries, benefits, plus expenses	$
Facilitators’ travel and living
Other presenters’ salaries, benefits, plus expenses
Other presenters’ travel and living
External and vendors’ facilitator fees
External and vendors’ travel and living
Training facilitators and presenters on technology and facilitation skills for online and virtual classroom delivery*
On-the-job coaches’ salary and benefits
Talent development support staff salary and benefits
Technical support staff salaries and benefits
Participants’ travel and living
Costs for others taking over participants’ work
Registration expenses
Opportunity cost
Meals and breaks
Facility expenses
Equipment expenses (including software)
Licensing of software or groupware*
Additional bandwidth and server costs*
Shipping
Materials and supplies
Overhead allocation
Miscellaneous expenses
Subtotal	$
Evaluation Costs
Salary and benefits of talent development staff	$
Travel and living
Salary and benefits of participants
Salary and benefits of managers and peers
Contractor’s fees plus expenses
Licensing and groupware costs for online and virtual data collection*
Printing
Other services
Office supplies and expenses
Equipment expenses
Overhead allocation
Miscellaneous expenses
Subtotal	$
Grand Total	$

*Indicates costs associated with online or virtual classroom courses

Reprinted with permission from Performance Advantage Group, 2016.

Step 7: Determine the Impact and Calculate the ROI

The impact is the shift in the business metric. In the call center example, the impact is the reduction in turnover. ROI goes a step further in that it:

• determines the associated costs for the training program

• isolates the variables to determine the impact training has on the shift in the business metric

• determines the dollar value of the benefit

• completes the calculation:

ROI is the return on the investment in training. It represents how much additional net dollar benefits are accrued to the organization for each dollar spent on the training. Remember that an ROI above 100 percent adds a positive dollar value to the organization—an ROI of 112 percent means the organization received $12 for each dollar spent on training.

You already know that reducing turnover per quarter by four at the call center has an annual dollar value of $122,400. If we assume that the course was a two-day initiative and the total cost was $45,250, we can calculate the ROI:

That is, for each dollar spent on this training program, the organization gets a return of $70.50 above the total cost.

Step 8: Communicate the Results

Who is interested in what information? Not all information is of equal value to all audiences. Generally, the talent development staff is most interested in Level 1 data, which are available immediately following the training initiative. At this time, the talent development staff is getting ready for the next delivery, so this information allows them to look at the information and make decisions (and if necessary, changes) regarding delivery strategies, learning activities, or content. In other cases, the information focuses on where the talent development staff may need to gather additional information.

Level 2 information is primarily for the talent development staff and managers of the participants. The immediate results of tests and assessments are provided to participants as feedback on their learning and practice. The talent development staff uses this information for program revision (content and instructional strategies) and decisions around the quality of facilitation. The managers of the participants use this information to see if their employees learned anything as a result of their investment: Did learning take place? Were the participants able to demonstrate the skills? Have participants’ abilities improved?

The primary audience for Level 3 is the participants’ managers. Because they are sending their people to the training, they want to know if it transfers to the job. What is the utility of the training? Is the training being used on the job? If so, how much of the training is being transferred? They also want to know of any barriers to transfer so they can take action to minimize or remove these barriers. Likewise, they want to know what enablers to reinforce to enhance transfer.

This does not mean that the talent development staff isn’t interested in Level 3 evaluation. There is rich information regarding how to improve transfer through better design, involvement of the field in the design and development process, and facilitation of the training. Although environmental issues have some talent development implications, there are also nontraining issues that affect transfer. Identifying these issues and sharing that information with the appropriate people (managers and others in talent development) can help build a more supportive environment.

Client would be interested in Level 3 to get an insight into the potential change that could support their objectives. If transfer takes place and the business metric moves in a positive direction, your clients receive help and benefits. The workforce is improving in those areas—that’s good news! Alternatively, if transfer does not take place, the clients can help identify and implement training and nontraining strategies to make a positive change in the environment.

Level 4 evaluation results are of great interest to your clients, because they invested heavily in the training initiative. Even if the training is a corporate-sponsored program, managers still send their employees and help fund it. So, clients want to know what they are getting for their investment (ROI), the extent to which the business metric moved (impact), and how the training can be provided more cost-effectively. Level 4 evaluation provides this information.

Your talent development line managers are also interested in Level 4 results. They build and sustain relationships with their clients and must seek continued funding for talent development initiatives. Positive and significant impact and ROI results help position them to do this. It’s clear that different audiences are interested in different information. The following guidelines provide assistance:

• Align the information with the interests and needs of the target audience.

• Provide the complete information in a concise format.

• Provide the information in a timely manner so that your client or talent development management can use it to make relevant decisions.

• Make recommendations based on the research.

• Communicate the information in an objective fashion—do not offer biased opinions or exaggerations.

• Provide an executive summary containing the most important information and recommendations.

• Present the information in an appropriate format for the audience. This could be a written report, verbal discussion, presentation supported by media, executive format, and so forth.

Other Measures of Impact

ROI is the most popular way to measure impact. A couple other ways are discussed below.

Cost-Benefit Ratio

Sometimes referred to as the benefit-to-cost ratio or BCR, this approach to evaluating the return for the investment in training is a relatively easy method. It compares program benefits (not net benefits as in the ROI formula) with program costs. Assuming you have collected the data, determined the dollar value of the benefit, and calculated the total program costs, the formula is:

A CBR of 1 means that one dollar is returned for each dollar invested. A CBR greater than 1 means the program gained money. For example, if the total benefits of a program are $2,654,987 and the total costs are $377,486, then the CBR is:

In other words, for every dollar invested, $7.03 in benefits were returned.

Payback

Payback refers to the length of time to recover the initial investment. According to Schmidt (2015), the payback period is the point in time when the cumulative cash inflows (for training this would be the dollar value of the benefit or savings) exactly balance cumulative outflows or outlay. It shows how long the program must be offered before it pays for itself. As a general rule, if the payback period is an acceptable length of time, the organization will be more inclined to accept the project. A shorter payback period is preferred; it is viewed as a less risky investment because the invested monies are recovered and available quicker for other investments. The figure is usually reported in years and months: payback period = 3.7 years.

To calculate the payback, the cumulative cash flow (taking into consideration the sum of cash inflows and outflows for the current and all preceding years) must be tracked and summed. The payback period is when the cumulative cash inflows and outflows are equal. Generally payback is an estimation; to be more exact, you’ll need to do further analysis or assume that the cash flows are spread evenly throughout the year.

Phillips (2012) seeks to simplify the analysis by assuming a training program has a certain life, which can be estimated during the design and development phase and is influenced by the audience size, number of participants per offering, and the number of offerings per year (throughput). The formula is:

The 0.55 years is about 28.6 weeks (0.55 x 52 weeks = 28.6).

Where the total original investment is the sum of all original costs (from the cost worksheet Table 7-7) and the net annual savings is the expected or actual annual savings. If the annual net savings is not constant, the average over the expected life of the program could be used.

Getting It Done

Chapter 7 introduced you to the world of Level 4 evaluation—impact and ROI. Level 4 data can show how much the training contributed to the shift in the business metric identified in the initial business analysis and recorded on the evaluation plan. Certainly, carrying out high-level evaluation involves challenges for the talent development department, but the process positions the department as a strategic partner of the client.

Exercise 7-1 can help you think about building upon the evaluation plan for one of your courses to include Level 4 evaluation.

Exercises 7-2, 7-3, and 7-4 present case studies. See if you can determine the ROI for each one. The solutions are in Appendix B.

Exercise 7-1. Level 1, 2, 3, and 4 Evaluation Plan

Evaluation Plan: Level 1 (Reaction), Level 2 (Learning and Application), Level 3 (Transfer and Environment), and Level 4 (Impact and ROI)

Business Metric(s):

Reprinted with permission from Performance Advantage Group, 2016.

Exercise 7-2. Case Study: Use of Estimates

TCR is a Fortune 500 company with 268 retail outlets in 35 states that sells tires to the consumer market. It has been implementing a strategy of planned growth through acquisitions and mergers to extend its reach and product portfolio. While sales are generally good throughout the year, sales are higher during the spring and winter months due to weather conditions. (This is more true in some parts of the country than others.) Having completed their growth strategy, TCR now faces the challenge of rebuilding its infrastructure, including talent development.

To ease the management burden and drive consistent systems and processes, TCR instituted a single performance tracking system for all retail outlets. The system makes it possible to track sales by individual and report out by store and region. The company also developed a profiling system that allows it to study a store’s traffic pattern and other critical factors. This system will better allow it to manage each store and geography.

TCR wants to institute a single sales process throughout all stores in the hopes that this strategy, coupled with incentives and coaching, will increase store sales. The curriculum includes basic selling skills, product knowledge, up-selling and cross-selling across the portfolio, and addedvalue selling to increase margins. The company embarked on an aggressive program rollout, hitting critical mass of sales associates in 207 stores within six months.

Four months later, average sales for those who received training increased by $27,482 per month. The sales of the stores that have not yet received training have also increased. There is a 12 percent margin on sales, but further analysis shows that some stores significantly outperformed others in increased monthly sales. Senior management wants to know the affect the training program had on the increase in store performance.

Management elected to use a random sample of the retail outlets to study the effect of the training program. A questionnaire was sent to sales associates and store managers to determine the amount of influence four factors—sales training, incentives, management coaching, and reduced competition—had on increased sales. Those responding were asked to allocate 100 percent of influence across the four factors and to indicate their level of confidence in their estimates. Tables 1 and 2 present the average research results for the sales associates and store managers, respectively.

What is the annualized dollar impact of sales training for all stores receiving the training based on the responses from the sales associates and store managers? (See Appendix B for the solution.)

Table 1: Sales Associates

Table 2: Store Managers

Exercise 7-3. Calculating ROI: A Meeting Management Case Study

Communications R Us is a manufacturing organization producing components for car stereos and CD players. The organization has 4,000 employees, including 35 supervisors. Middle management has noticed that some supervisors appear to be spending an inordinate amount of time in meetings. This has adversely affected their time to do some operational aspects of their jobs and focus on their people who need coaching. The middle managers contacted you, the training specialist, to help reduce the amount of time supervisors are spending in meetings.

In response to their request, you developed a time sheet for the supervisors to use to log their time in meetings for a month. This exercise showed two distinct groupings: 15 managers spent between six and nine hours in formal meetings, while the remaining 20 managers averaged 20 hours. Correlating performance against the managers showed that those spending less time in meetings were also performing better overall.

Based on these data, and with the support of the middle managers, you designed and delivered a one-day meeting management course for the 35 managers in two identical sessions. The 20 managers with excessive time spent in meetings were in one of the two sessions, which were completed four months ago. You have been asked to calculate the ROI for the session involving the 20 managers.

Through data collection you determine:

• Prior to the training, the 20 managers spent an average of 20 hours per month in meetings.

• Their average annual salary is $74,074.

• The company benefit package is valued at 35 percent of salary.

• Following the training, the 20 managers now average eight hours per month in meetings.

To calculate ROI, you must first determine the program costs. Use these assumptions to complete the table that follows:

• There are 240 workdays in a year.

• There are 1,920 hours in a work year.

• A workday is eight hours.

• Two days of administrative support is required for each offering.

Meeting Management Training Program Costs	Cost per Session
Program design and development was $17,000
Facilitation (assume no preparation time) (annual fully loaded salary of $65,000)
Participant time
Materials ($35/participant)
Administrative support (annual fully loaded salary of $45,000)
Total Cost

Based on the information provided and the following equation, what is the session’s ROI?

What is the cost-benefit ratio?

Exercise 7-4. Calculating ROI: Weston Marketing

Weston Marketing provides telemarketing services for a large catalog sales company. To support the effort, Weston has a call center staffed with 50 agents to provide telemarketing services for their coverage area. The call center’s responsibilities include outbound sales, customer service, billing, and negotiations. There are a total of five supervisors for the call center operations. Unlike many call centers of this nature, Weston is experiencing relatively high turnover of 20 percent.

Weston’s management is concerned about the turnover rates. It views the call center as a primary interface with the customer base that is critical for good customer service and customer retention. Management knows that it provides good initial training, excellent benefits for the size of the company, good working conditions, and some opportunity for advancement. However, exit interviews allude to management practices as a reason for the turnover.

The talent development department, with management’s support, decided to conduct a brief employee satisfaction survey to try to determine the causes of turnover in the call center. The survey went to all call center agents and returned an average score of 76 out of 100, which is not too bad. However, an item analysis indicated some problems around management practices: The average on these questions was 53. The findings were provided to all employees with management’s promise to address critical issues.

To address the issue, the talent development department, which is led by Jim Willis (who has an annual fully loaded salary of $92,000), elected to provide a two-day training course on management practices aimed at reducing turnover. This training was followed by one day of personal coaching for each supervisor by the course facilitator.

Turnover is tracked quarterly at Weston, and three months following the program the turnover rate was showing a slight decline. After a year, the turnover rate had settled at 8 percent. The talent development department wanted to take some credit for helping to solve this key problem. However, to do that Jim had to demonstrate the contribution that the training program had on the change in turnover. Because the company bases many decisions on ROI, Jim was determined to provide his contribution in those terms.

To determine the program’s ROI, Jim had to find out the program costs and benefits. First he determined the costs: The vendor program cost was $800 per participant, with some extra customization for $3,500. The facilitator was local and charged $1,100 per day. To help manage the program’s flow, lunch was provided for the participants, the facilitator, and Jim for $8.50 per person. The supervisors’ average annual salary is $57,000, with a benefit package valued at 30 percent of salary. With the exception of senior management, all employees receive the same benefit package. The administrative support for the program was only one day. The training administrator has a base salary of $26,500 per year.

After several days of talking to internal personnel and reviewing external studies, Jim determined that the cost of turnover for an agent was about equivalent to a fully loaded annual salary. Agents average an annual salary of $26,250, and HR records indicate that Weston employees average 220 work days and 1,940 hours per year.

Jim knew that management would be skeptical if his department took credit for the reduction in turnover. He was sure that someone would ask what else could influence turnover in addition to the training program. So, to address this concern he conducted some internal research and determined no other major changes or initiatives had influenced the call center during the period he was analyzing.

Questions

1. What factors go into determining the cost of turnover?

2. What is the program’s ROI? (Remember,

3. What cost factors were not addressed?

4. What else could be affecting turnover?

5. How could you factor that impact in?