CHAPTER 19

Implementing the Four Levels of Evaluation

Jim Kirkpatrick and Wendy Kayser Kirkpatrick

The purpose of all training is to enhance on-the-job performance through consistent application and to positively affect organizational results. The Kirkpatrick Model, often referred to as the Four Levels of Training Evaluation, is one of the primary methods used to ensure that training is being applied and contributing to results.

How do you best apply the Kirkpatrick Model? The most effective way is to consider all four levels during the training design and development process. Waiting until after the program ends to determine how learning will be applied on the job and how impact on the business will be measured generally means that these activities will not occur reliably.

In this chapter, you will learn about the four levels of evaluation and how they have been enhanced with the New World Kirkpatrick Model. Guidelines for their effective application are described using the Kirkpatrick Foundational Principles.

The Kirkpatrick Model

Donald Kirkpatrick (1924–2014) is credited with creating the Kirkpatrick Model in the mid-1950s while writing his PhD dissertation. His goal was to effectively measure the impact of the management development programs he was teaching at the University of Wisconsin Management Institute. His work became known and later published by a trade journal in the late 1950s, and worldwide use grew organically over the following six decades. Today, the Four Levels of Evaluation is the most highly recognized, used, and regarded method of evaluating the effectiveness of training programs (Figure 19-1).

In 2010, Kirkpatrick’s son, Jim, and his business partner, Wendy, enhanced the Kirkpatrick Model to accomplish the following goals:

•  Incorporate the forgotten or overlooked teachings of Don Kirkpatrick.

•  Correct common misinterpretations and misuse of the model.

•  Illustrate how the model applies to modern workplace learning and performance.

The resulting New World Kirkpatrick Model honors and maintains the time-tested four levels of evaluation and adds new elements to help people to operationalize them effectively (Figure 19-2).

The New World Kirkpatrick Model—Levels 1 and 2

The first two levels of the New World Kirkpatrick Model are also known as effective training. These levels measure the quality of the training and the degree to which it resulted in knowledge and skills that can be applied on the job. The training function uses Levels 1 and 2 to internally measure the quality of the programs they design and deliver.

Level 1: Reaction

Level 1 is the degree to which participants find the training favorable, engaging, and relevant to their jobs.

In a 2016 study conducted by ATD, 88 percent of organizations evaluated Level 1 Reaction for their training. The current investment in gathering this type of data is far greater than the importance this level dictates, and occurs at the cost of measuring Levels 3 and 4, which would yield data more meaningful to the business (those levels were measured only by 60 percent and 35 percent of organizations, respectively). Sadly, these numbers have improved only a couple of percentage points during the preceding six years.

The New World Kirkpatrick Level 1 Reaction has three dimensions: customer satisfaction, engagement, and relevance.

Customer Satisfaction

Customer satisfaction refers to the degree to which participants find the training favorable. For example, did they find it enjoyable and free of distractions from the learning itself? Was it taught in a way that is comfortable and helpful for them? This dimension is the least important and tends to be overevaluated.

Engagement

Engagement refers to the degree to which participants are actively involved in and contributing to the learning experience. Engagement levels directly relate to the level of learning that is attained.

Personal responsibility and program interest are both factors in the measurement of engagement. Personal responsibility relates to how present and attentive participants are during the training. Program interest is more commonly the focus, including how the facilitator involved and captivated the audience.

Relevance

Relevance is the degree to which training participants will have the opportunity to use or apply what they have learned on the job. This is important to the ultimate training value because even the best training is a waste of resources if the participants have no application for the content in their everyday work.

Tips for Implementing Level 1

L&D professionals should remember that:

•  Level 1 evaluation efforts should be matched to the degree of the course or program’s importance to the organization.

•  Level 1 is the least important evaluation measure to stakeholders, so keep it as brief and efficient as possible.

•  A variety of methods is useful when measuring Level 1, including formative (that is, during the class) observation. Save surveys and interview questions for the higher, more important levels.

Level 2: Learning

The New World Kirkpatrick Level 2 Learning is the degree to which participants acquire the intended knowledge, skills, attitudes, confidence, and commitment based on their participation in the training program.

Knowledge and Skill

Knowledge is the degree to which participants know certain information, as characterized by the statement, “I know it.” Skill is the degree to which they know how to do something or perform a certain task, as illustrated by the statement, “I can do it right now.”

Many organizations make the costly mistake of inaccurately diagnosing poor performance as a lack of knowledge or skill. Underachievers are often returned to training with the belief that they do not know what to do, when the more common cause of substandard performance is a lack of motivation or other environmental factors.

Attitude

Attitude is defined as the degree to which training participants believe that it will be worthwhile to implement what is learned during the training program on the job. It is characterized by the statement, “I believe this will be worthwhile to do on the job.”

Incorporating discussion about what is expected on the job and the why allows learners to develop the right beliefs about what you are asking of them.

Confidence

Confidence is defined as the degree to which training participants think they will be able to do what they learned during the training program once they’re back on the job. It is characterized by the statement, “I think I can do it on the job.”

Addressing confidence during training brings learners closer to the desired on-the-job performance. It can proactively surface potential application barriers so they can be resolved.

Commitment

Commitment is defined as the degree to which learners intend to apply the knowledge and skills learned to their jobs. It is characterized by the statement, “I intend to do it on the job.” Commitment relates to learner motivation by acknowledging that even if the knowledge and skills are mastered, participants must still put forth daily effort to use the information or perform the skills.

Tips for Implementing Level 2

L&D professionals should remember:

•  Measure Level 2 during formal training using quizzes, activities, demonstrations, and discussions.

•  Use pretests and post-tests only when the program will be modified to fit the pretest results, or when stakeholders specifically request this information.

•  Keep Level 2 measurement focused so resources stay in line with the relatively low importance of Level 2 when compared with Levels 3 and 4.

The New World Kirkpatrick Model: Levels 3 and 4

Levels 3 and 4, referred to as training effectiveness, encompass:

•  On-the-job performance and subsequent business results that occur, in part, due to training and reinforcement

•  Demonstration of the value that the training program has contributed to the organization

Level 3: Behavior

Level 3 is the degree to which participants apply what they learned during the training program when they are back on the job. The New World Level 3 Behavior consists of critical behaviors, required drivers, and on-the-job learning.

Critical Behaviors

Critical behaviors are the few, specific actions that, if performed consistently on the job, will have the biggest impact on the desired results. While there are thousands of behaviors an employee might perform on the job, critical behaviors are those that have been identified as the most important to achieving organizational success.

Required Drivers

Required drivers are processes and systems that reinforce, monitor, encourage, and reward the performance of critical behaviors on the job (Figure 19-3).

Organizations that use accountability and support systems to reinforce the knowledge and skills learned during training can expect as much as 85 percent application on the job. Conversely, companies that rely primarily on training events alone to create good job performance achieve around a 15 percent success rate (Brinkerhoff 2006).

Required drivers are the key to accomplishing the desired on-the-job application of what is learned during training. They also decrease the likelihood of people falling through the cracks or deliberately crawling through the cracks if they are not interested in performing the required behaviors.

Active execution and monitoring of required drivers is perhaps the biggest indicator of program success for any initiative.

On-the-Job Learning

On-the-job learning recognizes two facts of the modern workplace:

•  Most learning takes place on the job. A 2017 study showed that 56 percent of learning came from job-related experiences, 25 percent came from social sources such as interactions with peers and managers, and 19 percent came from formal training activities (Training Industry 2018).

•  Personal responsibility and motivation are key partners to external support and reinforcement efforts for optimal performance.

Creating a culture and expectation that individuals are responsible for maintaining the knowledge and skills to enhance their own performance will encourage individuals to be accountable and feel empowered. On-the-job learning provides an opportunity for employees and their employers to share the responsibility for good performance.

Tips for Implementing Level 3

L&D professionals should:

•  Have a pretraining conversation with the managers of the people who will attend training. Jointly determine the critical behaviors that need to occur on the job for them to view the training as time well spent.

•  Design learning objectives around critical behaviors.

•  Create job aids that help participants perform these critical behaviors on the job. Introduce them during training and use them during hands-on activities.

•  Design post-program follow-up when you create the training materials, which will ensure that this task gets accomplished. Take advantage of technology to schedule and automatically send reminders, refreshers, and encouraging messages.

•  Make a note in your calendar (or set up automatic reminders) to check in with some or all of the training participants after they have had a reasonable amount of time to try their new behaviors on the job. Ask how it is going and if they need any additional resources or support to be successful.

•  Make it part of your training design and development process to create the post-program implementation and support plan. Building this structure increases the likelihood that the resources that go into training will produce a measurable increase in performance.

Level 4: Results

Level 4 holds the distinction of being the most misunderstood of the four evaluation levels. It is the degree to which targeted outcomes occur as a result of the training, support, and accountability package.

A common misapplication occurs when professionals or functional departments define results in terms of their small, individual area of the organization instead of globally for the entire company. This creates silos and fiefdoms that are counterproductive to organizational effectiveness because the resulting misalignment causes layers of dysfunction and waste.

Clarity regarding the true Level 4 result is critical—it is some combination of the organization’s purpose or mission and the financial reality of sustained existence. In a for-profit company, this means profitably delivering the product or service to the marketplace. In a humanitarian, government, or military organization that is not primarily focused on a financial outcome, it means accomplishing the mission within the available funding, allocations, or donations.

Every organization has just one Level 4 Result. A good test of whether the correct result has been identified is a positive answer to the question, “Is this what the organization exists to do, deliver, or contribute?” While this definition is straightforward, frustration with the seeming inability to relate a single training program to a high-level organizational mission is common. Business results are broad and long term. They are created through the culmination of countless efforts of people and departments, as well as environmental factors, and can take months or years to manifest.

Leading Indicators

Leading indicators help bridge the gap between individual initiatives and efforts and organizational results. They are defined as short-term observations and measurements suggesting that critical behaviors are on track to create a positive impact on the desired results. Organizations have several leading indicators that encompass departmental and individual goals, each contributing to the accomplishment of the highest-level results.

Common leading indicators include:

•  Customer satisfaction

•  Employee engagement

•  Sales volume

•  Cost containment

•  Quality

•  Market share

While leading indicators are important measurements, they must be balanced with a focus on the highest-level result. For example, a company with excellent customer satisfaction scores could go out of business if it did not maintain profitability, comply with laws and regulations, and keep its employees reasonably happy. Note that customer satisfaction is an example of a goal that does not provide an affirmative answer to the question, “Is this what the organization exists to contribute?” No organization exists simply to deliver customer service alone.

Tips for Implementing Level 4

L&D professionals should remember:

•  At the beginning of any initiative or training program, start by considering the highest-level result your organization is charged with accomplishing. Use this as your target for each effort in the initiative. If you cannot describe how the intended training would in some way positively affect your overall result or mission, you are not on the right track.

•  Every major training initiative should be tied to the highest goals and key directives of the organization. Here are a few ways to discover them:

  Read the about us section of the organization’s website.

  Look at mission and vision statements, as well as what types of messages are posted on the walls of the office.

  Ask your boss about the highest priorities or directives for each department that quarter or year.

  If appropriate, request to attend strategy and planning meetings, even if just as an observer. Or ask for a summary of what was discussed.

•  Once you are clear on the key initiatives and goals, look at the training programs that are consuming the most time, money, and resources. Is there a direct link between them? If not, re-evaluate whether training resources are being properly allocated.

The Kirkpatrick Foundational Principles

The Kirkpatrick Foundational Principles are the key beliefs underpinning Kirkpatrick evaluation:

•  The end is the beginning.

•  Return on expectations (ROE) is the ultimate indicator of value.

•  Business partnership is necessary to bring about positive ROE.

•  Value must be created before it can be demonstrated.

•  A compelling chain of evidence demonstrates your bottom-line value.

Principle 1: The End Is the Beginning

Effective training and development begins before the program even starts:

It is important that the results are at the organizational level and defined in measurable terms so that all involved can see the initiative’s ultimate destination. Clearly defined results will increase the likelihood that resources will be used most effectively and efficiently to accomplish the mission.

Attempting to apply the four levels after an initiative has been developed and delivered makes it difficult, if not impossible, to create significant training value. All four levels must be considered at every step in the program’s design, execution, and measurement.

Principle 2: ROE Is the Ultimate Indicator of Value

Return on expectations is what a successful training initiative delivers to key business stakeholders demonstrating the degree to which their expectations have been satisfied. When executives ask for new training, many learning professionals retreat to their departments and begin designing and developing suitable programs. While a cursory needs assessment may be conducted, it is rarely taken to the extent to which expectations of the training’s contribution to Level 4 Results are completely clear.

Stakeholder expectations define the value that training professionals are responsible for delivering. Learning professionals must ask questions to clarify and refine stakeholder expectations on all four Kirkpatrick levels, starting with the organizational Level 4 Results and the leading indicators for the requested program. Determining the leading indicators upon which the success of an initiative will be measured is a negotiation process in which the training professional ensures that the expectations are satisfying to the stakeholder and realistic to achieve with the resources available.

Once stakeholder expectations are clear, learning professionals then need to convert them into observable, measurable leading indicators by asking the question, “What will success look like to you?” It may take a series of questions to arrive at the final indicators of program success.

Agreement surrounding leading indicators at the beginning of a project eliminates the need to later attempt to prove the value of the initiative. It is understood from the beginning that if the leading indicator targets are met, the initiative will be viewed as a success.

Principle 3: Business Partnership Is Necessary to Bring About Positive ROE

Research has validated that training events in and of themselves typically produce about 15 percent on-the-job application. To increase application and therefore program results, additional actions must be taken before and after formal training.

Historically, the role of learning professionals has been to accomplish Levels 1 and 2, or just to complete the training event alone. Not surprisingly, this is where they spend most of their time. Producing ROE, however, requires a strong Level 3 execution plan. Therefore, it is critical not only to call upon business partners to help identify what success will look like, but also to design a cooperative effort throughout the learning and performance processes to maximize results.

Before the training program begins, learning professionals need to partner with supervisors and managers to prepare participants for training. Even more critical is the role of the supervisor or manager after the training ends. They are the key people who reinforce newly learned knowledge and skills through support and accountability. The degree to which this reinforcement and coaching occurs directly correlates to improved performance and positive outcomes.

Principle 4: Value Must Be Created Before It Can Be Demonstrated

Up to 90 percent of training resources are spent on the design, development, and delivery of training events that yield the previously mentioned 15 percent on-the-job application. Reinforcement that occurs after the training event produces the highest level of learning effectiveness, followed by activities that occur before the learning event, yet each typically garners only 5 percent of the training time and budget.

Many training professionals put most of their resources into the part of the training process that produces the lowest level of business or organizational results. They spend relatively little time on the pretraining and follow-up activities that translate into the positive behavior change and subsequent results (Levels 3 and 4) that organizations seek.

Formal training is the foundation of performance and results. To create ultimate value and ROE, however, practitioners must focus on Level 3 activities. To create maximum value within their organizations, learning professionals must redefine their roles and extend their expertise, involvement, and influence into Levels 3 and 4.

Principle 5: A Compelling Chain of Evidence Demonstrates Your Bottom-Line Value

The training industry is on trial, accused by business leaders of consuming resources that exceed the value delivered to the organization. A chain of evidence includes data, information, and testimonies at each of the four levels that, when presented in sequence, act to demonstrate the value obtained from a business partnership initiative (Figure 19-4).

Following the Kirkpatrick Foundational Principles and using the Kirkpatrick Model will create a chain of evidence that demonstrates the organizational value of the entire business partnership effort. It consists of quantitative and qualitative data that sequentially connects the four levels and shows the ultimate contribution of learning and reinforcement to the organization.

When L&D professionals work in concert with their key business partners, this chain of evidence supports the partnership effort and demonstrates the organizational value of working as a team to accomplish the overall mission. The chain of evidence serves to unify the learning and business functions, not to isolate training or set it apart. This unity is critical for Level 3 execution, which is where organizational value is produced.

When presenting a chain of evidence, keep in mind what is most important to the stakeholder audience. Data at Levels 3 and 4 is of most interest. Data related to Levels 1 and 2 should be limited unless a detailed report is requested specifically. Gather data and information to form your chain of evidence using the Kirkpatrick Blended Evaluation Plan tool found on the handbook website at at ATDHandbook3.org.

Final Thoughts

The Kirkpatrick Model, when implemented correctly, is an effective and time-tested way to support learning transfer and evaluate training impact on the business. Consider each of the four levels throughout the training design and development process. Determine what information is required to show that the training program improved job performance and key organizational results.

For mission-critical initiatives, create a strong Level 3 implementation plan and a Level 4 tracking and measurement strategy. Use evaluation time, money, and resources sparingly on Levels 1 and 2 for all programs.

Following these guidelines will maximize your training results and create the most organizational impact with the resources invested.

About the Authors

Jim Kirkpatrick, PhD, is the senior consultant for Kirkpatrick Partners. He is a thought leader in training evaluation and the creator of the New World Kirkpatrick Model. Jim is passionate about assisting learning professionals in redefining themselves as strategic business partners to remain a viable force in the workplace. He co-authored three books with his late father, Don Kirkpatrick, and four books with Wendy Kayser Kirkpatrick, including Kirkpatrick’s Four Levels of Training Evaluation. Jim can be reached at [email protected].

Wendy Kayser Kirkpatrick is the president and founder of Kirkpatrick Partners. She draws on experience in training, retailing, and marketing to make her programs relevant and impactful with measurable results. Wendy co-authored four books with Jim Kirkpatrick, including Kirkpatrick’s Four Levels of Training Evaluation. She can be reached at [email protected].

References

ASTD (American Society for Training and Development). 2009. The Value of Evaluation: Making Training Evaluations More Effective. Alexandria, VA: ASTD Press.

ATD (Association for Talent Development). 2016. Evaluating Learning: Getting to Measurements That Matter. Alexandria, VA: ATD Press.

Brinkerhoff, R.O. 2006. Telling Training’s Story: Evaluation Made Simple, Credible, and Effective. San Francisco, CA: Berrett-Koehler Publishers.

Kirkpatrick, D.L. 2010. Evaluating Human Relations Programs for Industrial Foremen and Supervisors. St. Louis, MO: Kirkpatrick Publishing.

Kirkpatrick, D.L., and J.D. Kirkpatrick. 1993. Evaluating Training Programs: The Four Levels, 1st ed. San Francisco: Berrett-Koehler Publishers.

Kirkpatrick, D.L., and J.D. Kirkpatrick. 2005. Transferring Learning to Behavior. San Francisco: Berrett-Koehler Publishers.

Kirkpatrick, J.D., and W.K. Kirkpatrick. 2016. Kirkpatrick’s Four Levels of Training Evaluation. Alexandria, VA: ATD Press.

Training Industry. 2018. “Deconstructing 70-20-10.” Training Industry, June. trainingindustry.com/Deconstructing_70-20-10.

Recommended Resources

Kirkpatrick, J.D., and W.K. Kirkpatrick. 2009. Kirkpatrick Then and Now. St. Louis, MO: Kirkpatrick Publishing.

Kirkpatrick, J.D., and W.K. Kirkpatrick. 2010. Training on Trial. New York: AMACOM.

Kirkpatrick, J.D., and W.K. Kirkpatrick. 2013. Bringing Business Partnership to Life: The Brunei Window Washer. Newnan, GA: Kirkpatrick Publishing.

Kirkpatrick, J.D., and W.K. Kirkpatrick. 2021. “Stumped on How to Measure DEI Training?” TD, October.