Donald L. Kirkpatrick

On how it all started…

I was at the Management Institute at the University of Wisconsin in Madison, teaching a course on Human Relations for Supervisors. I had my MBA, and I decided I was going on for a PhD at the the School of Education there. I decided that as long as I am teaching in that program, I am going to evaluate it. So I was concerned about two things: the reaction of the people attending our program and learning effectiveness. To evaluate reaction, we used comment sheets and were conscious that participants better go back to their companies saying good things about the programs because they are our customers. I called that a measure of customer satisfaction. We did those every time, and we were sure that people went back and said good things about the program. Sometimes we even had an individual in higher management come and attend a seminar just to see what this was and then he would go back and make decisions about whether to send the supervisors. So it was important to measure that, and I called it reaction. We wanted to measure reaction to the program, and we did that regularly. We did not do much learning evaluation, obviously we were trying to teach knowledge and skills and change some attitudes, but we had not done anything in terms of evaluating it. As a part of my dissertation, I did some pre- and posttests to measure the learning effectiveness of the program and did quite a bit to find out that the tests were significant and valid. The dissertation was basically about those two things: reaction and learning.

After my dissertation, I decided I’d better do a little more evaluation and so I decided I’d better go back on the job and see whether the people are using what we have taught them. I would typically ask the supervisors what happened when they got back to their companies: “What is your boss going to say?” Most of them had never even given it a thought. Most of them said, “I don’t know.” One guy said, “I think I know what my boss is going to say: ‘You have been gone and now your work has piled up. I hope you had a good time, but let’s get back to work.’” I asked, “Did any of you talk to your bosses before you came?” And a few of them had, but most of them were sent there because the personnel department had signed them up to attend the program. At that time, I thought, well it is important that when they go back, they do something about the training and I was beginning to look at behavior. So I wrote a little booklet and sent it to the superintendents and department heads who were sending their supervisors. And I said, before anyone attends a program, sit down with the people who are attending and say, “Here is an opportunity to learn, I’m all for it, I’ve approved it, and have a good time in Madison. But, when you come back, I am going to ask you, ‘What did you learn and how can I help you apply it?’” I think that made all the difference in the world, in terms of supervisors’ motivation. So I started to get into behavior. Then I did some research between my dissertation and the time the articles were published in the ASTD journal five years later. I went into companies and measured behavior change, and I also got into in the area of results and I never gave it much thought. Five years later, the editor of the ASTD journal at that time, Bob, called me and said, “Don, I understand you’ve done some research on evaluation. Would you write an article for us?” I said, “Bob, I’ll tell you what I’ll do. I’ll write four articles.” So at that particular time, when I wrote those articles, I called it reaction, learning, behavior, and results. That was 50 years ago that these articles were printed. I am telling you it is unbelievable what happened after that. People started calling it the “Kirkpatrick Model.” I never called it the “Kirkpatrick Model.” They started calling it the “Four Levels.” I never called it the “Four Levels.”

I hoped that they would learn them. That was my only hope. I hoped that I had given them something that would help them evaluate their programs effectively. I had no idea it was going to be translated into foreign languages and used all over the world.

On how training evaluation has changed over the years…

At the time, almost every company was using reaction; that was about it. Some were measuring training in terms of learning. But if they got good reactions from the people, if the people went back satisfied and said good things about the training, then the trainers felt good because the participants said great things about them—that was pretty much the way it was when I was at the Management Institute at the University of Wisconsin, back when I started in 1949.

On the progress the profession has made in embracing evaluation…

On LinkedIn, I put up a discussion question a few months ago and said, “The four levels are 50 years old now. What do you think? Are they still viable? Are they still in good order or does something need to be changed about them?” Nearly everyone who replied said they are just as good now as they were then. The problem is that people are concentrating on Levels 1 and 2, and there’s not enough activity at Levels 3 and 4. I think that’s true, and it’s going to be true in many companies until training people begin to get the pressure from the jury up above that says that’s what we are going to base your budget on. I think that is why James and Wendy’s forthcoming book Training on Trial is going to have a real impact. They are going to realize that the jury up there is going to look at other things and here’s what you can do to be ready for that. How can you be sure that when the jury makes the budget decision that they are going to approve the budget? Because we have proved to them, through examples and through other kinds of things, that we have done the things that should have been done.

The whole idea of a corporate university has taken the training department out of human resources, leaving it by itself. It’s finally gotten the status that reports directly to top management. Chief learning officers are very conscious of what is going on. Training has received a lot more prestige in organizations and in the country, and I think the people up there are concerned with not only Levels 1 and 2, but also with Levels 3 and 4.

On how executives view learning and development and investment in it...

What about the executives in those years, way back in the 1950s, 60s, 70s, 80s, and 90s, what kind of evaluation was taking place? Most companies were telling their top people to justify their budget by reporting this is how many programs we ran, this is how many people attended, this is the reaction we got, and we have measured some learning. So, we have done this for the company, and the executives were pretty happy with that. They thought, “Gee, that’s great.” But then, I wrote something in my dissertation that I will never forget, “By the way, at this particular point most companies are concerned with just reaction and learning but the day of reckoning is coming.” This was in the 1959 articles; the day of reckoning is coming when top management will look for more than that. They are going to look for whether people applied training on the job and whether they got better results.

The whole idea now is that the jury up above is looking at not only Levels 1 and 2 but also at Levels 3 and 4. Before, maybe they were satisfied; they’d heard good things about the program. People said, “Yeah, I enjoyed the program; I got a lot from it and so on.” And that was good enough. But it’s not good enough any more, I can assure you that’s the case in practically every company I’ve run across.

On how evaluation makes a difference in the perception held by executives of training and development…

In this era of “return on expectations,” what we’re talking about is that we start with the end. We’re encouraging companies to go to top management ahead of time and say, “OK, you want a leadership program; now what do you expect to accomplish in a leadership program? How are you going to measure success so we know what to do, what to look for, and what to concentrate on?” What we need to do is to start with them and ask, “What do you expect? You want the leadership program; that was not our idea. Therefore, what do you want? Do you want less turnover? Do you want more productivity? Do you want higher morale? Do you want better customer service? Do you want more customers? What do you think a leadership program can accomplish?” With that kind of thing and maybe a little negotiation between training and top management, we’ll get to a point where we’d better be sure that people behave in ways to accomplish these goals. So ahead of time, we have a pretty good idea of what we are trying to accomplish.

On what the future holds for measurement and evaluation…

I think it is pretty well set now that the four levels are still going to remain the basics all over the world. But the emphasis now is going to be the implementation of Level 3 and Level 4 because of the jury up there and because of the kind of pressure on training departments not only to have programs that people like that they learn from, but also to have programs that people will use when they go back and thus have an impact on the business and the organization. We still have to get good reactions and tell people about how good the program was. We still have to measure learning. We who are going to implement effective training in an organization have to partner with top management starting from the beginning, talking to them about what to expect, and concentrating on them helping us. One of the chapters I added to the book Implementing the Four Levels was on how to get managers on board. This is key. They have got to get managers on board right from the beginning.

About Donald L. Kirkpatrick

Donald L. Kirkpatrick is professor emeritus at the University of Wisconsin and honorary chairman of Kirkpatrick Partners. In 1959, the publication of his seminal articles on training evaluation created the foundation for the Kirkpatrick Four Level Evaluation Model. He is the author of seven books on training evaluation and human resource topics. He has been a volunteer president of ASTD, and he was honored with the 2003 ASTD Lifetime Achievement in Workplace Learning and Performance Award.