Mary L. Broad
On how it all started…
My whole professional career has been around training and then moved into the general performance improvement field. My main work employment has been with various federal agencies. The last 10 years or so before I retired in 1993, I was the director of training and then human resource development at the Defense Information Systems Agency—a fascinating place to work. Since my retirement, I have done a variety of consulting, and the theme of evaluation has always been important. We need to know the outcomes and the results of the work that we do. I went back and looked at my first publication with John Newstrom, Transfer of Training, back in 1992 and found that, yes, we were paying attention to evaluation back then, but sort of as a pro forma thing that everyone needs to do. We did not focus too much on it in terms of what we were urging managers to do, which was really to enrich the workplace to help people apply what they have learned. I think that my real focus on evaluation, as a really high priority, probably occurred not too long after 1992. I was elected to the Board of Governors of ASTD in 1993. Gary Rummler was also on the board—a guy who has been extremely important in the International Society for Performance Improvement, the other professional association with a huge focus on performance. Performance is not so much what people learn to do, but it is what they actually do. We can help them with that performance—often through training—but also by helping managers to enrich the workplace. That has been my focus ever since. Evaluation is a strong partner with performance because managers need to know how things have worked out—what the results have been. Since that time, the whole process of evaluation has just loomed larger and larger in my mind, and I think it is more and more important that anyone who is in any organization begin to get a feeling for what people need to do to be able to perform well and how we can measure that. So that has been my trajectory in terms of appreciating evaluation.
On how training evaluation has changed over the years…
I grew up, as so many of us have, with the Kirkpatrick framework as the foundation of things. As training and performance improvement get integrated with the way organizations work, we are going to come up with a whole roster of evaluation focuses related to timeliness, accomplishments, savings, satisfaction, and all kinds of measures that will be available and just built in as the particular situation requires so that we won’t have to do the same kinds of analysis to get evaluations done. There will be some patterns, approaches, and ways of thinking about it that are way beyond me right now, but certainly the way people are inventing and making inroads into extremely fruitful new ways of doing business, I am sure things will change dramatically. The whole principle of evaluation—of measuring where we are, where we are going, did we get it done, what’s still missing—all of those kind of questions are never going to go away, and I think we are going to be able to build them in to the way people do their work in a much more smooth and integrated way.
On the progress the profession has made in embracing evaluation…
People in our field have some work to do to help organizations recognize the value of looking at people through a performance lens. Performance refers to the outcomes, the results, the payoff. Managers usually care a whole lot more about these things than they care about whether somebody actually attended a particular training program. So, the outcomes and how we can support the organization in achieving its strategic outcomes are where we can get managers’ attention and become better at partnering with them in meeting their goals.
On how evaluation makes a difference in the perception held by executives of training and development…
It is up to us to educate organizations that we can be more than trainers. We can really help them with other aspects of performance. People in our business are probably the few who really pay attention to how well people can do their work. If we show that we really can tune it up and help managers who have a challenge on their hands get something done through improved performance, if we can begin to find those instances where it really paid off, those managers will be some of our best advocates for getting support for dealing with their performance problems. So I think it comes back in a way to people in our field beginning to see themselves differently, beginning to present themselves as partners with management, trying to find instances in the organization where someone can really use their help and working with that person to get some sort of success and then publicizing it. I think we have a lot to do on our own behalf, to market ourselves—perhaps in a different way—as not just presenting training, even very good training, but as helping managers get the performance they want out of the people who work for them. That can get managers’ attention and real interest.
We talk about Level 3 evaluations frequently in our line of business, meaning how much people are actually using what they have learned on the job. One point I wanted to make is that Level 3 evaluations are so often considered another aspect of evaluating the training program, the effectiveness of the training program. I think that is totally wrong. Level 2 evaluation, that is, what did people learn in the course of the training event, that is where we learn about the effectiveness of the training. Level 3 actually measures how much the workplace has changed to support the use of that training; it is much more a measure of what the stakeholders in the workplace have done than it is an evaluation of the training.
On what the future holds for measurement and evaluation…
Well, the future is a real challenge, but now may be a good time for people in the field to lay the groundwork for partnerships. In general, I think evaluation has got to become more recognized as an absolutely essential tool. Now, we have some good tools, like the works of good people like Jack Phillips, to help show the payoff—even to show in monetary terms that if we do these kinds of things in these ways then the outcome is really going to be valuable. Now that we can make those kinds of outcomes clear, I think we have some tools to show managers how they can save money, how they can make money, how they can make things work much better, and how they can get where they need to go. I think that the more training becomes on the job and on demand, the more evaluation needs to be embedded in the way the training is designed so we don’t have a separate set of decisions such as: “We’re going to do this training, should we evaluate it?” Training design and evaluation should be absolutely integrated together. Evaluation is built into the way that the training is presented, is dealt with, and the data are collected.
About Mary L. Broad
Mary L. Broad, PhD, is a certified performance technologist who has served on the ASTD Board of Directors. She is the author and editor of several books, including Transfer of Training: Action-Packed Strategies to Ensure High Payoff from Training Investments (1992, with John W. Newstrom); Beyond Transfer of Training: Engaging Systems to Improve Performance (2005); and In Action: Transferring Learning to the Workplace (1997). As leader of her company, Performance Excellence, she helps organizations improve human performance systems through strategic planning, performance technology, and the transfer of learning for high-priority performance improvement intervention. She is an experienced future-search facilitator for public and private sector clients.