1

Evaluation for You and for Your Client

What’s Inside This Chapter

This chapter provides an overview of the book, establishing the premise that evaluation is a process that starts with the learning design and continues all the way to assessing organizational impact. You’ll learn:

• how Evaluation Basics can support and enhance your skills as an evaluator or designer

• how to locate information in the book using the chapter-by-chapter outline

• how to use the icons as guides to special material in the book.

1

Evaluation for You and for Your Client

How Evaluation Basics Can Help You

Evaluation can be beneficial to you as a course designer or evaluator. Information gleaned from evaluation can be used to improve your courseware, improve facilitation skills, and position your learning experiences as value-added. Furthermore, evaluation can help you build relationships with internal and external clients, and gain support from internal clients.

The evaluation challenge is one of balance. You must do enough analysis to meet your own needs while also meeting the needs of your client. Too much evaluation is a waste of effort; not enough inhibits good decision making. This is why evaluation planning is so important. As part of the design process, you determine the initial business metric (the data you will track, such as number of sales, number of defects, or turnover rates), what evaluation level of information to gather, when to gather that information, and how it will be used.

Evaluation is about quality and making learning better. The process—which is supported by methods, tools, and instruments—starts at the beginning of the design and continues long after the learners return to their jobs. It is not something you simply do at the end of a course; it is something you do before, during, and after.

What’s New in This Edition?

The second edition has expanded content in many areas. Conceptually, there is a discussion on “flipped evaluation,” which has Level 4 as the starting point, supporting the idea that training should support business objectives and align with business operations. While the evaluation planning model is retained, there is additional content around the benefits of the plan, key questions, how training evaluations can go wrong, developing instruments for evaluating training, and tips for measuring return on investment (ROI) and other measures of impact. To support the evaluation effort, we present a peer evaluation instrument, a facilitation assessment, a work product to support transfer, and a simplified project plan.

Other unique features of this edition include several aspects of the online, blended, and virtual classroom delivery formats and the implications for evaluation. These ideas are interspersed throughout the book. Also included is a discussion on the certification of internal training programs, including links to evaluation. Because practice and application is important to learning, this edition also features many new exercises and case studies.

How This Book Is Organized

Your success as an evaluator depends on your ability to gather information, analyze that information, and make and implement recommendations based on your findings. You must identify what you want to know and how you will use the information; that is, you need to identify beforehand what decision you need to make. Each chapter in Evaluation Basics focuses on a critical aspect of developing and implementing an evaluation plan. Here’s a summary of the nine chapters in Evaluation Basics:

1. “Evaluation for You and for Your Client.” Right now you are engaged in an overview of the book, establishing the premise that evaluation is a process that starts with the learning design and continues all the way to assessing organizational impact. Also described in this chapter are the icons used in the book and how they can help you get the most out of the information presented.

2. “An Overview of Evaluation.” This chapter presents an understanding of evaluation, the purposes and benefits of evaluation, reasons why talent development professionals tend not to conduct evaluations, and an overview of Kirkpatrick’s four levels of evaluation.

3. “Evaluation and the Design Process.” In this chapter, the focus is on linking evaluation to course design. This chapter also presents criteria for determining the best way to evaluate a course and establish an evaluation plan.

4. “Level 1: Audience Reaction.” This chapter presents common errors and ways to improve Level 1 evaluations.

5. “Level 2: Learning and Application During Training.” This chapter discusses what is included in a Level 2 evaluation, various aspects of testing and assessments to measure learning and practice, and instructional strategies to support Level 2 evaluation.

6. “Level 3: Transfer to the Job and the Environment.” In this chapter, the business case and requirements for transfer are discussed, along with the barriers to transfer, as well as the strategies and instruments that support the transfer of new knowledge and skills to the job. Transfer, in its most basic form, is the idea that what was learned in one situation (training) is being applied in another situation (on the job).

7. “Level 4: Impact and ROI.” This chapter links back to the initial design phase, when the business metric was established for the evaluation. Level 4 evaluation reveals the impact (change in the business metric) training has had on the organization. Then, the chapter presents the steps to conducting an ROI evaluation and discusses ways to isolate the effects of training from other variables. Finally, it shows how to calculate ROI.

8. “Evaluation Biases and Communicating the Results.” This chapter begins with a discussion of the various biases that can affect the quality and reliability of your information. The issue of bias cuts across all levels of evaluation. This discussion is followed by an outline of an evaluation report. The chapter concludes with the development of a communication plan. Once the study is complete, the results need to be communicated to appropriate audiences in an acceptable format. The communication plan provides a structure to develop the message for different audiences.

9. “A Final Thought.” This chapter begins by revisiting the link between evaluation and design and development. This is followed by a discussion of how to demonstrate value to your client. The chapter closes by examining your readiness for conducting a more extensive evaluation of your training programs. Perhaps your organization is not in a position to undertake ROI evaluation, but is ready to take the next step in enhancing your evaluation efforts.

In addition to the chapter material, you’ll find some additional information in the back matter. Appendix A provides several evaluation instruments that can help you evaluate facilitators and courseware (regardless of delivery method). Appendix B provides the answers to the case studies found throughout the book. The additional resources section lists some supplementary reading materials related to evaluation and other aspects of talent development.

Each chapter opens with a quick access guide—What’s Inside This Chapter—to introduce you to the contents of the chapter. Use this section to identify the information it contains and, if you wish, skip ahead to the material most useful to you.

The final section of each chapter—Getting It Done—offers you a chance to practice some of the concepts discussed in the chapter and provides closing tips to help you apply what you have learned.

This book strives to make it as easy as possible for you to understand and apply its lessons. Icons throughout the book help you identify key points that can mean the difference between a coaching success or failure.

	Basic Rules
	These rules cut to the chase. They are unequivocal and important concepts for evaluators.

	Noted
	This icon is used to give you more detail or explanation about a concept or a principle. It is also occasionally used for a short but productive tangent.

	Think About This
	These are helpful tips to help you prepare for future evaluation conversations.

Let’s Go!

Evaluation requires some planning and diligence, but it allows you to make a difference in the quality and value of the learning experiences you provide for your clients as you enhance your credibility. Evaluation allows you to document the value that your programs provide for your organization.

This book presents some theory, but the main intent is to provide practical guidelines, methods, models, and instruments to help you become a more effective evaluator. Evaluation Basics will reveal the essentials of planning and implementing an evaluation plan.

So, use what works for you and your organization, and make it happen. Good luck.

Getting It Done

This chapter provides a road map to Evaluation Basics and outlines some ways the material in this book can help you evaluate your programs in a meaningful way. Now it’s time to think about some initial issues and practices regarding evaluation to see where you and your organization are in terms of evaluating programs. Exercise 1-1 can help you structure your thoughts around evaluation.

Exercise 1-1. Are You Ready to Move Ahead With Evaluation?

1. The first section of this exercise addresses your organization’s current level of evaluation and readiness to go further in its evaluation efforts. The second section deals with your current evaluation practices. Read each statement and indicate whether your current evaluation practices support the statement (yes) or not (no). You may also indicate “not sure” if you don’t have the information to make the decision.

2. Based on the initial assessment of your organization’s evaluation readiness, develop three specific actions to support that readiness or help overcome inertia.

Actions:

•

•

•

3. Based on the initial assessment of your own evaluation practices, develop three specific actions to enhance those practices.

Actions:

•

•

•