Chapter 21

Implementing and Sustaining a Measurement and Evaluation Practice

Debi Wallace

In This Chapter

This chapter describes how to develop, implement, and sustain a measurment and evaluation practice focused on results. Upon completion of this chapter, you will be able to

• identify the critical components of a comprehensive assessment, measurement, and evaluation strategy
• create an action plan with critical milestones for successful implementation
• develop a plan to gain support for measurement and evaluation in your organization.

 

There are many paths you could have taken to get to this point of aspiring to develop and implement a comprehensive measurement and evaluation practice in your organization. You could have received a mandate from a senior leader to justify the budget for training in your organization and are currently sifting through volumes of data on participation, staffing, and financials to react to this request. You desire to start running the business of training in such a way that demonstrates the value of learning and development to your organization, seeking to position yourself with the information you need in advance of being asked. You could be concerned about whether your learning programs are truly targeting the right issues and performance goals and want to ensure that training is prescribed and delivered when appropriate, along with a plan to affect results. You could be involved in a major change initiative in your organization and want to ensure that team members don’t revert to previous processes or behaviors, but rather apply what they’ve learned on their jobs to achieve measurable results. You may have already engaged in one or more measurement projects, or you may just be at the beginning of your journey. Regardless of what brought you to this point, developing and implementing assessment, measurement, and evaluation processes as a systemic part of the learning process will improve the learning and development team’s effectiveness and reputation as valued business partners. Carefully consider the following to develop the right plan for your company.

Create a Strategy for Measurement in Your Organization

Creating a strategy for measurement in your organization is important. To do so, you need to determine what you want to do, identify what to avoid, and develop specific outcomes.

Develop Goals

Determining what you want to be able to do is a critical first step. Carefully select colleagues and leaders in your learning organization that you know are both supportive of and skeptical about measuring and evaluating learning, and collaborate with them to answer the following questions:

• What do you want to know?
• Why are you interested in measuring learning?
• Who wants to hear (or do you want to tell) about measurement results, and why?
• How will measurement information be used? What decisions will be made?
• When is measurement needed?

Identify What You Want to Avoid

Equally important to knowing what you want to be able to do is to identify what to avoid. By deliberately exposing any barriers or baggage in your organization that could hinder your progress with measurement and evaluation, you will save time and significantly increase your chances of sustaining what you’re about to work hard to design and implement. Answer the following questions with this same group of colleagues and leaders:

• Have there been previous measurement efforts that have left organizational baggage behind?
• Are there particular elements of fear with partners or stakeholders?

Develop Specific and Measurable Outcomes

Finally, you should develop specific and measurable outcomes you expect to achieve. This final step in defining your goals will allow you to not only define success, but will also assist you as you define the organizational scope of your efforts. Questions to consider include

• What are you currently experiencing in which you expect to see a difference?
• How do senior leaders currently view learning and development processes, programs, services, and results?
• What type of partnership do you experience with business unit leaders when implementing a new learning program?
• What are you hoping to achieve with programs, curriculum, and overall learning results?

Once you’ve compiled a draft of your goals, test them with colleagues in learning and development, human resources, and business unit leaders. Then, make revisions based on their input. Use these goals to equip you as you design a comprehensive measurement and evaluation practice tailored to your organization’s needs and gain much needed advocacy and support.

Gain Advocacy and Support

It is likely that during your work to define goals for measurement and evaluation in your organization, you identified individuals in learning and development; broader human resources; and hopefully, business units eager to begin working with information that equips them to partner and improve their group’s performance. These individuals can become early advocates and champions to provide the support necessary to secure resources and make this work an organizational priority.

Develop Individual and Organizational Experience

To gain individual and organizational experience, select a challenging problem or opportunity to measure, creating a chain of impact from the learning event all the way through to business impact and ROI. Use results from this effort at appropriate points in time along the way to “show” rather than “tell” colleagues and leaders about the value of measurement and evaluation. Consider the following questions to help you select your first project:

• Where is the organizational or business unit pain?
• Can operational or strategic objectives be clearly defined?
• Are senior leaders eager and committed to resolving this problem or pursuing this opportunity?
• Will solving this problem or opportunity require a large investment of time and/or money?
• Do data exist or can you collect data to analyze?
• How many people will be involved or affected?
• Have skill or knowledge gaps been identified as part of the problem or opportunity?

As you assess the problem or opportunity, ensure that skill or knowledge needs to be acquired by your target audience. If not, the program may be a good candidate for evaluating from an overall human performance improvement perspective, but it won’t help you show the value of training and development. Additionally, being able to link the program to specific business unit objectives, having access to obtain or collect related data, and getting active senior leader interest and commitment are critical for your first measurement and evaluation effort.

Select a Subject Matter Expert with Whom to Partner

Once you’ve selected a project, engaging a subject matter expert with whom to partner provides important experience to guide the work, as well as produces results that are often more credible in the early stages of implementation. These experts can be engaged in a way that enables your long-term capability by not only leading this initial project, but also by coaching and mentoring individuals in your organization as they develop necessary skill and knowledge. Consider the following questions when selecting a subject matter expert:

• Do you have experts internally that can partner with you to implement your study?
• Do you want this expert to transfer the ability to you or other team members to complete similar analysis in the future? If so, what type of training and implementation support do they offer?
• Does the expert have a documented and proven business process, or is it a theoretical model?
• Does the expert use processes that position you to gain credibility as a business person, producing results your leadership team can relate to, or use language that are unique to human resources?
• Does the expert have resources and a proven track record to ensure he or she will deliver on their commitments to you and your organization?

Seek experts who use a process that is both simple and consistent with other processes your business unit leaders are familiar with. If you’re calculating a metric like the return-on-investment of a human performance improvement program, you should use the same formula an executive would use to calculate ROI on a business unit initiative. If multiple interventions are developed and executed to achieve the desired performance improvement (for example, training, plus a change in a compensation plan, plus new marketing tools), expect the expert to have valid methods to measure and attribute that portion of the improvement resulting from training and the other interventions separately. Whomever you partner with should be able to clearly and concisely articulate how their approach will achieve your goals, the process(es) they will use during the engagement, key deliverables and related timing, and demonstrate flexibility given your unique set of circumstances. Be skeptical of experts who build themselves up by tearing other experts down, and always ask for and review relevant examples and references.

Secure Senior Leader Involvement and Engagement

Having access to individuals who either have specific subject matter expertise in the business unit (for example, product specialists), or being assured of leadership support to both set expectations and inspect results are critical to any successful training and development effort. For measurement and evaluation efforts, it can make the difference between having or not having access to the information you need to engage. For any measurement project you participate in, be sure to answer the following questions:

• Who do you need access to in the business unit to offer expert input and insight for planning efforts?
• What types of communications do you need to contract with senior leaders for in advance?
• How can you demonstrate strong business acumen versus overwhelming with “HR speak”?
• How will information be used at each step to focus on improving results?

Consider asking the most senior leader involved in your program to introduce the learning opportunity either in writing or by kicking off the session. Contract with them to make the program a topic of discussion at lead team meetings before, during, and after the learning event. Set expectations with them for specific communications, and solicit their support for securing assistance from other teams if necessary to collect and report on required data.

Follow Through with Concise and Targeted Communication

How you communicate results has both short- and long-term effects on your implementation. Because some colleagues may have had experience with measurement processes that didn’t equip them with information they could tangibly use to facilitate performance improvement, or they may view measurement as an academic process and not a business process, it’s very important to develop communications that “show,” not just “tell.” Use measurement results along the way to inform and equip leaders to intervene if necessary where there is concern and to encourage and reward where things are progressing as planned. Carefully plan and execute your communications considering the following questions:

• What types of data will you collect along the way?
• How will you communicate the “so what”? Given all the data you’re likely to have available, what do they really mean, and what do the various partners or stakeholders need to know?
• How will you use findings to affect the probability of a positive change in performance and business results?
• When do partners and stakeholders need to have the information to affect results?
• How will you share overall final results with partners and stakeholders, and who needs to be involved?
• In the event the program’s results are negative, how and to whom will you communicate? Will this change your overall communication plan?

Finally, don’t forget about those individuals you identified in step one who could be classified as “skeptics” for pursuing this important work. If they remain skeptical, their opinions could create unnecessary obstacles, or even sabotage your efforts. You should deliberately seek opportunities to engage them in the work in such a way that they experience the business value of information produced, perhaps even equipping them with information that will allow them to facilitate critical communications at some appropriate point in the project. It is not uncommon for them to become some of the greatest and most outspoken champions, with relevant and compelling testimonies that develop advocacy and support.

Identify Necessary Processes and Adapt Them to Your Organization

Assessment, measurement, and evaluation span the entire performance improvement cycle from the initial assessment of the performance issue, to the final analysis and reporting of business results. If your processes are not integrated, you risk designing and implementing programs that are not linked with the problem or opportunity and will struggle to develop an effective measurement plan. Consider figure 21-1 for overall performance improvement, and answer the following questions to discover where your organization should develop more robust processes to support this work:

• How do you know what leaders are aspiring for, or where the organization’s pain points are?
• How do you identify opportunities to improve efficiency or effectiveness?
• What reactions have you gathered from human resources colleagues or business unit partners and stakeholders about how performance issues are assessed? Is the current situation viewed as effective or ineffective, and why?
• How do you conduct training needs assessments? Are they aligned to specific organizational or business needs?
• At what point in the training design and development process do you plan for measurement and evaluation?
• Are objectives developed for the various types of performance, that is, readiness, learning, on-the-job application, business impact, and ultimately ROI?
• Are measurement data collected, analyzed, reported on, and used with partners throughout the performance improvement cycle, or only at the end when final results are determined?

Maintaining a focus on overall performance improvement as you identify and adopt required processes is critical to your success. Remember, you want the processes to support the work and the information gained from applying the processes to be used to achieve improved performance for your clients and partners.

Identify Resources and Roles Needed to Support Implementation

You’ll require a variety of resources to implement your measurement strategy, including human resources and technology.

Identify Human Resources Required to Support Your Strategy

The implementation of a results-based approach may require updated roles and responsibilities for all individuals and groups involved in initiating, designing, developing, delivering, and supporting learning in the organization. Human resources partners must assess performance problems and opportunities and prescribe relevant performance improvement interventions targeting specific business measures. During the design and development of training, designers and vendors must develop materials directly linked to the measurable and criterion-referenced performance and learning objectives. Facilitators must focus their presentations, exercises, and activities on relevant business issues and collect information required to measure readiness and learning. Managers must support and reinforce the learning process by expecting and inspecting behavior change. The ultimate responsibility for the success of a learning opportunity must rest with business unit leaders, and the learning organization must support them with performance data to guide them to specific action. Consider the following questions to determine the effect on roles in your organization:

• Who is currently involved in assessing performance issues, measuring human reource initiatives, or consulting with partners and stakeholders?
• Will business unit leaders be required to take different actions or behave differently?
• What skill sets are needed by those involved to effectively engage in the new processes and use results produced?
• How will processes for assessing, measuring, and evaluating programs be linked to existing work processes?
• How will you ensure that these new processes become systemic in your organization?

Identify Technology Required to Support Data Collection and Communication

Technology has enabled efficient data collection in most companies, while mergers and acquisitions have created data quality issues because of multiple systems of record, data redundancy, and lack of consistent data standards. Having data to analyze is required, and should be collected from various sources. Consider the following questions to identify potential technology needs to support data collection:

• How do you collect survey data? Do you have access to raw data that can be integrated with other types of human resource and performance data?
• How do you collect assessment data and who are your sources for this information?
• How will you collect performance data? Do you need special access to secure it?
• How will you track costs? What will you include to “fully load” expenses?

Carefully identify, plan, and budget for technology to ensure you are able to collect or secure the data you need. To learn about deciding on the appropriate technology for your evaluation needs, read chapter 22, “Selecting Technology to Support Evaluation.”

Determine the Effect on Organizational Structures

To implement and sustain a comprehensive measurement and evaluation practice, human resources must be aligned and equipped to offer support and develop expertise. Consider the following questions to determine the best approach for your organization:

• Is centralized leadership necessary to support consistent implementation and application?
• How will you keep the work close enough to the business units to facilitate necessary customization and flexibility, yet ensure consistent enterprise results?
• If you choose to have centralized and decentralized human resources, how will they work together to achieve and support your overall goals?

Develop Implementation Plan and Execute

At this point, you have considered a wide range of issues to define your goals for measuring and evaluating learning in your organization, gaining advocacy and support for the work, and have identified new processes and resources required to support your implementation. You are ready to develop an implementation plan covering the following milestones to ensure a successful execution:

• secure approval for strategy
• communicate to all partners and stakeholders involved
• secure resources (human and financial) as necessary
• develop and implement new governance and process
• develop and implement required learning
• develop and implement new technology
• develop rollout plans and measurement criteria for each business unit
• address any barriers to implementation as necessary
• create and support the measurement community
• share status updates and success stories throughout the organization
• communicate results to partners and stakeholders
• continuously improve your process.

This work is about so much more than collecting data and producing reports; rather, it is about equipping individual learners and leaders in your company with the information they need to know, whether they are well positioned to succeed or are at risk of not achieving desired results. Many great ideas and valiant efforts result in little more than good intentions and significant time spent trying to implement them. Diligently consider the questions in this chapter and the checklist in table 21-1, and allow your discoveries to establish a reasonable plan for using measurement and evaluation results to drive greater accountability and improved results in your company’s learning organization.

Table 21-1. Checklist for Implementing and Sustaining a Measurement Practice

---

1. Create a strategy for measurement in your organization.

2. Gain advocacy and support.

3. Identify necessary processes and adapt them to your organization.

4. Identify resources and roles needed to support implementation.

5. Develop implementation plan and execute.

 

Knowledge Check

Align each activity listed below with one of the following key components of a comprehensive assessment, measurement, and evaluation practice implementation. Check your answers in the appendix.

Key Components:

1. Create a strategy for measurement in your organization.

2. Gain advocacy and support.

3. Identify necessary processes and adapt them to your organization.

4. Identify resources and roles needed to support implementation.

5. Develop implementation plan and execute.

Activities:

	Secure senior leader involvement and engagement.
	Create and support a measurement community.
	Develop measurement and evaluation processes.
	Develop goals.
	Select a subject matter expert to partner with.
	Identify technology required to support data collection and communication.
	Develop and implement required learning.
	Integrate new processes with existing program design and development processes.
	Develop specific and measurable outcomes.
	Determine the impact on organizational structures.
	Develop individual and organizational experience.
	Identify what you want to avoid.
	Develop rollout plans and measurement criteria for each business unit.
	Develop assessment processes.
	Identify human resources required to support your strategy.
	Follow through with concise and targeted communication.

About the Author

Debi Wallace has more than 23 years’ experience in leading and evaluating strategic human performance interventions in the financial services industry, along with a strong analytics, learning and development, human resources, and communications background. She has led the strategic and tactical development and implementation of an enterprise assessment, measurement, and evaluation practice to measure the impact of learning and other human resource programs, conducting dozens of linkage and ROI studies since 1995. She most recently created and led Wachovia’s Advanced Analytics and Research Practice in HR Workforce Analytics as a senior vice president in their human resources division, where she partnered with other leaders to set Wachovia’s strategic analytical direction for HR-related information, striving to consistently demonstrate how to use HR data and measurement systems to drive fact-based decision making. Her team led HR business impact analysis projects, as well as complex enterprise analytics initiatives. She regularly speaks at industry conferences and has published a number of case studies on related topics. She can be reached at [email protected].

Additional Reading

Blanchard, K. (2002). Zap the Gaps. New York: HarperCollins.

Phillips, J. J. (1998). Implementing Evaluation Systems and Processes. Alexandria, VA: ASTD.