Chapter 9
Planning the Evaluation
Few things are more important in a consulting process than the initial planning and, in particular, the planning for evaluation processes. When attempting to measure the success of consulting after the project is complete—as an add-on, follow-up assignment—the need for more planning is often obvious. The initial planning involves several key issues explored in this chapter.
Initial Planning for Success
An important ingredient in the success of the use of the ROI methodology is to properly plan for the ROI study early in the consulting cycle. Appropriate up-front attention will save much time later when data are collected and analyzed, thus improving accuracy and reducing the cost of the ROI study. Planning also avoids any confusion surrounding what will be accomplished, by whom, and at what time. Four planning documents are necessary and should be completed before the consulting project is initiated.
Data Collection Plan
Figure 9.1 shows a completed data collection plan for the work-at-home consulting project described previously. The project was initiated to reduce turnover, reduce office expenses, and improve productivity. An ROI calculation was planned to show the value of this project.
Figure 9.1 Data Collection Plan
This document provides a place for the major elements and issues regarding collecting data for the five evaluation levels. In the first column, specific project objectives are stated. In the second column, the specific measures or data descriptors are indicated when they are needed to explain the links from objectives to the data collection. In the next column, the specific data collection method is briefly described using standard terminology. Next, the source of the data is entered. Data sources will vary considerably but usually include participants, team leaders, company records, and/or the client. In the next column, the timeframe for data collection is listed, usually referenced from the beginning of the project. Finally, the responsibility for data collection is noticed.
The objectives for reaction usually include the desired reactions to the consulting project and suggested improvements. Planned actions may be included here. Reaction data may be collected at different intervals. In this example, feedback is taken only at one time, when the work-at-home project is implemented.
Because Level 2 evaluation focuses on the measures of learning, specific objectives include those areas where participants are expected to learn new tasks, procedures, technology knowledge, skills, or processes. The data collection method is the specific way in which learning is assessed, in this case using a self-assessment questionnaire. The timing for Level 2 evaluation is during the implementation of the project.
For application evaluation, the objectives represent key areas of application, including significant on-the-job activities and implementation steps. In this example, the methods include questionnaires, surveys, and monitoring company records. This information is usually collected a matter of months after the implementation. Because responsibilities are often shared among several groups, including the consulting staff, it is important to clarify this issue early in the process. The timing depends on the scope and nature of the project, and is usually in the range of three weeks to three months after the launch of the project.
For impact evaluation, objectives focus on business impact measures influenced by the consulting project. The measures/data column includes the specifics and may provide a hint about the location of the measure. For example, if one of the objectives is to improve productivity, a specific measure would indicate how productivity is actually measured. In the example, productivity is measured for each group, processors and examiners. The preferred evaluation method is performance monitoring, though other methods may be appropriate. The sources of data utilized at this level are the company records. The timing depends on how quickly the intervention can generate a sustained impact on the three measures—usually a matter of months after the consulting project is completed. In this example, data are collected at six-month intervals. A project evaluator is responsible for data collection at this level. If appropriate, an ROI objective (Level 5) is included.
The data collection plan is an important part of the evaluation strategy, and it should be completed prior to moving forward with the consulting project; the plan is completed before pursuing an ROI evaluation. The plan provides a clear direction of what types of data will be collected, how they will be collected, where they will be collected, when they will be collected, and who will collect them.
ROI Analysis Plan
Figure 9.2 shows a completed ROI analysis plan for the same project. This planning document is the continuation of the data collection plan presented in Figure 9.1 and captures information on several key items that are necessary to develop the actual ROI calculation. In the first column, significant business impact data items are listed. These items will be used in the ROI analysis. The method for isolating the effects of consulting is listed next to each data item in the second column. For most projects, the method will be the same for each data item, but there could be variations. For example, if no historical data are available for one data item, then trend line analysis is not possible for that item, although it may be appropriate for other items. In this example, a control group arrangement was feasible, and a trend line analysis was also considered. Participant estimates were used as a backup.
Figure 9.2 ROI Analysis Plan
The method for converting data to monetary values is included in the third column. In this example, office expenses are converted to monetary values with two approaches: using costs in the company records and collecting expert input from the staff directly involved in the process. The cost categories planned for capture are outlined in the fourth column. Instructions about how certain costs should be prorated are noted here. Normally the cost categories will be consistent from one consulting project to another. However, a specific cost that is unique to this consulting project is also noted. The anticipated intangible benefits expected from this project are outlined in the fifth column. This list is generated from discussions about the project with sponsors, subject matter experts, and other stakeholders.
Communication targets are outlined in the sixth column. Although there could be many groups that should receive the information, four target groups are always recommended: senior management, consulting participants, managers of participants, and the consulting staff. Each of these four groups needs to know about the results of the ROI analysis. Finally, other issues or events that might influence the success of the consulting project are highlighted in the last column. Typical items include the capability of participants, external issues, the degree of access to data sources, political influence, and unique data analysis issues.
The ROI analysis plan, when combined with the data collection plan, provides detailed information on calculating the ROI, illustrating how the process will develop from beginning to end. When completed, these two plans should provide the direction necessary for the ROI evaluation and should integrate with the overall project plan.
Communication Plan
Although communicating the results of a consulting project is often the most neglected step, it is the most important. A golden rule of evaluation: If you ask for the data, do something with them. In order to do something constructive, the right people must know the results. The plan goes from describing target audiences to purpose, communication time line, distribution channel, and responsibility. The plan also includes a place for communication status. Figure 9.3 shows the communication plan for the work-at-home project. In this example, the results were communicated to the board of directors.
Figure 9.3 Communication Plan
Project Plan
The final planning document is the project plan, which maps each step of the evaluation process. Figure 9.4 shows an example. This plan is a culmination of the previously described plans, plus the detailed steps. The project plan begins with project approval and ends with communication and follow-up activities. As you complete each chapter in the book, the steps in the project plan will become clear. The first step is the project go-ahead, followed by problem definition, business alignment, and setting objectives. The project plan includes these steps, as well as steps to design the data collection instruments and to administer the data collection process. The final section of the project plan includes any follow-up activity after you have communicated results.
Figure 9.4 Project Plan
The project plan helps track resources. Accounting for resources expended while implementing the project is a plus. It will save you time as you go. There are a variety of useful tools available to help plan a comprehensive consulting project. It allows you automatically to account for the costs of people involved in the evaluation, as well as the cost of other resources. However, spreadsheet tools such as Microsoft Excel work just fine, or your organization may have an internal project planning tool. Microsoft Word is even a sufficient tool to develop a project plan, although it does lack some of the calculation capabilities. Nevertheless, the plan should represent an operational tool. Develop it by using the best tool for you and your team. It will save you time, money, and frustration!
ROI Planning Meeting
The plans are developed, completed, and finalized in a project planning meeting (although some of the input will be known and developed before this meeting). This crucial meeting brings together the various stakeholders. Four important issues must be addressed:
- Who should be involved in the planning meeting?
- What are the success factors? (credible sources, access to data, etc.)
- What is the agenda?
- Who will approve the project plan?
Planning Participants
Participants of the planning meeting vary according to project. It is essential for the consultant to participate. This person will drive the consulting project. Next is the evaluator who will actually collect and analyze the data. The evaluator(s) could be the consultant or a member of the consulting team. In the work-at-home example, it is the person who manages the HR analytics process inside the organization. The client, who would represent the interests of the particular business unit, would also be involved in the meeting. In the case of the work-at-home project, it is the HR executives and others on the HR team who are familiar with the various issues with working at home. A subject matter expert who understands the work-at-home process should participate, as well as another who understands the dynamics of working at home. Finally, someone from finance and accounting should be involved, ideally.
Success Factors
To make the planning meeting successful, several issues have to be addressed or be in place. This meeting must involve people with credible sources of information. Sometimes for a planning session like this, representatives are sent who are not necessarily the best and most credible people for the task. Access to the data is another issue. When business impact is needed, it should be readily available to the team, or at least the capture process should be known to the group. Also, the group must cover all the issues, so no key participants can be missing. In terms of duration, this meeting can last anywhere from an hour for small projects to four hours for more involved projects. The key is to move swiftly and have the data prepared ahead of time, if possible. It is important to consider the output to be a draft at this point, which could be adjusted when additional information is known.
Agenda
The meeting agenda is straightforward. The purpose of the meeting must first be explained. Next on the agenda are the objectives of the project. The project description itself can be detailed, as are the objectives for the particular elements of the project. For a solution-focused project, such as the work-at-home example, the objectives need to be detailed. Objectives should be developed ahead of time. Then a step-by-step description of the data collection plan, the analysis plan, the communication plan, and the project plan are presented. In each case, it is helpful to have documents that are almost fully completed before the meeting so that the team is merely filling in the gaps. After the planning documents are complete, a quick review of the next steps would be in order.
Approval
Now that the plans are complete, the adjustments are made based on any other feedback. The key is to seek commitment from key stakeholders so that they understand what their role will be and the roles of those whom they represent. Their commitment is critical. Then, perhaps one of the most important points is to get the client to sign off on the project. Signing off can be a simple process in which you obtain agreement for the plan of action. It is helpful in this discussion to point out those things that might be considered weaknesses in the project, controversies around the project, or the difficult parts of the project. Finally, if budgets need to be secured (many times they do), they need to be secured at this time. The project planning meeting often yields an estimate of project cost. In all, this is one of the most important meetings related to a particular consulting project.
Final Thoughts
This chapter presents planning for the evaluation of a consulting project. Four evaluation planning tools are introduced: the data collection plan, ROI analysis plan, communication plan, and project plan. It is also important to integrate an effective change management process into the overall project plan, including stakeholder management, issue resolution approach, and implementation planning and transfer of ownership activities. When the ROI process is thoroughly planned, taking into consideration all potential strategies and techniques, it becomes manageable and achievable. The next few chapters focus on the major elements of this process.