Chapter 24

Evaluating Leadership Development

Emily Hoole and Jennifer Martineau

In This Chapter

This chapter details a case study about a leadership training program for public libraries, the successes of the program, and the methods and processes used to evaluate it. Upon completion of the chapter, you will be able to

• recognize how to design an evaluation that targets behavior change and organizational impact as a result of a leadership development intervention
• identify how to successfully integrate multiple measures and perspectives to build a rich description of the program and triangulate the evaluation data
• become acquainted with how to benchmark programs against best practices in the field of leadership development for program improvement
• recognize how to present results to key stakeholders in a way that is aligned to their needs.

 

Background

The Executive Leadership Institute (ELI) program is a unique development opportunity provided by the Urban Libraries Council (ULC) and funded by the Institute for Museum and Library Sciences (IMLS) to libraries across the country. ULC contracted with the Center for Creative Leadership’s Evaluation Center to conduct an external evaluation on the ELI program.

First, here is a bit about the program itself and the situation that led to the design and implementation of the ELI program. The context in which libraries and library leaders function is changing rapidly due to technological and societal changes. As society changes, so must libraries. For their institutions to remain relevant, library leaders will have to survive in competitive political and economic environments, manage in the context of changing information technology, guide significant organizational restructuring initiatives, and respond to customer and community dynamics.

Urban Libraries Council’s ELI program used an action-learning model to help library leaders adapt to the changing environment in which they operate. ELI participants (called fellows in this program) engaged in challenging projects that provided an effective and appropriate practice ground for leadership development over an 11-month period. Twenty fellows engaged in learning lab projects in partnership with a sponsor from their library, three intensive face-to-face sessions, coaching, a national peer community, and reflective practice.

The overarching goals of the ELI program were to

• increase the leadership capacity and diversity of future public library leaders on a national basis
• support an expansive national network of new library leaders
• create opportunities for local library experimentation with new organizational structures and strategies that support leadership development
• create a national model and brand of effective public library leadership development.

Through the evaluation, we concluded that ELI was a powerful developmental and transformational program for individuals who participated. Fellows changed their behavior; experienced increased confidence in their leadership; and significantly shifted their viewpoints, vision, and professional approaches. The libraries that participated also experienced positive and potentially powerful shifts in service provision and strategy. Clearly, the communities these libraries serve are reaping the benefit from the fellow’s and library’s participation in ELI—through new programs and services, greater efficiency, and a library better connected to the community. The library field will also continue to benefit from the ELI program for years to come as these fellows influence others within their organization and across the field to provide strong leadership for addressing the many challenges libraries face in today’s fast-changing society.

Guideline 1: Articulate Outcomes of the Leadership Development Intervention

ULC made it a priority to clearly articulate the outcomes of the intervention at each level because behavior change and organizational impact were desired targets of the program.

Articulating outcomes is critical, and the ideal timing for evaluation planning is alongside the needs assessment and design of the leadership development initiative itself. Integrating evaluation with program design allows for the outcomes of the program to be properly identified so that the program will be designed in a way that is likely to achieve the intended outcomes. Although CCL was not involved with ULC or the program director during the needs assessment and design phases, they articulated the skills areas and competencies for the individual leaders and specified the desired impact on the organizations.

At the point the evaluator is engaged, he or she should work closely with the program’s sponsors to understand the program and its goals, outcomes, and objectives before designing an evaluation that will measure the extent to which these are reached. We try to cocreate with stakeholders the objectives for the evaluation itself, as well as the outcomes for the program.

The three primary objectives of the ELI evaluation are

• Impact. Provide evidence of leadership development of fellows, institutional changes, and community impact.
• Benchmarking. Identify the degree to which ELI makes use of processes, methodologies, and approaches that are considered best practice and/or are supported by the research on formal leadership development programs.
• Program improvement. Provide feedback that enables ULC to redesign the program for increased impact.

The evaluation design was based on two common frameworks. The first was Kirkpatrick’s four-level model (1998), focused on reaction, behavior change, and impact. The second was Brinkerhoff’s Success Case Method (2003), intended to understand the use of program content in relation to the outcomes, identify facilitators and barriers, and further document stories of impact.

In determining whether the ELI program’s intended effect was achieved, they identified the following desired outcomes for participating fellows and libraries during the program’s design phase (and were therefore assessed through the evaluation):

• increase the leadership capacity of future public libraries. Build skills in political relations, community planning, problem solving, and collaboration. Targeted competencies include
  — understanding the larger context of public libraries in communities
  — working effectively with community partners and leaders
  — working effectively as change agents in complex initiatives
  — having a vision about the future of the public library
• develop an effective learning community and valuable national network of colleagues, specifically relating to peer support and information exchange
• experiment with new library structures and strategies for leadership development within participating libraries
• evaluate learning lab projects for individual learning, organizational restructuring/revitalization, and new ways of connecting with the community
• create a national model and brand for public library leadership development.

Guideline 2: Think in Multiples

When evaluating behavioral and organizational changes, think in multiples. Multiple perspectives and multiple measures can be used to establish the type and magnitude of change to a greater degree than a single assessment of participant perspectives.

The design of the ELI evaluation used multiple methods and gathered multiple perspectives to substantiate the findings. Evaluations that collect data through multiple methods (for example surveys, interviews, reviews of existing project documentation) and from multiple perspectives (for example, program participants, participants’ managers, program sponsors, organizational executives) are stronger than evaluations that do not because they are triangulating the data. That is, they are asking questions in multiple ways from multiple people to get the best representation possible of what actually occurred as a result of the program. When the data from multiple methods and multiple perspectives include consistent themes, the evaluators can make statements about the outcomes of the program with more confidence.

This evaluation included the fellows of two cohorts, as well as executives and sponsors related to both cohorts.

The evaluation design included the following components:

• online survey to gather data on behavior change, individual impact, organizational impact, value and usage of program content and components, utilization and value of the peer learning community, and learning lab project from two cohorts of fellows (N=38, response rate of 93 percent)
• online survey of sponsors and library CEO/executive directors for the two cohorts on behavior change, individual impact, organizational impact, learning lab project, and changes to the library’s approach to leadership development (N= 36, response rate of 71 percent)
• identification of success cases and fellows who experienced less success:
  — interview fellows (21 interviewed)
  — interview CEO/executive directors and sponsors (eight CEO/executive directors and five sponsors interviewed)
• for benchmarking, review of program materials and interviews with ULC staff and program director.

Data were collected over several months using online survey software (see figure 24-1). Various software can be used—three of the more commonly known and used programs are SurveyMonkey, SurveyGizmo, and Zoomerang.

Through the online survey, fellows provided self-ratings, CEO/executive directors rated one or more fellows (some libraries had fellows in both cohorts), and sponsors rated the fellow they worked with during the intervention. We conducted one-hour telephone interviews with the fellows identified from the survey data and their corresponding CEO/executive directors and sponsors to gain multiple perspectives on the program’s effect and relevant organizational and contextual variables.

We analyzed survey data and broke data down into ELI III and ELI IV data (referring to two cohorts). We compared frequencies and averages from the survey across cohorts and raters as well as to frequencies from the interview data. We content analyzed and themed interviews. We supported key evaluation questions by triangulating the data from both the survey and interview data (that is, do the data from both methods support the conclusions that are drawn regarding the answer to the question?).

We examined differences in data at the cohort level from various perspectives. Next, we considered changes in program design from one cohort to the next, along with differences in the composition of each cohort. This helped program staff understand if design changes had the desired effect for the subsequent cohort.

Figure 24-1. Example from Participant Survey

Reflect on your ELI experience and provide a rating of your current ability with regard to the following leadership behaviors and then rate how much your ability changed as a result of the ELI program.

 

Current ability rating scale: No ability, Not much ability, Some ability, Great ability, Very great ability

Changed ability rating scale: No change, A little change, Some change, A lot of change, A great amount of change

 

1. Lead organizational change: Ability to set direction, manage, and support organizational change.

2. Influence, leadership, power: Ability to inspire and promote a vision; persuade, motivate, and influence others.

3. Manage organizational politics: Ability to understand and negotiate political boundaries.

4. Solve problems and make decisions: Ability to analyze problems, formulate solutions, and make clear decisions.

5. Gather and understand information, identify problems: Ability to seek information, create order out of large quantities of information, and get to the heart of a problem.

6. Take risks, innovate: Ability to see new opportunities and consistently generate new ideas, effectively introduce and create needed changes, and think outside the box.

7. Think systemically: Ability to demonstrate understanding of the whole organization and how to move initiatives forward in that context.

8. Act systemically: Ability to effectively navigate the politics and processes of the organization, establish effective collaborative relationships, and create alliances throughout the organization.

9. Demonstrate openness to influence and flexibility: Ability to invite and consider ideas different from own, accept criticism, and listen with an open mind.

10. Demonstrate adaptability: Ability to adapt to changing community conditions and open to new ideas and methods.

Using the rating scale provided, please indicate the level of change that has occurred as a result of your participation in the ELI program.

Level of change rating scale: Significantly decreased or worsened, Decreased or worsened, Remained about the same, Increased or improved, Significantly increased or improved

My...

• self confidence has

• understanding of my strengths and limits as a leader has

• ability to effectively delegate work has n

• ability to effectively manage projects has n

• ability to gain the support of colleagues and peers has

• ability to gain the support of my superiors has

• ability to build effective relationships with others has

• ability to meet future leadership challenges has n

• appreciation of the value of diverse perspectives has

• openness to feedback has n

• vision of success for the leadership project has n

• confidence to take risks has n

• understanding of the national context of public libraries in communities has

• understanding of the context of my public library in my local community has

• ability to work effectively with community partners and leaders has

• ability to work effectively as a change agent in complex initiatives has

Using the rating scale provided, indicate the level of change that occurred in the part of the organization for which you are responsible.

In the part of the organization for which I am responsible...

Level of change rating scale: Significantly decreased or worsened, Decreased or worsened, Remained about the same, Increased or improved, Significantly increased or improved

• employee commitment to the profession of public librarianship has

• employee performance has

• the ability to mobilize people to support a change agenda has

• levels of collaboration have

• communication within the library has n

• communication outside the library has

• innovation has

• attitude toward customer service has

• staff ability to work with community stakeholders has

• staff ability to identify the community it serves and make appropriate decisions in how it serves has

• staff ability to promote a culture of inclusiveness and diverse participation has

• staff ability to communicate the library’s public value has

Would you consider the leadership project to have been a successful leadership learning experience?

Describe the effect of the leadership project on your library.

Describe the effect of your leadership project on your community.

How did your leadership project result in new ways of working inside the library?

Did the leadership project result in new ways of connecting with your community stakeholders?

If yes, why? It not, why not?

Have you remained in contact with any of your peers from the ELI program?

With how many of your ELI peers do you remain in contact?

Please rate the following statements regarding the peer network you developed as a result of the ELI program:

Rating scale: Strongly Disagree, Disagree, Neither Disagree nor Agree, Agree, Strongly Agree

My ELI fellows provide me with valuable personal support.

My ELI fellows provide me with valuable professional support.

I exchange career advice with my ELI fellows.

I exchange library service-related advice with my ELI fellows.

I exchange leadership advice with my ELI fellows.

How would you rate the value of the following program components to your development as a leader?

Rating scale: Not valuable, Somewhat valuable, Very valuable, Exceptionally valuable

Sponsor

Coaching

Leadership project

Situational assessments

Leadership interviews

Peer learning circles

Audio conferences/conference calls

Virtual community (community zero)

Director-in-residence

CEO/sponsor interview questions

Please rate your use of the content you learned during the ELI program.

 

	Applied the content—very effective/helpful	Applied the content—not very effective/helpful	Have not applied the content but plan to	Have no plans to apply the content	Content is not applicable to my situation
Change management
Political skills
FIRO-B
MBTI
360 assessment
Project management
Reflection
Priority and time management
Action learning

CEO/sponsor interview questions

• How, if at all, do you think the fellow’s leadership capacity has increased as a result of participating in the ELI program overall?

• To what extent, if at all, and in what ways has the program affected the fellow’s understanding of the larger context of public libraries in communities?

• To what extent, if at all, and in what ways has the program affected the fellow’s ability to work effectively as a change agent in complex initiatives?

• To what extent and in what ways has the program affected the fellow’s vision about the future of the public library?

• To what extent, if at all, and in what ways has the ELI program affected the fellow’s political relations skills?

• To what extent, if at all, and in what ways has the ELI program affected the fellow’s community planning skills?

• To what extent, if at all, and in what ways has the ELI program affected the fellow’s skills in managing organizational change initiatives?

• To what extent, if at all, and in what ways has the ELI program affected the fellow’s problem-solving skills?

• To what extent, if at all, and in what ways has the ELI program affected the fellow’s collaboration skills?

• To what extent, if at all, and in what ways has the ELI program affected the fellow’s negotiation skills?

• What extent, if at all, has participation in the ELI program helped change or revitalize your library’s organizational structure? What do the new structures look like? What realignments have been made, if any?

• To what extent, if at all, has participation in the ELI program helped change or revitalize your library’s strategic direction? (New priorities? Shift in priorities?)

• To what extent, if at all, has participation in the ELI program helped change or revitalize your library’s ways of conducting business? (Customer service? Service strategy changes?)

• What impact, if any, has the ELI experience had on ways your organization works with community partners?

• What impact, if any, did the leadership initiative have on community relations and/or the ways the library connects with the community?

• What impact, if any, has the change initiative had on your constituencies?

• To what extent, if at all, did the leadership lab project raise the library’s visibility or credibility in the community?

• To what extent, if at all, has participation in the ELI program changed the ways in which your organization supports leadership development?

• What, if any, new strategies or resources have you put in place to support leadership development?

• How were fellows supported? Have you continued to support their development with new strategies or resources?

• To what extent, if at all, did the leadership lab project assignments contribute to changes in the way that you support leadership?

• To what extent, if at all, has the ELI program affected you, personally, as a leader?

• From your perspective, what are the most effective elements and approaches of ELI?

• Do you have any suggestions for how the ELI program can be improved to increase efficiency of its delivery and implementation? To increase its impact?

 

Using Technology

We used technology throughout the evaluation process from data collection to analysis. As a primary data collection method, we used online surveys, along with telephone interviews. We recorded the interviews and transcribed them for analysis. Because the quantitative analyses required for this report were not complex, we used a spreadsheet calculator rather than a statistical program.

Guideline 3: Benchmark Against Best Practices

Benchmarking a program against evidenced-based best practices can identify areas of strength and weaknesses for purposes of program improvement and replication.

A key purpose of the evaluation was to compare the ELI program to best practices in leadership development. To what degree does ELI use processes, methodologies, and approaches that are considered best practice and/or are supported by research on formal leader development programs?

To address this question, we first identified 10 elements associated with effective leader development programs. We drew these elements from accumulated knowledge from research and practice at CCL (Center for Creative Leadership, 2008; McCauley and Van Velsor, 2004) and from recent studies examining best practices in leadership development in organizations (American Management Association, 2005; APQC, 1998, 2006; Bersin Associates, 2008; Hewitt Associates, 2005). Next, we sought evidence that these best practice elements were presented in ELI, relying on documents describing the initiative and on interviews with the ELI program director and ULC staff.

The 10 best practice elements we identified are

• clearly articulated program goals linked to business needs
• extra developmental attention and resources for high-potential leaders
• careful selection of participants based on criteria to ensure that the program is a good fit for the participants
• program design that integrates general principles of adult learning and the particular needs of the target audience
• frameworks that provide participants with an understanding of the components of effective leadership (for example, competency models, leadership behaviors)
• reliance on challenging experiences and relationships as major drivers of learning, supported by opportunities for feedback and reflection
• strategies to transfer learning to the back-home context
• accountability for development
• evaluation and continuous improvement of the initiative
• an organizational context that supports the program.

We reviewed ELI’s use of best practices within each element and then discussed in the final report. Figure 24-2 presents an example from the final report.

The evaluation determined that the ELI program represents exemplary best practices in five out of the 10 elements, is on par with best practices in three additional elements, and uses some best practices in two elements. The areas in which ELI does not currently represent exemplary best practices use represent opportunities for future improvement in designing or reimagining leadership development for future interventions.

Figure 24-2. Example from ELI’s Use of Best Practices Final Report

Element 3: Careful selection of participants based on criteria to ensure that the program is a good fit for the participants.

ULC has articulated the characteristics of an appropriate fellow in its program descriptions and application materials:

 

• has the potential (within five years) to be part of the library’s senior leadership team

• understands challenges faced by today’s public libraries

• indicates an interest in learning, personal change, and institutional change

• has experience in/knowledge of basic managerial skills (supervising, budget, planning, etc.)

• has credibility in the institution and enthusiastic backing of the library director

• is willing and able to commit at least two days a week (cumulative over 10 months to work on the leadership challenge).

 

These characteristics are used by the program advisory team (which is made up of current and retired urban library directors). However, libraries do not just nominate individuals to attend the program; their application includes the project the individual will work on and an internal sponsor for the individual and project. Thus, applications are also judged in terms of the appropriateness of the project and sponsor. Although this can make the selection process more complicated (including the need to coach libraries as they are submitting applications), it does have the potential for creating more organizational support for the program.

Although applications are rated on the fellow characteristics above and on the appropriateness of project and sponsor, there is not a strict reliance on quantitative assessments. Lengthy discussions among advisory team members make use of the in-depth knowledge contained within the group in arriving at a decision through sense-making (rather than strictly analytic) processes.

 

Guideline 4: Present Results to Key Stakeholders in a Way That Is Aligned to Their Needs

When reporting evaluation results, it is critical to know the audience and their preferences in terms of the type and level of information shared, report format, report length, and so forth. During the needs assessment process, we determined that two audiences for the evaluation results would need different levels of information in their reports. Thus, two written reports were created—a comprehensive report for staff, program designers, and others for lessons learned and program improvement and an executive summary on the results of the program to share with a wider stakeholder group and the general public. Figure 24-3 presents an example of a pie chart that illustrates CEO and sponsor perceptions of the increases they have seen in fellows’ abilities since participating in the ELI. It is followed by a brief paragraph noting some of the overall takeaways from this particular set of questions, and then by a series of tables illustrating the open-ended comments relative to the quantitative data.

The report also broke out the data by cohort to understand the different perspectives of each participant group and how their program experience was shaped by the design of the program. This is shown in figure 24-4.

The ELI III cohort found the peer learning circle component of the program more valuable than the ELI IV cohort, who rated the virtual community much higher in terms of value. This outcome can be explained by the way the peer circles were organized in each cohort group and the significant enhancements made to the virtual community prior to ELI IV. Analyzing and presenting the data in this way made it apparent that for all participants the leadership project and coaching were critical program components.

By using various pie charts, bar charts, tables, and text, the report addressed multiple learning needs of the target audience for the evaluation.

How the Results Were Used to Make Decisions about the Program

The results of the evaluation were used to shape what ULC hoped would be the next generation design for library transformation. The ELI program was effective at individual level change for participants and for facilitating increased connection between the library and community through new or improved services and involvement, but it did not have the effect on organizational transformation that ULC believed to be necessary for libraries in the 21st century. The next generation design has a greater organizational focus involving a larger group of individuals for development in transformational efforts.

Figure 24-3. CEO and Sponsor Perceptions of Increases in Fellows’ Abilities

The graph above represents the percentage of CEOs/sponsors interviewed who indicated that a skill had increased in the area indicated for one or more fellows as a result of the program.

All of the categories represented in the pie chart are key areas targeted by the program, and fellows and CEO/sponsors were asked how the program affected the fellow. From the interview data, the areas of greatest skill increases were

• collaboration skills

• workiing effectively with community partners and leaders

• political relations skills

• managing organizational change initiatives.

 

Interview Data on the Key Program Outcomes (21 Fellows, 13 CEOs/Sponsors) Collaboration Skills To what extent, if at all, and in what ways has the ELI program affected the fellows/your collaboration skills?
Fellows	CEO/Sponsors
Increased Skills (16 comments): “Definitely one of the strongest areas that has been worked on both inside and outside the organization. I am not only working outside the organization and with nonprofits, I also started collaborating with other city departments. As a result, I now feel very comfortable calling on the assistance of other departments.” Reinforced Skills (2 comments ):"I've always been pretty good at this, but I learned to collaborate and still move forward on the agenda to be addressed.” No Change (3 comments): "I've always been a strong collaborator.”	Increased Skills (9 comments): “Definitely improved. Both were required to work with units outside their regular day work and had to work across departments. What was amazing was that they were able to manage the projects in a way they never had to do before.” Other (4 comments): “Already very collaborative and her skills may have been enhanced but I think it was always a part of her makeup.”

 

Knowledge Check

You have been contracted to design and conduct an evaluation of a leadership development program that is in the early stages of needs assessment and planning. This is a program that is being designed by a regional telecommunications company for approximately 50 “critical leaders” in seven different functions. The commonality between them is that they are all responsible for helping the organization expand its product offerings to meet market demand, and none of the leaders are the heads of their function—they all have a vice president to whom they report. There are several key groups of stakeholders: the leaders themselves; the functional vice presidents who are supporting the initiative; the executive team of the company, which includes the president, the seven functional vice presidents, and several executive vice presidents; the talent office, which is responsible for designing and conducting the program; and ultimately the company’s board of directors.

What steps would you take to plan the evaluation, including its design, implementation, and reporting phases? What questions do you need to ask of the stakeholders? Check your answers in the appendix.

About the Authors

Emily Hoole, PhD, is the director of the Evaluation Center at the Center of Creative Leadership (CCL). She works on client initiatives, new product development groups, and with internal CCL teams on identifying organizational and leaders’ needs, clearly articulating program goals, objectives, and outcomes and evaluating the initiatives for improvement and individual and organizational impact. Prior to CCL, she was the director of evaluation at the Rapides Foundation in central Louisiana. Hoole also served as the executive director of the United Way of Harrisonburg and Rockingham County in Virginia. She holds a doctorate in assessment and measurement from James Madison University. She can be reached at [email protected].

Jennifer Martineau, PhD, is the group director of Global Research, Innovation, and Evaluation at the Center of Creative Leadership (CCL). She is responsible for setting CCL’s strategy for these functions in coordination with the current and future needs of clients. She played a foundational role in establishing CCL’s Evaluation Center through creating a 360-degree evaluation assessment, designing and launching an evaluation framework used to guide evaluation studies, serving as lead evaluator on many leadership evaluation projects, and hiring talented evaluators. In her evaluation work, her clients include international for-profit and nonprofit organizations, school systems, and governmental agencies. She can be reached at [email protected].

References

American Management Association. (2005). Leading into the Future: A Global Study to Leadership 2005–2015. New York: Authors.

APQC. (1998). Leadership Development: Building Executive Talent. Houston, TX: Authors.

APQC. (2006). Leadership Development Strategy: Linking Strategy, Collaborative Learning, and Individual Leaders. Houston, TX: Authors.

Bersin Associates. (2008). High-Impact Leadership Development. Oakland, CA: Authors.

Brinkerhoff, R. O. (2003). The Success Case Method: Find Out Quickly What’s Working and What’s Not. San Francisco: Berrett-Koehler.

Center for Creative Leadership. (2008). Leadership Development Framework. Unpublished intranet knowledge asset.

Hewitt Associates. (2005). Research Highlights: How the Top 20 Companies Grow Leaders. Lincolnshire, IL: Authors.

Kirkpatrick, D. L. (1998). Evaluating Training Programs, 2nd ed. San Francisco: Berrett-Koehler.

McCauley, C. D. and E. Van Velsor. eds. (2004). The Center for Creative Leadership Handbook of Leadership Development, 2nd ed. San Francisco: Jossey-Bass.

Additional Resources

Hannum, K. M. and J. W. Martineau. (2008). Evaluating the Impact of Leadership Development. San Francisco: Pfeiffer.

Hannum, K. M., J. W. Martineau, and C. Reinelt eds. (2007). The Handbook of Leadership Development Evaluation. San Francisco: Jossey-Bass.

Russ-Eft, D. and H. Preskill. (2009). Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. Philadelphia: Perseus.