Evaluation Approaches

Even a brief look at the research literature about evaluation reveals numerous options for planning and implementing a program evaluation. For the most part this diversity is the result of the context in which the evaluation takes place, the requirements of the program managers or funders, the knowledge and skill of the evaluators, and the stakeholders' preference from among evaluation perspectives discussed in the next section.

Evaluation Perspectives

Broadly, program evaluations can be viewed as framed by utilitarian and intuitionist/pluralist perspectives (House, 1980). A utilitarian approach makes judgments based on the notion of the greatest good for the greatest number of people. In contrast, an intuitionist/pluralist perspective seeks the widest representation of values from diverse populations.

Specific assumptions of the utilitarian perspective are that (1) the audience for the evaluation is a predetermined decision maker, either the managerial elite or mass consumers; (2) the evaluations rely on predetermined standards as a means of evaluating social utility of the greatest good for the greatest number; and (3) data and performance criteria relate to the total system rather than to individuals, program staff, or program recipients (Hamilton, 1997). Most utilitarian approaches rely heavily on specific mandated designs, the identification of objectives and standards, and the definition of variables in observable terms, placing an emphasis on primarily quantitative data (Gredler, 1996, p. 41).

In contrast, assumptions of the intuitionist/pluralist perspective differ as follows: (1) the audiences for the evaluation are all the individuals associated with the innovation or program, not just the managers or decision makers; (2) the role of evaluation is to reflect diverse perceptions of program worth, not to apply a uniform standard of value; and (3) the experiences of individuals associated with the program are important to understanding how a program works and what its impacts are, instead of summarizing or generalizing the data and performance criteria to the program as a whole (Gredler, 1996). “Likewise, the subjective utility of something is based on personal judgment and personal desires. Each person is the best judge of events for himself” (House, 1993, p. 56). Qualitative data play a much larger role in these evaluations.

Historical Foundations

A pioneer in the development of evaluation as a professional field, Robert Stake (1967) identified these program elements in his “countenance model” in order to capture the complexity of educational or other social programs:

1. Program antecedents (resources): the plans, personnel, supplies, and funds that are available to design and implement a program
2. Program transactions (implementation): how a program operates and functions, including its routine and nonroutine activities
3. Program outcomes (impacts or effects): the short- and long-term actual effects of the program's transactions, whether these are expected or unexpected

According to Stake (1967), these three elements form the organizational basis for a matrix of evaluation process which include both program description and judgment of quality. To describe a program, evaluators gather data to determine the amount of congruency between a program's intended and observed antecedents, transactions, and outcomes, while making note of any discrepancies. Evaluators also identify standards of excellence for use in making judgments about a program's elements (also see the Selecting Standards section of this chapter). To implement an evaluation examining all of these aspects of a program can be an expensive and time-consuming undertaking; a program's managers, funders, or sometimes both groups may choose to request an evaluation that places a priority on one or more of these elements as needed for their purpose.

For example, an evaluation of a teen pregnancy prevention program might assess the plans, personnel, and funds available to design and implement the program (its antecedents) to determine if the community has sufficient resources available. Another evaluation might focus on whether that program is being implemented as planned (its transactions): Are the activities (such as public information events and free condom distributions) being planned and carried out in the most effective and logical locations? Alternatively, evaluators could focus on whether or not teens are using the program's activities as intended and on the rates of teen pregnancies over time (its outcomes).

Over the past thirty-five years, Daniel Stufflebeam and his colleagues at the Western Michigan University Evaluation Center have developed another framework for guiding evaluations of programs, projects, personnel, products, institutions, and systems, particularly those aimed at effecting long-term, sustainable improvements—the CIPP Evaluation Model (Stufflebeam, 2003). The name is an acronym for the fundamental components of the model, to address the following questions:

• Context: What needs to be done? What assets are there? What needs or problems can be addressed by the program? What are the political dynamics in this setting?
• Input: How should it be done? What are the competing strategies, work plans, and budgets of the selected program approach?
• Process: Is it being done? How well is the program implementing its strategies and work plans?
• Product: Did it succeed? Did the program have an impact on its targeted recipients? What are the quality and significance of the program's outcomes? To what extent are the program's contributions successfully continued over time? How successfully can the program be adapted and applied elsewhere?

More information about the CIPP Model is available from the Evaluation Center Web site listed among the resources at the end of the chapter.

This model has been used effectively for a wide range of program evaluations in such areas as education, transition to work, training and personnel development, welfare reform, nonprofit organization services, community development, community-based youth programs, and community foundations. One of these evaluations, of the Consuelo Foundation's Ke Aka Ho'ona self-help housing and community development program, will serve as the representative study in this chapter.

Many other evaluation framework options are evident in a search of the research literature, such as goal-free evaluations, cost-benefit analyses, empowerment evaluations, needs assessments, and so forth. These do not represent rigid methodologies, but rather idealized approaches (House, 1980). Because the various evaluations are developed to address different needs, and because each evaluation is unique, the evaluator must identify what approach is useful in a specific situation. As Worthen, Sanders, and Fitzpatrick (1997, p. 68) note,

Thus strict adherence to only one approach is unlikely. Evaluators tend to be eclectic in using these models, choosing and combining concepts from the various approaches to fit the particular situation. Recognizing that there are important differences in the contexts and activities of individual social services and educational programs, evaluators must tailor their approach to meet the needs of the stakeholders (Fitzpatrick, Sanders, & Worthen, 2003).