Agency Records (Administrative Data)

Existing agency records have been by far the greatest source of performance data that most government agencies use. These records are the primary source of output data such as data on program costs and the number of units of physical output produced by a program (e.g., number of repairs, number of records processed, gallons of water treated, tons of garbage collected). They also sometimes provide data on service outcomes, for example, the number of reported crimes (as an indicator of successful crime prevention and school attendance and graduation rates.

Governments have increasingly measured response times to citizens' requests for services, such as response times to calls reporting fires and crimes. Public agencies have also begun to measure response times for many other public services, such as time to answer complaints, issue a driver's license, complete housing and building inspections. Response-time data can be obtained from agency records, though agencies may need to modify their procedures in order to record the times of initial requests and times to completion of the needed work.

Recent technology developments, such as in big data, have led to considerable attention to the potential for data sharing among agencies. This can enable an agency to obtain outcome information on a regular, timely basis from other agencies that have outcome information on the first agency's customers. An example is the use by social service agencies that are providing prevention services to adults or juveniles at risk for criminal behavior. The social services agency seeks regular information from police departments or courts on clients' subsequent criminal behavior.

A major hurdle to obtaining agency record data from other agencies is the need to ensure the confidentiality of customer information. This will likely require negotiations and memos of understanding among sharing agencies.

Customer Surveys

Surveys enable public administrators to obtain systematic feedback on a number of outcome indicators for almost all public services. Customers are, by definition, the recipients of government services, and their ratings of these services should be of critical importance to public officials.⁵

Other forms of citizen feedback are also important to public officials, such as complaint counts (available through agency administrative records) and at public hearings and similar sessions. However, these sources are not likely to be sufficiently representative of the views of the total set of customers as can be obtained through properly done surveys. (Not everyone knows how to complain or is willing to complain.)

The use of surveys by federal, state, and local governments has increased considerably in recent years. Some states sponsor annual surveys (perhaps conducted by a state university). ICMA's Center for Performance Measurement since the late 1990s has been reporting annual comparison data on basic municipality service performance indicators for a number of city and county governments. As part of this effort, each participating local government uses a standardized questionnaire.

Surveys can seek information on all households in a community or only citizens or businesses that are customers of a particular service. They can obtain customer ratings of specific characteristics of service quality, such as timeliness, accessibility, adequacy, and the dignity with which specific services are provided. Surveys can provide other intermediate outcome data, such as changes in customer behaviors and practices (e.g., environmental protection behavior and improved personal health care practices). And they can obtain information on end outcomes, such as current employment status, current health, and satisfaction with recreation and library services.

Household surveys can provide information on the percentage of households that have not used particular public services or facilities (such as parks or public transit) and can also ask for reasons for nonuse. Agency records can sometimes provide counts of the number of uses of controlled-access facilities. However, such counts do not indicate how many different individuals or families used a facility or service, only how often it was used. Nor do agency records generally provide demographic information on nonusers or on reasons for nonuse.

Surveys can seek explanatory information. Questionnaires can ask respondents who gave poor ratings to a particular characteristic to identify why they gave that rating. Such information can provide public administrators valuable leads to needed improvements.

A major concern in the survey community has been technology advances in communication. Telephone surveys are finding it difficult to get people to respond, such as when people screen calls and use cell phones or smart phones rather than land lines.

The major constraint on wider regular use of surveys is their potential cost. This concern is driven by the belief that surveys need to be conducted by survey organizations, which can be quite expensive. For use in regularly tracking service quality, however, elaborate surveys are seldom likely to be needed. The cost in time and money can be alleviated by these steps:

• Limit the size of the questionnaire, perhaps to two pages. Those surveyed are more likely to respond to a short survey, especially if the agency asks the respondents for their help in improving the service.
• Use mail surveys, providing stamped, addressed return envelopes. For surveys of an agency's customers, agencies will often have, or can readily obtain, contact information on their customers.
• Consider web-based surveys. These are likely to become options as more and more persons have ready access. Surveys of businesses are likely to be readily administered by electronic means.
• For agencies with large numbers of customers, use random sampling. Samples of even one hundred customers or households, randomly selected, can be informative for a program if the issues being investigated do not require a high degree of precision. For example, a survey that receives a response from one hundred clients out of two hundred questionnaires mailed to a random sample of clients (a 50 percent return rate) is likely to be more accurate than one that gets one thousand responses from a mailing list of five thousand clients (a 20 percent return rate).
• Consider combining surveys sent to citizens that include questions on multiple services to spread survey costs among programs or agencies. However, surveys of customers of a single agency or program can provide more detailed information.

Inexpensive software programs are available to help with questionnaire construction and for tabulating and reporting the results of surveys. This means that information can be processed quickly and efficiently once the responses have been received. However, follow-ups to nonrespondents are likely to be needed to obtain an adequate response rate, say, 50 percent.

Trained Observer Ratings

Trained observer procedures use observers taught to use standardized rating procedures to rate physically observable characteristics of a service.⁶ A number of cities have used such ratings to track street cleanliness, led by New York City, which has been tracking street cleanliness since the 1970s. Such ratings have also been used to assess the physical conditions of road and bridges, traffic signs and signals, public buildings, and parks and playgrounds.

The purpose of the procedure is to provide reliable objective ratings so that different observers making ratings at different times would give approximately the same rating to observed conditions. The ratings might be made by employees (such as in New York City), volunteers, or student interns trained in the use of rating scales.

An attractive feature of such ratings of physical conditions is that they can identify in real time where physical problems exist and can even be linked to work orders for correcting those problems. Another feature is that the agency can map conditions showing the relative extent of deficiencies in various parts of its jurisdiction (e.g., by using various degrees of shading or coloring), as did the maps that New York City included for many years in the annual Mayor's Management Report. Agencies with geographic information system capability should be readily able to produce such exhibits.

Trained observer procedures have been used in some human services agencies to estimate the extent of improvement in the condition of specific clients (e.g., clients in developmental disability programs). Rating scales are established for various conditions that indicate a client's functioning levels relevant to the particular conditions of the customers being evaluated. The observers might rate a customer on each condition at intake and again at one or more points after the client has received services. The ratings for all clients are tabulated to determine the percentage of clients whose condition improved by various degrees.

Physical Measurements

Some governments use physical measurements to measure selected physical outcomes of some services, such as the quality of water after treatment by a water supply or wastewater treatment facility, air and water pollution levels, and road conditions. For example, state and local transportation agencies use a variety of devices attached to or carried in cars that record vertical displacement accurately, providing reliable measurements of road conditions.

Focus Groups

Focus groups do not provide statistical data for tracking service quality. However, they can be useful in providing information on what service quality characteristics are of concern to customers and, thus, what should be measured. Sessions are held with perhaps eight to twelve persons (such as customers) to obtain feedback about a service. Focus group participants might be asked two primary questions: What do you like about the service? What do you not like about it?

Focus groups can also be useful after data have been obtained to help the agency interpret why unexpected results occurred. This particular use of focus groups, however, appears to be rare.

Providing Timely Data

The timeliness of performance measurement information is critical to the usefulness of information. Performance measurement systems should be able to provide information on performance progress throughout the year. This will allow public administrators to identify problems, take timely action, and subsequently assess whether their actions have led to improvement.

In the early days of performance measurement, formal reporting from performance measurement systems tended to be infrequent, typically annually. Quarterly reporting occurred in some states such as Texas, where the quarterly reporting was required by the legislature. Starting about 2010, Congress and the Office of Management and Budget (OMB) began requiring federal agencies to report quarterly on “agency priority goals” but only annually on other goals.⁹ Federal agencies can, of course, report performance more frequently if only for internal use.

Managers make many program decisions throughout the year and need performance information that is as current as possible. They should be able to quickly pull the latest performance indicator data from their computers, smart phone, or other devices.

The frequency of data collection and reporting depends on the specific performance indicator. The values of some indicators might not change appreciably, or might only be needed, or might not be feasible to collect, more than once a year. Other indicators are most useful if measured weekly; others need less frequent measurement, such as monthly, quarterly, or annually. Data on street cleanliness, for example, might be provided weekly or monthly, as in New York City. Quarterly reporting of major performance information appears to be becoming a common performance reporting interval.

Some governments have recently begun conducting surveys of households within their jurisdictions on a regular basis to obtain feedback on the quality of their services. Most often these surveys are conducted annually or less frequently, not a timely schedule for program managers. To reduce this problem, a program can split its annual survey effort into parts. For example, a quarter of the total number of sample households might be surveyed every three months. Although the data obtained each quarter will yield less precise findings for the quarter, the increased frequency of feedback is likely to be considerably more useful to administrators and will identify seasonal differences.

For some services, a survey questionnaire can be sent to all of a program's customers, especially if their number is not large, rather than having to use random samples. This would enable managers to obtain up-to-data information more frequently.

Delayed delivery of data also is sometimes delayed for time to clean, process, and tabulate the data after all returns have been received. Technology advances should enable delay times to be significantly reduced.

Disaggregating Outcome Data

This enables public administrators to examine outcomes for different categories of customers and service situations. Performance measurement systems suffer greatly from “aggregationitis.” Performance measurement systems typically provide only aggregate information on their performance indicators, especially their outcome indicators. Likely to be considerably more useful to managers is disaggregated performance information, that is, information broken out into relevant groupings. Breakouts, especially of outcome data, can provide public administrators and other officials with considerably more specific information on where their programs are working, where not, and under what conditions. This information can provide important clues on where problems exist and the corrective action needed. It can also help identify inequities.

Public administrators are likely to find particularly useful the following breakouts of outcome indicator values:

• Various customer demographic groups, such as by age, income, race/ethnicity, educational level, and household composition. Such breakouts can be important for assessing equity concerns.
• Geographical categories, such as outcome data broken out by region, state, county, city, service district, neighborhood, census tract, and so on.
• Service providers. It is likely to be very helpful it outcome information on their own customers (as well as aggregate data across units) is available to each unit's manager and supervisor. This applies to such individual service units as each facility, each program office, and so on. Outcome indicators can also be broken out by individual case workers. Another application is to prove outcome information on each contractor.
• Type and amount of service provided to individual customers. Program administrators should find it highly useful to break out some outcome indicators by mode of service delivery. This should be particularly helpful to administrators who want to be innovative and try different approaches to service delivery. A program might be applying different amounts or different types of assistance to different parts of its workload. If the program records which approaches are used for each element of the workload, a computer can then readily tabulate the performance data for each approach. Even better, to provide more convincing evidence that the approach with the best outcomes caused the outcome, the program would assign each different service delivery procedure randomly to each customer.
• Client difficulty. Another breakout that is often important is by level of difficulty of the incoming workload (whether the workload consists of human customers, roads, bodies of water, or something else). Differences in the difficulty of a program's workload can have a major impact on the effectiveness, and costs, of individual programs and services. If level of difficulty is not explicitly considered, the performance data can be misleading.

Differences in difficulty of the incoming workload can directly affect outcomes in most, if not all, services and programs. For example, in comparing the percentage of successful outcomes from one year to the next, the percentages may actually have been better for both difficult-to-help and easy-to-help patients, but aggregate performance data might indicate a worsening of the aggregated outcomes of both patient categories. The breakout by difficulty provides public administrators with a much different story than if only the aggregate outcome was considered. This can occur because a larger proportion of difficult-to-help clients came in for help during that period, causing the aggregate data to look bad. A public manager would likely come to the wrong conclusion if he or she considered only the aggregate success rate. A similar problem will occur if comparisons across service units are made. Different units will likely have larger proportions of difficult-to-help clients than others.

A classic example is the percentage of children available for adoption for whom adoptions occurred. Adoption agencies have generally found it considerably easier to place healthy white babies than older minority children or children with physical or mental problems. Adoption data can be broken out for each relevant demographic group in order to provide a more accurate picture of placement performance. In this example, customer demographic characteristics are used as proxies for difficulty.

Public agencies might categorize their incoming workloads by a small number of different levels of difficulty (perhaps three or four levels). At, or shortly after, intake, supervisors might examine the information on each incoming customer, assign a difficulty category to the customer, and enter it into a computer for subsequent calculations as to the outcomes for each difficulty category.

The use of risk-adjusted indicator values, such as of mortality rates in hospitals, is a version of difficulty-related measurement. Here, rather than grouping customers into qualitatively determined difficulty categories, difficulty levels are calculated statistically using more sophisticated approach to measuring difficulty.

These breakout categories apply to services that directly serve humans and those that do not. Road maintenance programs might break out outcomes: road composition, location, average daily traffic categories, amount of truck traffic, urban versus rural roads, and regions with different weather conditions.

To obtain breakout information on outcomes, the program needs to enter the appropriate characteristics into the database for each workload unit (such as each customer) and link that information to the source that contains the outcome information on each of those customers. The computer then makes such calculations as the number and percentage of customers in each breakout category with each outcome level.

Deeper drilling down can also be done. A manager might want more targeted information, such as identification of the outcomes of women of a specific ethnicity and age group who lived in a specific neighborhood as compared to men.

Formal performance reporting will likely require a considerably limited set of outcome information, such as for an agency's dashboard. However, in the near future, many managers are likely to have access from their own electronic devices to be able to query the device through drop-down menus and hyperlinks to pull up a wide variety of disaggregated outcome information.

Regularly Seeking Explanations for Unexpectedly Poor or Unexpectedly Good Outcomes

Many outside factors, in addition to a program's own efforts, can affect the outcomes sought by a program, factors sometimes called contextual factors.

Few public agencies have required substantive explanatory information as a regular part of performance reporting, especially for unexpectedly poor and unexpectedly good, outcomes. This information can be useful to program managers in providing clues as to why the measured outcomes have occurred and what might be done. Such an examination should be considered part of a modern performance measurement system today, even if resources permit only a small effort.

While asking for explanations is common, this is usually done on an informal basis. Requiring explanations for unusual performance levels can alert programs and their staffs to focus routinely on this key purpose of performance measurement of raising questions when unusual results occur.

Texas, for example, requires agencies in their quarterly performance reports to provide “explanations when actual performance of key measures varies five percent or more from targeted performance.”¹⁰ Few other governments at any level have such formal requirements.

Explanatory information might be obtained from examining:

• Context factors such as special economic or weather data
• Information from customer (or field employee) focus groups
• The mix of customers to assess whether a larger proportion of more difficult-to-help customers had entered the service
• Findings from open-ended questions contained in customer surveys that ask respondents to indicate why they had given poor ratings to any of the rating questions. More in-depth studies such as program evaluations (though these are likely to require additional resources and are likely to be appropriate only for programs with large budgets and particularly important consequences).

Only the last of the above sources is likely to involve significant cost or specialized help.

Providing Basic Analysis of the Information

The data in regular performance reports too often have been little used other than as an accountability tool (such as for the organization's website and public consumption). The increasing pressure for evidence-based programs and policies has been growing, in part because of pressure from OMB.¹¹

Performance measurement systems should provide at least some basic analysis of the data, even if only by asking someone to summarize each report and identify highlights. Such summaries enable managers to focus on problem areas and potential needed actions.

In the future, most public agencies of almost any size are likely to be able to identify one or more persons to act as analysts. Increasingly, young professionals coming from schools of public administration and public policy have gained knowledge of basic analytical (and computer) tools. In addition to preparing the summaries and highlights, they can do a search for explanations for unexpected levels of good or poor outcomes.

Key questions for public administrators reviewing performance data are whether the current level of performance is good or bad, whether performance is improving or worsening and to what extent, and why unexpected outcomes have occurred. Administrators typically compare the current level of performance to benchmarks. Following are some comparisons that managers are likely to find useful when considering the latest values for each indicator:

• Values for one or more past reporting periods. Examining multiple past reporting periods can identify trends that may help managers obtain a better perspective than only comparing the data for one previous period. When values are reported frequently during the year, this can identify seasonal effects.
• Targets. The federal Government Performance and Results Act of 1993 requires agencies to set annual targets (“goals”) for each of its performance indicators for the coming year. Many states and local governments also set targets, if only as part of their annual budget process. For internal management purposes to enable regular midyear reviews, targets would be provided for segments of the year, such as quarterly or monthly.
• Data disaggregated for different demographic and service groups. Such comparisons are typically done only as special studies or on an as-needed basis. However, disaggregations examined on a regular basis as part of the performance measurement system can be highly useful in helping pinpoint for whom and where poorer outcomes are occurring and for identifying equity issues.
• Data disaggregated by work groups. Service outcomes and efficiency might be compared among different police, fire, sanitation, and road maintenance districts. Or comparison can be made among any type of facility where the agency has more than one such facility, such as mental health clinics or parks and recreation facilities, post offices, or correctional facilities.
• Similar jurisdictions. This approach is useful if reasonably comparable performance indicator data are available for reasonably similar communities and customers. Examples of outcome data for which comparisons are generally available are crime rates and traffic accident rates. Reasonable comparisons are most often appropriate for programs for which the federal or state government has supported development of common definitions and common data collection procedures for the performance indicators.

Tracking Postservice Outcomes

This need applies to any service where a major desired outcome for the person is expected to be sustained after completion of the program's services. The needed outcome indicator would look something like: “Number, or percent, of customers who X months after completing service had the desired improved condition.” For example, social service and health programs often provide treatment but do not follow up afterward to see whether the customer is still maintaining the benefits. If the benefits are not present, say, twelve months later, the service could be a waste of resources and should be reviewed. If customer surveys are used to obtain this information, respondents could also be asked for their ratings as to the extent to which the service had contributed to the improvement and be asked for suggestions for improving the service.

In some program areas, such follow-up has been required by federal agencies, such as in some employment and substance abuse treatment programs.

The primary barrier to following up former customers is the expected data collection cost and effort. This legitimate concern needs to be balanced against the potential value of the information for avoiding spending future resources on unsuccessful services and the ability to help identify improvements to the service. Following are some ways to reduce after-exit survey costs:

• Link the follow-up to an after-care checkup to find out not only how the former customer is doing but also whether the customer needs further help.
• Obtain contact information from the customer before exit.
• Use electronic surveys for customers with such access. Otherwise use the postal service mail.
• Obtain outcome data from other agencies rather than a survey. This is becoming more feasible because of the ability of computer technology to handle big data. A common example is seeking state unemployment insurance data to obtain postservice employment information for employment programs.

Reporting Performance Information Well

Modern data visualization technology has ramped up the ability to easily report performance data in visually attractive ways. The use of color, bar charts, and so on can attract readers to key data information. Dangers include the all-too-present temptation to overdo it, such as providing overly crowded visuals, cryptic labels, overly small font sizes, and coloring that makes key information hard to read.¹²