Working a Mission-Critical Problem in a Federal Agency
Jac Fitz-enz
A federal agency that I will identify as Research Economic Services (RES) has been chartered by the U.S. Congress to conduct what industry would call “competitor intelligence.” This phrase implies that the agency should gather data on foreign countries that include economic, military, political, cultural, and other factors. The data are then formatted for a variety of purposes and distributed to many federal departments and agencies.
RES has been criticized in recent years for the quality, timeliness, and cost of its services. It has undergone studies by the General Accounting Office, Inspector General, House Select Committees, and outside consultancies, all of which were looking for solutions to these problems. In the fall of 2008, Human Capital Source (HCS) met with representatives of RES to discuss how the principles of predictive management (our HCM:21 model) might be applied to this situation.
The Human Resources Systems
RES is staffed by a combination of civilian and military personnel. It also contracts with outside research service providers. Civilian and military systems of RES personnel are dissimilar in several ways that make performance measurement a crippling problem. For instance, pay plans are different for the civilian and military personnel. Personnel development operates on two different principles, in that military personnel receive a great deal of formal training whereas civilian personnel more likely grow through on-the-job experience, or what some call “scar tissue.” They do receive technical training.
In addition, there is no objective measure of organizational success or failure. The absence of a dependent variable makes standard statistical analysis extremely difficult, if not impossible. During our background research for this assignment, we were told that some RES projects succeeded or failed for reasons outside of its personnel’s control. This was not a face-saving excuse; it turned out to be true in a number of cases. The bottom line was that the problems could not be solved solely through changes in HR systems or even management’s behavior. The history of RES’s formation had saddled it with inherent structural inhibitors that could be cured only through political means. Nevertheless, we believed that some improvements were possible if we could find a way to set a quasi-dependent variable and correlate the behaviors and processes with it.
Project Manager as Mission Critical
The most mission-critical position in RES is that of project manager (PM). Typically, projects are multiyear in length and involve use of advanced technology, some of which must be developed along the way. To make matters worse, PMs do not always see a project through from beginning to end. This is particularly true of the military cadre, whose assignments usually are no more than four years before they return to their service parent.
Owing to the complexity of a project’s communication, employee commitment, ongoing engagement, personnel development, and retention are all conundrums. One of the previous studies identified organizational structure as an inhibitor. The HR function was charged with improving its systems to offset at least some of the organizational restraints.
The challenge was to find specific actions—that is, policy changes, training and development, interventions, performance management systems—that could be linked to project improvements. Absent quantitative objectives, we had to specify some visible, measurable antecedents, which hypothetically would drive improvements.
Creating a Target
Analytic work normally requires a goal or target, or a dependent variable against which one can relate many interactive variables. For example, it is expected that successful performance is measureable in terms of individual or group actions. When there are no past or current quantitative performance targets, it is difficult, if not impossible, to link action to result at the project level. Lacking that target, what can be done to assess success?
As a starting point, our HCS team decided to create a surrogate dependent variable that we labeled “organizational effectiveness” (OE). It was hypothesized that if we were able to link the interactive variables to certain objective points within OE we could at least move forward with concrete recommendations. The OE target was defined as follows: A successful project (SP) is a function of organizational effectiveness. In turn, OE results from the interaction of facilitating (F) and inhibiting (I) organizational, technological, and human (individual) variables.
Specifically, the key terms were:
Organization (Org): Processes, systems, policies, and structures related to project manager talent management
Technology (Tech): Project manager knowledge and ability to manage state-of-the-art investigative technology
Human (Hum): Attraction, deployment, compensation, development, and retention of talent
Expressed as an equation, we had:
SP = f (OE)
OE = f [(OrgF + OrgI) + (TechF + TechI) + (HumF + HumI)]
The Analytic Process
The HCS staff met first with the HR managers to be briefed in detail. From these meetings in early 2009 a study plan was designed. It was to consist of a data-gathering phase that started with personal interviews of all key management personnel, including the director and deputy director and outside individuals who had past personal experience with RES. This entailed nearly thirty initial contacts, with some follow-up clarifying discussions.
Based on the data from the interviews, HCS designed a survey instrument that would be given to and responded by approximately 250 management personnel in the agency. Also, a document review was done of studies and reports on RES from the past ten years, and this review included forty-two documents of considerable length. When stacked one upon another, the pile was nearly a foot high. After all data collection was completed, we applied statistical procedures to discern the meanings and relationships between interview and survey data. These results were compared to data gleaned from the document review.
The lack of a dependent variable made it impossible to carry out normal quantitative predictive analysis at the individual level. Using standard statistical programs, we could not tie independent variables such as training, tenure, parent organization, education, performance management, or several other factors to a successful project. Nevertheless, we were able to find strong correlations between the beliefs of various knowledgeable groups on personal characteristics, training, and on-the-job experiences of those judged to be successful PMs. We cross-correlated the views from military and civilian personnel and from individuals at three levels of management, as shown in Figure 10.1.
Survey results showed a high level of belief that success as a PM depended on three competencies:
Leadership
Integrity
Decisiveness
On the matter of work experience, on-the-job experience—working up from small to large projects—was deemed to be the best predictor of success. Formal organizational training was viewed as useful but not critical. This could be, in part, because there is no discernable career path to PM. Mentoring was also seen as useful but, for the same reason, not as critical. Responses were consistent also across gender, age, education, and total experience (inside and outside RES). When we ran a three-cluster solution across the categories of organization, supervisory level, and RES experience, the results were the same.
What this told us was that the opinions across groups were similar. This was a surprise to management, however. Because of the outside criticism that RES had received, there was a feeling that the organization was breaking apart. Yet while there were certainly differences on some topics in various groups, we found more similarity than difference.
In total, the process took six months. A report was submitted, and subsequently a briefing was delivered to top management.
Figure 10.1. Agency and supervisory-level comparisons.
Findings and Recommendations
In order to predict with precision what makes a successful PM, any HR operation needs dependent variables at the individual, group, and organizational level, as mentioned earlier. This means that RES needed to develop a measurement system that functions at three levels: strategic, operational, and future focused. Specifically, it must:
Clearly articulate strategic imperatives, link operating objects to those strategic imperatives, and develop a set of leading indicators to keep the organization focused on emerging trends and requirements.
Allow for quantitative analysis of individual performance.
The outcome measures to accomplish this needed to be both short-and long-term. Short-term measures would require a dedicated effort to determine the conditions, goals, and dependencies of success in selected projects. The measures would identify fundamental processes, stages, and outcomes. Long-term measures would create a composite variable for a successful project: ratings of on-time, on-budget, and on-spec. Each rating would be weighted and combined to form a successful project state. A composite variable would then be assigned to each individual working on a project.
Conclusions and Lessons Learned
At the point of this writing, the RES management has not decided what to do with the findings. As mentioned, politics play a major role in this case. Persons in superior positions have not determined the long-term fate, design, or role of the agency.
As for HR situations in which dependent variables do not exist and there are apparent differences of opinion, there is, nonetheless, no need to concede that improvements cannot be identified and implemented. As General George Patton said, “A good plan today is better than a perfect plan tomorrow.” “Analysis paralysis” has killed many a good idea. There are seldom as many valid data as one would like for making decisions. Nevertheless, as the saying goes, when confronted with lemons, make lemonade. That is, surrogate dependent variables can be constructed and tested. Where there is a will to change and improve, analytic techniques—some standard and some innovative—can be applied to advantage. The goal is not extreme precision but, rather, improved operations.