Case Study: Estimation as a Tool

Directions

You can read the case independently of the material from the chapter. If you choose to read the case study first, then read it once quickly in “story-reading mode” simply to set the context for the information in the chapter. Then read the chapter material thoroughly, and finally reread the case study in an analysis mode.

Case Summary

Description

A software project manager must make an estimate for a job and figure out what issues are influencing the estimate. He must figure out what are good inputs into the process so as to make a good estimate.

Case Study Objectives

After reading the case study, you will be able to do the following:

• Understand examples of some estimation techniques

• Understand that there are different phases in which to estimate and how estimation during the planning phase might be improved

• Understand how some estimation techniques might be set up to better support an organization

Subjects Covered

Use of historical data to improve estimation, overall metrics in estimation, understanding metrics versus data issues, and how to look for similarity in projects

Setting

This case will provide methods for estimating with minimal data, for obtaining base data that is usable in a software development environment, and for quickly collecting data and converting it into metrics.

Terminology

Junior engineer: Engineers with less than five years of experience

Metrics: A relative understanding of how data can be used to evaluate some activities

Productivity numbers: A measure of how much project work the company gets

Senior engineer: Engineers with more than five years of experience

The Case

Scenario

Marvin Saymore was being asked to estimate his first full project after starting his new job with Transad. He had to give his boss an estimate, and he was not sure what the primary estimation issues on his projects were. The question was whether estimates were driven by events that were unavoidable or were because of mistakes in estimating past projects. He began to reflect on what he had learned back in school about estimation and what might influence the Washington Transit Authority (WTA) project.

Marvin thought that when people in the company tried to estimate, they tried, in some sense, to predict the future. Marvin was not sure whether this was really true as the new WTA project was being estimated. He thought that estimates were more based on the past and that they were a best guess of what might happen, not a prediction. With that “best guess” came some risk and some probability of success or failure. He had been taught that humans are creatures of habit and tend to do things more or less the same unless a major change occurs in the environment. In a sense, Transad had more or less been the same for the last 100 years of development, so change was not necessarily a big issue for the managers. So, based on this understanding, history should be a good predictor of future activities.

However, Marvin was still left with deciding what past, or, more accurately, what data from the past, he should use. There always seems to be much data, but if you do not know what the data means or represents, then it is hard to decide how to use it. For his purpose, Marvin would accept that data used to follow the progress of a project were valid measurements. Making his decisions more difficult on how to estimate the project was that Marvin had been taught to use more than one way of estimating. This, he believed, would allow him to compare estimates and see which ones seemed more reasonable and in some ways see whether he was aware of all the major issues on the project. Dramatically different estimates for the same project using different techniques might indicate hidden issues or future problems. Marvin realized that to understand the history for projects, he would have to understand the metrics the organization collected on projects in the past.

Marvin first discovered, as he took over the reins as the new software manager at Transad, that metrics were fine for other people but not necessarily for his group. He found out that a number of metric efforts had been proposed, but none had been maintained for a long period of time or were used to predict future performance. The company almost exclusively depended on the expert judgment of the senior engineers and the senior managers in the company for estimates. This would have probably been all right if historical data had been used independently to validate the estimates, but this was never done. In addition, the historical data that was available was mainly “bean counter” data, that is, number of hours on the project, hours preproduction, hours postproduction, and repair hours in the field. The preproduction did not specifically break out any software events, only how many hours were used.

One of the first issues Marvin had to resolve was whether the “bean counter” data was usable in some way. He needed to develop a way to obtain accurate data, which he could use to compare with the historical data he had available. He considered this accurate data, which was once related to his projects, as “metrics.” He used a simple technique over a short period of time that would enable him to understand clearly how each type of engineer spent his time. Once he had enough of this information, he might then be able to use it to validate the overall estimates on the project. Marvin wanted to see whether he could establish a correlation between the bean counter data and real data on his development projects. Table 3-4 gives the resulting data that Marvin obtained.

Table 3-4: Engineer Project Hours for Currently Operating Projects at Transad

What immediately became apparent is that accounting for all the time the engineer spent was something that would take a little work. Next, it was also apparent that there was a large difference between how much time the senior engineers and the junior engineers were getting to work on what people considered “real” project work. There seemed to be an inverse relationship between an engineer’s experience and the length of time that the engineer was allowed to work on a project. Marvin wondered whether this interpretation of the data was correct or whether there was another reason that the experienced engineers spent less time on real project work.

Without productivity numbers, these values may or may not have a major impact on the estimates, but Marvin’s gut feeling was that something was very much out of “whack.”

Just Like the Last One

Next, Marvin began to examine the company concept of “sameness” that was used by engineers to evaluate requirements. A number of recent projects had been bid to customers as being exactly the same with “minor” changes. As Marvin discussed this with his engineers, he found that this sameness was not quite the same for everyone involved. He decided to conduct a direct requirement-by-requirement examination of a new project compared with the “base” project, which was being changed. Requirements were directly compared by one of his senior engineers, and any differences other than syntax were noted. There was also an attempt to evaluate how different the requirement was from the original baseline project.

On one of the “same” projects, about half the original requirements had been modified, and about half of these modifications involved significant or moderate changes. In the eyes of management, this indicated a “same” project.

Significant Events

One other piece of information that Marvin began to put together for his projects was the idea of whether there might be some significant events during projects that were contributing to a large number of hours. In other words, Marvin was looking for the 80/20 effect on his project estimates. As he reviewed the data, he found that whenever the hardware group proposed a new hardware configuration for a project solution, there were a certain number of hours needed to integrate the software and new hardware and to integrate the new hardware into the system. Table 3-5 has a number of the data points that Marvin was able to develop from the historical data.

Table 3-5: Group Hours Spent on New Hardware Implementations for Transad Projects

Marvin now felt armed with a number of new elements to help with his estimation efforts. He planned to update his estimation process so that he could incorporate his new knowledge. He had to figure out how best to explain to his boss that he had a new tool to better estimate the WTA project. He thought that by using the information in Table 3-5 he could produce at least two if not three more techniques to help the expert judgment that had been used by Transad to estimate projects in the past. Now all he had to do was convince Enrique, his boss, that this was the right thing to do.

Case Problem

Marvin encounters the following problems in the case study:

• Are the decisions to use the estimation techniques or the information that Marvin has at his disposal effective?

• What issues are most affecting the estimation decisions Marvin has to make?

Case Analysis

Analysis Activity

Analysis Questions

1. What are the major decisions that the manager must make to help him with his estimation problem?

2. What would you have done differently if you were in the manager’s place?

3. What estimation techniques from the case can you identify that are being used? Were the decisions for these techniques supported in the case?

4. What information about estimation did you gather that you did not already know?

5. What are the decisions that Marvin needs to make?

• What are the PEAK inputs for these decisions?

• What are the alternative solutions and risks associated with these decisions?

Conclusions

As you look at the decisions that were made concerning estimates in the case, one of the key concepts is that your best estimates must be based in accurate metrics. To ensure accurate metrics, you must make sure to verify that the data feeding the metrics is valid. Although decisions concerning estimates can be evaluated with PEAK, the knowledge can be corrupted if the metrics being used are not correct. “Garbage in, garbage out” is a well-understood concept. But garbage can be hidden in metrics very easily, and the impact on estimation along with the consequences is severe.

Key Ideas to Remember

The PEAK model can help clarify the inputs and show that a decision based on faulty input data is also likely to be faulty.

The decision to put new functionality in hardware rather than in software is not the best decision because it is based on faulty input data. The experience and the assumptions are flawed in the decision:

(P) Problem: New functionality is needed.

(E) Experience: It’s easy to add hardware to the system.

(A) Assumptions: It’s easier and less costly to add hardware than to add software to a system.

(K) Knowledge: Basically, for a hardware company, software is harder to manage.

Solution: Add a new hardware component to the system to meet the functionality.

Assumed Risk: The cost to get software running on new hardware is not well understood and may be greater than expected. The recent cost model shows the cost to be about 5,000 hours to integrate software with new hardware.

The estimation decision to split the engineers’ time also needs to be reviewed. The information being used shows all time being equal, but the concept of context switching is ignored. Machines can switch jobs back and forth, but humans cannot easily do this. Application of the PEAK model reveals the flawed assumptions associated with this estimation decision:

(P) Problem: Projects are late because engineers are overloaded.

(E) Experience: The company has many projects all in different phases of development.

(A) Assumptions: Engineers can support more than one project without much impact.

(K) Knowledge: Senior engineers will most quickly resolve problems with products in the field.

Solution: Split the engineers’ time as needed to solve problems.

Assumed Risk: Junior and senior engineers may not actually get the same amount of time on a project, and therefore using senior engineers to solve field problems may be an issue.

The PEAK model helps you evaluate the estimation decisions that were made and look for where problems with the inputs can contribute to issues with the solutions.

Case Study: When a Team Runs a Race

Directions

You can read this case independently of the material from the chapter. If you choose to read the case study first, then read it once quickly in “story-reading mode” simply to set the context for the information in the chapter. Then read the chapter material thoroughly, and finally reread the case study in an analysis mode.

Case Summary

Description

The manager Halley realizes that a project is going to be late well in advance of the date when the project is due. He has to figure out whether the data is accurate enough to make his claim, what he can do about the problem, and what made it happen. He wants to understand the decisions that were made that created the problem in the first place so that he can improve his decision making in the future.

Case Study Objectives

After reading the case study, you will be able to do the following:

• Identify a number of estimation techniques used in this case study. You will also be able to state whether the techniques were used correctly or incorrectly and what the people in the case study did wrong.

• Understand how to identify “the haze” of data that normally surrounds a project and what you have to do to get the proper information for correct estimates.

• Provide mechanisms to build your case for improving estimation within your own organization.

Subjects Covered

Estimation bias, perception, and delivery schedule

Setting

A software project manager cannot figure out why his projects seem to be off schedule. He is being asked to provide status but thinks he does not understand the key issues that govern his estimates. He has to do a better job of explaining his estimates.

Key Concepts

Many factors influence the decisions around estimates:

• Looking at the problem from multiple directions can help with understanding.

• The problems are seldom easy, or they would have long ago been solved.

• First impressions seldom provide you with enough information for in-depth analysis.

Terminology

Loaded cost: The organization cost per unit of time, which includes not only salary but also infrastructure support for engineers, such as benefits, building, and environment

TC: Train control

WTA: Washington Transit Authority

The Case

Scenario

When a relay team runs a race, the last team member always seems to get the blame for the team being late. It does not matter that everyone before him dropped the baton or took their time. There is an expectation that the last person, the anchor, can make up for anything. In the development of complex integrated systems, this last “person” usually seems to be software.

Halley Anderson had a tough day ahead. He knew that today he was going to have to tell his manager that the Washington Transit Authority (WTA) project was going to be late. This might sound unusual because the project was not due for another six months, but nonetheless it was going to be late.

This all started back when the WTA project began. Halley had done a detailed estimate of the project after the Train Control (TC) group had received a notice to proceed on the contract. He wanted to make sure that the proper resources were assigned and that the proper risks were observed in the project. Halley had used a number of techniques to estimate the cost of this project, but the assumptions made in one of them were now being reviewed.

The estimation had accounted for the fact that every time new hardware was used on a project, there appeared to be a fixed number of engineering hours needed to get the software operational on the hardware. This additional time almost seemed like a constant and had to be verified a number of times by Halley, but he started to believe it was not only the software but the time to get the software installed and working in these new hardware environments that made this development time constant. In other words, Halley thought the problem was with the hardware group.

The hardware group was where Halley’s boss, Enrique Wattsman, had come from. Enrique had a “soft spot” for the hardware group. How could Halley blame the hardware group for the problem when Enrique’s view of software was put to him as follows?

Enrique said, “Here is what I view about software,” while drawing on the board (Figure 3-2).

Figure 3-2: Halley’s boss’s view of software development

Halley replied, “But at least we might want to break it down into design and then implementation (code) and test. Design is the creation of the solution.”

Enrique then said, “Well, I guess we could add that, but the percentages would stay the same.”

This left Halley with a problem and maybe a solution. Enrique had the view that software was “easy” and not related to the other things that the company did. For example, Enrique did not say that requirements followed availability of hardware. He assumed they were independent. Was this an opening for Halley to explain to Enrique the relationship, or was this just another “thing” the last runner (software) would have to overcome?

Halley had calculated that it would take about 5,000 engineering hours to get software operational on a new hardware board. He had used this information for the WTA estimate but had not made his manager aware of this number. It was somewhat scary to tell Enrique that every time his engineers decide to add a piece of hardware, it cost the company $500,000 dollars. (The loaded cost for engineers at Transad was about $100 per hour.)

The question was how he could make his manager and everyone else aware of this so that basically the “last runner” was not blamed again for always being late.

Halley had made friends with the head of the hardware group. Calvin Smith was nice but knew his department was contributing to some of the problems that recent projects had experienced at Transad. He always had engineers in the field debugging problems and always seemed to be in Enrique’s office explaining why this or that was late. His engineers were overworked, and he could not see any clear way out of his problem. Calvin’s problems and Halley’s problems were related. If the hardware were late, the software would be late.

One issue in particular bothered both managers. They both understood that they had about 11 months to develop the system; however, a couple of months before the end of that period, if they needed to make changes, it became more and more difficult. Although the number of changes usually decreased during the life of the project as the product was being integrated and finalized, changes could become more frequent. When the engineers understood the most about the product, they were being allowed less and less opportunity to implement changes that would improve the product.

Halley and Calvin came in on a weekend and decided to look at the data. A diagram that Halley and Calvin put together made the problem immediately clear (Figure 3-3). Although technically the managers were given a certain amount of time to produce their solutions, in reality the time was much shorter. This was because the time needed by manufacturing overlapped with the time needed for development. This overlap had probably started occurring in response to the company needing to win more projects by shortening the overall time to deliver a product to the customer. Although on paper this looked good and probably meant that overall development schedules could be compressed, the reality was that new development time overall had been cut from the understood 10 to 11 months down to 8 to 9 months. This began to explain why projects were coming in later and later at Transad. However, of course, the bidding was showing the same amount of time as before, and the costs were holding—to some extent.

Figure 3-3: Manufacturing and development relationship

The need to overlap manufacturing with development was a logistical need to meet the demands of new contracts. If you have to build a thousand units and it takes you a fixed amount of time to build each unit, then divide the number of units by the time and back up the calendar. Transad was still thinking in a mode that is common to many hardware manufacturing companies rather than with an awareness that hardware depends on software as well as on creative processes such as hardware and software design.

Calvin now understood why he was in so much trouble. His normal lead time on parts and development of boards was longer than the time he could make changes. He was expected to deliver new hardware before he could have it fully tested. Halley used this opening to let Calvin in on a secret, the 5,000-hour cost. If Calvin was having trouble getting parts and if when he finally did get the components built it would take another 5,000 engineering hours to load the software, project delay was inevitable.

Halley suggested that the problem was that the system had to be product-lined. The company could not afford, except for highly specialized conditions, to add new hardware. The cost would be too high to get the new component in place. The specifications would have to be available and the interfaces would have to be defined by the second or third month of the project for the TC group to be able to make the new board work. Two to three months seemed too short for the hardware group. Even with this understanding, there was still a problem in making this newly discovered internal deadline of an eight- or nine-month delivery to meet the manufacturing issues. Halley would have to assign approximately five or six engineers to work on the software for any new board.

In some ways, Transad had limited its options. If Halley could convince the systems engineers to use “off-the-shelf” hardware, then projects could begin to be on time. It was too late for the WTA project, but Halley would insist on the change for the future.

The WTA project had a new component in the form of a board the company had never made before. It was called the Brake Feedback Monitor (BFM). The device might make a nice product-line element in the future, but on this project, it created a deadline that Halley needed the hardware group to meet.

At the kickoff meeting, Halley announced the need for the new BFM hardware by March 27. At the meeting, this date was some three months away.

Halley said, “In the planning phase of WTA, it has become apparent that this new board is going to be difficult to handle. Historically, it has taken a lot of time to install software on new hardware, so can we make sure to get the boards we need for test, by March 27?”

Calvin quickly replied, “Yeah, sure, no problem.”

Enrique said, “Calvin, let’s make sure. Your guys have not been doing very well lately.”

Calvin responded, “What are you talking about? I have three guys in the field right now. We are handling more than our share of the work. They are doing pretty damn good in my eyes . . . .”

This discussion between Calvin and Enrique was typical. It came to a head when one evening after most of the engineers had left. Enrique came into Halley’s office and asked, “One year! How can it take someone one year and still not get it done?”

Halley replied, “I am not sure I understand.”

Enrique then said, “Calvin’s guys have been working on a board for the Fusser project, and they have had an action item to fix it for a year. They know what the solution is but have not worked on it, and it is due tomorrow. So, a year after we tell the customer that the problem will be fixed, I am going to have to tell them, “A little longer, please.’ A year . . . a whole year!”

Halley realized how all the projects in the company were to some extent interrelated, and this meant that problems on one project could impact development on another project.

Shortly after this, Calvin left the company. He did not think he could deal with the hardware group problems logically or in an objective fashion anymore.

In his last conversation with Calvin, Halley said, “Calvin, I understand you are leaving. I am sorry to see you go.”

Calvin replied, “Save your self, man; this place is never going to get better.”

The important day had come. How was Halley supposed to approach his boss and tell him that today, March 28, represented a one-day slip in the WTA schedule? And the day after that would represent another day’s slip. As of today, the WTA project was going to be late. Halley thought about the famous Fred Brooks quote, “How does a project become one year late? ...One day at a time” (Brooks 1995, 153).

Case Problem

Halley encounters the following issues and decisions related to project estimates:

• What are the major issues impacting the WTA project that may cause Halley’s decisions concerning estimates to be wrong?

• What decisions might Halley have made to try to deal with the impacts to his estimates when software is embedded in a larger hardware system?

Results

The hardware for the WTA project was delivered a day late. The project was able to get the software into production roughly on time, after all the issues, about a week late because of other issues also found on the project. The project overall was about 14 days late because of a subcontractor on a subcomponent that interfaced with the main motor control system. In general, the estimation decisions were some of the best seen on any new project at the company in some time.

Case Analysis

Analysis Activity

Analysis Background

First Case Problem

(P) Problem: What does Halley have to do to ensure that the project is not late?

(E) Experience: The Transad standard development process was in place, and they were in the middle of an 11-month effort.

(A) Assumptions: Everything will arrive on schedule because even if the hardware is delivered late, they can make up time in software.

(K) Knowledge: Because software is waiting on the new hardware, software appears to be running late “as usual.”

Solution: They need to work harder in software.

Assumed Risk: They may not be able to make up time in software and get the project back on schedule.

Second Case Problem

(P) Problem: What can Halley do to minimize the impact to his estimates when the software is embedded in a larger hardware system?

(E) Experience: Halley understands the Transad production environment for hardware or software integration.

(A) Assumptions: Software is just like hardware and does not cost as much as the hardware to change.

(K) Knowledge: Hardware takes a certain amount of time to manufacture. Halley knows the 5,000-hour cost to integrate hardware and software.

Solution: They can build the hardware and fit in the software as needed.

Assumed Risk: Software problems, especially those associated with integration or the need to change, may affect production schedules. The estimates may not account for this possibility.

Analysis Questions

1. Why is Halley concerned about the WTA project? What key issues should Halley be aware of that would help him understand the issues that affect his estimates?

2. Whose fault was the one-year failure to deliver the hardware solutions, Calvin (the head of hardware) or Enrique (the head of the division)? Why?

3. What makes it one or the other’s fault, or would you blame it on someone or something else?

Conclusions

As you examine the assumptions that you have to make to produce estimates, you need to be aware of when assumptions are directly in opposition to the solutions or options produced by your decisions. You want to minimize the assumptions that increase the risks to the solutions or verify the assumptions and convert them into knowledge. The chapter discussion shows that assumptions are the largest generator of bad decisions. You are forced to make the assumptions because sometimes you don’t have the needed data until later in the project. Although, for estimation, the PEAK model will mostly show that assumptions are a problem, other factors such as the availability and accuracy of knowledge are also issues.

An example might be the one-year problem in the case. The risk was that free time might not be available, but the assumption was that free time would be available sometime within a year. When the original estimate of one year was put forth, everyone assumed that the time would become available. No one verified this assumption, and the knowledge that this issue was pending was probably not provided to the engineers who had to work on it. The project went forward with an estimate decision based on an assumption that may not hold.

The main principle that Halley learned that would influence his decisions about estimates in the future is that some costs, such as the 5,000 hours needed to integrate software into new hardware, are fixed. If these fixed costs were better understood, his estimation process could be improved. Halley increased his knowledge for future estimate decisions. He also learned that his knowledge about the production process, actual costs, and estimation needed to be made available to everyone in the organization if the group was going to be successful.

Key Ideas to Remember

At first glance, it seems that the problem for many project managers is that software is always late. As the title of the case indicates, the software people “run the anchor leg” of the project race. They usually handle the integration of the product and, in doing so, are the last people to work on it. Therefore, they may be expected to make up for any previous deficiencies that have occurred earlier in the project. The estimation decisions made early in the project might be completely invalid if this is not taken into account.

In practice, being 14 days late on an 11-month (220 working days) project is a very small amount, but understanding what might make a project late is critical to knowing how to keep the project on schedule. Many factors can influence this understanding, and taking the time to reflect can be critical to making the right decisions.

When Halley and Calvin stepped back from “running the race” of managing their projects and reflected, they began to understand a much deeper issue that was influencing their projects. Their reflection effort increased their knowledge for future PEAK decisions about estimates. They learned that one of the problems they had thought was the fault of the software engineering organization was actually related to the manufacturing requirements and the order of performing the related tasks in the compressed schedule. This issue, which was not completely under their control, indicates the complexity of factors that can influence estimation and schedule planning for a project. After this experience, Halley was better prepared to defend his estimate decisions.