For projects with an iterative development process, the macro level of product development would contain phases such as analysis, design, code, test, and customer validation although at the micro level, selected components may still be going through the iterative cycles near the back end of the project schedule.

For each macro phase of the development process and project schedule, there is a set of data, both quantitative and qualitative, that gauges the development progress, helps to surface problems, and can provide a predictive indication of final product quality. Previous chapters contain many examples of phase-specific metrics and data. In general, fewer data and metrics are available in the early phases of the development cycle. Those very early indicators are also less representative of final product quality than those at the back end of the development cycle. For example, the frequency of system crashes and hangs during the system test phase indicates better how the product will perform in the field than the number of defects found during unit testing. This does not mean quality assessments at the early cycles of the project are less important. One needs to make sure that the project is on track at every major phase in order to achieve the desirable final outcome. For example, positive indicators from the requirements and design phases mean that the stability and predictability of the back end of the development process will be better.

Suppose we are conducting a quality assessment for Project Checkpoint Review 1 (PR1) in the project depicted by Figure 15.1. The data to be gathered and assessed would be related to requirements and design such as progress of design complete, coverage and effectiveness indicators of design reviews, and so on. If one is conducting an assessment for PR2, then indicators pertaining to the status of coding activities and code integration into the system library, and builds and releases of drivers for testing will be pertinent. It is important to plan ahead the indicators, metrics, and information you intend to rely on for your assessment at various checkpoints. If you have a metrics program in place and have been tracking the necessary data on an ongoing basis, conducting quality assessments will be much easier. However, if you are starting from scratch at this point, don’t despair. There are always data, information, and observations available that one can gather and analyze even when a metrics program is not in place. This is also a good time to start such a program and to demonstrate the value-added of the tracking system and the metrics.

The preceding discussion implies quantitative data. Qualitative data is equally important, and at times even more so. We gather much of our qualitative data through one-on-one interviews or small group discussions. Information gathered via formal meetings such as the presentations from functional development teams are useful but usually need in-depth follow-up. We first determine who we want to talk to, then we prepare a list of what questions we want to ask. To determine the “who,” think about the following:

To develop the list of questions, use both specific and open-ended questions. Open-ended questions are often the most useful. Here are some examples:

This last question helps to put the person’s comments into perspective. Asking people to compare the current release to a specific past release puts all qualitative data into a similar frame of reference. During the preparation phase, we determine which past release or releases would be best for such comparison. For organizations without historical data for comparison or analysis of metric levels and trends, quality assessment planning may not be easy. For quality indicators that are well practiced in the industry (e.g., defect removal efficiency), targets can be based on industry benchmarks and best practices (Jones, 2000).

Figure 15.2 shows a list of quality indicators for quality assessment at various project checkpoints. The list includes both quantitative and qualitative indicators.

Figure 15.2. Quality Indicators by Checkpoint and Development Phase

For quantitative analysis tools such as control charts, trend charts, histograms, pareto diagrams, and scatter diagrams or statistical techniques ranging from simple tabulation analysis to sophisticated multivariate methods are all fair game. It is our experience that simple techniques can be very powerful and most of the time sophisticated statistical techniques are unnecessary. The key point is to garner useful information from the data. As discussed in previous chapters, we found that using the effort/outcome paradigm is particularly useful in assessing in-process metrics. Of course, the data gathered must include both effort indicators and outcome indicators in order to apply this approach, and this should be a consideration in the planning and preparation phase. At the least, from raw data to useful information, some meaningful comparisons with relevant baseline, plan, or a previous similar product need to take place.

When analyzing the data, it is always good practice to pay particular attention to anything unusual. Good questions to ask in such situations are, “What more can I learn about this?” and “How can I put this into perspective?” Figures 15.3, 15.4, and 15.5 include examples of data that bear further investigation. In Figure 15.3, Team A was significantly behind plan in its functional test and Component X had not even started its testing. In Figure 15.4, the defect arrival pattern of the current project dif-fered from that for previous comparable projects. Was the higher defects volume in the early part of the defect curve due to more effective testing and better progress? Was the testing effectiveness and progress about the same as previous project at this point in the development cycle? In Figure 15.5, the test plan S-curve shows an unusual and potentially unachievable pattern.

Figure 15.3. Data on Functional Tests that Beg for Further Investigation

Figure 15.4. A Defect Arrival Pattern that Deviates from Historical Data

Figure 15.5. A Test Plan S Curve Showing an Unusual Pattern

For the qualitative evaluation, information from the interviews and open-ended probing can be classified, grouped, and correlated with existing knowledge and findings from quantitative analyses. The strongest proponents of quantitative methods argue that without metrics, an assessment is just another opinion. While quantitative data is important, our experience indicates that effective quality assessments are characteris-tically based on cross validation of findings and observations of both quantitative data and qualitative evaluation. Expert opinions also carry special weight. To that regard, the assessor should be equipped with acute observations to delineate whether the input he or she is getting is true expert opinion or opinion clouded by other factors. For example, opinions of the quality of the project may be optimistic by the development manager and pessimistic by the testing manager. It is not uncommon that at project checkpoint review meetings, the status of the project goes from excellent to poor, or vice versa, in just a few moments depending on the order of presentations by the development, support, testing, and service groups.

Evaluation of qualitative data is based on expert judgment and cross validation. For quantitative indicators, you may want to use predetermined criteria to ensure consistency. The following are sample criteria for evaluation of quantitative indicators:

The following are sample criteria for a qualitative indicator (plan change):

The following shows sample criteria for an indicator that may require both qualitative and quantitative evaluation (design status):

In summarizing the key issues and concerns, a quick analysis of the potential impacts of the identified problem areas can help rank the issues properly. For instance, the discovery of several low-severity problems in one area might not be a major concern, but a potential installation problem that customers will run into first thing when they install the product could be a very big deal. To put the information into perspective, one might compare a potential problem to a similar problem that occurred with a competitor’s product or a discovery in a past beta test. Furthermore, in summarizing data, don’t forget to identify what’s done right. This information can be every bit as useful as the problem areas. If an incremental improvement in one component’s code inspection process that resulted in nearly problem-free testing for that component during functional test, this could potentially provide a major breakthrough for the quality improvement effort of the entire team.

We found the format in Table 15.1 useful for summarizing and displaying the results. Each row shows a different quality parameter, listed in the first column. We often include key findings from the metrics or comments and information from interviews in the “observations” column. The final column shows an assessment for each parameter. At each interview, we ask for a “thumbs up” or “thumbs down” of the project compared with a previous similar project, and an overall assessment with regard to the project’s quality goals. However, it’s the assessor’s overall equalizing judgment that goes on the final assessment, as shown in the table.

Table 15.1 shows only a sample of the parameters and their assessment summary. The set of parameters for a quality assessment should include all pertinent attributes of the project’s quality objectives and development activities associated with those attributes. Some of the parameters may be phase-specific and others applicable for most of the development cycle. (See Figure 15.2 for a list of parameters.)

In each assessment we provide an overall assessment as the “bottom line.” The overall assessment should be developed with regard to the quality, function, and schedule objectives. In other words, “What is the likelihood that the product will meet quality objectives with the current content and schedule?” The overall assessment should be an integrated element in the project risk management process.

It is important to develop criteria for each level of the scale that you can clearly communicate along with your final assessment. It is useful to develop criteria that can be used over time and across multiple assessments. The following is an example of an overall quality assessment scale.

Figure 15.6 displays potential quality assessment ratings over the project checkpoint reviews for two scenarios. Apparently the scenario of steadily declining assessment rating (from red to green) is more favorable. This trend might occur when a company is developing a cutting-edge product. In any project, the risks and unknowns could be very high early on, resulting in an overall assessment of “Red.” Ideally, as the project progresses, the risks are addressed and problems resolved, thus improving the product’s potential for meeting quality objectives.

Figure 15.6. Scenarios of Quality Assessment Ratings of a Project over Time

The second scenario is undesirable not only because the final rating is poor, but also because the ratings worsen over time and initial ratings suggest low risk. While it is entirely possible for a project risk to increase (loss of key personnel would be one example), one should examine early positive ratings closely. It can be difficult to identify risks early in a project, but failure to do so can result in false positive ratings. In the early phases of a project, there are few concrete indicators, much less quantitative metrics, and it is human to assume no news is good news. The challenge to the quality professionals who conduct quality assessments is to make use of all fuzzy information and murky indicators to come up with a candid assessment.