Product Scoping

Selecting which features to spend development resources on is a key task for product managers, referred to as product scoping. The product manager must decide which features to invest development effort in and which to exclude from the project. In order to analyze how well a company was able to identify the feasible set of features at an early stage for later development and release, researchers collected data on which features were selected for development. Further, they analyzed which features made it into the final release, and which were excluded during later stages of development, for example, due to changing market situations or competing products. Excluding features late in the process implies that development effort is wasted.

The researchers defined several relevant metrics. Basic ones include the average share of excluded features, and the time to birth and time to removal of a feature. However, the overall view was hard to communicate until they presented the feature survival chart, visualizing features and their status change in an x-y–diagram, see Fig. 1. The features are listed on the y axis—one row for each feature—and time and milestones are marked on the x axis. Feature status is color coded: red/dark for excluded features, and two shades of green/grey for inserted features, depending on the source of the feature request.

Through the visualization chart, managers got an overview of the scoping of features and were able to connect major decisions in the project to the scope, marked with numbers in Fig. 1, and further explained by Wnuk et al. [1]. For example, number 1 in Fig. 1 represents a significant scope reduction (features turning from green to red), related to a cancellation of the products from the product line. At number 3, new features were included, while others were excluded in parallel. This was the desired behavior at the company, since development resources were limited. On the contrary, at number 4, new features were included late in the project, which is related to significant risks. This analysis helped them rethink the scoping process to make it more efficient.

Regression Test Selection

Selecting test cases for regression testing of a new software version is the second case study on visualization. In a product-line project, there are several dimensions that are relevant to analyze to support the choice of regression tests, for example (1) levels of abstraction (unit, integration, system, etc.), (2) regression testing as the system evolves over time with new versions, and (3) testing over different product variants, sharing a common code base [3]. We collected data on a number of test cases, test case executions, failed tests, etc. However, for test designers and managers, none of these metrics were sufficient to support their regression test selection. The many dimensions made it hard to understand which test cases were run at which level, version, and variant.

We prototyped a visualization tool that enables the users to see multiple views of the data, and to combine several dimensions, see Fig. 2 for an example of a subset of an industrial software system [2]. Each dot represents one test coverage item, and the color coding signals the frequency of the metric under analysis, for example, the number of executed test cases. In this case, red means an insufficient number of test cases for the item under test, green means a sufficient number of test cases, and blue indicates the range in between. The example shows that version 3.0 was well tested in the system test, but insufficiently in the unit test. More unit testing and less system testing was conducted in versions 3.2 and 3.3, while integration testing remains the same across the releases.

We evaluated the tool in three industrial focus groups with experienced test managers and consultants. The users were generally in favor of the visualization prototype, while they provided detailed feedback on which data to visualize, and how. They wanted the x–y position to represent a meaning, and requested more dynamic interaction in the views than our prototype could manage.

These case studies clearly show the power of visualization. Graphs may help stakeholders get an overview of data, which is tailored to support their decisions. In the feature scoping case, some managers interpreted all descoped features as a waste, which might not be the case. The organization may have learned something during the work with the feature, although the visualization clearly signals it as a waste. This message has to be brought together with the visualization, so the graphs are not over-interpreted.