6.2.1. Special case of the cost of defaults

A common metric is measuring the “cost of defects” by dividing testing expenses (design, execution, fix, retest, etc.) by the number of defects identified. Note that such a measurement is not linear. As soon as we have a reduced number of defects, the unit default cost becomes very high while the quality is in fact very good. We do not recommend using this metric.

6.2.2. Special case of defects

The defect sheets or reports are sources of important metrics: it is by measuring the number of defects per component that we will identify the aggregates of defects and that we will evaluate the level of efficiency of the activities. It is therefore important that all defects, even the most benign, are entered into the defects management tool, and that the data is filled in correctly.

It is obvious that nobody will introduce in the defects management tool a defect corrected immediately (e.g. a developer who corrects their code); however, this will not make it possible to identify the components having undergone difficulties of realization or design.

Completion of defect report fields – for example, quality characteristic, root cause, introduction phases, type of code (legacy or new), etc. – often leaves something to be desired. When statistics are performed on defects, this poor quality of data limits the lessons that could be drawn from these defects reports, whether in terms of root causes, quality characteristics to promote, design process to improve or technical debt in the code.

6.2.3. Accuracy or order of magnitude?

A project is always in constant evolution, so the measurements that are taken and reported always represent a past vision and not a current vision to the nearest second. When we measure the number of test cases run to show progress at the project meeting this afternoon, we will have – at best – a difference of a few hours of work. The same will apply for identified anomalies and corrected anomalies.

If past information is provided, it may be necessary to define when the measurement took place. If we want to provide predictions about the future, it will be necessary to weight the calculated value with a plus or minus uncertainty factor (e.g. +/- 10%). This solution will allow us to protect ourselves from future uncertainties.

The uncertainty rate should be based on an analysis of the differences identified in the past between predictions and realizations. We can consider that “all other things being equal” there is no reason that velocity, effectiveness, efficiency or quality level would show a very significant improvement. An order of magnitude should therefore be able to meet the need. To understand the difficulties of accurate forecasts, refer to Taleb (2008).

To use a quote from Tom Gilb, “a metric is better than no metric at all, because it can always be improved”.

6.2.4. Measurement frequency

Trends are calculated by taking several measurements over time. The periodic reporting and analysis of measurements are important elements allowing the test manager to measure the quality of the product, the level of progress of the tests and the efficiency of the processes.

However, for testers, taking measurements and reporting them daily (or according to another frequency) to the test manager is a significant burden. Similarly, the recovery, merging and analysis of the reported measurements are also heavy activities for the test manager. It is therefore recommended to try to automate as much as possible the activities of reporting measurements and analyzing metrics. Take care that the measurements taken are not distorted by other unmeasured elements (e.g. drop in the number of anomalies detected over a period of time, when the tests could not be carried out on this period).

Depending on the phases of the project, a weekly or daily frequency may be necessary. For some aspects, monthly reporting may be sufficient. The frequency of measurement must adapt to the frequencies of reporting on the project.

6.2.5. Using metrics

Regardless of the metrics used, it is essential that this data is used to improve processes and/or the product, and is never used to punish people or teams. In addition, it is important to use several metrics so as not to be misled by the use of a single metric.

There are also different levels of metrics: some will be used to refine the understanding of other metrics and will be kept at the test team level; others will relate to the project as a whole and will be escalated to higher levels.

A measurement without a reference is useless: a value can be close to or far from the desired target. It is therefore necessary to associate each metric with a range or target, which will make it possible to compare the measured values with the predicted values. We will then be able to define whether the current state is above or below the planned target and therefore what decisions to take if necessary.

The metrics will therefore be able to measure the progress of the processes (design and test of the system-of-systems) as well as the progress of the product (the system-of-systems itself and its various functionalities).

It is possible to use metrics on:

– components and their design processes regardless of the methodology;

– systems and component integration, as well as integration processes;

– functionalities of the system-of-systems, with a detail by criticality;

– evaluation of the deliverables and the processes that are at their origin.

As mentioned before, we need to set target values for each metric, before we even start measuring.

Above we can see the progress chart of an activity at two points in time. The forecasts are fixed, while the actual evolves over time.

6.2.6. Continuous improvement of metrics

The initial estimate – the target – may have been incorrectly estimated initially. So the forecasts will be wrong and it will be necessary to reassess them. This re-estimate is necessary to ensure that the data reported is correct and reflects reality.

The initial estimate may need to be modified, for example, to reflect variations in workload, performance, resources, scope or timelines. An image taken before the reassessment can serve as a reference and be compared – during the feedback meeting at the end of the project – with subsequent data.

6.3.1. FTR: first time right

Derived from first time fix right, which ensured that a correction did not generate other anomalies, first time right focuses on the ability to provide a good quality component that does not need to be reworked. This metric measures the efficiency of the development–test couple, by identifying the number of round trips (correction iterations) between the development teams and the test teams before the component is considered good enough to be put into production.

Among the criteria to include, we have:

– traceability from/to requirements;

– presence and completeness of the documentation;

– results of test case executions available and correct;

– external interfaces with correct and functional components.

FTR is measured, during system tests or acceptance tests, by identifying per component the number of times it is re-tested following the identification of an anomaly. The “components” can be software bricks or functionalities. The objective (the target to be reached) is 1, that is, each component is tested once and only once before being considered correct.

In an Agile world, this implies respecting all the criteria for defining DONE and that the component must no longer be modified or retested.

6.3.2. Coverage rate

To correctly evaluate a product, it is necessary to ensure that the product has been completely tested. This involves several simultaneous levels of coverage, including:

– coverage of risks by test cases, with an increase in number of test cases (and/or number of test techniques to be used) if the risks are high;

– coverage of requirements, use cases and user stories, by a sufficient number of tests to ensure that the requirements, use cases and user stories do what they are supposed to do, and do not do what that they are not supposed to do. This often involves the use of multiple test techniques and many test cases per requirement;

– feature coverage, where we will ensure that all features are properly tested and validated, often in decreasing order of priority and criticality;

– coverage of quality characteristics (see ISO25010) according to the level of risk associated with each characteristic or sub-characteristic. Here too we will have the use of multiple test techniques and a large number of test cases;

– coverage of environments insofar as the real environments may be different from the environments used during the tests;

– code coverage, where we will ensure the quality of the code and compliance with coding rules, the absence of dead code, etc.;

– component coverage where we will ensure that all components – in their latest version – have been correctly and exhaustively tested. By component we will have, for example, screens, functions, reports or interfaces;

– the coverage of test techniques, where we will ensure that all the various test techniques provided are correctly applied in order to find as many defects as possible.

To measure the coverage rate, we will need to measure the percentage of tests passing – for a category – compared to the total number of tests for that category. This involves ensuring the presence of a link – traceability – between the criteria to be covered and the test cases covering these criteria. These links are easy to set up when they are defined at the start of the test design; on the other hand, they involve a significant workload if it is a question of adding these indicators after the start of the test design. Some tools allow traceability – often tree-like – from requirements to test cases.

The information provided during the reporting must include the coverage rates, to link the information collected to all the requirements initially requested.

6.3.3. Code churn

Code change over time is calculated as the sum of lines of code added, removed or changed. Although this can help measure code rewrites – fixes, refactoring, etc. – it is necessary to relate these measures to the additions, modifications or corrections, of functionalities. Agility suggests scheduling consolidation sprints, where code is refactored and simplified, so these sprints will have higher code churn rates than others where new features are designed. A “code churn” ratio that is not associated with corrections or evolutions can be synonymous with the desire for perfectionism by the design teams.

However, care must be taken when calculating the “code churn”, insofar as:

– the number of lines of code varies according to the languages and is not synonymous with functionalities unlike function points;

– the calculation does not take into account the size of the design team, therefore the volume of modified code;

– the calculation is done using configuration management tools, so everything that is not managed in configuration is not taken into account;

– the calculation does not take into account the evolutions or the complexity of the functionalities.

The measurement of “churn” in the business (customer abandonment rate, for example, in percentage of abandonment of subscriptions) is important to measure if it is linked to new versions or deliveries of the product. It can indeed indicate a lack of quality in the product.

6.4.1. Effectiveness metrics

6.4.1.1. DDE: defect detection efficiency

The DDE metric measures the defect detection efficiency of the various processes used.

The principle is that our testing activities must find the maximum number of defects and only allow a very small number of defects to pass to later phases. This metric can be evaluated at various stages and at the full system-of-systems level. We can also use this metric to assess the effectiveness of subsequent testing activities.

It is possible to weight this metric according to the level of criticality of the defects. This will identify whether the “faults found during testing” correspond to higher levels of criticality than the “faults found subsequently”.

6.4.1.2. DRE: defect removal efficiency

Once faults have been found, it is important to remove them. This is where the defect removal efficiency (DRE) metric comes in; DRE is the sum of the defects corrected before delivery divided by the sum of the defects corrected before delivery and the defects reported by customers or users during the first three months of use.

It is important to note that applications that have a DRE greater than 95% tend to have shorter cycles and lower costs (including maintenance costs) than applications with a lower DRE.

It is not mandatory to correct all identified defects; it is a business decision, so the DDP will be superior to the DRE. However, the business decision not to correct anomalies will have a negative impact on the perceived level of quality.

The DRE target value should be close to 99.5%, while it is often around 85% according to Jones (2018).

6.4.1.3. CRI: coverage of risks

The risks identified should be covered – depending on their level of criticality – by one or more tests. The risks the stakeholders have decided to ignore should not be counted.

It is also necessary here to weight this CRI metric by the level of criticality of the risks. We note that we can have several tests for the same risk. The target for risk coverage should be 100% for all critical, major and significant risks.

6.4.1.4. ART: automated regression testing

A criterion of confidence for the systems is the percentage of automated regression tests (ART). It is calculated as follows:

Warning: this metric does not consider the total number of test cases to be considered, nor the level of coverage of the requirements, risks or structure of the component under test. Similarly, we have no information on the size of automated tests compared to the size of non-automated tests.

The target for the ART number will vary by criticality and development model. In the case of DevOps, the ART should be close to 100%.

6.4.1.5. TDD and design effectiveness

Agile design using TDD (or ATDD or BDD) provides useful information for the test manager, including:

MBR: modifications before refactoring, which allows us to see whether the basic principle of TDD (small tests followed by small pieces of code) has been respected.

RFT: ratio between functional and test lines of code should be close to 1.

It is important to note that this efficiency rate (RFT) can also be calculated for programs where the size is calculated in function points. Estimations based on function points (FPs) are generally much more accurate than those based on lines of code (LOC).

6.4.2. Efficiency metrics

Efficiency is the ability of a process to perform well at a given type of work. For a test process, it is the ability to find the most defects and information with the least test effort. As with efficiency metrics, the importance of defects – their criticality and impact – will have to be considered.

6.5.1. Quality model metrics

We have seen previously the various aspects covered by the quality standards. Each can be measured and KPIs can be defined. Many KPIs are present in ISO25010 (2011) and ISO9126-4 (2004), as well as in Kan (2003). We can also find many statistical values to have points of comparison in Jones (2000, 2008) and Jones and Bonsignour (2012).

6.6.1. Baseline

We can take data available in the industry as a baseline and compare our teams to those that served as a source of this data. However, this comparison does not reflect the skills of our teams, nor the specificities of our organization or our projects. We can therefore decide to measure for a certain time (a week, a sprint or a month) in order to consider the measurement taken as the reference level, the baseline.

From this point on, all subsequent measurements will be compared to this initial measurement. Of course, this initial measurement may be incorrect and not reflect the tasks to be performed; this will lead us to correct our baseline. We will need to ensure that our baseline is consistent. If our baseline is too above or too below industry standard values, we will need to ensure its validity.

6.6.2. Historical data

We could assume that historical data is correct by default since it represents what really happened. Unfortunately, this is not the case because often these projects do not measure all the necessary information. For example, fault measurements frequently only start from system test levels. So anomalies in the design phases – anomalies in requirements, specifications or architecture – and detected by reviews are not reported. Anomalies in the code, detected by static testing or code review, are often corrected without being noted and anomalies in the context of the use of TDD cannot be used because the principle is to create anomalies and then the minimum code for these anomalies is corrected.

So we have significant gaps in the historical measurements we have taken on our previous systems-of-systems. To this, we must add the lack of measurements of related activities – administration, users, maintenance, operations, etc. – related to the use of these systems over time. All these measures are often distributed in other cost centers and are not aggregated with the development costs and the costs of ownership (TCO, Total Cost of Ownership).

As Jones (2017) demonstrates, and based on several thousand projects analyzed over the past 40 years, historical data does not consider a lot of data from load associated with software development. Among the data not measured and to be associated with the developments as well as the maintenance of systems, we have:

– unpaid overtime (up to 25% of the effort);

– allocation of expenses to another project (up to 20% of the effort);

– user workloads to be assigned to the project, such as acceptance testing, requirements elicitation interviews, etc. (up to 50% of effort);

– specialists (testers, architects, database managers, technical writers, etc.) temporarily assigned to the project and not taken into account in the measures (up to 45% of the effort);

– costs of monitoring and reporting metrics (up to 10% of the effort);

– supervision costs (up to 15% of the effort);

– meetings, training and other travel (up to 25% of the effort).

We understand that the lack of measurement of these expenses in past projects can make it difficult to compare them with forecasts for future projects, which would include these factors. However, this consideration is necessary in order to be able to correctly anticipate the size, duration and effort of projects, especially systems-of-systems, which are often very large.

6.6.3. Periodic improvements

Whether in the context of agile projects or for sequential projects, it is necessary to continuously improve the effectiveness and efficiency of our processes. This involves – periodically – comparing the reported metrics of our tasks and processes to the values defined as the baseline for these tasks and processes.

One way to identify if it is necessary to improve processes is to identify whether the defects found during a phase could have been found before this phase. If so, the earlier phases should be improved. Another thing that can be improved is the effort required to identify a type of defect, with the goal of reducing that effort without reducing the ability to identify anomalies.

Improvements can therefore relate to:

– effectiveness: finding more defects;

– efficiency: finding defects with less effort.

6.7.1. Internal test reporting

The information that the test manager will need is more detailed and comprehensive than that provided to stakeholders. Indeed, the test manager must have a detailed vision of the progress of the tests and be able to synthesize this information for the other stakeholders.

Among the information to follow, the test manager should include:

– the workload planned and carried out on the test project, or even on each of the tasks of the test project;

– the progress in terms of test design (test cases, scenarios and test procedure, etc.) because these elements will have an impact on the tests that can be executed later;

– progress on the provision of test environments, interfaces with other applications for integration tests;

– the provision or generation of the test data necessary for the tests to be executed; this includes setting up the accounts and users that will be used during testing;

– monitoring of risk coverage and requirements according to the level of RPN identified; the objective here is to ensure that the testing effort is consistent with the level of priority;

– the progress of requirements coverage by the designed tests, then – during the test execution phase – by the executed tests;

– the monitoring of defect detection rates, their distribution by software component, product or system and their impact and severity levels; this information will allow us to measure the density of defects (according to the volume of the components) and to identify components concentrating many defects;

– monitoring of the correction times for identified anomalies, to check whether the SLAs (Service Level Agreement) are respected and to identify both the backlog of anomalies and the average correction time. It may be necessary to split this according to the criticalities of the defects;

– analysis of anomalies to identify the main root causes and propose improvement actions in advance of the phase such as static analysis or reviews.

6.7.2. Reporting to the development team

Reporting by the test manager to the development team should include:

– information on the components, software, products or systems that are affected by anomalies;

– a measurement of the number of anomalies found by component and by version in order to define the technical debt associated with each component;

– information on the quality level of each version or release of the software components submitted for testing;

– an indication of the level of quality of the test inputs and the root causes of the anomalies identified.

The reporting expected by the test manager from the development team concerns the planned delivery dates of software components, versions and releases, and the supply of corrections to identified anomalies.

Other information such as the location of anomalies, the average duration of correction of anomalies or the level of quality of the deliverables provided may be part of the information provided by the test manager.

6.7.3. Reporting to the management

The reporting intended for management and the hierarchy mainly concerns:

– compliance with commitments, milestones, charges and deadlines;

– identification of risks, hazards and problems;

– impacts on other systems, environments and domains (if applicable).

Compliance with deadlines can be done by time-on-time diagrams (see Figure 6.3), which allow us to view the milestone reports.

The identification of risks and their states of mitigation is also to be reported to management because certain risks are to be dealt with at hierarchical levels higher than that of the project, and because provisions must probably be anticipated. Risk reporting can be carried out as defined in section 5.6 of Homès (2012).

The identification of impacts on other systems, environments and domains makes it possible to identify potential coordination problems between actors and/or services. The reporting of these elements can be done via shared schedules associated with each of the environments.

6.7.4. Reporting to the clients or product owners

In general, customers and their representatives (e.g. product owners in Agile environments) are focused on:

– obtaining requirements or user stories or features in the production environment in order to obtain the expected benefits;

– identification of applicable limitations or constraints;

– firm delivery dates to manage production milestones (e.g. training, deployment, marketing campaigns, etc.).

It is therefore important to ensure that:

– Traceability, from the contractual requirements and references to the test cases executed on a given version, is correctly established, so that this information reflects the proofs provided and complete. In the context of systems-of-systems subject to regulation, these proofs of design, testing and qualification are necessary to obtain the contractual and regulatory agreement prior to marketing.

– Defects or limitations are identified, as well as the actions that will or will not be implemented to mitigate them. This information must be seen both at the level of the impacts on each of the components, subsystems or systems, and at the level of the end-to-end functionalities of the system-of-systems.

– Milestones and deadlines are realistic rather than reflecting a commercial or marketing wish.

Other more detailed elements may need to be identified, such as:

– detailed percentage of coverage of the requirements, allowing us to visualize the progress;

– quality of deliverables (e.g. the number and type of defects identified in the deliverables entering a task and which should have been identified by previous tasks), making it possible to direct continuous improvement efforts;

– progress of the tests in the phases of analysis, design, implementation of the tests, to ensure that the test activities progress correctly in parallel with the design activities of the components to be tested and that the execution of the testing can begin upon delivery in test environments;

– degree of progress based on the criticality or priority of the requirements or user stories;

– volume of defects as well as their type and location, in order to identify the clusters of defects in the components as well as the components that deserve to be rewritten;

– impacts on the other components, subsystems, systems and on the system-of-systems, both from a functional point of view and from the points of view of exchanged flows, performance, security, volume and capacity.

6.7.5. Reporting to the direction and upper management

Reporting for management and hierarchy is similar to that for customers, but must also identify:

– the risks and their evolution (increase or reduction) on the project;

– the evolution of the use of resources in relation to the progress of activities;

– resource needs – hardware, software, products, tools and personnel – and team training, etc. in order to anticipate the periods of the project when these elements will be necessary.